1.Synchronously:

• 线性执行
• 同步方法
• 耗时长

2.Multiprocessing if it’s CPU bound:

• The multiprocessing library gives each process its own Python interpreter and each their own GIL(Global Interpreter Lock).

• allows you to create programs that can run concurrently (bypassing the GIL)

• use the entirety of your CPU core

    import multiprocessing
  
    def concurrent_multi_process():
        dids = []
        with concurrent.futures.ProcessPoolExecutor() as executor:
            results = executor.map(call_out_data, dids)
  
    def call_out_data(did):
        requests.adapters.DEFAULT_RETRIES = 5
  
        url = "http://202.205.91.21:60001/nrmv/canInfo"
        headers = {
            'Content-Type': 'application/json'
        }
  
        payload = json.dumps(
        {
            'appId': "",
            "did": did,
            "startTime": 'test',
            "endTime": '20220610000000',
            "limit": 1000000000,
            "nextPageStartRowKey": "",
            "reverse": 'false',
            "showColumns": ["`did`", "`3014`", "`2602`", "`2603`", "`2204`"],
            "token": ""
        })
        response = requests.request("POST", url, headers=headers, data=payload)
        print(response)
  
        if (response.status_code == 200 and 'data' in response.json()):
            data = response.json()['data']
            return data
  

3.Threading if your program is network bound:

• it’s perfect for I/O operations such as web scraping because the processor is sitting idle waiting for data.

    import threading
  
    def thread_process():
        dids = []
        with concurrent.futures.ThreadPoolExecutor() as executor:
            results = executor.map(call_out_data, dids)
  

针对中农的大数据服务器，测试了请求3645 次 不同类型下的性能比较，可以看出，使用 thread 方式效果最优秀

ModuleNotFoundError: No module named ‘XXX’ in Python

The Python “ModuleNotFoundError: No module named ‘XXX’” occurs when we forget to install the XXX module before importing it or install it in an incorrect environment. To solve the error, install the module by running the pip install XXX command.

Open your terminal in your project’s root directory and install the XXX module.

notice:

👇️ in a virtual environment or using Python 2

pip install XXX

👇️ for python 3 (could also be pip3.10 depending on your version)

pip3 install XXX

👇️ if you get permissions error

sudo pip3 install XXX

👇️ if you don’t have pip in your PATH environment variable

python -m pip install XXX

👇️ for python 3 (could also be pip3.10 depending on your version)

python3 -m pip install XXX

👇️ for Anaconda

conda install -c anaconda XXX

The Python error “ModuleNotFoundError: No module named ‘XXX’” occurs for multiple reasons:

• Not having the XXX package installed by running pip install XXX.
• Installing the package in a different Python version than the one you’re using.
• Installing the package globally and not in your virtual environment.
• Your IDE running an incorrect version of Python.
• Naming your module XXX.py which would shadow the official module.
• Declaring a variable named XXX which would shadow the imported variable.

If the error persists, get your Python version and make sure you are installing the package using the correct Python version.

For example, my Python version is 3.10.4, so I would install the XXX package with pip3.10 install XXX.

pip3.10 install XXX

👇️ if you get permissions error use pip3 (NOT pip3.X)

sudo pip3 install XXX

If the “No module named ‘XXX’” error persists, try restarting your IDE and development server / script.

You can check if you have the XXX package installed by running the pip show XXX command.

👇️ check if you have XXX installed

pip3 show XXX

👇️ if you don’t have pip setup in PATH

python3 -m pip show XXX

The pip show XXX command will either state that the package is not installed or show a bunch of information about the package, including the location where the package is installed.

If the package is not installed, make sure your IDE is using the correct version of Python.

If you have multiple Python versions installed on your machine, you might have installed the XXX package using the incorrect version or your IDE might be setup to use a different version.

For example, In VSCode, you can press CTRL + Shift + P or (⌘ + Shift + P on Mac) to open the command palette.

Then type “Python select interpreter” in the field.

Your IDE should be using the same version of Python (including the virtual environment) that you are using to install packages from your terminal.

👇️ check if you have XXX installed

pip3 show XXX

👇️ if you don’t have pip setup in PATH

python3 -m pip show XXX

👇️ uninstall XXX

pip3 uninstall XXX

👇️ if you don’t have pip setup in PATH

python3 -m pip uninstall XXX

👇️ install XXX

pip3 install XXX

👇️ if you don’t have pip setup in PATH

python3 -m pip install XXX

related blog link:

https://bobbyhadz.com/blog/python-no-module-named-numpy