浅谈 Python 中 Synchronously, Multiprocessing, Threading 区别

1.Synchronously:

• 线性执行
• 同步方法
• 耗时长

2.Multiprocessing if it’s CPU bound:

• The multiprocessing library gives each process its own Python interpreter and each their own GIL(Global Interpreter Lock).

• allows you to create programs that can run concurrently (bypassing the GIL)

• use the entirety of your CPU core

    import multiprocessing
  
    def concurrent_multi_process():
        dids = []
        with concurrent.futures.ProcessPoolExecutor() as executor:
            results = executor.map(call_out_data, dids)
  
    def call_out_data(did):
        requests.adapters.DEFAULT_RETRIES = 5
  
        url = "http://202.205.91.21:60001/nrmv/canInfo"
        headers = {
            'Content-Type': 'application/json'
        }
  
        payload = json.dumps(
        {
            'appId': "",
            "did": did,
            "startTime": 'test',
            "endTime": '20220610000000',
            "limit": 1000000000,
            "nextPageStartRowKey": "",
            "reverse": 'false',
            "showColumns": ["`did`", "`3014`", "`2602`", "`2603`", "`2204`"],
            "token": ""
        })
        response = requests.request("POST", url, headers=headers, data=payload)
        print(response)
  
        if (response.status_code == 200 and 'data' in response.json()):
            data = response.json()['data']
            return data
  

3.Threading if your program is network bound:

• it’s perfect for I/O operations such as web scraping because the processor is sitting idle waiting for data.

    import threading
  
    def thread_process():
        dids = []
        with concurrent.futures.ThreadPoolExecutor() as executor:
            results = executor.map(call_out_data, dids)
  

针对中农的大数据服务器，测试了请求3645 次 不同类型下的性能比较，可以看出，使用 thread 方式效果最优秀