Python请求操作系统错误104连接中断错误

Question

[嗨，我正在尝试使用python的requests模块命中API。 Api必须命中20000次，因为页面数大约为20000。每次命中数据大约需要10 mb。在该过程结束时，它会创建一个约100gb的json文件。这是我编写的代码

with open('file.json','wb',buffering=100*1048567) as f:
  while(next_page_cursor != ""):
    with request.get(url,headers=headers) as response:
      json_response = json.loads(response.content.decode('utf-8')) 
      """
      json response looks something like this 
      {
        content:[{},{},{}........50 dictionaries]
        next_page_cursor : "abcd"
      }
      """
      next_page_cursor = json_response['next_page_cursor']

      for data in json_response['content']:
        f.write((json.dumps(data) + "\n").encode())

但是在成功运行了几页之后，代码失败，出现以下错误：

Traceback (most recent call last):
  File "<command-1206920060120926>", line 65, in <module>
    with requests.get(data_url, headers = headers) as response:
  File "/databricks/python/lib/python3.7/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/databricks/python/lib/python3.7/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/databricks/python/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/databricks/python/lib/python3.7/site-packages/requests/sessions.py", line 686, in send
    r.content
  File "/databricks/python/lib/python3.7/site-packages/requests/models.py", line 828, in content
    self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
  File "/databricks/python/lib/python3.7/site-packages/requests/models.py", line 753, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: OSError("(104, \'ECONNRESET\')")', OSError("(104, 'ECONNRESET')"))

Answer 1

您需要使用response.iter_contenthttps://2.python-requests.org/en/master/api/#requests.Response.iter_content