Skip to content
This repository has been archived by the owner on Jan 26, 2021. It is now read-only.

Fix timeout on final data upload #46

Open
lucasmbrown opened this issue Jul 28, 2020 · 1 comment
Open

Fix timeout on final data upload #46

lucasmbrown opened this issue Jul 28, 2020 · 1 comment

Comments

@lucasmbrown
Copy link
Contributor

On the final upload, we time out. We're probably (newly) hitting this because we've added fields over time as well as added more data each day, since we upload all historical data which keeps growing.

We could do any of the following:

  1. Reduce the size of the data we're uploading by limiting fields, date ranges, etc

  2. Fork our gsheets uploader client to send a new timeout parameter as part of the requests usage.

  3. Change uploader clients (I know @wesmwoo has been looking at this already, since this client gspread uploads all data as text rather than numbers).

Or something else?

The full stack trace is:

Beginning to upload data to workbook 1s534JoVjsetLDUxzkww3yQSnRj9H-8QLMKPUrq7RAuc and tab All State Data...
Traceback (most recent call last):
  File "/Users/lucas/.virtualenvs/covid1/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/Users/lucas/.virtualenvs/covid1/lib/python3.6/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/Users/lucas/.pyenv/versions/3.6.9/lib/python3.6/http/client.py", line 1254, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Users/lucas/.pyenv/versions/3.6.9/lib/python3.6/http/client.py", line 1300, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Users/lucas/.pyenv/versions/3.6.9/lib/python3.6/http/client.py", line 1249, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Users/lucas/.pyenv/versions/3.6.9/lib/python3.6/http/client.py", line 1075, in _send_output
    self.send(chunk)
  File "/Users/lucas/.pyenv/versions/3.6.9/lib/python3.6/http/client.py", line 996, in send
    self.sock.sendall(data)
  File "/Users/lucas/.pyenv/versions/3.6.9/lib/python3.6/ssl.py", line 975, in sendall
    v = self.send(byte_view[count:])
  File "/Users/lucas/.pyenv/versions/3.6.9/lib/python3.6/ssl.py", line 944, in send
    return self._sslobj.write(data)
  File "/Users/lucas/.pyenv/versions/3.6.9/lib/python3.6/ssl.py", line 642, in write
    return self._sslobj.write(data)
socket.timeout: The write operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/lucas/.virtualenvs/covid1/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/Users/lucas/.virtualenvs/covid1/lib/python3.6/site-packages/urllib3/connectionpool.py", line 725, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/Users/lucas/.virtualenvs/covid1/lib/python3.6/site-packages/urllib3/util/retry.py", line 403, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/Users/lucas/.virtualenvs/covid1/lib/python3.6/site-packages/urllib3/packages/six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "/Users/lucas/.virtualenvs/covid1/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/Users/lucas/.virtualenvs/covid1/lib/python3.6/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/Users/lucas/.pyenv/versions/3.6.9/lib/python3.6/http/client.py", line 1254, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Users/lucas/.pyenv/versions/3.6.9/lib/python3.6/http/client.py", line 1300, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Users/lucas/.pyenv/versions/3.6.9/lib/python3.6/http/client.py", line 1249, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Users/lucas/.pyenv/versions/3.6.9/lib/python3.6/http/client.py", line 1075, in _send_output
    self.send(chunk)
  File "/Users/lucas/.pyenv/versions/3.6.9/lib/python3.6/http/client.py", line 996, in send
    self.sock.sendall(data)
  File "/Users/lucas/.pyenv/versions/3.6.9/lib/python3.6/ssl.py", line 975, in sendall
    v = self.send(byte_view[count:])
  File "/Users/lucas/.pyenv/versions/3.6.9/lib/python3.6/ssl.py", line 944, in send
    return self._sslobj.write(data)
  File "/Users/lucas/.pyenv/versions/3.6.9/lib/python3.6/ssl.py", line 642, in write
    return self._sslobj.write(data)
urllib3.exceptions.ProtocolError: ('Connection aborted.', timeout('The write operation timed out',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/lucas/repos/covid-exit-strategy/main.py", line 220, in <module>
    extract_transform_and_load_covid_data(post_to_google_sheets=True)
  File "/Users/lucas/repos/covid-exit-strategy/main.py", line 214, in extract_transform_and_load_covid_data
    credentials=credentials,
  File "/Users/lucas/repos/covid-exit-strategy/covid/load.py", line 20, in post_dataframe_to_google_sheets
    col_names=True,
  File "/Users/lucas/.virtualenvs/covid1/lib/python3.6/site-packages/df2gspread/df2gspread.py", line 147, in upload
    wks.update_cells(cell_list)
  File "/Users/lucas/.virtualenvs/covid1/lib/python3.6/site-packages/gspread/models.py", line 909, in update_cells
    body={'values': values_rect},
  File "/Users/lucas/.virtualenvs/covid1/lib/python3.6/site-packages/gspread/models.py", line 235, in values_update
    r = self.client.request('put', url, params=params, json=body)
  File "/Users/lucas/.virtualenvs/covid1/lib/python3.6/site-packages/gspread/client.py", line 67, in request
    headers=headers,
  File "/Users/lucas/.virtualenvs/covid1/lib/python3.6/site-packages/requests/sessions.py", line 590, in put
    return self.request('PUT', url, data=data, **kwargs)
  File "/Users/lucas/.virtualenvs/covid1/lib/python3.6/site-packages/google/auth/transport/requests.py", line 450, in request
    **kwargs
  File "/Users/lucas/.virtualenvs/covid1/lib/python3.6/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/lucas/.virtualenvs/covid1/lib/python3.6/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/Users/lucas/.virtualenvs/covid1/lib/python3.6/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', timeout('The write operation timed out',))
@lucasmbrown
Copy link
Contributor Author

I don't think we use the All Data sheet for anything for the website right now, although it has been a handy reference occasionally. So we could also choose option

  1. Stop uploading the CDC Guidance (All Data) sheet

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant