markPendingRemoval fails with 500 Internal Server Error

[abhidg]

Obtained on Peru ingestion running on dev:

Downloading CSV content from https://cloud.minsa.gob.pe/s/Y8w3wHsEdYQSZRp/download
Download finished
Guessing mimetype of https://cloud.minsa.gob.pe/s/Y8w3wHsEdYQSZRp/download
Could not determine mimetype
detecting encoding of retrieved content.
Source encoding is presumably {'encoding': 'UTF-8-SIG', 'confidence': 1.0, 'language': ''}
Uploaded source content to s3://epid-sources-raw/5f6bb1ce0b414482cc5580a0/2021/07/05/2025/content.csv
Invoking parser (parsing.peru.peru)
Extracting fields from event {'env': 'dev', 's3Bucket': 'epid-sources-raw', 'sourceId': '5f6bb1ce0b414482cc5580a0', 's3Key': '5f6bb1ce0b414482cc5580a0/2021/07/05/2025/content.csv', 'sourceUrl': 'https://cloud.minsa.gob.pe/s/Y8w3wHsEdYQSZRp/download', 'uploadId': '60e36aa163c405cb2a1b2849', 'dateFilter': {}, 'dateRange': {}}
Retrieving service account credentials from s3://epid-ingestion/covid-19-map-277002-0943eeb6776b.json
Retrieving raw data from s3://epid-sources-raw/5f6bb1ce0b414482cc5580a0/2021/07/05/2025/content.csv
Raw file retrieved at /tmp/tmpirc7jarz
500 Server Error: Internal Server Error for url: https://dev-data.covid-19.global.health/api/sources/5f6bb1ce0b414482cc5580a0/markPendingRemoval
Updating upload via https://dev-data.covid-19.global.health/api/sources/5f6bb1ce0b414482cc5580a0/uploads/60e36aa163c405cb2a1b2849
Traceback (most recent call last):
  File "./retrieval/retrieval.py", line 308, in <module>
    run_retrieval(tempdir=(TEMP_PATH if len(sys.argv) == 1 else sys.argv[1]))
  File "./retrieval/retrieval.py", line 296, in run_retrieval
    invoke_parser(
  File "./retrieval/retrieval.py", line 212, in invoke_parser
    importlib.import_module(parser_module).event_handler(payload)
  File "/app/./parsing/peru/peru.py", line 145, in event_handler
    return parsing_lib.run(event, parse_cases)
  File "/app/retrieval/../common/parsing_lib.py", line 411, in run
    common_lib.complete_with_error(
  File "/app/retrieval/../common/common_lib.py", line 101, in complete_with_error
    raise exception
  File "/app/retrieval/../common/parsing_lib.py", line 373, in run
    requests.post(mark_pending_url, headers=api_creds, cookies=cookies).raise_for_status()
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://dev-data.covid-19.global.health/api/sources/5f6bb1ce0b414482cc5580a0/markPendingRemoval

[abhidg]

Failing with 504 Gateway timeout now

...
Source encoding is presumably {'encoding': 'UTF-8-SIG', 'confidence': 1.0, 'language': ''}
Uploaded source content to s3://epid-sources-raw/5f6bb1ce0b414482cc5580a0/2021/07/12/2205/content.csv
Invoking parser (parsing.peru.peru)
Extracting fields from event {'env': 'dev', 's3Bucket': 'epid-sources-raw', 'sourceId': '5f6bb1ce0b414482cc5580a0', 's3Key': '5f6bb1ce0b414482cc5580a0/2021/07/12/2205/content.csv', 'sourceUrl': 'https://cloud.minsa.gob.pe/s/Y8w3wHsEdYQSZRp/download', 'uploadId': '60ecbc729a8ad464c39c9b32', 'dateFilter': {}, 'dateRange': {}}
Retrieving service account credentials from s3://epid-ingestion/covid-19-map-277002-0943eeb6776b.json
Retrieving raw data from s3://epid-sources-raw/5f6bb1ce0b414482cc5580a0/2021/07/12/2205/content.csv
Raw file retrieved at /tmp/tmpkrevny_d
504 Server Error: GATEWAY_TIMEOUT for url: https://dev-data.covid-19.global.health/api/sources/5f6bb1ce0b414482cc5580a0/markPendingRemoval
Updating upload via https://dev-data.covid-19.global.health/api/sources/5f6bb1ce0b414482cc5580a0/uploads/60ecbc729a8ad464c39c9b32
Traceback (most recent call last):
File "./retrieval/retrieval.py", line 313, in <module>
run_retrieval(tempdir=(TEMP_PATH if len(sys.argv) == 1 else sys.argv[1]))
File "./retrieval/retrieval.py", line 301, in run_retrieval
invoke_parser(
File "./retrieval/retrieval.py", line 217, in invoke_parser
importlib.import_module(parser_module).event_handler(payload)
File "/app/./parsing/peru/peru.py", line 145, in event_handler
return parsing_lib.run(event, parse_cases)
File "/app/retrieval/../common/parsing_lib.py", line 411, in run
common_lib.complete_with_error(
File "/app/retrieval/../common/common_lib.py", line 101, in complete_with_error
raise exception
File "/app/retrieval/../common/parsing_lib.py", line 373, in run
requests.post(mark_pending_url, headers=api_creds, cookies=cookies).raise_for_status()
File "/opt/pysetup/.venv/lib/python3.8/site-packages/requests/models.py", line 943, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 504 Server Error: GATEWAY_TIMEOUT for url: https://dev-data.covid-19.global.health/api/sources/5f6bb1ce0b414482cc5580a0/markPendingRemoval

[abhidg]

Since marking cases for deletion and deleting them is not possible synchronously for millions of records, here's a new workflow for non-UUID sources:

1. batchUpsert sets pendingDeletion: false for new uploads
2. Data export, filtered downloads filter for cases that do not have pendingDeletion set
3. A periodic script runs that checks for all uploads more recent than the upload currently in the sourceId (for non-UUID sources, each ingestion will be a single uploadId), and selects the one with the highest number of cases that is greater than the current case count. If there are no such uploadIds, set all pendingDeletion false to true. If an uploadId exists, set everything in the sourceId other than that uploadId to pendingDeletion True, and unset pendingDeletion from this uploadId.
4. Cleanup, either immediately, or via another periodic script that deletes all cases with pendingDeletion True

This way, everything with pendingDeletion unset is the current data, pendingDeletion false is incoming data, and pendingDeletion true is data marked to be deleted.

[abhidg]

Asynchronous deletion is being tracked in #2002, closing this as we are not using this approach anymore.