Reprocess candidate-totals-by-batch files when contests are edited

[arsalansufi]

Overview

In testing multi-contest batch audits, I found an issue able to affect not only multi-contest batch audits but also single-contest batch audits - the fact that we don't reprocess uploaded candidate-totals-by-batch files when contests are edited. The number of things that can go wrong because of this has just grown with multi-contest batch audits.

For example, adding a contest after uploading candidate-totals-by-batch files can render those files incomplete and actually crash Arlo the next time Arlo fetches jurisdictions:

INFO:werkzeug:127.0.0.1 - - [06/Mar/2024 17:57:53] "GET /api/election/c8472cc3-e746-4285-ad41-b7abd2b7a760/jurisdiction HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/home/arsalan/.cache/pypoetry/virtualenvs/arlo-Lrf6nvlW-py3.8/lib/python3.8/site-packages/sentry_sdk/integrations/flask.py", line 88, in sentry_patched_wsgi_app
    return SentryWsgiMiddleware(lambda *a, **kw: old_app(self, *a, **kw))(
  File "/home/arsalan/.cache/pypoetry/virtualenvs/arlo-Lrf6nvlW-py3.8/lib/python3.8/site-packages/sentry_sdk/integrations/wsgi.py", line 140, in __call__
    reraise(*_capture_exception(hub))
  File "/home/arsalan/.cache/pypoetry/virtualenvs/arlo-Lrf6nvlW-py3.8/lib/python3.8/site-packages/sentry_sdk/_compat.py", line 54, in reraise
    raise value
  File "/home/arsalan/.cache/pypoetry/virtualenvs/arlo-Lrf6nvlW-py3.8/lib/python3.8/site-packages/sentry_sdk/integrations/wsgi.py", line 133, in __call__
    rv = self.app(
  File "/home/arsalan/.cache/pypoetry/virtualenvs/arlo-Lrf6nvlW-py3.8/lib/python3.8/site-packages/sentry_sdk/integrations/flask.py", line 88, in <lambda>
    return SentryWsgiMiddleware(lambda *a, **kw: old_app(self, *a, **kw))(
  File "/home/arsalan/.cache/pypoetry/virtualenvs/arlo-Lrf6nvlW-py3.8/lib/python3.8/site-packages/flask/app.py", line 2464, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/arsalan/.cache/pypoetry/virtualenvs/arlo-Lrf6nvlW-py3.8/lib/python3.8/site-packages/werkzeug/middleware/proxy_fix.py", line 169, in __call__
    return self.app(environ, start_response)
  File "/home/arsalan/.cache/pypoetry/virtualenvs/arlo-Lrf6nvlW-py3.8/lib/python3.8/site-packages/flask/app.py", line 2450, in wsgi_app
    response = self.handle_exception(e)
  File "/home/arsalan/.cache/pypoetry/virtualenvs/arlo-Lrf6nvlW-py3.8/lib/python3.8/site-packages/flask/app.py", line 1867, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/home/arsalan/.cache/pypoetry/virtualenvs/arlo-Lrf6nvlW-py3.8/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/home/arsalan/.cache/pypoetry/virtualenvs/arlo-Lrf6nvlW-py3.8/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/arsalan/.cache/pypoetry/virtualenvs/arlo-Lrf6nvlW-py3.8/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/arsalan/.cache/pypoetry/virtualenvs/arlo-Lrf6nvlW-py3.8/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/arsalan/.cache/pypoetry/virtualenvs/arlo-Lrf6nvlW-py3.8/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/home/arsalan/.cache/pypoetry/virtualenvs/arlo-Lrf6nvlW-py3.8/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/arsalan/.cache/pypoetry/virtualenvs/arlo-Lrf6nvlW-py3.8/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/arsalan/Workspace/code/arlo/server/auth/auth_helpers.py", line 225, in wrapper
    return route(*args, **kwargs)
  File "/home/arsalan/Workspace/code/arlo/server/api/jurisdictions.py", line 609, in list_jurisdictions
    json_jurisdictions = [
  File "/home/arsalan/Workspace/code/arlo/server/api/jurisdictions.py", line 610, in <listcomp>
    serialize_jurisdiction(election, jurisdiction, round_status[jurisdiction.id])
  File "/home/arsalan/Workspace/code/arlo/server/api/jurisdictions.py", line 179, in serialize_jurisdiction
    min_num_ballots = max(
  File "/home/arsalan/Workspace/code/arlo/server/api/jurisdictions.py", line 180, in <genexpr>
    min_num_ballots_for_contest(contest) for contest in contests
  File "/home/arsalan/Workspace/code/arlo/server/api/jurisdictions.py", line 164, in min_num_ballots_for_contest
    sum(
  File "/home/arsalan/Workspace/code/arlo/server/api/jurisdictions.py", line 165, in <genexpr>
    tally[contest.id][choice.id]
KeyError: '051dcd66-33ff-424f-bacc-6b9ac165edee'

This PR addresses this issue by reprocessing candidate-totals-by-batch files when contests are edited. While this reprocessing happens in the background, the initial clearing of jurisdiction.batch_tallies fields happens synchronously.

Testing

• Added automated tests
• Tested manually

[arsalansufi]

We're already reprocessing candidate-totals-by-batch files when ballot manifests are updated. This PR just moves that reprocessing logic to a shared location and applies it when updating contests as well

[arsalansufi]

I'm gonna go ahead and merge this PR but have opened an issue to track the addition of a warning message for audit admins, per discussion in Slack: #1897

Planning on getting to this before March batch audits