2002 async deletion

[abhidg]

This doesn't have the workflow/Batch action to actually periodically run, will add it once this is merged and list is set to True on prod/dev.

• Remove markPendingRemoval endpoints and rename pendingRemoval->list
• Only export cases with list = True
• data: Update endpoints to only return cases with list=True Migrate to asynchronous deletion for non-UUID sources #2002
• Add script to mark uploads periodically Migrate to asynchronous deletion for non-UUID sources #2002

[codecov-commenter]

Codecov Report

Merging #2004 (5a7e13d) into main (cf2689b) will decrease coverage by 8.67%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main    #2004      +/-   ##
==========================================
- Coverage   75.42%   66.74%   -8.68%     
==========================================
  Files          24      129     +105     
  Lines         940     4421    +3481     
  Branches      150     1181    +1031     
==========================================
+ Hits          709     2951    +2242     
- Misses        231     1470    +1239     

Impacted Files	Coverage Δ
...rving/data-service/src/controllers/preprocessor.ts	95.71% <ø> (ø)
data-serving/data-service/src/index.ts	95.45% <ø> (ø)
data-serving/data-service/src/model/case.ts	100.00% <ø> (ø)
verification/curator-service/api/src/index.ts	88.52% <ø> (-0.28%)	⬇️
...curator-service/ui/src/components/BulkCaseForm.tsx	10.21% <ø> (ø)
...ion/curator-service/ui/src/components/CaseForm.tsx	30.55% <ø> (ø)
data-serving/data-service/src/controllers/case.ts	82.02% <100.00%> (ø)
...ion/curator-service/api/src/controllers/sources.ts	65.94% <0.00%> (-16.22%)	⬇️
...rving/data-service/src/geocoding/remoteGeocoder.ts	100.00% <0.00%> (ø)
...n/curator-service/ui/src/components/TermsOfUse.tsx	25.00% <0.00%> (ø)
... and 103 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cf2689b...5a7e13d. Read the comment docs.

[iamleeg]

don't we want to use poetry everywhere now?

[iamleeg]

(besides which, it'd be good to pin the version)

[abhidg]

I prefer requirements.txt for simple one liner scripts with few dependencies, can change to poetry if we want to maintain consistency. Advantage of requirements.txt is the setup code is one-liner unlike poetry which uses a curl | python style of installation.

[iamleeg]

It doesn't look like there's any default value for list supplied anywhere in the API layer. Should there be? I'm guessing it should be false (or the type should be bool?). Then no cases are displayed until the periodic script has fixed up the values by connecting them to the correct uploads. And when we land this, some manual step will need to list:true all the existing cases.

Is that correct?

[abhidg]

It doesn't look like there's any default value for list supplied anywhere in the API layer. Should there be? I'm guessing it should be false (or the type should be bool?).

Currently there is no default value. Only list=true is considered true; this avoids changing a lot of code which has no knowledge of the list parameter; with the idea being that prune_uploads orchestrates the line list marking by itself. The curator UI has been changed to use list=true when adding or uploading cases. I couldn't see any optional type marker for the current fields, it seems that is set using required: true in the schema.

Then no cases are displayed until the periodic script has fixed up the values by connecting them to the correct uploads. And when we land this, some manual step will need to list:true all the existing cases.

Yes, tracking this in #2002