[WIP] Improved robustness of concurrent schema updates

[brimoor]

As explained in the @todo, this is a bit of a hack to specifically avoid issues when concurrently modifying a dataset's schema. However, the underlying problem still exists whenever list fields other than DatasetDocument.sample_fields and DatasetDocument.frame_fields are concurrently edited without first reloading.

I think it makes sense to go ahead and merge this particular patch because schema updates are by far the most likely case where concurrent list edits may arise, since in many workflows dataset objects may be held in-memory for long periods of time without reloading them. Unlike samples, which are generally only loaded + modified on-demand.

[codecov]

Codecov Report

Patch and project coverage have no change.

Comparison is base (d0c3dff) 15.56% compared to head (4ad5b8f) 15.56%.

Additional details and impacted files

@@           Coverage Diff            @@
##           develop    #3186   +/-   ##
========================================
  Coverage    15.56%   15.56%           
========================================
  Files          564      564           
  Lines        69319    69319           
  Branches       681      681           
========================================
  Hits         10791    10791           
  Misses       58528    58528           

Flag	Coverage Δ
app	15.56% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[allenleetc]

Still orienting but LGTM. One question -- theoretically race conditions are still possible as the dataset doc is not locked between reload() and the subsequent schema update (not sure exactly where that happens)?

[brimoor]

Still orienting but LGTM. One question -- theoretically race conditions are still possible as the dataset doc is not locked between reload() and the subsequent schema update (not sure exactly where that happens)?

yes theoretically there still could be issues if two processes update the schema within the few lines of code between the reload() I added here and when the db transaction has happened. However, this is substantially better than status quo, where this issue can happen if any concurrent updates happen between the time when the dataset is loaded in one process and when a schema change is made (which could be hours or days).

I'd like us to properly address this issue as per #3187; ;this is just attempting to place a bandaid on the most likely case.

stale

[brimoor]

Closing in favor of #3308.