Check for missing fovs after run is complete

[camisowers]

If you haven't already, please read through our contributing guidelines before opening your PR

What is the purpose of this PR?

Closes #399. Checks that bin/json files are generated for all FOVs outlined in the run file.

Also addresses the watcher edge case where a run crashes or does not output the bin/json files of one of the fovs in the run file. Currently the watcher notebook will hang forever and ever, since our current FOV timeout system is based on how long it takes for the full bin file to be completely generated (only triggered once it is written with size zero).

How did you implement your changes

Create a helper function to add to notebook 3f and also implement a run callback in watcher.

Add parallel function file_timer() that checks the time elapsed since the last FOV file was written. If the elapsed time is greater than 3x the normal FOV timeout period, the watcher function will call the run callbacks and then exit.

Remaining issues

• add error check in watcher tests
• add notebook implementations

[camisowers]

@alex-l-kong Before I implement any more watcher testing, will you take a look? I just realized that with the watcher now checking for completion based on the FOVs provided in the json file, this new callback might not ever raise an error. Is there a scenario where the run callbacks will get called even when a FOV was skipped?

[alex-l-kong]

missing_fov_check should still error out properly since the watcher also marks FOVs that time out as completed, however, these FOVs won't have any extracted data. The function does need to account for one additional edge case.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[srivarra]

Looks pretty good, just a clarification question.

[alex-l-kong]

Minor detail about the missing FOV timeout print statement.

[ngreenwald]

Looks good