Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

aggregate checkpoint information in a summary file #150

Merged
merged 6 commits into from
Nov 14, 2018

Conversation

jchiang87
Copy link
Collaborator

This code maintains an sqlite db of checkpoint progress that's separate from the individual checkpoint pickle files so that the image simulation status of many subprocesses running in parallel on a node can be extracted from a single source. This uses multiprocessing.Pipe connections to send the checkpoint summary info from the single sensor pool workers to the aggregator function that runs asynchronously in its own processing pool.

@villarrealas
Copy link

Looks like this above and beyond what I was hoping for! Just to confirm a couple of things to make sure I am reading correctly though...

  1. This uses its own processing pool inside the imSim instance. In practice, does this mean the # of hardware threads used is now # sensors + 1 for sqlite syncing?

  2. Just confirming that the resulting file outputs to the same directory as the checkpoints with the same naming convention (but with .sqlite format) and contains entries of # objects drawn, # objects total, and the detector name? Is this detector name in the imSim convention that we make calls currently, or is that going to be changing?

@villarrealas
Copy link

I have also done my best to look over the code itself and it looks solid to me. That being said, I don't do the most object oriented programming, so Tom should probably give a second look and see if something jumps out to him.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants