Seg Array Optimization

[CPBridge]

A few improvements focused on improving the efficiency of building segmentation arrays from loaded Segmentation objects.

This was achieved through:

• Re-ordering loops in the construction of the seg_frames_matrix to avoid searching through the large _frames_lut more than necessary.
• Creating a specialized case for the combine_segments option, rather than implementing it as a postprocessing of the standard routine. This reduces memory consumption significantly as it is now O(1) rather than O(n) in the number of segments requested.
• Being more careful with dtypes, and allowing the user to specify a dtype
• Adding an option to skip checks for overlapping segments (skip_overlap_checks), which results in a further modest speed-up.

@pieper you should now find that your example from #202 runs significantly faster and uses much less memory than before when combine_segments=True (especially in the case where the full set of segments is requested).

I went a little way down the path of JITing the methods with numba, but it actually slowed things down. There may be a bit more performance improvement possibly using threading, but we are hitting on some numpy limitations

[pieper]

Thanks @CPBridge ! Let me know if you need me to test before you merge or i can just wait until @hackermd has had a look through it.

[CPBridge]

Thanks Steve. If you do have time to test on your hardware at some point during the process that would be very helpful. I think before or after @hackermd has a look would be fine, I don't expect the core numerical process to change much

[CPBridge]

A couple of other points to note:

• It will still be slow to load the file initially, that will require a fix in pydicom.
• Please try with and without the skip_overlap_checks option and see how this affects things

[hackermd]

Thanks @CPBridge. I will take a look.

It will still be slow to load the file initially, that will require a fix in pydicom

See pydicom/pydicom#1734

[hackermd]

A few suggestions regarding style.

[CPBridge]

@hackermd I have refactored out all the database logic and I think this is ready to go

[CPBridge]

@hackermd I have addressed the various comments, mostly just accepting all suggestions. I have attempted to improve the docstrings for the _SegDBManager class