Large data mode needs more granular checkpointing during clustering

[jason-c-kwan]

This issue is to record Brian Couger's request that large data mode should checkpoint during clustering so that it won't waste too much time if the pipeline has to be resumed in a HPC environment. Below are the notes I took during the recent Autometa strategy meeting, going into a potential way of implementing.

We start with a set of datapoints that is to be clustered.

1. Cycle through EPS values
2. Decide upon the “best” EPS value
3. Take out “good” bins
4. Go back to 1 (until no more “good” bins are yielded).

Checkpointing:

Every time 4 finishes, record good bins somewhere, and note which contigs are left.
Also delete all EPS tables when 4 finishes
For resume - just start 1 at the contigs you have left

Every time 1 finishes, record a table of the groupings for a given EPS value, remember which EPS value is “next”.
For resume: Note that tables already exist, read them into memory and start clustering algorithm at the next EPS value.

[evanroyrees]

The checkpointing behavior you have outlined above is not quite the same as the checkpointing behavior implemented in the linked PR, but the linked PR behavior will allow resuming from where binning stopped (at the taxon iteration level).