Skip to content

Data feeding restructure #69

@Psy-Fer

Description

@Psy-Fer

with the changes in how dorado-server processes batches, it is causing performance issues.

I need to change the current static pipeline of

1. get batch
2. mutate batch into compatible structure
3. send that to basecaller
4. wait for it to be basecalled
5. get the reads back from the basecaller and process them
6. send to the writter queue.

to something like

1. get batch
2. mutate to compatible structure
3. send to basecaller
4. check if reads are ready, if so, process them, otherwise repeat 1-3
5. if a batch comes back, process it and send to writter
6. have a max number of batches "in flight" limit for 1-3

This should fix the batch waiting shinanigans caused by the new "optimisations" caused by ONT.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions