LabelBOT

[gkourie]

TODO

Backend

• Adjust request format for storing image annotations: The store method in ImageAnnotationController requires a label_id in the request, but when LabelBOT is active, a request with a feature vector should be accepted without label_id.
• Attach additional labels to the response: Since LabelBOT may suggest multiple labels, these labels should be included in the response appropriately.
• Check if HNSW index exists before running ANN if not, switch to KNN.
• Temporarily drop the HNSW index and rollback the transaction: When switching to exact KNN, the HNSW index must be temporarily dropped, and the transaction should be rolled back.
• Handle the case when LabelBOT returns no result because there are no annotations are associated with any labels of the project.

Frontend

• On/Off button for LabelBOT.
• LabelBOT can't be activated if:
  • neither of the label trees contains any labels.
  • a label is selected
  • or maybe deselect the selected label automatically when LabelBOT is activated?
  • ..
• LabelBOT is active:
  • Labels can't be selected.
  • refreshing the page deselects the selected label, should LabelBOT be turned off when refreshing?
  • ..
• LabelBOT logic:
  • allow drawing without selecting a label.
  • use own canvas to extract the correct region of interest (without color adjustment, etc..).
  • point to shape conversion with SAM or PTP.
  • cache onnx model (see onnx cache)
  • ..
• LabelBOT indicator:
  • yellow: initializing.
  • green: ready.
  • blue: computing.
• Top-3 labels popup:
  • progress bar for first label selection.
  • typeahead to select other label.
  • deleting the top 1 label annotation should close the popup immediately.
  • draggable popup around the annotation for better visibility when multiple popups are close to each other.
• ONNX URL & new API Endpoint

[mzur]

Please don't implement all the logic as helper methods. Since the logic is used only in the single controller (for now), implement it as (protected) controller methods there.

Also please us camelCase for variable names.

Finally, please fix the failing checks (except the test-current check which fails because biigle/largo is not there). The lint checks can be run locally with composer lint and composer fix.

[mzur]

Just minor comments left now. We have to make sure that we get the exact queries here. Maybe you can dump the two queries as I explained below and post them here for comparison?

[mzur]

Looks fine so far (ignoring the missing TODOs). Please go ahead 😉

[gkourie]

okay, thanks.

[mzur]

Where do I get the model file, i.e. which did you use? This should be mentioned in the config file (see below).

I will review the UI once it is implemented according to the mockup in #988. If you want feedback before that, we should meet and you can show me what you currently have.

We also need a migration file for the HNSW index. Maybe this migration could also implement a progress bar that shows the progress of the index creation? This also has to be mentioned in the upgrade instructions of the release. We should prepare a text for that. See here for an example of upgrade instructions.

[mzur]

Add a URL where to get the file. We also need to mention this (i.e. admins have to put the model manually somewhere) in the upgrade notes of the new BIIGLE version.

[mzur]

A note about the model file. We can release the code to generate the ONNX in this repository. The actual ONNX file can be published as a release asset. The DINOv2 model weights were released under an Apache-2.0 license. We have to add the license reference and link to the original model file to the respective BIIGLE release. Example text:

The DINOv2 ONNX file was derived from the ViT-S/14 distilled (?) model weights, which are licensed under an Apache-2.0 license (© 2023 Meta AI Research, FAIR). The code to generate the ONNX file can be found here.