Introduce catalog + ndcg

[maximilianwerk]

This PR contains the following changes:

• add catalog to Labeler and tuner API
  • for Labeler: catalog performs best if it is a DocumentArrayMemmap, since no copying of data is necessary.
  • for tuner: no special requirements of the datatype
• changed how the frontend requests new Documents from the backend. The decision, which Documents are shown next lies with the backend now.
• toydata (both QA and FMNIST) return now a data generator and a catalog. If pre_init_generator is set to False, it will return a callable, which will return the generator. The precalculation of the catalog takes a little longer than before. This makes test take longer.
• Tuner can now compute metrices. For now hits and NDCG is implemented.
• the result of Tuner.fit will now be a TunerStats object. This object allows easy printing and saving to file.
• fixed a whole lot of tests in order to respect the new interfaces.

TODO:

• docstrings

[maximilianwerk]

If someone wants to play around and hasn't the dataset locally: https://github.com/jina-ai/workshops/blob/main/pokedex/get_data.sh

[maximilianwerk]

@bwanglzu please check, if the implementation regarding GPU is correct and is the way to do. It finally passes the test, but I am not sure about standards.

[JoanFM]

I think u should never rely on match. I think u should have some reserved key in tags

[maximilianwerk]

For now this is our data structure in finetuner. That might change in the future.

[bwanglzu]

i see some logs using gpu, while others using cpu, looking into the code

[bwanglzu]

ignore the screenshot above, it's using gpu.

[tadejsv]

I have yet to review this in more detail, but I think that the design needs to be refactored. Namely:

• with this addition, we have two evaluation loops: one that mirrors the training one, and one for computing metrics. We should only have one, otherwise we are computing embeddings twice for no good reason.
• related, there should only be one evaluation dataset
• the metrics we compute should replace the ones from "old" evaluation" loop (i am removing them in my PR anyway) and be added to return dict, not just logged

[codecov]

The author of this PR, maximilianwerk, is not an activated member of this organization on Codecov.
Please activate this user on Codecov to display this PR comment.
Coverage data is still being uploaded to Codecov.io for purposes of overall coverage calculations.
Please don't hesitate to email us at success@codecov.io with any questions.

[maximilianwerk]

Beware, that passing a callable into the tuner.fit function does only provide any benefit, if the catalog is also passed.

[tadejsv]

I think the main problem of how we design evaluation/metrics remains. Right now, we first run the eval loop, which computes the embeddings, and then run metrics computation, which computes embeddings again (in a more efficient way). I think this can be refactored so that:

• embeddings are computed after training for the whole catalog
• both eval loop and metrics computation take these pre-computed embeddings.

Also, we need to add batching to embeddings computation for the catalog - with larger datasets the whole computation simply won't fit into memory.

Not sure if this is feasible before wednesday, but should be done soon

[tadejsv]

Suggested change

	embeddings = self.embed_model(tensor)
	with torch.inference_mode():
	with torch.inference_mode():
	embeddings = self.embed_model(tensor)

Will be done later.

[tadejsv]

Looks good 👍