Competitions feature

[kirahowe]

Changelogs

This branch contains all of the changes to Polaris client required to support competitions in the hub, including:

• uploading new competition datasets and benchmarks to the hub
• retrieving competition datasets and benchmarks
• evaluating competition benchmarks

Some key notes about the structure of the whole system (i.e. including Polaris hub and supporting services):

• Competition datasets and competition benchmarks are new entities in the system. They behave very similarly to current datasets and benchmarks except for how results are evaluated.
• Evaluation logic is shared with regular benchmarks.
• Preparing competitions (splitting the target labels into a separate dataset) is done in a separate, standalone service.
• Evaluating competition benchmarks is done in a separate, standalone service.

Corresponding changes to the hub and the implementation of these new external services will be discussed in separate PRs in the relevant repos.

---

Checklist:

• Was this PR discussed in an issue? It is recommended to first discuss a new feature into a GitHub issue before opening a PR.
• Add tests to cover the fixed bug(s) or the newly introduced feature(s) (if appropriate).
• Update the API documentation if a new function is added, or an existing one is deleted.
• Write concise and explanatory changelogs above.
• If possible, assign one of the following labels to the PR: feature, fix or test (or ask a maintainer to do it for you).

---

Discussion

The original design proposal is discussed in this doc, with further details in this initial project proposal doc. This implementation closely follows the agreed-upon design in the first linked doc.

Briefly, this feature allows users to launch a competition on Polaris Hub, which are essentially regular benchmarks with hidden test labels. Predictions and evaluation results associated with a particular competition can be uploaded to the hub for a given period of time, after which the competition benchmark labels are made available and the dataset is converted to a "regular" dataset on Polaris Hub. A typical user workflow would involve uploading a dataset to polaris hub, launching the competition, and then collecting results.

Competition evaluation logic is shared with regular benchmarks, and there has been some refactoring to facilitate re-using this code. Note: Currently competition benchmarks accept only predictions, not probabilities. Support for probabilities was adding part way through the development of the competitions feature and will need to be implemented in the evaluation service before competitions can support probabilities.

Other large and potentially interesting changes to discuss are highlighted below in comments in the relevant context.

Deployment

We will need to coordinate deployment of these changes to make sure the the corresponding changes to the hub and other services are available before we release a version of the client containing these changes. No existing functionality should break, but the newly available competition-related APIs will not work until the updated hub and other services are live.

[jstlaurent]

I'll let Cas comment on the domain-specific changes, he's better placed to evaluate. It looks pretty from my part. I did have some questions and comments. I have one concern in particular where the competition upload user experience might not be great.

[cwognum]

Monumental work and a long time coming! Thank you @Andrewq11 and @kiramclean ! 🚀

[cwognum]

Thanks once more @Andrewq11 and @kiramclean ! 🚀

I'm just left with some minor comments and question. The main difficulty may come from the checksum. Since I made those changes to the checksum, I'm of course happy to help if so! 😄

I did break my head over the predictions type in reviewing this. This is becoming increasingly complex and having a single interface where the predictions are standardized would give me peace of mind. In another PR though!

[jstlaurent]

This is good stuff, very nice! A few things here and there, nothing major. I do think with the multiplication of artifact types, we should start to centralize their relevant attributes in one place, rather than having conditionals all over the code.