Competitions feature

docs/api/competition.dataset.md

@@ -0,0 +1,5 @@
+::: polaris.dataset.CompetitionDataset
+    options:
+        filters: ["!^_"]
+
+---

docs/api/competition.evaluation.md

@@ -0,0 +1,7 @@
+::: polaris.evaluate.CompetitionPredictions
+
+---
+
+::: polaris.evaluate.CompetitionResults
+
+---

docs/api/competition.md

@@ -0,0 +1,3 @@
+::: polaris.competition.CompetitionSpecification
+
+---

docs/api/evaluation.md

@@ -1,3 +1,12 @@
+::: polaris.evaluate.ResultsMetadata
+    options:
+        filters: ["!^_"]
+
+---
+
+::: polaris.evaluate.EvaluationResult
+
+---
 
 ::: polaris.evaluate.BenchmarkResults
 

docs/tutorials/competition.participate.ipynb

@@ -0,0 +1,253 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "40f99374-b47e-4f84-bdb9-148a11f9c07d",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "# Participating in a Competition\n",
+    "\n",
+    "<div class=\"admonition abstract highlight\">\n",
+    "    <p class=\"admonition-title\">In short</p>\n",
+    "    <p>This tutorial walks you through how to fetch an active competition from Polaris, prepare your predictions and then submit them for secure evaluation by the Polaris Hub.</p>\n",
+    "</div>\n",
+    "\n",
+    "Participating in a competition on Polaris is very similar to participating in a standard benchmark. The main difference lies in how predictions are prepared and how they are evaluated. We'll touch on each of these topics later in the tutorial. \n",
+    "\n",
+    "Before continuing, please ensure you are logged into Polaris."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "3d66f466",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "remove_cell"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# Note: Cell is tagged to not show up in the mkdocs build\n",
+    "%load_ext autoreload\n",
+    "%autoreload 2"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "id": "9b465ea4-7c71-443b-9908-3f9e567ee4c4",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "\u001b[32m2024-08-09 18:05:23.205\u001b[0m | \u001b[32m\u001b[1mSUCCESS \u001b[0m | \u001b[36mpolaris.hub.client\u001b[0m:\u001b[36mlogin\u001b[0m:\u001b[36m267\u001b[0m - \u001b[32m\u001b[1mYou are successfully logged in to the Polaris Hub.\u001b[0m\n"
+     ]
+    }
+   ],
+   "source": [
+    "import polaris as po\n",
+    "from polaris.hub.client import PolarisHubClient\n",
+    "\n",
+    "# Don't forget to add your Polaris Hub username below!\n",
+    "MY_POLARIS_USERNAME = \"\"\n",
+    "\n",
+    "client = PolarisHubClient()\n",
+    "client.login()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5edee39f-ce29-4ae6-91ce-453d9190541b",
+   "metadata": {},
+   "source": [
+    "## Fetching a Competition\n",
+    "\n",
+    "As with standard benchmarks, Polaris provides simple APIs that allow you to quickly fetch a competition from the Polaris Hub. All you need is the unique identifier for the competition which follows the format of `competition_owner`/`competition_name`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "4e004589-6c48-4232-b353-b1700536dde6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "competition_id = \"polaris/hello-world-competition\"\n",
+    "competition = po.load_competition(competition_id)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "36f3e829",
+   "metadata": {},
+   "source": [
+    "## Participate in the Competition\n",
+    "The Polaris library is designed to make it easy to participate in a competition. In just a few lines of code, we can get the train and test partition, access the associated data in various ways and evaluate our predictions. There's two main API endpoints. \n",
+    "\n",
+    "- `get_train_test_split()`: For creating objects through which we can access the different dataset partitions.\n",
+    "- `evaluate()`: For evaluating a set of predictions in accordance with the competition protocol."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d8605928",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "train, test = competition.get_train_test_split()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e78bf878",
+   "metadata": {},
+   "source": [
+    "The created test and train objects support various flavours to access the data."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7b17bb31",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# The objects are iterable\n",
+    "for x, y in train:\n",
+    "    pass\n",
+    "\n",
+    "# The objects can be indexed\n",
+    "for i in range(len(train)):\n",
+    "    x, y = train[i]\n",
+    "\n",
+    "# The objects have properties to access all data at once\n",
+    "x = train.inputs\n",
+    "y = train.targets"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5ec12825",
+   "metadata": {},
+   "source": [
+    "Now, let's create some predictions against the official Polaris `hello-world-competition`. We will train a simple random forest model on the ECFP representation through scikit-learn and datamol, and then we will submit our results for secure evaluation by the Polaris Hub."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "902353bc",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import datamol as dm\n",
+    "from sklearn.ensemble import RandomForestRegressor\n",
+    "\n",
+    "# Load the competition (automatically loads the underlying dataset as well)\n",
+    "competition = po.load_competition(\"polaris/hello-world-benchmark\")\n",
+    "\n",
+    "# Get the split and convert SMILES to ECFP fingerprints by specifying an featurize function.\n",
+    "train, test = competition.get_train_test_split(featurization_fn=dm.to_fp)\n",
+    "\n",
+    "# Define a model and train\n",
+    "model = RandomForestRegressor(max_depth=2, random_state=0)\n",
+    "model.fit(train.X, train.y)\n",
+    "\n",
+    "predictions = model.predict(test.X)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1a36e334",
+   "metadata": {},
+   "source": [
+    "Now that we have created some predictions, we can construct a `CompetitionPredictions` object that will prepare our predictions for evaluation by the Polaris Hub. Here, you can also add metadata to your predictions to better describe your results and how you achieved them. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "2b36e09b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from polaris.evaluate import CompetitionPredictions\n",
+    "\n",
+    "competition_predictions = CompetitionPredictions(\n",
+    "    name=\"hello-world-result\",\n",
+    "    predictions=predictions,\n",
+    "    github_url=\"https://github.com/polaris-hub/polaris-hub\",\n",
+    "    paper_url=\"https://polarishub.io/\",\n",
+    "    description=\"Hello, World!\",\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5ff06a9c",
+   "metadata": {},
+   "source": [
+    "Once your `CompetitionPredictions` object is created, you're ready to submit them for evaluation! This will automatically save your result to the Polaris Hub, but it will be private. You can choose to make it public through the Polaris web application. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e684c611",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "results = competition.evaluate(competition_predictions)\n",
+    "\n",
+    "client.close()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "44973556",
+   "metadata": {},
+   "source": [
+    "That's it! Just like that you have partaken in your first Polaris competition. Keep an eye on that leaderboard and best of luck in your future competitions!\n",
+    "\n",
+    "The End.\n",
+    "\n",
+    "---"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.2"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/tutorials/custom_dataset_benchmark.ipynb

@@ -393,7 +393,7 @@
    },
    "outputs": [],
    "source": [
-    "from polaris.hub.client import PolarisHubClient\n",
+    "# from polaris.hub.client import PolarisHubClient\n",
     "\n",
     "# NOTE: Commented out to not flood the DB\n",
     "# with PolarisHubClient() as client:\n",
@@ -491,11 +491,11 @@
      "evalue": "1 validation error for MultiTaskBenchmarkSpecification\ntarget_cols\n  Value error, A multi-task benchmark should specify at least two target columns [type=value_error, input_value='LOG SOLUBILITY PH 6.8 (ug/mL)', input_type=str]\n    For further information visit https://errors.pydantic.dev/2.4/v/value_error",
      "output_type": "error",
      "traceback": [
-      "\u001B[0;31m---------------------------------------------------------------------------\u001B[0m",
-      "\u001B[0;31mValidationError\u001B[0m                           Traceback (most recent call last)",
-      "\u001B[1;32m/Users/cas.wognum/Documents/repositories/polaris/docs/tutorials/custom_dataset_benchmark.ipynb Cell 25\u001B[0m line \u001B[0;36m3\n\u001B[1;32m      <a href='vscode-notebook-cell:/Users/cas.wognum/Documents/repositories/polaris/docs/tutorials/custom_dataset_benchmark.ipynb#X33sZmlsZQ%3D%3D?line=0'>1</a>\u001B[0m \u001B[39mfrom\u001B[39;00m \u001B[39mpolaris\u001B[39;00m\u001B[39m.\u001B[39;00m\u001B[39mbenchmark\u001B[39;00m \u001B[39mimport\u001B[39;00m MultiTaskBenchmarkSpecification\n\u001B[0;32m----> <a href='vscode-notebook-cell:/Users/cas.wognum/Documents/repositories/polaris/docs/tutorials/custom_dataset_benchmark.ipynb#X33sZmlsZQ%3D%3D?line=2'>3</a>\u001B[0m benchmark \u001B[39m=\u001B[39m MultiTaskBenchmarkSpecification(\n\u001B[1;32m      <a href='vscode-notebook-cell:/Users/cas.wognum/Documents/repositories/polaris/docs/tutorials/custom_dataset_benchmark.ipynb#X33sZmlsZQ%3D%3D?line=3'>4</a>\u001B[0m     dataset\u001B[39m=\u001B[39;49mdataset,\n\u001B[1;32m      <a href='vscode-notebook-cell:/Users/cas.wognum/Documents/repositories/polaris/docs/tutorials/custom_dataset_benchmark.ipynb#X33sZmlsZQ%3D%3D?line=4'>5</a>\u001B[0m     target_cols\u001B[39m=\u001B[39;49m\u001B[39m\"\u001B[39;49m\u001B[39mLOG SOLUBILITY PH 6.8 (ug/mL)\u001B[39;49m\u001B[39m\"\u001B[39;49m,\n\u001B[1;32m      <a href='vscode-notebook-cell:/Users/cas.wognum/Documents/repositories/polaris/docs/tutorials/custom_dataset_benchmark.ipynb#X33sZmlsZQ%3D%3D?line=5'>6</a>\u001B[0m     input_cols\u001B[39m=\u001B[39;49m\u001B[39m\"\u001B[39;49m\u001B[39mSMILES\u001B[39;49m\u001B[39m\"\u001B[39;49m,\n\u001B[1;32m      <a href='vscode-notebook-cell:/Users/cas.wognum/Documents/repositories/polaris/docs/tutorials/custom_dataset_benchmark.ipynb#X33sZmlsZQ%3D%3D?line=6'>7</a>\u001B[0m     split\u001B[39m=\u001B[39;49msplit,\n\u001B[1;32m      <a href='vscode-notebook-cell:/Users/cas.wognum/Documents/repositories/polaris/docs/tutorials/custom_dataset_benchmark.ipynb#X33sZmlsZQ%3D%3D?line=7'>8</a>\u001B[0m     metrics\u001B[39m=\u001B[39;49m\u001B[39m\"\u001B[39;49m\u001B[39mmean_absolute_error\u001B[39;49m\u001B[39m\"\u001B[39;49m,\n\u001B[1;32m      <a href='vscode-notebook-cell:/Users/cas.wognum/Documents/repositories/polaris/docs/tutorials/custom_dataset_benchmark.ipynb#X33sZmlsZQ%3D%3D?line=8'>9</a>\u001B[0m )\n",
-      "File \u001B[0;32m~/micromamba/envs/polaris/lib/python3.11/site-packages/pydantic/main.py:164\u001B[0m, in \u001B[0;36mBaseModel.__init__\u001B[0;34m(__pydantic_self__, **data)\u001B[0m\n\u001B[1;32m    162\u001B[0m \u001B[39m# `__tracebackhide__` tells pytest and some other tools to omit this function from tracebacks\u001B[39;00m\n\u001B[1;32m    163\u001B[0m __tracebackhide__ \u001B[39m=\u001B[39m \u001B[39mTrue\u001B[39;00m\n\u001B[0;32m--> 164\u001B[0m __pydantic_self__\u001B[39m.\u001B[39;49m__pydantic_validator__\u001B[39m.\u001B[39;49mvalidate_python(data, self_instance\u001B[39m=\u001B[39;49m__pydantic_self__)\n",
-      "\u001B[0;31mValidationError\u001B[0m: 1 validation error for MultiTaskBenchmarkSpecification\ntarget_cols\n  Value error, A multi-task benchmark should specify at least two target columns [type=value_error, input_value='LOG SOLUBILITY PH 6.8 (ug/mL)', input_type=str]\n    For further information visit https://errors.pydantic.dev/2.4/v/value_error"
+      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
+      "\u001b[0;31mValidationError\u001b[0m                           Traceback (most recent call last)",
+      "\u001b[1;32m/Users/cas.wognum/Documents/repositories/polaris/docs/tutorials/custom_dataset_benchmark.ipynb Cell 25\u001b[0m line \u001b[0;36m3\n\u001b[1;32m      <a href='vscode-notebook-cell:/Users/cas.wognum/Documents/repositories/polaris/docs/tutorials/custom_dataset_benchmark.ipynb#X33sZmlsZQ%3D%3D?line=0'>1</a>\u001b[0m \u001b[39mfrom\u001b[39;00m \u001b[39mpolaris\u001b[39;00m\u001b[39m.\u001b[39;00m\u001b[39mbenchmark\u001b[39;00m \u001b[39mimport\u001b[39;00m MultiTaskBenchmarkSpecification\n\u001b[0;32m----> <a href='vscode-notebook-cell:/Users/cas.wognum/Documents/repositories/polaris/docs/tutorials/custom_dataset_benchmark.ipynb#X33sZmlsZQ%3D%3D?line=2'>3</a>\u001b[0m benchmark \u001b[39m=\u001b[39m MultiTaskBenchmarkSpecification(\n\u001b[1;32m      <a href='vscode-notebook-cell:/Users/cas.wognum/Documents/repositories/polaris/docs/tutorials/custom_dataset_benchmark.ipynb#X33sZmlsZQ%3D%3D?line=3'>4</a>\u001b[0m     dataset\u001b[39m=\u001b[39;49mdataset,\n\u001b[1;32m      <a href='vscode-notebook-cell:/Users/cas.wognum/Documents/repositories/polaris/docs/tutorials/custom_dataset_benchmark.ipynb#X33sZmlsZQ%3D%3D?line=4'>5</a>\u001b[0m     target_cols\u001b[39m=\u001b[39;49m\u001b[39m\"\u001b[39;49m\u001b[39mLOG SOLUBILITY PH 6.8 (ug/mL)\u001b[39;49m\u001b[39m\"\u001b[39;49m,\n\u001b[1;32m      <a href='vscode-notebook-cell:/Users/cas.wognum/Documents/repositories/polaris/docs/tutorials/custom_dataset_benchmark.ipynb#X33sZmlsZQ%3D%3D?line=5'>6</a>\u001b[0m     input_cols\u001b[39m=\u001b[39;49m\u001b[39m\"\u001b[39;49m\u001b[39mSMILES\u001b[39;49m\u001b[39m\"\u001b[39;49m,\n\u001b[1;32m      <a href='vscode-notebook-cell:/Users/cas.wognum/Documents/repositories/polaris/docs/tutorials/custom_dataset_benchmark.ipynb#X33sZmlsZQ%3D%3D?line=6'>7</a>\u001b[0m     split\u001b[39m=\u001b[39;49msplit,\n\u001b[1;32m      <a href='vscode-notebook-cell:/Users/cas.wognum/Documents/repositories/polaris/docs/tutorials/custom_dataset_benchmark.ipynb#X33sZmlsZQ%3D%3D?line=7'>8</a>\u001b[0m     metrics\u001b[39m=\u001b[39;49m\u001b[39m\"\u001b[39;49m\u001b[39mmean_absolute_error\u001b[39;49m\u001b[39m\"\u001b[39;49m,\n\u001b[1;32m      <a href='vscode-notebook-cell:/Users/cas.wognum/Documents/repositories/polaris/docs/tutorials/custom_dataset_benchmark.ipynb#X33sZmlsZQ%3D%3D?line=8'>9</a>\u001b[0m )\n",
+      "File \u001b[0;32m~/micromamba/envs/polaris/lib/python3.11/site-packages/pydantic/main.py:164\u001b[0m, in \u001b[0;36mBaseModel.__init__\u001b[0;34m(__pydantic_self__, **data)\u001b[0m\n\u001b[1;32m    162\u001b[0m \u001b[39m# `__tracebackhide__` tells pytest and some other tools to omit this function from tracebacks\u001b[39;00m\n\u001b[1;32m    163\u001b[0m __tracebackhide__ \u001b[39m=\u001b[39m \u001b[39mTrue\u001b[39;00m\n\u001b[0;32m--> 164\u001b[0m __pydantic_self__\u001b[39m.\u001b[39;49m__pydantic_validator__\u001b[39m.\u001b[39;49mvalidate_python(data, self_instance\u001b[39m=\u001b[39;49m__pydantic_self__)\n",
+      "\u001b[0;31mValidationError\u001b[0m: 1 validation error for MultiTaskBenchmarkSpecification\ntarget_cols\n  Value error, A multi-task benchmark should specify at least two target columns [type=value_error, input_value='LOG SOLUBILITY PH 6.8 (ug/mL)', input_type=str]\n    For further information visit https://errors.pydantic.dev/2.4/v/value_error"
      ]
     }
    ],

docs/tutorials/optimization.ipynb

@@ -72,7 +72,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Let's create a dummy dataset with two columns \n",
+    "# Let's create a dummy dataset with two columns\n",
     "rng = np.random.default_rng(0)\n",
     "col_a = rng.choice(list(range(100)), 10000)\n",
     "col_b = rng.random(10000)\n",
@@ -195,10 +195,10 @@
     "import zarr\n",
     "from tempfile import mkdtemp\n",
     "\n",
-    "tmpdir =  mkdtemp()\n",
+    "tmpdir = mkdtemp()\n",
     "\n",
-    "# For the ones familiar with Zarr, this is not optimized at all. \n",
-    "# If you wouldn't want to convert to NumPy, you would want to \n",
+    "# For the ones familiar with Zarr, this is not optimized at all.\n",
+    "# If you wouldn't want to convert to NumPy, you would want to\n",
     "# optimize the chunking / compression.\n",
     "\n",
     "path = os.path.join(tmpdir, \"data.zarr\")\n",
@@ -276,7 +276,7 @@
    ],
    "source": [
     "%%timeit\n",
-    "for batch in dataloader: \n",
+    "for batch in dataloader:\n",
     "    pass"
    ]
   },
@@ -314,7 +314,7 @@
    ],
    "source": [
     "%%timeit\n",
-    "for batch in dataloader: \n",
+    "for batch in dataloader:\n",
     "    pass"
    ]
   },
@@ -336,6 +336,7 @@
    "outputs": [],
    "source": [
     "from shutil import rmtree\n",
+    "\n",
     "rmtree(tmpdir)"
    ]
   },

mkdocs.yml

@@ -23,13 +23,19 @@ nav:
           - Zarr Datasets: tutorials/dataset_zarr.ipynb
           - Dataset Factory: tutorials/dataset_factory.ipynb
           - Optimization: tutorials/optimization.ipynb
+      - Competitions:
+          - tutorials/competition.participate.ipynb
   - API Reference:
       - Load: api/load.md
       - Core:
           - Dataset: api/dataset.md
           - Benchmark: api/benchmark.md
           - Subset: api/subset.md
           - Evaluation: api/evaluation.md
+      - Competitions:
+          - Competition Dataset: api/competition.dataset.md
+          - Competition: api/competition.md
+          - Competiton Evaluation: api/competition.evaluation.md
       - Hub:
           - Client: api/hub.client.md
           - PolarisFileSystem: api/hub.polarisfs.md

polaris/__init__.py

@@ -4,9 +4,9 @@
 from loguru import logger
 
 from ._version import __version__
-from .loader import load_benchmark, load_dataset
+from .loader import load_benchmark, load_dataset, load_competition
 
-__all__ = ["load_dataset", "load_benchmark", "__version__"]
+__all__ = ["load_dataset", "load_benchmark", "__version__", "load_competition"]
 
 # Configure the default logging level
 os.environ["LOGURU_LEVEL"] = os.environ.get("LOGURU_LEVEL", "INFO")