Advanced Tutorial
- Overview
-
Creating a Benchmark by Editor
- Step 1: Click on Management in the top right corner of Codabench's home page under Competitions.
- Step 2: Click on the Create button in the top right corner of Competition Management.
- Step 3: Fill in the Details tab content.
- Step 4: Fill in the Participant Tab.
- Step 5: Fill in the Pages Tab.
- Step 6: Fill in the Phases Tab.
- Step 7: Fill in the Leaderboard Tab.
- Step 8: Save and Publish the Benchmark
- Creating a Benchmark by Bundle
- Benchmark Examples
- How do I set up submission comments for multiple submissions?
Here is an advanced tutorial. If you are new to CodaBench, please refer to get started tutorial first. In this article, you'll learn how to use more advanced features and how to create benchmarks using either the editor or bundles. Before proceeding to our tutorial, make sure you have registered for an account on the Codabench website.
- Creating a Benchmark by Editor
- Creating a Benchmark by Bundle
- Benchmark Examples
- How do I set up submission comments for multiple submissions?
The image below is an overview of the benchmark creation process
In this chapter, I'll take you step by step through the Editor's approach to creating benchmark, including algorithm type and dataset type.
When you click on it, you will see the screen as shown in the screenshot below.
When you click on the Manage Tasks/Datasets button, you will see the screenshot shown below
Click the Add Dataset button in the diagram to upload the resource files needed to create the competition.
Creating a phase will require a bundle of the following types (.zip format), I'll give you more details on how to write these bundle later.
Now you just need to have this concept in your mind
Here are the screenshots of the 5 types of bundles after they were uploaded
Creating a benchmark through bundles is a much more efficient way than using editors.
Click on the link above to download the bundle in the screenshot.
After the bundle has been uploaded, you will see the screenshot shown below.
Example bundles for code & dataset competition can be found here:
https://github.com/codalab/competition-examples/tree/master/codabench
Iris Codabench Bundle is a simple benchmark involving two phases, code submission and results submission.
Two versions of the Automated Weakly Supervised Learning Benchmark:
Mini-AutoML Bundle is a benchmark template for Codabench, featuring code submission to multiple datasets (tasks).
