OmniLabelTools

A Python toolkit for the OmniLabel benchmark (https://www.omnilabel.org)

Main features:

• evaluation of prediction results
• visualization of ground truth and predictions
• extract basic statistics of the dataset annotations

Install | Dataset setup | Annotation format | Evaluate your results | License

Install

Install OmniLabelTools as:

git clone https://www.github.com/samschulter/omnilabeltools
cd omnilabeltools
pip install .

You can also install in developer mode:

pip install -e .

Dataset setup

Please visit https://www.omnilabel.org/dataset/download for download and setup instructions. To verify the dataset setup, you can run the following two scripts to print some basic dataset statistics and visualize some examples:

olstats --path-to-json path/to/dataset/gt/json

olvis --path-to-json path/to/dataset/gt/json --path-to-imgs path/to/image/directories --path-output some/directory/to/store/visualizations

Annotation format

In general, we try to follow the MS COCO dataset format as much as possible, with all annotations stored in one json file. Please see https://www.omnilabel.org/dataset/download and https://www.omnilabel.org/task for more details.

Ground truth data

{
    images: [
        {
            id              ... unique image ID
            file_name       ... path to image, relative to a given base directory (see above)
        },
        ...
    ],
    descriptions: [
        {
            id              ... unique description ID
            text            ... the text of the object description
            image_ids       ... list of image IDs for which this description is part of the label space
            anno_info       ... some metadata about the description
        },
        ...
    ],
    annotations: [        # Only for val sets. Not given in test set annotations!
        {
            id              ... unique annotation ID
            image_id        ... the image id this annotation belongs to
            bbox            ... the bounding box coordinates of the object (x,y,w,h)
            description_ids ... list of description IDs that refer to this object
	    },
        ...
    ]
}

Submitting prediction results

NB: The test server is not online at this time. Once online, prediction results are submitted in the following format:

[
    {
        image_id        ... the image id this predicted box belongs to
        bbox            ... the bounding box coordinates of the object (x,y,w,h)
        description_ids ... list of description IDs that refer to this object
        scores          ... list of confidences, one for each description
    },
    ...
]

Evaluate your results

Here is some example code how to evaluate results:

from omnilabeltools import OmniLabel, OmniLabelEval

gt = OmniLabel(data_json_path)              # load ground truth dataset
dt = gt.load_res(res_json_path)             # load prediction results
ole = OmniLabelEval(gt, dt)
ole.params.recThrs = ...                    # set evaluation parameters as desired
ole.evaluate()
ole.accumulate()
ole.summarize()

We also provide a stand-alone script:

oleval --path-to-gt path/to/gt/json --path-to-res path/to/result/json

The result json file follows the format described above.

License

This project is released under an MIT License.