Skip to content

Latest commit

 

History

History
147 lines (113 loc) · 4.24 KB

evaluation.md

File metadata and controls

147 lines (113 loc) · 4.24 KB
Raw

Evaluating Trackers

Results format

The TAO toolkit expects results in the same format as COCO, but with additional track_id and video_id fields. Specifically, results.json should have the following format:

[{
    "image_id" : int,
    "category_id" : int,
    "bbox" : [x,y,width,height],
    "score" : float,
    "track_id": int,
    "video_id": int
}]

Evaluation (toolkit)

The TAO toolkit provides code for evaluating tracker results.

import logging
from tao.toolkit.tao import TaoEval

# TAO uses logging to print results. Make sure logging is set to show INFO
# messages, or you won't see any evaluation results.
logging.setLevel(logging.INFO)
tao_eval = TaoEval('/path/to/annotations.json', '/path/to/results.json')
tao_eval.run()
tao_eval.print_results()

Evaluation (command-line)

TAO also comes with a higher-level evaluate.py script which incorporates various additional features for evaluation.

In all the examples below, let -

  • $ANNOTATIONS be the /path/to/annotations.json
  • $RESULTS be the /path/to/results.json
  • $OUTPUT_DIR be the /path/to/output/logdir.

We demonstrate some features below; for more, take a look at the config description in ./tao/utils/evaluation.py.

  • Simple evaluation, with logging to an output directory

    python scripts/evaluation/evaluate.py \
    $ANNOTATIONS $RESULTS --output-dir $OUTPUT_DIR \
  • Classification oracle

    python scripts/evaluation/evaluate.py \
    $ANNOTATIONS $RESULTS --output-dir $OUTPUT_DIR \
    --config-updates ORACLE.TYPE class

  • Track oracle (for linking detections)

    python scripts/evaluation/evaluate.py \
    $ANNOTATIONS $RESULTS --output-dir $OUTPUT_DIR \
    --config-updates ORACLE.TYPE track

  • Evaluate MOTA

    python scripts/evaluation/evaluate.py \
    $ANNOTATIONS $RESULTS --output-dir $OUTPUT_DIR \
    --config-updates MOTA.ENABLED True

  • Evaluate at (3D) IoU threshold of 0.9

    python scripts/evaluation/evaluate.py \
    $ANNOTATIONS $RESULTS --output-dir $OUTPUT_DIR \
    --config-updates EVAL_IOUS "[0.9]"

  • Evaluate at multiple (3D) IoU thresholds

    python scripts/evaluation/evaluate.py \
    $ANNOTATIONS $RESULTS --output-dir $OUTPUT_DIR \
    --config-updates \
        EVAL_IOUS "[0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95]"

  • Category agnostic evaluation

    python scripts/evaluation/evaluate.py \
    $ANNOTATIONS $RESULTS --output-dir $OUTPUT_DIR \
    --config-updates CATEGORY_AGNOSTIC True

  • Report evaluation by source dataset

    python scripts/evaluation/evaluate.py \
    $ANNOTATIONS $RESULTS --output-dir $OUTPUT_DIR \
    --config-updates EVAL_BY_DATASET True

Evaluation (challenge server)

For local evaluation, evaluate with steps above on the released validation set. When submitting test set results to the challenge server, follow same format for json files as mentioned above.

The server requires you to submit train, validation and test set results. We request you to submit these three json files for facilitating progress in the tracking community. However, if absolutely necessary, submit empty json files for train and validation. Create a .zip archive that deflates into the following files

./TAO_test.json
./TAO_train.json
./TAO_val.json

More details

Merged classes

A few classes from LVIS (v0.5) are merged in TAO, as they are nearly-synonymous. As such, methods are given credit for predicting any one of the merged classes. These classes are marked in the annotations, with a merged key in the category dictionary. The following code constructs the list of merged classes from the annotations file: https://github.com/TAO-Dataset/tao/blob/63d04f3b62bd0656614902206bd5e0d1e801dc26/tao/toolkit/tao/tao.py#L96-L104