Skip to content

Latest commit

 

History

History
executable file
·
54 lines (43 loc) · 2.22 KB

README.md

File metadata and controls

executable file
·
54 lines (43 loc) · 2.22 KB

QVHighlights Evaluation and Codalab Submission

Task Definition

Given a video and a natural language query, our task requires a system to retrieve the most relevant moments in the video, and detect the highlightness of the clips in the video.

Evaluation

At project root, run

bash standalone_eval/eval_sample.sh 

This command will use eval.py to evaluate the provided prediction file sample_val_preds.jsonl, the output will be written into sample_val_preds_metrics.json. The content in this generated file should be similar if not the same as sample_val_preds_metrics_raw.json file.

Format

The prediction file sample_val_preds.jsonl is in JSON Line format, each row of the files can be loaded as a single dict in Python. Below is an example of a single line in the prediction file:

{
  "qid": 2579,
  "query": "A girl and her mother cooked while talking with each other on facetime.",
  "vid": "NUsG9BgSes0_210.0_360.0",
  "pred_relevant_windows": [
    [0, 70, 0.9986],
    [78, 146, 0.4138],
    [0, 146, 0.0444],
    ...
  ],  
  "pred_saliency_scores": [-0.2452, -0.3779, -0.4746, ...]
}

entry description
qid int, unique query id
query str, natural language query, not used by the evaluation script
vid str, unique video id
pred_relevant_windows list(list), moment retrieval predictions. Each sublist contains 3 elements, [start (seconds), end (seconds), score]
pred_saliency_scores list(float), highlight prediction scores. The higher the better. This list should contain a score for each of the 2-second clip in the videos, and is ordered.

Codalab Submission

To test your model's performance on test split, please submit both val and test predictions to our Codalab evaluation server. The submission file should be a single .zip file (no enclosing folder) that contains the two prediction files hl_val_submission.jsonl and hl_test_submission.jsonl, each of the *submission.jsonl file should be formatted as instructed above.