PnLCalib: Sports Field Registration via Points and Lines Optimization

Extension of the previous paper "No Bells, Just Whistles: Sports Field Registration by Leveraging Geometric Properties"

🚀 Key Features

• Improved Keypoint and Line Detection
  Enhanced models for faster and more accurate detection of keypoints and line extremities.

• Upgraded Camera Calibration Pipeline
  Refined pipeline for better initial camera calibration performance.

• PnL Refinement Module
  Added a powerful refinement module that leverages detected field lines for non-linear optimization, boosting calibration accuracy.

Contents

• Abstract
• Model Weights
• Inference / Demo
• Experiments
  • Datasets
  • Metrics
  • Evaluation
• Training
• Requirements
• Citation

---

Abstract

Camera calibration in broadcast sports videos presents numerous challenges for accurate sports field registration due to multiple camera angles, varying camera parameters, and frequent occlusions of the field. Traditional search-based methods depend on initial camera pose estimates, which can struggle in non-standard positions and dynamic environments. In response, we propose an optimization-based calibration pipeline that leverages a 3D soccer field model and a predefined set of keypoints to overcome these limitations. Our method also introduces a novel refinement module that improves initial calibration by using detected field lines in a non-linear optimization process. This approach outperforms existing techniques in both multi-view and single-view 3D camera calibration tasks, while maintaining competitive performance in homography estimation. Extensive experimentation on real-world soccer datasets, including SoccerNet-Calibration, WorldCup 2014, and TS-WorldCup, highlights the robustness and accuracy of our method across diverse broadcast scenarios. Our approach offers significant improvements in camera calibration precision and reliability.

Overview of our proposed framework. Top: Training data generation pipeline. Beginning with SoccerNet annotations, we utilize field line extraction and ellipse fitting to establish a hierarchical structure for computing each set of keypoints. Bottom: Inference stage pipeline. The encoder-decoder networks produce heatmaps for keypoints and extremities of soccer field lines to extract their positions in the image space. The obtained keypoint set is augmented with intersections of lines generated by the second model to ensure a sufficient number of points. After initial calibration, our PnL refinement module is applied to further refine the calibration estimate by jointly using detected points and lines information.

Weights

Download the keypoints and line detection model weights for both single-view and multiple-view configurations. Our models are pre-trained on the SoccerNet distribution and further finetuned on the WC14 and TSWC datasets for enhanced performance.

Single-View

Model	Finetuned	Link
Keypoints	X	SV_kp
Lines	X	SV_lines
Keypoints	WC14	SV_FT_WC14_kp
Lines	WC14	SV_FT_WC14_lines
Keypoints	TSWC	SV_FT_TSWC_kp
Lines	TSWC	SV_FT_TSWC_lines

Multiple-View

Model	Finetuned	Link
Keypoints	X	MV_kp
Lines	X	MV_lines

---

Inference

Remove --pnl_refine argument for a base inference.

On video:

python inference.py --weights_kp "SV_kp" --weights_line "SV_lines" --pnl_refine --input_path "examples/iniesta_sample.mp4" --input_type "video" --display

On image:

python inference.py --weights_kp "SV_kp" --weights_line "SV_lines" --pnl_refine --input_path "examples/messi_sample.png" --input_type "image" --save_path "examples/messi_results.png"

---

Experiments

Datasets

SoccerNet-Calibration-V3:

To download the 2022 version, switch the downloader's task to "calibration".

from SoccerNet.Downloader import SoccerNetDownloader
mySoccerNetDownloader = SoccerNetDownloader(LocalDirectory="</nfs/data/soccernet>")
mySoccerNetDownloader.downloadDataTask(task="calibration-2023", split=["train","valid","test"])

See https://github.com/MM4SPA/tvcalib to download the camera-type annotations for SoccerNet-Calibration-2022.

WorldCup 2014

mkdir -p datasets/WC-2014/test && cd datasets/WC-2014/test
# Images and provided homography matrices from test split
wget https://nhoma.github.io/data/soccer_data.tar.gz
tar -zxvf soccer_data.tar.gz

See https://github.com/MM4SPA/tvcalib to download the additional segment annotations in SoccerNet-Calibration format.

TS-WorldCup

Download the TS-WorldCup dataset

---

Metrics

Segment Reprojection Error

See https://github.com/SoccerNet/sn-calibration for details on the evaluation metric.

IoU, Projection error and Reprojection error:

For the Homography Estimation evaluation, we adopt the approach outlined at https://github.com/ericsujw/KpSFR. Minor modifications of the used script can be seen in model/metrics.py

---

Evaluation

We provide scripts scripts/ to reproduce the paper's results for the presented approach. Make sure to change the dataset location on the bash scripts, default is set as "datasets/calibration-2023/", and the location of model weights, default is set as weights/MV_kp.

#Multi-view camera parameter estimation for SN-Calib-2023
chmod +x scripts/run_pipeline_sn23.sh
./scripts/run_pipeline_sn23.sh

#Single-view camera parameter estimation for SN-Calib-2022
chmod +x scripts/run_pipeline_sn22.sh
./scripts/run_pipeline_sn22.sh

#WorldCup 2014 camera parameter estimation task
chmod +x scripts/run_pipeline_wc14_3D.sh
./scripts/run_pipeline_wc14_3D.sh

#WorldCup 2014 homography estimation task
chmod +x scripts/run_pipeline_wc14.sh
./scripts/run_pipeline_wc14.sh

#TS-WorldCup homography estimation task
chmod +x scripts/run_pipeline_tswc.sh
./scripts/run_pipeline_tswc.sh

Task results will be printed on screen.

---

Training

Keypoint detection

python train.py --cfg config/hrnetv2_w48.yaml --dataset SoccerNet --root_dir datasets/calibration-2023/ --save_dir weights/MV_kp_exp1 

Line detection

python train_l.py --cfg config/hrnetv2_w48_l.yaml --dataset SoccerNet --root_dir datasets/calibration-2023/ --save_dir weights/MV_line_exp1 

Options for --dataset argument are: [SoccerNet, WorldCup2014, TSWorldCup]

Additional available arguments for both keypoint and line detection training:

• --cuda: CUDA device index (default: 'cuda:0')
• --batch: Batch size for train / val (default: 2)
• --num_workers: Number of workers for data loading (default: 4)
• --num_epochs: Number of training epochs (default: 200)
• --pretrained: Pretrained weights path
• --lr0: Initial learning rate (default: 0.001)
• --patience: Patience parameter for lr scheduler (default: 8)
• --factor: Reducing factor for lr scheduler (default: 0.5)
• --wandb_project: Wandb project name

---

Requirements

Conda Environment

conda env create -f PnLCalib.yml
conda activate PnLCalib

PIP

python -m venv PnLCalib
source env/bin/activate
pip install -r requirements.txt

---

Citation

@article{gutierrez4998149pnlcalib,
  title={Pnlcalib: Sports Field Registration Via Points and Lines Optimization},
  author={Guti{\'e}rrez-P{\'e}rez, Marc and Agudo, Antonio},
  journal={Available at SSRN 4998149}
}