Skip to content

nineblu/omni-res

Repository files navigation

Omni-RES

Python PyTorch

Omni-RES is a simple and lightweight codebase for the research of Omni-supervised referring expression segmentation, which currently supporting SimRES as base model. Later LAVT and ReLA models also will be updated.

Installation

  • Clone this repo
  • Create a conda virtual environment and activate it
conda create -n omni_res python=3.7 -y
conda activate omni_res
pip install -r requirements.txt
wget https://github.com/explosion/spacy-models/releases/download/en_vectors_web_lg-2.1.0/en_vectors_web_lg-2.1.0.tar.gz -O en_vectors_web_lg-2.1.0.tar.gz
pip install en_vectors_web_lg-2.1.0.tar.gz

Data preparation

  •  Follow the instructions of DATA_PRE_README.md to generate training data and testing data.
  •  Download the pretrained weights of backbone (vgg, darknet, cspdarknet, DResNet, etc.). Expect for DResNet, all pretrained backbones are trained on COCO 2014 train+val set while removing the images appeared in the val+test sets of RefCOCO, RefCOCO+ and RefCOCOg (nearly 6500 images). Please follow the instructions of DATA_PRE_README.md to download them.
  •  Also, we provide the necessary json files for training, the downloading url is https://anonymous.4open.science/r/omni_res_data

Training and Evaluation

  1. Config preparation. Prepare your own configs in configs, you don't need to rewrite all the contents in config every time.You can import the config as a python file to use the default configs. For example, to run 10% RefCOCO with omni-box,you can use simres_refcoco_omni.py as follows:
# your own config.py
from .common.dataset import dataset
from .common.train import train
from .common.optim import optim
from .common.models.simres import model

# Refine data path depend your own need
dataset.ann_path["refcoco"] = "./data/anns/refcoco.json"
dataset.image_path["refcoco"] = "./data/images/train2014"
dataset.mask_path["refcoco"] = "./data/masks/refcoco"
dataset.sup_ann_path["refcoco"] = "./data/anns/refcoco_0.1.json"
dataset.omni_ann_path["refcoco"] = "./data/anns/refcoco_0.9.json"
...
  1. Train the model. To start the training, you can input command as follows:
# Training 10% RefCOCO with omni-box (SimRES as base model)
python -m torch.distributed.launch --nproc_per_node 4 train_omni.py --config configs/simres_refcoco_omni.py

# Training 100% RefCOCO in fully supervised learning (SimRES as base model)
python -m torch.distributed.launch --nproc_per_node 4 train_sup.py --config configs/simres_refcoco_sup.py

# Training RefCOCO, RefCOCO+, RefCOCOg, Visual Genome with omni-box (SimRES as base model)
python -m torch.distributed.launch --nproc_per_node 4 train_vg.py --config configs/simres_refcoco_vg.py

The training logs, config.yaml and model checkpoints will be automatically saved under cfg.train.output_dir.

  1. Resume training. We support two resume training mode. You can resume from a specific checkpoint or resume from the latest checkpoint:
  • Auto resume from last.pth:
# config.py
from .common.train import train
train.auto_resume.enabled = True

Setting train.auto_resume.enabled=True, which will automatically resume from last_checkpoint.pth saved in cfg.train.output_dir.

  • Resume from a specific checkpoint
# config.py
from .common.train import train

# disable auto resume first
train.auto_resume.enabled = False

# modify the resume path
train.resume_path = "path/to/specific/checkpoint.pth"

Setting train.resume_path to the specific checkpoint.pth you want to resume from.

  1. Test the model.
python -m torch.distributed.launch --nproc_per_node 4 eval_engine.py --config configs/simres_refcoco_omni.py --eval-weights /path/to/checkpoint

License

This project is released under the Apache 2.0 license.

About

No description, website, or topics provided.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages