Real-time image saliency

See what your classifier is looking at! [PAPER]

Real-time saliency view

Run python real_time_saliency.py to perform the saliency detection on the video feed from your webcam. You can choose the class to visualise (1000 ImageNet classes) as well as the confidence level - low confidence will highlight anything that resembles or is related to the target class, while higher confidence will only show the most salient parts.

The model runs on a CPU by default and achieves about 5 frames per second on my MacBook Pro (and over 150 frames per second on a GPU).

Training

Run python saliency_train.py to start the training. By default it will train the model to perform the saliency detection on the ImageNet dataset for the resnet50 classifier, but you can choose your own dataset/classifier combination. You will need PyTorch wich cuda support, the training will be performed on all your GPUs in parallel. I also advide to run the script from iTerm 2 terminal so that you can see the images during traning.

Using pretrained model

from saliency_eval import get_pretrained_saliency_fn

sal_fn = get_pretrained_saliency_fn()

# get the saliency map (see get_pretrained_saliency_fn doc for details)
sal_map = sal_fn(images, selectors)

Requirements

pip install -r requirements.txt

Also, in case you don't have OpenCV3 installed run:

pip install opencv-contrib-python