SLEAP: Social LEAP Estimates Animal Poses

Looking to get some SLEAP?

We've released the next version of LEAP, with full support for multi-animal pose tracking in addition to all of the functionality of the original LEAP.

Check it out now at sleap.ai!

Related software

• DeepLabCut: The popular deep learning framework for pose estimation. Soon to support full multi-animal pose tracking!
• DeepPoseKit: Supports multi-animal pose tracking via top-down centroid detection (e.g., from Ctrax, idTracker, Tractor, etc.).
• Animal Part Tracker: Supports multi-animal pose tracking and lots of other functionality!

LEAP (deprecated)

Full movie: YouTube

This repository contains code for LEAP (LEAP Estimates Animal Pose), a framework for animal body part position estimation via deep learning.

Preprint: Pereira et al., bioRxiv (2018)

We are still working on documentation and preparing parts of the code. See the Features section below for an overview and status of each component.

We recommend starting with the Tutorial: Training Leap From Scratch.

Features

• Tracking and alignment code
• Cluster sampling GUI
• Skeleton creation GUI (create_skeleton)
• GUI for labeling new dataset (label_joints)
• Network training through the labeling GUI
• MATLAB (predict_box.m) and Python (leap.predict_box) interfaces for predicting on new data
• GUI for predicting on new data
• Training data + labels for main fly dataset used in analyses
• Trained network for predicting on main fly dataset
• Analysis/figure generation code
• Documentation
• Examples of usage

Installation

Pre-requisites

All neural network and GPU functionality is implemented in Python. The library was designed to be easy to use by providing commandline interfaces, but it can also be used programatically if the MATLAB GUIs are not required.

For the Python environment, we recommend Anaconda 5.1.0 with Python 3.6.4. Note that Python 2.x is not supported.

For GPU support, you'll want to first install the CUDA drivers with CuDNN and then install these packages:

pip install -Iv tensorflow-gpu==1.6.0
pip install -Iv keras==2.1.4

See the TensorFlow installation guide for more info.

If you don't have a GPU that supports CUDA, install the regular TensorFlow distribution:

pip install tensorflow

Please note that CPU execution will be MUCH slower (10-20x) than on a GPU. Consider investing in a GPU for your machine if you're thinking of using LEAP routinely.

Automated installation

To get started with using LEAP, open up MATLAB and download the repository:

>> !git clone https://github.com/talmo/leap.git

Then, install the package and add to the MATLAB path:

>> cd leap
>> install_leap

That's it!

To avoid having to run this function every time you start MATLAB, save the MATLAB Search Path after running it (or just add the leap subfolder to your permanent path).

Manual: MATLAB dependencies

GUIs and analyses are implemented in MATLAB, but is not required for using the neural network functionality implemented in Python.

We use MATLAB R2018a with the following toolboxes: Parallel Computing Toolbox, Statistics and Machine Learning Toolbox, Computer Vision Toolbox, Image Processing Toolbox, Signal Processing Toolbox.

All MATLAB external toolboxes are included in the leap/toolbox subfolder. Just add the leap subdirectory to the MATLAB Search Path to access all functionality:

addpath(genpath('leap'))

Manual: Python dependencies

The versions below were used during development of LEAP but other versions will also likely work.

Libraries required are easily installable via the pip package manager:

pip install -Iv numpy==1.14.1
pip install -Iv h5py==2.7.1
pip install -Iv clize==4.0.3

You will also need OpenCV 3 with Python bindings. We recommend using skvark's excellent precompiled packages:

pip install -Iv opencv-python==3.4.0.12

You can install this library as a python package by downloading this git repository:

git clone https://github.com/talmo/leap.git

then typing:

pip install -e leap # installs the leap directory using pip

If you are using anaconda to manage different python environments, it is highly recommended that you use this matlab script to manage your python envs within matlab: condalab (see their readme for how to use it)

Usage

Refer to the Tutorial: Training Leap From Scratch.

Preprocessing

GUI Workflow

1. Cluster sampling: Call cluster_sample from MATLAB commandline to launch GUI.
2. Create skeleton: Call create_skeleton from MATLAB commandline to launch GUI.
3. Label data and train: Call label_joints from MATLAB commandline to launch GUI.
4. Batch estimation: Coming soon.

Programmatic API

See leap/training.py and leap/predict_box.py for more info.

Contact and more information

Reach out to us via email: Talmo Pereira (talmo@princeton.edu)