Exercise counter with DeepLens

Original idea was to have camera which will count exercises on machines for bench press, lat pulldown, chest press etc. in gym. And mobile or web app which will show personal statistics. During development I got stuck many times on training, deployments, model conversions etc. I am not python developer and I didn't have any knowledge about machine learning. At the end I was able to create just simple model which can recognize barbell and from its moves it tries to count exercises. Logs about exercises are sent via IoT to Lambda which saves them to Elasticsearch. Unfortunetely I was not able to create any frontend. Exercises logs might be seen just in Kibana. If you ask I can add your IP address to firewall and send you Kibana endpoint to see results.

Model

I have trained ssd_resnet50 model on custom dataset (one class) with tools from Incubator-mxnet SSD example. I have some notes on training and model conversion

.json and .xml files are here in repository or might be downloaded from

• https://s3.amazonaws.com/deeplens-th-ex/bells-model5/ssd_resnet50_300-symbol.json
• https://s3.amazonaws.com/deeplens-th-ex/bells-model5/opt/mxnet_deploy_ssd_resnet50_300_FP16_FUSED.xml

Corresponding binary files are too big so they are on S3 (I have reviewed permissions, should be public now)

• https://s3.amazonaws.com/deeplens-th-ex/bells-model5/ssd_resnet50_300-0000.params
• https://s3.amazonaws.com/deeplens-th-ex/bells-model5/opt/mxnet_deploy_ssd_resnet50_300_FP16_FUSED.bin

Rest of project

As I have mentioned before I have used 2 lambdas, both managed by Serverless Framework. count-exercise is function to be deployed with Greengrass to DeepLens and pass-to-es is lambda triggered by IoT, which will pass data into Elasticsearch.

In Terraform folder there are other AWS resources - Elasticsearch Domain and EC2 Instance as proxy.

Instructions to run whole project

0. Prepare ES and proxy

cd terraform

Change params in default.tfvars to fit your account and credentials.

terraform plan -var-file default.tfvars
terraform apply -var-file default.tfvars

after proxy has started install nginx and configure as proxy

ssh ubuntu@PROXY_IP
sudo apt update && sudo apt install nginx
sudo nano /etc/nginx/sites-enabled/default

find location / section and change to

        location / {
                proxy_pass https://YOUR_ELASTICSEARCH_DOMAIN_ADDRESS.us-east-1.es.amazonaws.com/ ;
        }

*might be automatized with EC2 User Data

1. Prepare and deploy Lambdas

In count-exercise/serverless.yml change profile (line 24) and lambda role (line 68) and deploy code

cd count-exercise/
serverless deploy

In pass-to-es/serverless.yml change profile (line 24) IoT, VPC, Elasticsearch Domain details below line 69 and deploy

cd pass-to-es/
serverless deploy

*these steps might be automatized and connected to terraform

2. Create DeepLens project

• create model from s3://deeplens-th-ex/bells-model5/opt/
• create project using model above and function count-exercise-leafs-dev-count-exercise
• deploy to device

Images Demo - using non converted model

• checkout incubator-mxnet project
• download -symbol.json and -0000.params files to incubator-mxnet/example/ssd/model/
• run demo:

git clone https://github.com/apache/incubator-mxnet.git
cd incubator-mxnet/example/ssd
# download model
./demo.py --prefix model/ssd_resnet50_300 --network resnet50 --class-names barbell --data-shape 224 --thresh 0.4 --nms 0.3 --images img1.jpg,img2.jpg

Video Demo - using non converted model

Change path to model (line 21) and path to video (line 16) in count-exercise/video-demo.py and run

python3 count-exercise/video-demo.py

It will produce video resized to 224x224 with, barbell detection and counter.

You can check demo video

Project structure

./
├── artifacts # .json
├── count-exercise # lambda to be deployed to DeepLens
├── pass-to-es # lambda to pass data from IoT topic to Elasticsearch, probably should be renamed and used for ES-mobile interface
├── terraform # terraform for other resources
└── training # notes on training and converting model

What's next

I will take few days rest - it was really hard to work on such project with so many fails, expecialy after my normal job on weekends and evenings! But I definetely won't give up on it! I will walk through Python course, Machine Learning course and through mxnet & gluon tutorials. Meantime I will ask some gyms if I can place my camera there and will gather material for datasets. One friend of mine is really interested in project and offered help with dataset preparation.

Ideas:

• train better model to recognize at least person (maybe with pose), barbell weights, kettlebell, block-weights used on machines (easier and more general to watch then handles and other moving parts of machines)
• Exercise Recognition
  • position within camera view => match to exercise
  • from main model we will get timeseries of person + pose + weights positions which is perfect fit for another machine learning model
  • such model probably will be able to evaluate precision and effect of exercise (bodybuilding, strength or endurance). Which is valuable info for eventual customer.
  • How to find time/period of exercise - same exercises might be done fastly or slowly. (multi-scale sliding window - might be resource hungry, autocorrelation - exercise is not really periodic)
• Exact weight classification
  • Cut from detection
  • Once per serie?
• Solve detections "jumping".
  • It is physically impossible for weight to jump from one place to completely different and back from frame to frame or to radically change size. Unfortunately inference on video from camera does it.
  • maybe add some memory on size and position of last detection(s) to model (is it possible?)
  • or consider it as noise and clean it from signal
  • or maybe secondary model will take care of it (not probable)
• Authentication and how to connect person and exercise.
  • Person's face can't be seen during every exercise and it would be expansive (computationally and financially) to recognize with each exercise.
  • Camera directed to entrance which will detect faces and send them to AWS Rekognition
  • Some simplified model to person recognition/classification. Any suggestions? How does it AWS Rekognition on videos.
• Multiple Cameras cooperation

All these are about machine learning and technical stuff around DeepLens camera. I was thinking a lot about potential project. And I have also some notes on possible project. Unfortunately these thoughts was much much faster than actual progress on model and code. :-)