Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Custom pose classification tutorial #33

Open
khanhlvg opened this issue Oct 15, 2020 · 12 comments
Open

Custom pose classification tutorial #33

khanhlvg opened this issue Oct 15, 2020 · 12 comments
Assignees

Comments

@khanhlvg
Copy link

Goal:

  • Write a tutorial to explain how to train a custom pose classifier (e.g. to detect yoga pose)

Technologies used:

  • ML Kit's Pose Detection API for pose detection
  • TF and TFLite to train a custom model that takes ML Kit's model as input, and outputs classification logits. A model with 2 Dense layers should be enough.
  • Firebase for storing training data

Artifacts:

  • Data collection app
    • Capture images and use Pose Detection API to detect body landmarks
    • Store the landmarks to Firestore, together with user input labels
  • Model training notebook
    • Load the training dataset from Firestore
    • Build a Keras model that classify the pose (model inputs are the landmark from Pose Detection)
    • Convert the model to TFLite
  • Demo app
    • Use Pose Detection API to detect landmark from the camera input, and run classification using the TFLite model
    • Can be the same app as the data collection app
@sayakpaul
Copy link
Contributor

sayakpaul commented Oct 23, 2020

Summarizing the discussion between @khanhlvg and myself from today's meeting -

  • We first need to start with a demo mobile application (preferably Android or iOS) to first gather data. Here's how the application could be utilized -
    • Let the user first decide which pose (yoga pose for example) they would want to make.
    • Under the pose received from the above step the application would start capturing the landmarks (using ML Kit's Pose Detection API) via continuous video feeds from the camera. This way, for a particular user-defined pose we will have a good amount of different landmarks.
    • Pack the data and upload it to Firestore.

After sufficient data collection, we can utilize the data in order to perform pose classification. So, the problem statement becomes - given a set of body landmarks predict the pose.

I am happy to work on the model training part as well as the model conversion part for the pose classification part. Also, I can work with a mobile developer to help them develop the application such that we gather the data in an expected way.

@sayakpaul sayakpaul self-assigned this Oct 23, 2020
@Anil-matcha
Copy link

Hi @khanhlvg . I can help with building an android app to collect the data required and once it is done can help with training a model too. I am creating a basic repo for the android app here https://github.com/Anil-matcha/pose_detection_data_collector .
Kindly let me know if you have any other information to discuss

@khanhlvg
Copy link
Author

@Anil-matcha thank you for your interest! @sayakpaul is looking for someone who can develop the Android app for data collection so I think this can be a nice collaboration.

@sayakpaul WDYT?

@sayakpaul
Copy link
Contributor

sayakpaul commented Oct 26, 2020

Yeah exactly. @Anil-matcha and I have discussed about it and we are game :D

@anilmaddala
Copy link

@Anil-matcha @sayakpaul I would like to jump in on the project, is there any thing either of you need help with? I can help with on device inference for Android.

@sayakpaul
Copy link
Contributor

@anilmaddala you can collaborate @Anil-matcha on the Android part for now on. Once that is sorted out we can work on the next steps together i.e. collecting the data and building the downstream model and so on.

@sayakpaul
Copy link
Contributor

Is there an update you, folks, would like to share? @anilmaddala @Anil-matcha

@anilmaddala
Copy link

@sayakpaul I am working on the data collection part of the app. I have integrated the MLKit apis to get the Pose points, need to implement the persistance part. Can we do on device training from pose classification instead of doing the training on Firebase?

@sayakpaul
Copy link
Contributor

sayakpaul commented Dec 5, 2020

@anilmaddala thanks for the updates. Actually, after we collect the data, the plan is to train a shallow neural network externally with TensorFlow/Keras, convert it to TensorFlow Lite and then redeploy it.

But I am curious regarding the on-device model training part. Do you have a pipeline in mind?

@anilmaddala
Copy link

@sayakpaul we might have to still build a base model in cloud. I was thinking for users to test out the app on new poses, we can implement something like https://blog.tensorflow.org/2019/12/example-on-device-model-personalization.html for on-device training and inference.

@sayakpaul
Copy link
Contributor

sayakpaul commented Dec 8, 2020

@anilmaddala so the idea here is after collecting the poses, we will build a shallow model, convert it to TFLite and deploy that TFLite model to the application to perform on-device inference.

Now coming to personalized training (as described in the blog post you mentioned), in order for that to work we still need a master model first which we could transfer learn for a particular user. Our initial shallow model can act as a master model here. Tagging @khanhlvg for any inputs he might have.

Let me know if there's any doubt in that regard.

@SunAriesCN
Copy link

Well, it this google-research/poem may help for it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants