Moosic - Facial emotion detection based music recommendation system

Submission for Microsoft Engage 2022

This project is built as a part of Engage Mentorship Program 2022 by Microsoft.

Table of Contents

1. About the Project
2. Useful Links
3. Features
4. Dependencies
5. Instructions
6. System Architecture
7. Structure of this repository
8. Future Scope
9. References

About

This repository demonstrates an end-to-end pipeline for real-time Facial emotion recognition application along with reccommending music based on detected emotions. Done in three steps:

1. Face Detection: from the video source using OpenCV.
2. Emotion Recognition: using a model trained by using Mediapipe library.
3. Music Recommendation: Using detected emotion to create a search query on Youtube

The model is trained for 50 epochs and runs at 87% accuracy.

Useful Links

• Demo Video
• Sprint Document
• Design Document

Features

1. Landing Page

Seamless landing page filled with dark-theme.

2. Detection of various emotions like [Sad, Angry, Happy, Neutral, Surprise]

3. Detection of various gestures like [Hello, Thumbsup, Nope, Rock]

Dependencies

This project depends on Python and following packages which can be easily installed through requirements.txt file by running the following command: pip install -r requirements.txt

• Python 3.9.6
• pip 22.1.1
• streamlit 1.9.1
• streamlit-webrtc 0.37.0
• opencv-python 4.5.5.64
• mediapipe 0.8.10

Instructions

Testing Locally

• git clone https://github.com/khankhushi/Moosic
• Run pip install -r requirements.txt to install all dependencies.
• cd ./moosic
• streamlit run app.py
• The app is now running at http://localhost:8501
• While testing, wait for the model to detect your emotions and click on recommend button to get songs based on a particular emotion
• Emotion used previously are stored as cache and might cause an error in recommending music, delete detected_emotion.npy file in the directory to resolve this.
• Recommended music is loaded in next tab as a youtube search query.

System Architechture

Repository Structure

This repository is organised as:

• app This file contain the setup of final web app.
• model This file contains the trained model.
• Emotion Detection This folder contains python scripts to train the model.
• .streamlit This folder contains configuration files for the streamlit theme in Web App.

Future Scope

• Deploying the web app.
• Integration of an inbuilt music player using SpotiPy library, with spotify authentication.
• Addition of more gestures, and control of volume using gesture detection.
• Improved Reliablity and addition of User Feedback

References

• Emotional Detection and Music Recommendation System based on User Facial Expression - S Metilda Florence and M Uma