An application to support learning the Polish Sign Language
Opis w języku polskim znajduje się poniżej.
Watch the video about the application
Make sure you have Python 3.12.0 installed on your computer. If yes, you need to install the required packages. To do this, run the following command in terminal inside the project directory:
pip install -r requirements.txt
Make the migrations and migrate the database:
python manage.py makemigrations
python manage.py migrate
To create superuser:
python manage.py createsuperuser
To run the application run the following command in terminal inside the project directory:
python manage.py runserver
Goal of the project is to familiarize people with the Polish Sign Language as it's the means of communication for the 50 thousand Deaf people in Poland. The application allows users to learn the alphabet of PSL by watching videos of signs and practicing showing the gestures themselves.
The main part of the application is the sign recognition module, which uses a neural network to recognize the signs shown by the user. The training data was collected over half a year and consists of over 5000 arrays of hand keypoints (detected using MediaPipe library). The data was additionally augmented 11 times (the original data was randomly rotated, translated, and scaled 10 times) to increase the size of the dataset. Examples of the data can be seen below.
The neural network was trained using the TensorFlow library and during the training the dropouts and both Learning Rate Reduction and Early Stopping callbacks were used to prevent model overfitting to training data. Plot of the model accuracy over epochs and final confusion matrix can be seen below.
To make the process of learning the signs more engaging, the application also includes a game where the user is supposed to spell a word (displayed as an image) using the Polish Sign Language alphabet. For each correctly shown sign, the user receives a point and there is a bonus for completing the word. In settings, user can change which hand is used, how long the game lasts, and the difficulty level.
Statistics of each game are saved and can be accessed in the Django admin panel.
Apart from the sign recognition module, the application also allows user to learn the alphabet of the Polish Sign Language by watching videos and graphics. After selecting a letter in the application, proper video is displayed.
- Python,
- Django,
- TensorFlow,
- OpenCV,
- MediaPipe,
- JavaScript,
- HTML,
- CSS,
- Firebase.
Application was created by:
Also thanks to @kacper-g48, @Dawidson174 and @Krasa35 for their help with gathering the training data.
Aplikacja wspomagająca naukę Polskiego Języka Migowego.