human_posture_analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "abd4498e",
   "metadata": {},
   "source": [
    "## Human Posture Detection\n",
    "\n",
    "[<img src =\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\" align=\"left\">](https://colab.research.google.com/github/spmallick/learnopencv/blob/master/Posture-analysis-system-using-MediaPipe-Pose/human_posture_analysis.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c4b1872",
   "metadata": {},
   "source": [
    "Poor posture is a modern-day epidemic. Research has shown that children as young as 10 years of age are demonstrating spinal degeneration on x-ray. Certain postures, like forward head posture, have been linked to migraines, high blood pressure, and decreased lung capacity. In this notebook, you will learn to create an application to detect posture using mediapipe. We are going to use side view samples to perform analysis and draw conclusion. Practical application would require an webcam pointed at the side view of the person.\n",
    "\n",
    "### Workflow\n",
    " - Find point of interests (landmarks) to check angle.\n",
    " - Perform analysis on standard sample images.\n",
    " - Find threshold range for good and bad posture.\n",
    " - Apply on video/webcam input.\n",
    " \n",
    "### Goal\n",
    "To detect neck inclination and torso inclination as shown below.\n",
    "<br>\n",
    "<img src = \"https://learnopencv.com/wp-content/uploads/2022/03/mp-pose-03-posture-sitting-scaled.jpg\" align='center'>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "546c5937",
   "metadata": {},
   "outputs": [],
   "source": [
    "if 'google.colab' in str(get_ipython()):\n",
    "    !pip install opencv-python\n",
    "    !pip install mediapipe\n",
    "    !wget https://raw.githubusercontent.com/spmallick/learnopencv/blob/master/Posture-analysis-system-using-MediaPipe-Pose/input.mp4\n",
    "else:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43e5ebdc",
   "metadata": {},
   "source": [
    "## Import Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "45c4ca55",
   "metadata": {},
   "outputs": [],
   "source": [
    "import cv2\n",
    "import time\n",
    "import math as m\n",
    "import mediapipe as mp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "882e9fbe",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initilize medipipe selfie segmentation class.\n",
    "mp_pose = mp.solutions.pose\n",
    "mp_holistic = mp.solutions.holistic"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0773cf81",
   "metadata": {},
   "source": [
    "## Function to calculate offset distance\n",
    "The setup requires the person to be in proper side view. **`findDistance`** function helps us determine offset distance between two poinst. It can be the hip points, the eyes or shoulder. As these points are always more or less symmetric about the central axis. With this, we are going to incorporate camera alignment assistance in the script. Distnace is calculated using the distance formula.\n",
    "\n",
    "\n",
    "\\begin{align}\n",
    "distance =  \\sqrt{(x2 - x1)^2+(y2 - y1)^2}\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ceb4ca55",
   "metadata": {},
   "outputs": [],
   "source": [
    "def findDistance(x1, y1, x2, y2):\n",
    "    dist = m.sqrt((x2-x1)**2+(y2-y1)**2)\n",
    "    return dist"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9634fd09",
   "metadata": {},
   "source": [
    "## Function to calculate angle subtended by the line of interest to y-axis\n",
    "This is the primary deterministic factor for the posture. Using the angle subtended by the neck line and the torso line to y-axis. The neck line connects the shoulder and the eye, here we take shoulder as the pivotal point. Similarly the torso line connects hip and the shoulder, where hip is considered pivotal point. \n",
    "<br>\n",
    "\n",
    "<img src=\"https://learnopencv.com/wp-content/uploads/2022/03/mp-pose-05-neckline-inclination.jpg\" alt=\"MediaPipe pose neck inclination\" align=\"middle\" width=\"500\" height=\"600\">\n",
    "\n",
    "<br>\n",
    "\n",
    "Taking neck line as an example, here points are $P_1(x_1, y_1)$(shoulder), $P_2(x_2, y_2)$ (eye) and $P_3(x_3, y_3)$ (any point on vertical axis passing through $P_1$). \n",
    "<br>\n",
    "Hence, for $P_3$ x-coordinate is same as to that of $P_1$ and since $y_3$ is valid for all y, let's take $y_3 = 0$ for simplicity. <br>To find the inner angle of three points, we take vector approach. Angle between two vectors $\\vec{P_{12}}$ and $\\vec{P_{13}}$ is given by,\n",
    "\\begin{align}\n",
    "\\theta = \\arccos (\\frac{\\vec{P_{12}}.\\vec{P_{13}}}{|\\vec{P_{12}}|.|\\vec{P_{13}}|})\n",
    "\\end{align}\n",
    "Solving for $\\theta$ we get, \n",
    "\\begin{align}\n",
    "\\theta = \\arccos (\\frac{y_1^2 - y_1.y_2}{y_1\\sqrt{(x_2 - x_1)^2 + (y_2 - y_1)^2}})\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4492e989",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate angle.\n",
    "def findAngle(x1, y1, x2, y2):\n",
    "    theta = m.acos((y2 -y1)*(-y1) / (m.sqrt((x2 - x1)**2 + (y2 - y1)**2) * y1))\n",
    "    degree = int(180/m.pi)*theta\n",
    "    return degree"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c70dd6b4",
   "metadata": {},
   "source": [
    "## Function to send alert\n",
    "Use this function to send alerts when bad posture is detected. Feel free to get creative and customize as per your convenience. You can use telegram notifiction alert system, [chek out Telegram bot automation here](https://core.telegram.org/bots). Or you can take it up a notch by creating an android app."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e8db85c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "def sendWarning(x):\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71f81cc7",
   "metadata": {},
   "source": [
    "## Constants and Initializations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cb810117",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initilize frame counters.\n",
    "good_frames = 0\n",
    "bad_frames = 0\n",
    "\n",
    "# Font type.\n",
    "font = cv2.FONT_HERSHEY_SIMPLEX\n",
    "\n",
    "# Colors.\n",
    "blue = (255, 127, 0)\n",
    "red = (50, 50, 255)\n",
    "green = (127, 255, 0)\n",
    "dark_blue = (127, 20, 0)\n",
    "light_green = (127, 233, 100)\n",
    "yellow = (0, 255, 255)\n",
    "pink = (255, 0, 255)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cb0eb7cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize mediapipe pose class.\n",
    "mp_pose = mp.solutions.pose\n",
    "pose = mp_pose.Pose()\n",
    "\n",
    "# Initialize video capture object.\n",
    "# For webcam input replace file name with 0.\n",
    "file_name = 'input.mp4'\n",
    "cap = cv2.VideoCapture(file_name)\n",
    "\n",
    "# Meta.\n",
    "fps = int(cap.get(cv2.CAP_PROP_FPS))\n",
    "width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))\n",
    "height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))\n",
    "frame_size = (width, height)\n",
    "fourcc = cv2.VideoWriter_fourcc(*'mp4v')\n",
    "\n",
    "# Initialize video writer.\n",
    "video_output = cv2.VideoWriter('output.mp4', fourcc, fps, frame_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb920c58",
   "metadata": {},
   "source": [
    "## Processing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00af6213",
   "metadata": {},
   "source": [
    "Processing the image using mediapipe is fairly simple, after setting minimum detection confidence and minimum tracking confidence, we need to pass the RGB image to `pose.process()` method. The documentation can be found in this [**link**](https://google.github.io/mediapipe/solutions/pose). With the acquired result, we find the coordinates of specific landmarks. Then we find the angles suntended by the line of interset to y-axis and draw conclusion on the basis of results obtained from analysis script. The code is self explanatory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dcfa56b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Processing..')\n",
    "while cap.isOpened():\n",
    "    # Capture frames.\n",
    "    success, image = cap.read()\n",
    "    if not success:\n",
    "        print(\"Null.Frames\")\n",
    "        break\n",
    "    # Get fps.\n",
    "    fps = cap.get(cv2.CAP_PROP_FPS)\n",
    "    # Get height and width.\n",
    "    h, w = image.shape[:2]\n",
    "\n",
    "    # Convert the BGR image to RGB.\n",
    "    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n",
    "\n",
    "    # Process the image.\n",
    "    keypoints = pose.process(image)\n",
    "\n",
    "    # Convert the image back to BGR.\n",
    "    image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)\n",
    "\n",
    "    # Use lm and lmPose as representative of the following methods.\n",
    "    lm = keypoints.pose_landmarks\n",
    "    lmPose = mp_pose.PoseLandmark\n",
    "\n",
    "    # Acquire the landmark coordinates.\n",
    "    # Once aligned properly, left or right should not be a concern.      \n",
    "    # Left shoulder.\n",
    "    l_shldr_x = int(lm.landmark[lmPose.LEFT_SHOULDER].x * w)\n",
    "    l_shldr_y = int(lm.landmark[lmPose.LEFT_SHOULDER].y * h)\n",
    "    # Right shoulder\n",
    "    r_shldr_x = int(lm.landmark[lmPose.RIGHT_SHOULDER].x * w)\n",
    "    r_shldr_y = int(lm.landmark[lmPose.RIGHT_SHOULDER].y * h)\n",
    "    # Left ear.\n",
    "    l_ear_x = int(lm.landmark[lmPose.LEFT_EAR].x * w)\n",
    "    l_ear_y = int(lm.landmark[lmPose.LEFT_EAR].y * h)\n",
    "    # Left hip.\n",
    "    l_hip_x = int(lm.landmark[lmPose.LEFT_HIP].x * w)\n",
    "    l_hip_y = int(lm.landmark[lmPose.LEFT_HIP].y * h)\n",
    "\n",
    "    # Calculate distance between left shoulder and right shoulder points.\n",
    "    offset = findDistance(l_shldr_x, l_shldr_y, r_shldr_x, r_shldr_y)\n",
    "\n",
    "    # Assist to align the camera to point at the side view of the person.\n",
    "    # Offset threshold 30 is based on results obtained from analysis over 100 samples.\n",
    "    if offset < 100:\n",
    "        cv2.putText(image, str(int(offset)) + ' Aligned', (w - 150, 30), font, 0.9, green, 2)\n",
    "    else:\n",
    "        cv2.putText(image, str(int(offset)) + ' Not Aligned', (w - 150, 30), font, 0.9, red, 2)\n",
    "\n",
    "    # Calculate angles.\n",
    "    neck_inclination = findAngle(l_shldr_x, l_shldr_y, l_ear_x, l_ear_y)\n",
    "    torso_inclination = findAngle(l_hip_x, l_hip_y, l_shldr_x, l_shldr_y)\n",
    "\n",
    "    # Draw landmarks.\n",
    "    cv2.circle(image, (l_shldr_x, l_shldr_y), 7, yellow, -1)\n",
    "    cv2.circle(image, (l_ear_x, l_ear_y), 7, yellow, -1)\n",
    "\n",
    "    # Let's take y - coordinate of P3 100px above x1,  for display elegance.\n",
    "    # Although we are taking y = 0 while calculating angle between P1,P2,P3.\n",
    "    cv2.circle(image, (l_shldr_x, l_shldr_y - 100), 7, yellow, -1)\n",
    "    cv2.circle(image, (r_shldr_x, r_shldr_y), 7, pink, -1)\n",
    "    cv2.circle(image, (l_hip_x, l_hip_y), 7, yellow, -1)\n",
    "\n",
    "    # Similarly, here we are taking y - coordinate 100px above x1. Note that\n",
    "    # you can take any value for y, not necessarily 100 or 200 pixels.\n",
    "    cv2.circle(image, (l_hip_x, l_hip_y - 100), 7, yellow, -1)\n",
    "\n",
    "    # Put text, Posture and angle inclination.\n",
    "    # Text string for display.\n",
    "    angle_text_string = 'Neck : ' + str(int(neck_inclination)) + '  Torso : ' + str(int(torso_inclination))\n",
    "\n",
    "    # Determine whether good posture or bad posture.\n",
    "    # The threshold angles have been set based on intuition.\n",
    "    if neck_inclination < 40 and torso_inclination < 10:\n",
    "        bad_frames = 0\n",
    "        good_frames += 1\n",
    "\n",
    "        cv2.putText(image, angle_text_string, (10, 30), font, 0.9, light_green, 2)\n",
    "        cv2.putText(image, str(int(neck_inclination)), (l_shldr_x + 10, l_shldr_y), font, 0.9, light_green, 2)\n",
    "        cv2.putText(image, str(int(torso_inclination)), (l_hip_x + 10, l_hip_y), font, 0.9, light_green, 2)\n",
    "\n",
    "        # Join landmarks.\n",
    "        cv2.line(image, (l_shldr_x, l_shldr_y), (l_ear_x, l_ear_y), green, 4)\n",
    "        cv2.line(image, (l_shldr_x, l_shldr_y), (l_shldr_x, l_shldr_y - 100), green, 4)\n",
    "        cv2.line(image, (l_hip_x, l_hip_y), (l_shldr_x, l_shldr_y), green, 4)\n",
    "        cv2.line(image, (l_hip_x, l_hip_y), (l_hip_x, l_hip_y - 100), green, 4)\n",
    "\n",
    "    else:\n",
    "        good_frames = 0\n",
    "        bad_frames += 1\n",
    "\n",
    "        cv2.putText(image, angle_text_string, (10, 30), font, 0.9, red, 2)\n",
    "        cv2.putText(image, str(int(neck_inclination)), (l_shldr_x + 10, l_shldr_y), font, 0.9, red, 2)\n",
    "        cv2.putText(image, str(int(torso_inclination)), (l_hip_x + 10, l_hip_y), font, 0.9, red, 2)\n",
    "\n",
    "        # Join landmarks.\n",
    "        cv2.line(image, (l_shldr_x, l_shldr_y), (l_ear_x, l_ear_y), red, 4)\n",
    "        cv2.line(image, (l_shldr_x, l_shldr_y), (l_shldr_x, l_shldr_y - 100), red, 4)\n",
    "        cv2.line(image, (l_hip_x, l_hip_y), (l_shldr_x, l_shldr_y), red, 4)\n",
    "        cv2.line(image, (l_hip_x, l_hip_y), (l_hip_x, l_hip_y - 100), red, 4)\n",
    "\n",
    "    # Calculate the time of remaining in a particular posture.\n",
    "    good_time = (1 / fps) * good_frames\n",
    "    bad_time =  (1 / fps) * bad_frames\n",
    "\n",
    "    # Pose time.\n",
    "    if good_time > 0:\n",
    "        time_string_good = 'Good Posture Time : ' + str(round(good_time, 1)) + 's'\n",
    "        cv2.putText(image, time_string_good, (10, h - 20), font, 0.9, green, 2)\n",
    "    else:\n",
    "        time_string_bad = 'Bad Posture Time : ' + str(round(bad_time, 1)) + 's'\n",
    "        cv2.putText(image, time_string_bad, (10, h - 20), font, 0.9, red, 2)\n",
    "\n",
    "    # If you stay in bad posture for more than 3 minutes (180s) send an alert.\n",
    "    if bad_time > 180:\n",
    "        sendWarning()\n",
    "    # Write frames.\n",
    "    video_output.write(image)\n",
    "print('Finished.')\n",
    "cap.release()\n",
    "video_output.release()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}