Skip to content

Commit

Permalink
Merge pull request #101 from TechLabs-Berlin/main
Browse files Browse the repository at this point in the history
main to frontend
  • Loading branch information
iradzh authored Apr 8, 2023
2 parents 54446b5 + 5c8a3c0 commit 4367a9f
Show file tree
Hide file tree
Showing 11 changed files with 462 additions and 343 deletions.
12 changes: 8 additions & 4 deletions backend/data/poses.json
Original file line number Diff line number Diff line change
@@ -1,14 +1,18 @@
[
{
"name": "Lunges"
"name": "Lunges",
"componentName": "Lunges"
},
{
"name": "Warrior"
"name": "Warrior",
"componentName": "Warrior"
},
{
"name": "Chair"
"name": "Chair",
"componentName": "Chair"
},
{
"name": "Curls"
"name": "Bicep Curls",
"componentName": "Curls"
}
]
1 change: 1 addition & 0 deletions backend/routes/poses.js
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ router.get("/", (req, res) => {
poses.push({
id: pose.name,
label: pose.name,
componentName: pose.componentName,
});
});
res.send(poses);
Expand Down
180 changes: 180 additions & 0 deletions blog.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,178 @@
# **poseGenie: AI powered form-feedback tool in fitness and yoga **

Are you tired of suffering from poor posture and feeling frustrated with incorrect exercise form? These issues can cause a range of health problems and limit your performance. Not to mention, it can be difficult to determine if you’re executing exercises properly or which muscles should be targeted. This can lead to muscle imbalances, reduced workout effectiveness, and even painful injuries.

Thankfully, there’s a high-tech solution that will leave you feeling confident and energized during your workouts. Introducing PoseGenie - the all-in-one workout assistant that uses advanced human pose estimation technology to detect your exercise and provide you with valuable feedback and customized suggestions to enhance your performance. This incredible tool offers exercise metrics and helps you optimize your workouts by engaging the correct muscles, allowing you to quantify your progress and achieve your goals.

Say goodbye to frustrating workouts and hello to your new personal trainer & workout optimizer - PoseGenie!

_This project was carried out as part of TechLabs in Berlin (winter term 2023)_

 

# All tracks: initial phase
This section outlines the common discussion points that involved all track members at the planning phase:

- Which poses/exercises to implement, why and how

- How to learn the mechanics of the poses/exercises

- Which human pose estimation model to use

- Webapp or mobile app

- No UX designer team member in the team: how to proceed

- Realtime detection or prerecorded video

- To what extent the assistance/feedback/suggestions can be provided to the user in general

- To what extent the assistance/feedback/suggestions can be provided to the user in our MVP

- Audio feedback or written feedback

- A final report or an analysis on the fly

- A rep counter tool

- Which ML/DL models can be used on the data obtained from the user

- Trimming videos, detecting when the exercise starts


# 👩 Data Science and 🤖 Artificial Intelligence tracks: common points
Most of the time in our project phase, AI and DS teams worked together and this section outlines the common tasks of AI and DS.

- DS: Esma B. Boydas, Naiara Fernandez
- AI: Rashmi C. Dsouza, Ignatio C. Hidayat
- Tech Stack:Jupyter Notebook, Python, Mediapipe, Pandas, NumPy, Matplotlib, Scikit-learn

### Learning about 3D human pose estimation
At the initial stage of the project we have started to look into the dynamics of 3D human pose estimation models in Python. We have gathered important sources outlining the capabilities and uses of different computer vision models, such as the links below:

- A comprehensive guide to Human Pose Estimation:
https://www.v7labs.com/blog/human-pose-estimation-guide

- Human Pose Estimation Technology Capabilities and use cases:
https://mobidev.biz/blog/human-pose-estimation-technology-guide

- 3D Human Pose Estimation Experiments and Analysis:
https://www.kdnuggets.com/2020/08/3d-human-pose-estimation-experiments-analysis.html

- An easy guide for pose estimation with MediaPipe:
https://medium.com/mlearning-ai/an-easy-guide-for-pose-estimation-with-googles-mediapipe-a7962de0e944

- Squat analyzer with MediaPipe:
https://learnopencv.com/ai-fitness-trainer-using-mediapipe/

- Deadlift analyzer I:
https://saketshirsath.github.io/cv.github.io/

- Deadlift analyzer II:
https://github.com/SravB/Computer-Vision-Weightlifting-Coach

- Deep Learning approaches for workout repetition counting and validation:
https://www.sciencedirect.com/science/article/abs/pii/S016786552100324X#!

- Yoga Pose Estimation and Feedback Generation using Deep Learning:
https://www.hindawi.com/journals/cin/2022/4311350/

- Validity of an artificial intelligence, human pose estimation model for measuring single-leg squat kinematics:
https://pubmed.ncbi.nlm.nih.gov/36198251/

After taken a quick look at these resources, we have discussed the potential capabilities of our MVP from a python-backend perspective.

### Decision of the AI-model
We have started testing the Mediapipe landmarks. Everyone in these two teams made use of real-life examples and observed the motion detection. To test the limitations, we have also resorted to videos in which the human body parts moved out of frame. We have discussed the detection confidence and used different thresholds to check the best setting in terms of computational cost vs. accuracy. The consensus was to proceed with mediapipe since it provides a collection of pre-built components that can be easily customized, combined, and extended to develop computer vision and ML models.

### Implementation phase
We have implemented common calculator functions which would be regularly used by means of all exercises. Angle calculations are the most crucial steps in pose detection, and we resorted to different techniques to calculate them, such as angle between two lines (4 landmark points) vs between three points (3 landmarks).

We distinguished between *dynamic exercises* that are based on repeats and counts, (e.g. lunges, curls) and *static exercises* that require holding a position for a certain time (e.g. yoga poses). Depending on the specific exercise, different data analysis approach was taken. Each exercise is implemented as a separate class.

## Static Exercises

### Class Warrior (Warrior 2 yoga pose)

#### Data Analysis

To analyze Warrior 2 pose, 10 different indicative angles, calculated from the Mediapipe landmarks, are used: angles at elbows, knees and hips (6 angles), and angles arms and legs from horizontal (4 angles).

Given Warrior 2 pose is an asymmetric pose, user can be facing right or left, it first required a preliminary analysis on the leg position to determine if the user is performing a left or right warrior. Then, angles are analized to check if they are within predefined ranges for the pose to be considered "detected", if yes a timer is started. To implement the timer, CV2 timestamps of video frames are used, which gives approximate duration. In addition, feedback is given to users to get their angles closer to the "optimal" expected angles and to "improve" their pose.

All indicative angles and calculations are internally stored in Numpy Arrays and Python lists, then at the end of video they are converted to Pandas DataFrame for final analysis and graph plotting with MatplotLib.

#### Machine Learning Model

The initial approach described above, relies on "hard-coded" angle ranges to detect the pose. But this would have been a limitation if we wanted to easily expand the app to many more yoga poses (ideally an entire yoga flow sequence).

A basic Machine Learning (ML) model was implemented to detect additional poses. For the training of the ML model, >300 images downloaded from internet are used. The ML model data preparation and training is done in a Jupyter notebook, where from each image, 10 indicative angles are stored as features with a label (pose name). The ML model was trained with SciKitLearn, using Decision Trees, Random Forest, and SVC (linear and rbc kernel) models. Trained models are saved and can be loaded in the main python app. At the moment of submission, few poses were contained in the model, mountain pose, tree pose, and warrior 1 and 2 poses. A similar approach, but using Deep Learning algorithms is implemented in the different class "Chair" pose.

Right now, feedback is only given for Warrior 2 pose. Ideally, the app, would detect any of the poses within the ML model, and would provide also feedback to the app user. One caveat of a model trained with arbitrary images downloaded from the internet is that, the images do not necesarily represent an optimally-performed yoga pose, but instead provides a broad range of angles for pose detection in the app. Ideally two different set of images would be used to train two different models: 1) a broad range of arbitrary images representing of all levels of performers to detect the pose in the app (as implemented right now). 2) A second set of images with the poses performed by advanced practitioner, coaches, or instructurs, for a model used as a benchmark to provide feedback to user.

## AI - Neural Net Model to Predict Stationary Poses

Similar to the machine learning model above, instead of hard-coding the angles to detect whether the desired pose is achieved, we utilized Deep Learning using Neural Networks to make pose detection more polished and refined. We have trained it for detecting 5 yoga poses, of which only one is currently implemented in the app. Implementations of the 4 other poses will be done in the future.

### Data

For the prediction models, pictures readily available online and separated them into train and test folders, specifically using Lawrence Moroney’s Yoga Pose Classification datasets which are computer generated 300x300 full color images in 5 different poses, using Daz3D, were used. This image set contains all the five poses For further information: https://laurencemoroney.com/2021/08/23/yogapose-dataset.html

### Model

For the model, we used took a pre-trained neural network built using keras from https://github.com/harshbhatt7585/YogaIntelliJ. The model uses 3 dense layers (2 used for input and output) and 2 dropout layers and achieved a perfect accuracy score on both training and validation sets which had 1513 and 878 images respectively, each having 5 classes.

Alternatively, we tried adding a convolutional and max-pooling layer on top of the original model. Convolutional layers are a type of layer that are commonly used in Deep Learning models for image processing tasks. These layers operate on small regions of the input image, called kernels or filters, and extract local features such as edges, shapes, and textures. By applying multiple convolutional layers in a network, we can learn more complex and abstract features of the image hierarchy. Since we inputs we are passing into the model are one-dimensional arrays containing the landmark keypoints, a one-dimensional convolutional layer was used.

The result of this layers might be that the extracted features include information about the position, orientation, and movement of the different joints, as well as more abstract features such as the overall pose of the person and the context in which the pose occurs.

Pooling layers operate on the output of convolutional layers by reducing the spatial resolution of the feature maps. The most common pooling operation is Max-Pooling, which selects the maximum activation value within a small window of pixels. Pooling helps to reduce the number of parameters in the network, prevent overfitting, and increase the computational efficiency of the model.

We experimented with adding a convolutional and max-pooling layer on top of the model. However, it had slightly less accuracy than the original model. The optimizer we used was adam, a popular optimizer that uses adaptive learning rates to update the weights of the neural network during training.

### Original Model
![Untitled](https://user-images.githubusercontent.com/50834160/230712925-01290a20-2213-4bac-81a0-670a7561fc9c.png)
![Untitled](https://user-images.githubusercontent.com/50834160/230713121-3786f375-171b-428d-b343-23c78a9b483c.png)

Accuracy: 0.9977194666862488

Loss: 0.004214226733893156

### Original Model (with 1 Convolutional Layer & 1 MaxPooling Layer)
![output](https://user-images.githubusercontent.com/50834160/230713201-2475cb4a-7d6e-4671-81dd-32cf95c4d5de.png)
![output](https://user-images.githubusercontent.com/50834160/230713209-51f71d3d-d266-4e6a-b178-e9e5d8d6ede6.png)

Accuracy: 0.998859703540802

Loss: 0.008410756476223469

### Original Model (with 2 Convolutional Layers & 2 MaxPooling Layers)

Accuracy: 0.9965792298316956

Loss: 0.015103639103472233

Through this experiment, adding convolutional and pooling layers to a neural network may not always improve its performance. The reason for this could be due to various factors such as the complexity of the dataset, with ours being very simple, the size of the network, the number of training examples, and the hyperparameters used in the model.

## Dynamic Exercises
Dynamic exercises involves repetitions and a higher degree of movement, which adds another layer of complexity regarding the landmark detection, data collection, and post-processing of the user exercise. For this purpose, we have resorted to detecting & recording & processing the most important landmarks for the exercise.

*How does this work?* For instance, a lunge exercise is a lower body exercise that involves stepping forward or backward with one leg and bending both knees to lower the body. The key points for this dynamic exercise are thus the upper legs, which should be actively detected by the detection model. We call it *the hip-knee angle* and record it progressively throughout the exercise.

*What comes after recording the key landmarks?* Following the detection and storing the important angles, we apply an initial parameter-free sinusoidal fit as the repetitive move resembles a sine wave. After fitting, there are three important parameters come into play:

- val_minmax: the difference between the maximum and minimum angle reached during the exercise
- val_amp: the amplitude of the sine wave calculated after sinusoidal fitting
- val_time: the period calculated after sinusoidal fitting

Based on these parameters, the feedback is generated in three parts and plotted by matplotlib along with the 2D graphs involving the measurements:

- Definition: This part explains the concept of the hip-knee angle and how it can be analyzed to assess the user's form during the lunge exercise.
- Analysis: This part provides a detailed analysis of the user's performance during the exercise, including their range of motion (val_minmax), consistency of movements (val_amp), and timing per rep (val_time). The workout assistant also provides specific comments and feedback based on these parameters.
- Feedback: This part provides specific feedback to the user based on their performance, including suggestions for improvement and areas to focus on during future workouts.

 

# 🕸 Web Development

## ⚙ Backend
Expand Down Expand Up @@ -80,3 +255,8 @@ I implemented user authentication via Firebase to ensure secure and reliable pro
I worked closely with the backend developer to ensure successful project completion. This included merging changes to the GitHub repository and optimizing the codebase for a successful MVP.

 





114 changes: 61 additions & 53 deletions frontend/src/components/BtnComponent/PosesButtonsGroup.js
Original file line number Diff line number Diff line change
@@ -1,69 +1,77 @@
import React, { useState, useEffect } from "react";
import axios from "axios";
import "./ButtonComponent.css";
import WarriorComponent from "./WarriorComponent";
import LungesComponent from "./LungesComponent";
import CurlsComponent from "./CurlsComponent";
import ChairComponent from "./ChairComponent";
import ModalFeedback from "../05_update_video/ModalFeedback";

function PosesButtonsGroup({ onButtonClicked, onModalClosed }) {
const [selectedPose, setSelectedPose] = useState(null);
const [poses, setPoses] = useState([]);
const [selectedPose, setSelectedPose] = useState(null);
const [poses, setPoses] = useState([]);

useEffect(() => {
axios.get("/poses").then((res) => {
setPoses(res.data);
});
}, []);
useEffect(() => {
axios.get("/poses").then((res) => {
setPoses(res.data);
});
}, []);

const handleButtonClick = (value) => {
setSelectedPose(value);
onButtonClicked(value);
};
const handleButtonClick = (value) => {
setSelectedPose(value);
onButtonClicked(value.id);
};

const resetStates = () => {
setSelectedPose(null);
};
const resetStates = () => {
setSelectedPose(null);
};

const renderPoseComponent = () => {
if (selectedPose === "Lunges") {
return <LungesComponent />;
} else if (selectedPose === "Warrior") {
return <WarriorComponent />;
} else if (selectedPose === "Curls") {
return <CurlsComponent />;
} else if (selectedPose === "Chair") {
return <ChairComponent />;
}
};
const renderPoseComponent = () => {
if (selectedPose) {
try {
const PoseComponent =
require(`./${selectedPose.componentName}Component`).default;
return <PoseComponent />;
} catch (error) {
console.log(error);
return null;
}
}
};

const handleModalClosed = () => {
resetStates();
onModalClosed();
};
const handleModalClosed = () => {
resetStates();
onModalClosed();
};

return (
<div className="wrapper">
<div className="container">
<h2 className="title-h2">Choose an exercise</h2>
<div className="btn_container">
{poses.map((button) => (
<button
key={button.id}
onClick={() => handleButtonClick(button.id)}
className={selectedPose === button.id ? "active" : "inactive"}
>
{button.label}
</button>
))}
</div>
{renderPoseComponent()}
const Buttons = () => {
return poses.map((pose) => {
const buttonStatus =
selectedPose && selectedPose.id === pose.id
? "active"
: "inactive";

return (
<button
key={pose.id}
onClick={() => handleButtonClick(pose)}
className={buttonStatus}
>
{pose.label}
</button>
);
});
};

<ModalFeedback onModalClosed={handleModalClosed} />
</div>
</div>
);
return (
<div className="wrapper">
<div className="container">
<h2 className="title-h2">Choose an exercise</h2>
<div className="btn_container">
<Buttons />
</div>
{renderPoseComponent()}

<ModalFeedback onModalClosed={handleModalClosed} />
</div>
</div>
);
}

export default PosesButtonsGroup;
Loading

0 comments on commit 4367a9f

Please sign in to comment.