python⼈体动作识别_基于OpenPo的⼈体⾻骼多⼈动作实
时识别
Multi-person Real-time Action Recognition Bad-on Human Skeleton
Highlights: 9 actions; multiple people (<=5); Real-time and multi-frame bad recognition algorithm.
Updates: On 2019-10-26, I refactored the code; added more comments; and put all ttings into the config/config.yaml file, including: class of actions, input and output of each file, OpenPo ttings, etc.
Project: This is my final project for EECS-433 Pattern Recognition in Northwestern Univeristy on March 2019. A simpler version where two teammates and I worked on is here.
Warning: Since I ud the 10 fps video and 0.5s-window for training, you must also limit your video fps to be about 10 fps (7~12 fps) if you want to test my pretrained model on your own video or web camera.
Contents:
1. Algorithm
I collected videos of 9 Types of actions: ['stand', 'walk', 'run', 'jump', 'sit', 'squat', 'kick', 'punch', 'wave']. The total video lengths are about 20 mins, containing about 10000 video frames recorded at 10 frames per cond.
The workflow of the algorithm is:
Get the joints' positions by OpenPo.
Track each person. Euclidean distance between the joints of two skeletons is ud for matching two skeletons. See class Tracker in lib_tracker.py
Fill in a person's missing joints by the joints' relative pos in previous frame. See class FeatureGenerator in
lib_feature_proc.py. So does the following.swallow是什么意思
Add noi to the (x, y) joint positions to try to augment data.
中国春节纪录片U a window size of 0.5s (5 frames) to extract features.
Extract features of (1) body velocity and (2) normalized joint positions and (3) joint velocities.
Apply PCA to reduce feature dimension to 80. Classify by DNN of 3 layers of 50x50x50 (or switching to other classifiers in one line). See class ClassifierOfflineTrain in lib_classifier.py
Mean filtering the prediction scores between 2 frames. Add label above the person if the score is larger than 0.8. See class ClassifierOnlineTest in lib_classifier.py
For more details about how the features are extracted, plea e my report.
2. Install Dependency (OpenPo)
First, Python >= 3.6.
I ud the OpenPo from this Github: tf-po-estimation. First download it:
export MyRoot=$PWD
cd src/githubs
Follow its tutorial here to download the "cmu" model. As for the "mobilenet_thin", it's already inside th
e folder.
$ cd tf-po-estimation/models/graph/cmu
$ bash download.sh
Then install dependencies. I listed my installation steps as bellow:
conda create -n tf tensorflow-gpu
conda activate tf
cd $MyRoot
pip install -
pip install jupyter tqdm
pip install tensorflow-gpu==1.13.1
sudo apt install swig
cd $MyRoot/src/githubs/tf-po-estimation/tf_po/pafprocess
swig -python -c++ pafprocess.i && python3 tup.py build_ext --inplace
Make sure you can successfully run its demo examples:
cd $MyRoot/src/githubs/tf-po-estimation
python run.py --model=mobilenet_thin --resize=432x368 --image=./images/p1.jpg
3. Program structure
Diagram
Trouble shooting:
How to change features?
In utils/lib_feature_proc.py, in the class FeatureGenerator, change the function def add_cur_skeleton!
The function reads in a raw skeleton and outputs the feature generated from this raw skeleton as well as previous skeletons. The feature will then be saved to features_X.csv by the script s3_preprocess_features.py for the next training step.
How to include joints of the head?
You need to change the aforementioned add_cur_skeleton function.
I suggest you to write a new function to extract the head features, and then append them to the returned variable(feature) of add_cur_skeleton.
Plea read def retrain_only_body_joints in utils/lib_feature_proc.py if you want to add the head joints.
How to change the classifier to RNN?
There are two major changes to do:
First, change the aforementioned add_cur_skeleton. Instead of manually extracing time-rials features as does by the current script, you may simply stack the input skeleton with previous skeletons and then output it.
Second, change the def __init__ and def predict function of class ClassifierOfflineTrain in utils/lib_classifier.py to add an RNN model.
Main scripts
The 5 main scripts are under src/. They are named under the order of excecution:
src/s1_get_skeletons_from_training_imgs.py
src/s2_put_skeleton_txts_to_a_single_txt.py
src/s3_preprocess_features.py
src/s4_train.py
src/s5_test.py
The input and output of the files as well as some parameters are defined in the configuration file config/config.yaml. I paste part of it below just to provide an intuition:
class: ['stand', 'walk', 'run', 'jump', 'sit', 'squat', 'kick', 'punch', 'wave']
image_filename_format: "{:05d}.jpg"
skeleton_filename_format: "{:05d}.txt"
features:
window_size: 5 # Number of adjacent frames for extracting features.
s1_get_skeletons_from_training_imgs.py:
openpo:
model: cmu# cmu or mobilenet_thin. "cmu" is more accurate but slower.
img_size: 656x368 # 656x368, or 432x368, 336x288. Bigger is more accurate.
input:
images_description_txt: data/source_images3/
images_folder: data/source_images3/
output:
images_info_txt: data_proc/raw_skeletons/# This file is not ud.
detected_skeletons_folder: &skels_folder data_proc/raw_skeletons/skeleton_res/
viz_imgs_folders: data_proc/raw_skeletons/image_viz/
s2_put_skeleton_txts_to_a_single_txt.py:
input:
# A folder of skeleton txts. Each txt corresponds to one image.
detected_skeletons_folder: *skels_folder
内审员资格证书
output:
# One txt containing all valid skeletons.
all_skeletons_txt: &skels_txt data_proc/raw_skeletons/
s3_preprocess_features.py:
input:
all_skeletons_txt: *skels_txt
output:
procesd_features: &features_x data_proc/features_X.csv
excite 翻译procesd_features_labels: &features_y data_proc/features_Y.csv
s4_train.py:
input:
procesd_features: *features_x
procesd_features_labels: *features_y
output:
model_path: model/trained_classifier.pickle
For how to run the main scripts, plea e the Section 4. How to run: Inference and 6. How to run: Training.
4. How to run: Inference
Introduction
The script src/s5_test.py is for doing real-time action recognition.
The class are t in config/config.yaml by the key class.
The supported input includes video file, a folder of images, and web camera, which is t by the command line arguments --data_type and --data_path.
The trained model is t by --model_path, e.g.:model/trained_classifier.pickle.
The output is t by --output_folder, e.g.: output/.
cassadee popeThe test data (a video, and a folder of images) are already included under the data_test/ folder.
An example result of the input video "exerci.avi" is:
output/exerci/
├── skeletons
│ ├──
│ ├──
takemedicine
│ └── ...
└── video.avi
Also, the result will be displayed by cv2.imshow().
Example commands are given below:
Test on video file
python src/s5_test.py \
-
-model_path model/trained_classifier.pickle \
--data_type video \
影视化妆学校voanews--data_path data_test/exerci.avi \
--output_folder output
Test on a folder of images
python src/s5_test.py \
--model_path model/trained_classifier.pickle \
--data_type folder \
--data_path data_test/apple/ \
--output_folder output
Test on web camera
python src/s5_test.py \
--model_path model/trained_classifier.pickle \
--data_type webcam \
--data_path 0 \
--output_folder output
5. Training data
Download my data
推荐信格式>jingle bell
Follow the instructions in data/download_link.md to download the data. Or, you can create your own. The data and labelling format are described below.
Data format
Each data subfolder (e.g. data/source_images3/jump_03-02-12-34-01-795/) contains images named as 00001.jpg, 00002.jpg, etc.
The naming format of each image is defined in config/config.yaml by the ntence: image_filename_format: "{:05d}.jpg".
The images to be ud as training data and their label are configured by this txt file: data/source_images3/
A snapshot of this txt file is shown below:
jump_03-02-12-34-01-795
52 59
72 79
kick_03-02-12-36-05-185
54 62
In each paragraph,
the 1st line is the data folder name, which should start with "${class_name}_". The 2nd and following
lines specify the
staring index and ending index of the video that corresponds to that class.
Let's take the 1st paragraph of the above snapshot as an example: jump is the class, and the frames 52~59 & 72~79 of the video are ud for training.
Class
The class are t in config/config.yaml under the key word class. No matter how many class you put in the training data (t by the folder name), only the ones that match with the class in config.yaml are ud for training and inference.
6. How to run: Training
First, you may read
Section 5. Training data
