Adrian Bulat Computer Vision / Machine Learning
Back to research

Human pose estimation via Convolutional Part Heatmap Regression

Adrian Bulat, Georgios Tzimiropoulos

European Conference on Computer Vision (ECCV), 2016

Architecture diagram: an image enters a part detection network; its heatmaps are stacked with the image and passed to a regression network to locate body joints.
Figure 1 from the paper: a part detection network produces heatmaps, which are stacked with the image and refined by a regression network to localize body joints.

Abstract

This paper is on human pose estimation using Convolutional Neural Networks. Our main contribution is a CNN cascaded architecture specifically designed for learning part relationships and spatial context, and robustly inferring pose even for the case of severe part occlusions. To this end, we propose a detection-followed-by-regression CNN cascade. The first part of our cascade outputs part detection heatmaps and the second part performs regression on these heatmaps. The benefits of the proposed architecture are multi-fold: It guides the network where to focus in the image and effectively encodes part constraints and context. More importantly, it can effectively cope with occlusions because part detection heatmaps for occluded parts provide low confidence scores which subsequently guide the regression part of our network to rely on contextual information in order to predict the location of these parts. Additionally, we show that the proposed cascade is flexible enough to readily allow the integration of various CNN architectures for both detection and regression, including recent ones based on residual learning. Finally, we illustrate that our cascade achieves top performance on the MPII and LSP data sets.

Key contributions

  1. Detect, then regress

    Connect a body-part detection network to a regression network that jointly refines the location of the joints.

  2. Context for occluded parts

    Stack detection heatmaps with the input image so the regression network can use visible parts to infer occluded joints.

  3. Flexible CNN architecture

    Implement the cascade with VGG-based fully convolutional networks and with residual networks.

  4. 2016 benchmark results

    Report 89.7% MPII PCKh@0.5 and 90.7% LSP PCK accuracy in the original study.

Models

Original pretrained checkpoints for the MPII and LSP benchmarks.

These models use Torch7/Lua. The original repository documents setup with Torch7 and Python 2.7. Its demo automatically downloads the selected checkpoint and dataset when needed; consult the usage and storage requirements before running it.

MPII model

Trained on MPII. Reported MPII accuracy in the 2016 paper: 89.7% PCKh@0.5.

Torch7 (.t7) · 579 MiB

Download weights Details

MPII + LSP model

Trained using MPII and LSP. Reported LSP accuracy in the 2016 paper: 90.7% PCK.

Torch7 (.t7) · 579 MiB

Download weights Details

Run the original demo

th main.lua --help
th main.lua -dataset lsp

Benchmark datasets

The paper evaluates human pose estimation on the existing MPII Human Pose and Leeds Sports Pose (LSP) benchmarks. MPII uses head-normalized Percentage of Correct Keypoints at a threshold of 0.5 (PCKh@0.5); LSP uses Percentage of Correct Keypoints (PCK).

See the MPII dataset project for its data and evaluation protocol. The original demo repository documents how to run and evaluate the provided MPII and LSP models.

Citation

If you use this work, please cite:

@inproceedings{bulat2016human,
	title={Human pose estimation via Convolutional Part Heatmap Regression},
	author={Bulat, Adrian and Tzimiropoulos, Georgios},
	booktitle={ECCV},
	year={2016}
}