Physics-based Human Motion Estimation and Synthesis from Videos

Paper Review of Physics-based Human Motion Estimation and Synthesis from Videos. The original paper can be found here.

Overview

Human motion synthesis is an essential field with relevance to graphics, gaming, and robotics. This paper breaks away from the typical reliance on large-scale motion capture and instead trains human motion synthesis directly from monocular RGB videos.

Contributions

1. A smooth contact loss function for physics-based pose estimation.

2. A fusion of visual pose estimation with physics-based optimization.

Method

This is a four-step process:

1) Estimate 2D and 3D body joints at each video frame using a monocular pose estimation model.

2) Transform 3D body joints to relative body part rotations using inverse-kinematics.

3) Refine initial motion estimates using physics-based optimization for physically plausible and temporal coherent motion.

4) Process all videos as above and use the resulting motions to train the motion synthesis model.

3D Pose Estimation

The 3D pose $p^{pe} \in \R^{J×3}$ and 2D pose $p^{pe,2d} \in \R^{J×2}$ are estimated at each frame.

Motion Representation and Initialization

$\theta_t^{root} = (\displaystyle\sum_{\tau=0} ^t \Delta \theta_{\tau}^{root,yaw})*\theta_t^{root,xy}$

Motion Optimization

Assume that a temporally evenly spaced sequence of motion frames $\{q_t\}_{1:T}$ and contact forces$\{f_c^t \}_{1:T}$ are given, where $q_t$ and $f_{t}^c$ represent the generalized coordinates and global contact forces of the body at time $t$. The loss function consists of three main parts:

$L_{physics}(q_t,f_{t}^{c}) = L_{dynamic} + L_{contact} + L_{penetration}$

Total loss function, $L_{total} = L_{pose} + L_{physics}+L_{smooth}$, consists of physics loss, pose estimation loss, and smoothness regularization. The physics loss $L_{physics}$ itself includes dynamics loss, contact loss, and penetration loss.

Physics Loss: $L_{dynamics}$

Dynamics loss ($L_{dynamics}$): It calculates the dynamics loss using the Recursive Newton Euler algorithm.

Using $f_t^r$ we can calculate the dynamics loss by comparing it to the actual forces on the character.

Recursive Newton Euler algorithm which exploits the sparsity structure induced by the kinematic tree and we use finite difference approximations for the time derivatives of $q(t)$.

$f_t^r(q(·))= M\ddot{q}_t + C \dot{q}_t + g$

$M$: inverse dynamics computation involving mass matrix

$C\dot{q}_t$: centrifugal and coriolis forces

$g$: gravity

Instead of leaving $f_t^a$ as yet an- other optimization variable, the optimal value of $\Beta f_t^a$ can be easily chosen by assuming no limits on actuation force. Magnitude of joint actuation is implicitly limited by penalizing acceleration of 3d joint positions and rotations, described later.

$L_{dynamics} = w_{dynamic}||f_t^r − \Beta f_t^a − J^{T} f_t^c||^2$

$J^T$: maps all the contact forces from the contact points onto the full space

$\Beta$ : joint actuation

Physics Loss: $L_{contact}$

Full-body inertia is accurately accounted for in the inverse dynamics loss and does not make use of centroidal approximations as in prior work. Contact forces are assumed to be exerted only by the feet at 4 different contact points.

The contact cost penalizes violation of Signorini’s conditions for contact

$L_{contact} = \displaystyle \sum_{i}^{n_c}c_{t,i} (w_e||e_{t,i}||^2 + w_{\dot{e}}||\dot{e_{t,i}}||^2)$

$e_{t,i} ∈ \R^3$ is the minimum displacement between the $i^{th}$ end effector position and contact surface and its time derivative $\dot{e_{t,i}}$ is also included to prevent slip

$c_{t,i} = \frac{1}{2} ( \tanh (k_1 || f_{t,i}^c|| - k_2) + 1)$

Physics Loss: $L_{penetration}$

As such the method can generate motions with penetrating objects without contact force. To avoid this, a separate term is used to explicitly penalize interpenetration

$L_{penetration} = w_{pen} \displaystyle \sum_i^{n_c}max( \{d_{t,i} + k_{margin}, 0 \})^2$

$d_{t,i}$ : signed distance of the contact surface at $i^{th}$ end effector which is negative if it is penetrating

Pose fitting loss ($L_{pose}$) measures the motion error in terms of local 3d keypoints deviation, global camera projected 2d keypoint deviations. A kinematic acceleration penalty ensures motion smoothness.

$L_{physics} = w_{dynamic}||f_t^r − \Beta f_t^a − J^{T} f_t^c||^2 \ + \ \displaystyle \sum_{i}^{n_c}c_{t,i} (w_e||e_{t,i}||^2 + w_{\dot{e}}||\dot{e_{t,i}}||^2) \ + \ w_{pen} \displaystyle \sum_i^{n_c}max( \{d_{t,i} + k_{margin}, 0 \})^2$

The pose fitting loss $L_{pose}$ we use is common in human shape estimation. It measures the motion error in terms of local 3d keypoints deviation, global camera projected 2d keypoint deviations.

We also use a kinematic acceleration penalty to ensure our motions are smooth.

$L_{smooth} = \frac {1}{n_{joints}} (w_{\ddot \theta} || \ddot{\theta}||^2 + w_{\ddot {p}} || \ddot {p_t}|| ^2)$

$\ddot{p_t}$ : global linear acceleration of the joints

Experimental Results

The paper explores whether the physics loss improves the accuracy and physical plausibility of pose estimation and compares the proposed method against other physics/temporal pose estimation methods.

Several metrics were used to measure 3D keypoint velocity, jittering in the motion, foot floating artifacts, and foot global z position error.

Evaluation metrics

It measures the difference in 3d keypoint velocity magnitude between the ground truth motion and the predicted motion which illustrates the amount of jittering present in the motion and is computed as follows:

$\hat{Jit} = || \hat{p_t} - \hat{p_{t-1}}||$

$Jit^{GT} = || p_t^{GT} - p_{t-1}^{GT}||$

$e_{smooth} = \displaystyle\sum_t^T \sum{joints} || p_t^{GT} = p_{t-1}^{GT}||$

To evaluate foot floating artifacts, we compare foot global $z$ position error ($e_{foot,z}$) on ground truth

$e_{foot,z} = mean(|\hat{p}_{foot,z} - p_{foot,z} ^ GT|)$

$e_{foot,vxy} = mean(||\Delta_t \hat{p}_{foot,xy} - \Delta_tp_{foot,xy}^{GT}||)$