py-feat/au_to_mesh

AU → MP-mesh PLS (visualization)

Predicts the 478-vertex MediaPipe FaceMesh deformation from 20 AU intensities. Trained with pose covariates and pose×AU interactions for cleaner AU coefficients; deployed as a 23-d input (AU + pose), with pose=0 at inference for the standard canonical-frontal visualization.

Backs py-feat's plot_face_mesh(au=...) / predict_face_mesh — the AU→geometry inverse of the detector.

Current default: au_to_mesh_pls_v6.npz.

Versions

file	detector AU space	held-out R²	notes
au_to_mesh_pls_v6.npz	Detectorv2 v2.8	0.472	current default
au_to_mesh_pls_v5.npz	Detectorv2 v2.5	0.362
au_to_mesh_pls_v4.npz	Detectorv2 v2.4	0.244	also bundled in feat/resources
au_to_mesh_pls_v3.npz	Detectorv2 (24-AU)	—
au_to_mesh_pls_v2.npz	v1 (img2pose + xgb AU)	0.244	original

Match the PLS version to your detector. Each model is fit on the AU distribution of a specific detector; pairing v6 with a pre-2.8 detector (or an older PLS with 2.8) maps AU values through a distribution the model was not fit on. py-feat ≥ 2.1.3 ships v2.8 weights and defaults to v6, so the pairing is correct out of the box. All older versions remain selectable via load_face_mesh_viz_model(model_version=...).

v6 training data

• 672,970 frames / 35,664 CelebV-HQ videos run through the v2.8 release checkpoint (soupGen, the mean of stage-3 epochs 2/6/19 — the same weights shipped as face_multitask_v28.safetensors).
• Targets are the detector's own predicted 478-vertex mesh, so the mapping is self-consistent with the AU space it is driven by.
• Pose-filtered to |yaw| ≤ 40°, |pitch| ≤ 30° → 608,513 frames; top 1% by max anchor residual dropped as alignment outliers → 602,427 frames / 33,920 videos fit.
• Per-frame Umeyama similarity Procrustes + iterative GPA to a population-mean template, on 12 stable anchors (forehead 10/9/8/151, nose bridge 6/168/197/195, outer canthi 33/263, inner canthi 133/362). Removes (R, s, t).
• No per-subject neutral subtraction — absolute aligned coordinates, the Cheong / py-feat-tutorial-06 recipe.

v6 method

• PLSRegression(n_components=83, scale=True).
• Inputs during fitting: [20 AU | 3 pose | 60 pose×AU] = 83. Only the 23-row AU+pose slice is deployed; pose is zero at inference, so the interaction terms vanish and the shipped model is a pure AU→shape map.
• Linearized to an explicit (coef, intercept) on the identity basis (linearization error 1.3e-15), then rigidly mapped into the v4 canonical frontal frame so every version renders at the same orientation and scale.

v6 performance

Variance-weighted R² = 0.4725 ± 0.0045, 5-fold GroupKFold by video (no video appears in both train and test), across all 1434 output dims. MAE = 0.0047 in aligned units.

Full k sweep (PLS components are nested, so one fit per fold yields the whole curve):

k	R²
10	0.4172
20	0.4505
40	0.4656
60	0.4706
66 (knee)	0.4716
83 (shipped)	0.4725

Against v5 at its own published grid points: 0.4505 vs 0.347 (k=20), 0.4656 vs 0.358 (k=40), 0.4706 vs 0.361 (k=60), 0.4725 vs 0.362 (k=83).

Fold count is immaterial at this sample size — 3-fold and 5-fold agree to ≤0.0005 across the entire curve.

Two changes contribute to the gain over v5 and are not separately identified: a better-conditioned v2.8 AU space, and a corrected extraction path (v5's inputs were center-cropped 256→224, while v2.5+ models and py-feat's shipped inference both resize — worth 0.05 mean absolute AU error and 4.6° of head pose).

R² is modest in absolute terms because 20 AU intensities cannot fully describe a 1434-d mesh deformation. For visualization, qualitative AU-direction correctness matters more, and v6's deformations are both more accurate and more anatomically localized than v5's (several v5 AUs smeared displacement across the whole face).

Inference

import numpy as np
m = np.load("au_to_mesh_pls_v6.npz")
au = np.zeros(20); au[m["au_columns"].tolist().index("AU12")] = 1.0  # smile
pose = np.zeros(3)                                                   # [pitch, yaw, roll]
x = np.concatenate([au, pose])                                       # (23,)
flat = x @ m["coef"] + m["intercept"]                                # (1434,)
# IMPORTANT: layout is axis-major [all x | all y | all z], NOT interleaved
mesh = np.stack([flat[:478], flat[478:956], flat[956:]], axis=1)     # (478, 3)
# Render with mediapipe.solutions.face_mesh.FACEMESH_TESSELATION

Or through py-feat:

from feat.plotting import plot_face_mesh, predict_face_mesh
plot_face_mesh(au={"AU12": 1.0})          # defaults to v6

AU inputs are probabilities in [0, 1]. Values above 1.0 are outside the model's input domain (the detector never emits them); because the model is linear, driving past 1.0 only scales the deformation and adds no information.

predict(au=0) equals mean_aligned_mesh by construction, so the "no AU given" neutral render and the AU=0 prediction agree.

Applying pose post-hoc (optional)

The model is trained with pose covariates but at inference users typically pass pose=0 for the canonical frontal deformation. To render at a chosen head pose, apply a rigid transform after prediction:

from scipy.spatial.transform import Rotation
R = Rotation.from_euler("xyz", [pitch, yaw, roll]).as_matrix()   # (3, 3)
posed_mesh = mesh @ R.T
# Or for re-projection onto an image: s * (mesh @ R.T) + t

Two ways to control pose:

• Render-time rotation (recommended): set pose=0, rotate post-hoc as above. The AU-driven deformation stays canonical and pose is purely a viewing transform.
• Pose-conditioned prediction: pass non-zero pose into the input vector to get the model's residual pose-correlated bias in addition to the rigid rotation needed at render time.

Convention notes:

• MP canonical frame: y-axis up, x-axis to the subject's left, z-axis out of the face. Right-handed.
• Rotation.from_euler("xyz", ...) is intrinsic xyz (R = Rx·Ry·Rz).
• With MP's 4×4 facial_transformation_matrix (M): posed = (np.concatenate([mesh, np.ones((478, 1))], axis=1) @ M.T)[:, :3]

File format

NPZ with:

coef               (23, 1434) float32  linear weights, rows match input_columns
intercept          (1434,)    float32  bias (== mean_aligned_mesh, flattened)
input_columns      (23,)      str      AU01..AU43, Pitch, Yaw, Roll
au_columns         (20,)      str      the AU subset
pose_columns       (3,)       str      the pose subset
mean_aligned_mesh  (478, 3)   float32  population mean canonical mesh ("AU=0" face)
reference_anchors  (12, 3)    float32  Procrustes reference
anchor_indices     (12,)      int32    MP indices used as anchors
n_components       ()         int32
model_card         ()         str

Loader: np.load("au_to_mesh_pls_v6.npz") — no extra dependencies.

Note v2 additionally carried mean_low_au_mesh and training_metadata; v4+ do not.

Figures (v6)

Each AU driven to 1.0, its full in-range maximum.

The au_solid_*.png and au_compare_overlay.png files in this repo are from the original v2 model and are retained for history only.

Citation context

Adapts Cheong et al. 2023 / py-feat tutorial 06 (E. Jolly): affine-aligned 68 dlib landmarks + pose → 20 AUs via PLS. Direction inverted (AU+pose → mesh) and scaled to MediaPipe's 478-vertex mesh on wild-celebrity video.

License

MIT for this artifact. Note the upstream detector that generated its training targets carries research-only restrictions.