PG-XFormer

Pose-guided cross-modal exercise recognition with a safety-gated feedback layer.

PG-XFormer fuses a CTR-GCN skeleton stream and a VideoMAE video stream with symmetric cross-attention to recognise physiotherapy exercises from a short clip and its 2D pose. It is the recognition backbone of the paper "PG-XFormer: Pose-Guided Cross-Modal Exercise Monitoring with Agentic Feedback". This repository holds the trained weights for every configuration and seed reported in the paper; code, split definitions, per-clip predictions, and the feedback audit are in PG-XFormer-artifacts.

All PG-XFormer resources are grouped in the PG-XFormer collection.

Model details


Inputs	16 RGB frames at 224 × 224 (values in [0, 1]) and 16 × 17 COCO-17 2D keypoints, pelvis-centred and divided by the shoulder-to-hip length
Skeleton stream	CTR-GCN, 7 blocks (64 → 256 channels), temporal kernel 9 — 1.9M parameters, ~1 GFLOP/clip
Video stream	VideoMAE-base (Kinetics-pretrained), lower 8 of 12 blocks frozen — 86M parameters, ~361 GFLOPs/clip
Fusion	4 layers of symmetric skeleton↔video cross-attention (8 heads, width 512), then concatenation, LayerNorm, mean-pool
Output	10 InfiniteRep exercises: arm raise, bicycle crunch, bird-dog, curl, fly, leg raise, overhead press, push-up, squat, superman
Size / cost	114.7M parameters, 416 GFLOPs per clip; ~85 ms per clip on a 6 GB consumer GPU (batch 1, fp16)

Checkpoints

model.safetensors is the default checkpoint (cross-attention, seed 0; fixed in advance as the initialisation of the clinical experiments, not chosen by test score). checkpoints/synthetic/<run>.safetensors holds the other runs, named <fusion>_<objective>_s<seed> (ce: cross-entropy; align: + InfoNCE, λ = 0.1). Held-out test results (70 clips, seven per exercise):

Run	Macro-F1 (%)	ECE
`xattn_ce_s0` (default, `model.safetensors`)	95.6	0.036
`xattn_ce_s2`	100.0	0.003
`xattn_align_s2`	94.2	0.046
`concat_ce_s2`	97.2	0.027
`concat_align_s2`	97.0	0.026
`bottleneck_ce_s1`	97.1	0.025
`bottleneck_ce_s2`	98.6	0.018
`video_ce_s1` / `video_ce_s2` (video only)	94.1 / 98.6	0.046 / 0.017
`skeleton_ce_s1` / `skeleton_ce_s2` (skeleton only)	66.1 / 63.5	0.109 / 0.146

Runs without a listed checkpoint (e.g. xattn_ce_s1) were trained before checkpoint saving was enabled; their per-clip predictions are in the artifacts repository.

Usage

pip install "torch>=2.4" "transformers>=5" safetensors huggingface_hub numpy

from huggingface_hub import hf_hub_download
import importlib.util, sys
spec = importlib.util.spec_from_file_location("pgx", hf_hub_download("FatimahEmadEldin/PG-XFormer", "pgxformer_load.py"))
pgx = importlib.util.module_from_spec(spec); spec.loader.exec_module(pgx)

model, E = pgx.load_pgxformer("xattn_ce_s0")      # or e.g. "concat_ce_s2"
logits = model(video, keypoints)                  # video: (B,16,3,224,224); keypoints: (B,16,17,2)

keypoints must be normalised with E.normalise_keypoints. The checkpoints use the transformers>=5 VideoMAE parameter naming.

Training

Data: synthetic InfiniteRep renders (archive), 700 clips of ten exercises; instance-disjoint, class-stratified split of 560 / 70 / 70 clips (split files in the artifacts repository). Keypoints were estimated from the RGB frames with an off-the-shelf 2D pose estimator.
Procedure: AdamW with cosine schedule, 20 epochs, effective batch 8 (4 × 2 accumulation), fp16 autocast; fusion learning rate 1e-4 (concatenation 3e-4); checkpoint selected on the development split; three seeds per configuration.

Evaluation (mean ± s.d. over three seeds)

Configuration	Macro-F1 (%)	ECE
Skeleton only (CTR-GCN)	65.9 ± 2.4	0.131 ± 0.020
Video only (VideoMAE)	96.6 ± 2.3	0.031 ± 0.015
Concatenation fusion	97.6 ± 0.8	0.025 ± 0.009
Bottleneck fusion	96.2 ± 2.9	0.027 ± 0.010
Cross-attention fusion	98.5 ± 2.6	0.017 ± 0.017

Fusion is significantly better than the skeleton stream (McNemar p < 1e-18). Differences between the fusion variants and video alone are not statistically significant on the 70-clip test set; cross-attention is significantly better calibrated than bottleneck fusion. Adding the InfoNCE term did not help (98.5 → 96.2%).

Clinical transfer (KIMORE). With a leakage-free five-fold protocol, synthetic pretraining gave no benefit over KIMORE-only training after fine-tuning (93.1 vs. 93.7% macro-F1), although frozen synthetic features reached 72.4% under a linear probe (chance 20%). KIMORE-adapted weights are not released because KIMORE is distributed by its authors on request.

Feedback layer

In the paper the recogniser is paired with a kinematic error report and a language-model cue generator enclosed by a rejection gate (clean and critical reports answered deterministically) and an output verifier (cues naming unreported body regions are replaced). The gate removed unrequested advice on clean repetitions (73.4% → 0% for Qwen3-235B) and made Qwen3-0.6B / 1.7B equally safe. Its code and every generated cue are in the artifacts repository.

Intended use and limitations

Research on exercise recognition and safe automated feedback. Not a medical device and not validated for clinical decision-making; no clinician has rated the feedback.
Trained only on synthetic renders of ten exercises; accuracy on real footage, other exercises, or other camera set-ups is unknown. The 70-clip test set limits statistical power.
Keypoint quality matters: the skeleton stream is weak on estimated synthetic keypoints.

Licence

CC BY-NC 4.0, inherited from the VideoMAE-base weights used for initialisation. InfiniteRep is a synthetic dataset by Infinity AI (research use with attribution).

Citation

Citation to be added on publication. Please also cite InfiniteRep, KIMORE (Capecci et al., IEEE TNSRE 2019), CTR-GCN (Chen et al., ICCV 2021), and VideoMAE (Tong et al., NeurIPS 2022).