Thanks for the first field report — an RTX 3080 20G on Windows finishing 512×288×39 in 20 minutes is genuinely useful data, and your log shows the engine doing exactly what it was designed to do.

One clarification on CUDA: your GPU was in use. The line

wgpu GPU path: on (NVIDIA GeForce RTX 3080 / Vulkan, discrete, weight budget 19229 MB)

means cortiq drives the 3080 through Vulkan — the same silicon CUDA would reach, with no CUDA toolkit to install. That is deliberate: one small binary that runs NVIDIA, AMD, Intel and phone GPUs with the same shaders.

Why your run is ~7× slower than the reference card (172 s on an RTX PRO 6000 96 GB): the file is 25.2 GB and your weight budget is ~19 GB, so not every layer stays resident on the card — part of the model streams from system RAM every step, and that dominates. The probe lines in your log are the engine measuring its way around that honestly: gemm-nt went to CPU (transfer-bound over PCIe on your box), matmat-wide went to GPU (3× faster than your CPU). Each op class runs where it is actually faster on your machine, not where a flag says it should.

Two practical knobs while iterating:

• --width 256 --height 160 --frames 22 renders in ~29 s on the reference card — expect a couple of minutes on the 3080. Good for prompt iteration before the big render.
• --height 256 instead of 288 roughly halves the video-VAE decode: the tiler covers a 288-px edge with two overlapping 256-px tiles, so you pay 512 rows for 288 rows of picture. An edge of exactly 256 is one tile.

On CUDA-enabled tooling: the output is standard MJPEG+PCM in an AVI plus a WAV, so it drops into any downstream pipeline as-is. A native CUDA backend is not planned, but cooperative-matrix (tensor-core) kernels for this model's GEMMs are the next GPU headroom on the roadmap — that is the gap you are feeling, not a missing API.

It’s really rewarding to see a project evolve from an idea into something that actually works. It feels like building a house for a long time and finally starting to move the furniture in :)

I’m still improving the format, but the solid foundation is already in place.

hello, i see this clipproj-minimax-h3.

ComfyUI-ClipProj is a new custom node that replaces MiniMax H3's Qwen3-VL-32B text encoder with a much smaller Qwen3-VL-4B plus a learned linear projection, cutting the conditioning VRAM footprint from 15.7 GB to 4.5 GB

and cmf here,

Prompt encoder — Qwen3-VL-32B truncated to 50 layers, 51.5 GB → 12.2 GB. It is the largest single component of the file and it runs once per generation.

By integrating ClipProj, it can shrink the final file size even more so the whole model fits entirely in VRAM. That eliminates the heavy latency from constant memory swapping, which sounds like a great angle.

Btw, is there any open-source code or reference document available for the tech using to handle models in CMF project ? For example, VAE-related stuff—it seems most video VAEs use f16 quantization and end up being over 4.8GiB . I'm planning to build one from scratch, so any help would be greatly appreciated!

Great news: over the last week I have improved the video-generation protocol by approximately 7×.

The optimization work is ongoing, but this is already a substantial step forward for practical inference speed.

You can review the available code and implementation details in the CMF repository:
https://github.com/infosave2007/cmf

Regarding ClipProj: I will try to integrate it, build a compatible version, and optimize it for CMF. If it works well, it may further reduce the memory footprint and help keep the full model in VRAM.

@bingoct — the compact build is up, and your report is what it came out of.

You were paging: 25.2 GB of weights through a ~19 GB budget on a 20 GB card, which is where the 20 minutes went. So the fix had to be the file, not the kernels.

mmh3-turbo-clipproj4b-q4tp.cmf — 13.16 GB. Whole-run VRAM peak measured at 15.1 GB, so your 3080 holds it resident with room to spare. Nothing streams.

It is the ClipProj idea you pointed at. The Qwen3-VL-32B prompt encoder is replaced by Qwen3-VL-4B tapped at layer 24 plus the learned projection, and everything else is untouched — DiT and both VAE decoders copied through byte for byte, still four bits. No two-bit anywhere; that build stops following the prompt and is not what you want.

What the substitution actually costs, measured rather than eyeballed — the conditioning the DiT receives, cosine against the real 32B on the same prompt:

Honest about the difference: it renders the same scene with plainer set dressing. Your corgi still flips the pancake — it leaves the surface and comes back — but the 32B's griddle and patterned wallpaper become a white plate and a flat ground. Clip and audio are here: ab_clipproj.mp4, and the three-way comparison is at the top of the model card.

Running it

Needs cortiq 0.5.73 or newer — the packing and the projection are both new:

cargo install cortiq-cli
# or grab a prebuilt binary: https://github.com/infosave2007/cmf/releases/latest
cortiq --version

hf download infosave/MiniMax-H3-Turbo-cmf mmh3-turbo-clipproj4b-q4tp.cmf --local-dir .
cortiq verify mmh3-turbo-clipproj4b-q4tp.cmf

cortiq animate mmh3-turbo-clipproj4b-q4tp.cmf ^
  --prompt "A corgi in a chef hat flipping a pancake, sizzling sounds and a cheerful bark." ^
  --width 512 --height 288 --frames 39 --steps 4 --seed 42 ^
  --out corgi.avi

(That is Windows line-continuation; on Linux/macOS use \.) You get corgi.avi and corgi.wav beside it — no ffmpeg involved.

Two things worth knowing:

• The GPU is not opt-in any more. cortiq probes this file's own first qkv weight against the host at startup and takes the device arm only if they agree, so CMF_MMH3_GPU=1 is no longer needed. CMF_MMH3_GPU=0 still forces the host path if you want to compare.
• --height 256 instead of 288 is close to free. The video VAE decodes in 256-pixel tiles and grows the overlap, so a 288-pixel edge costs six tiles where 256 costs three — for 89% of the pixels. It roughly halves the decode.

On CUDA: still no, and it is not planned to change soon — the device path is Vulkan through wgpu, which is why it ran on your card at all without a CUDA toolkit. Vulkan is what your driver already ships.

If you do run it, I would like the numbers: RUST_LOG=info prints a per-stage breakdown, and a 3080 20 GB is exactly the card this build exists for. My own timings are from a 3090 and they drifted enough between runs that I would rather have yours than quote mine.

cmd

➜ nvidia-smi --query-gpu=driver_version --format=csv,noheader
610.62

➜ cortiq --version
cortiq 0.5.73

➜
$env:RUST_LOG="info";$env:CMF_GPU_ADAPTER=1;$env:CMF_MMH3_GPU=1;`
cortiq animate mmh3-turbo-clipproj4b-q4tp.cmf `
  --prompt "A corgi in a chef hat flipping a pancake, sizzling sounds and a cheerful bark." `
  --width 512 --height 288 --frames 39 --steps 4 --seed 42 `
  --out corgi_clipproj4b.avi

logs

2026-08-14T01:52:15.709051Z  INFO cortiq_core::format: Opened CMF v2: minimax-h3-av | 2086 tensors | 0 masks | vocab embedded | 14130.0 MB
2026-08-14T01:52:15.987276Z  INFO cortiq_engine::tokenizer: Tokenizer loaded: 151669 vocab, 151387 merges, 26 added, eos=Some(151643)
encode 0/1 (0.3s)
encode 1/1 (1.9s)
2026-08-14T01:52:18.349425Z  INFO wgpu_hal::vulkan::adapter: Found 6 cooperative matrix configurations supported by wgpu
2026-08-14T01:52:18.945789Z  INFO cortiq_engine::gpu_wgpu: wgpu GPU path: on (NVIDIA GeForce
RTX 3080 / Vulkan, discrete, weight budget 19229 MB, q1 8 rows/128 threads)
2026-08-14T01:52:18.992870Z  INFO cortiq_engine::gpu_wgpu: pipeline cache: empty, will compile (C:\Users\bingo\AppData\Local\Temp\cortiq-pipelines-fcab4206908346fe.bin)
denoise 1/4 (26.2s)
denoise 2/4 (39.1s)
denoise 3/4 (51.9s)
denoise 4/4 (64.7s)
video vae 0/1 (64.7s)
video vae 1/1 (95.4s)
audio vae 0/1 (95.4s)
audio vae 1/1 (103.8s)
2026-08-14T01:53:59.552187Z  INFO cortiq_engine::videogen: stages: prepare 0.3s · text encode 1.6s · denoise 62.8s · video vae 30.6s · audio vae 8.5s
corgi_clipproj4b.avi: 512x288, 39 frames at 24 fps (1.62s), 1.62s of 32000 Hz stereo, 2.2 MB
in 105.1s
corgi_clipproj4b.wav: stereo PCM

monitor snippet

put together a handy script with help from ai to monitor gpu-related process usage via nvidia-smi.

09:53:29 57% 14.6Gi 󱐋190.6W 󰈸55℃
         └─ cortiq 󰻠162.5% 13.8Gi
09:53:32 50% 14.6Gi 󱐋186.7W 󰈸55℃
         └─ cortiq 󰻠116.0% 13.9Gi
09:53:34 61% 14.6Gi 󱐋185.2W 󰈸55℃
         └─ cortiq 󰻠130.9% 13.9Gi
09:53:36 51% 14.6Gi 󱐋182.9W 󰈸55℃
         └─ cortiq 󰻠219.6% 13.8Gi
09:53:38 58% 14.6Gi 󱐋192.3W 󰈸55℃
         └─ cortiq 󰻠152.3% 13.8Gi
09:53:40 58% 14.6Gi 󱐋194.6W 󰈸55℃
         └─ cortiq 󰻠126.4% 13.9Gi
09:53:42 51% 14.6Gi 󱐋193.2W 󰈸56℃
         └─ cortiq 󰻠135.3% 13.9Gi
09:53:44 51% 14.6Gi 󱐋185.9W 󰈸56℃
         └─ cortiq 󰻠205.8% 13.9Gi
09:53:46 53% 14.6Gi 󱐋189.8W 󰈸56℃

comment

Thanks so much for the swift update and support!

The drop in execution time is insane—from over 1,200s down to around 100s—while keeping the main animation remarkably intact. Resource-wise, a 16GB VRAM GPU should handle this smoothly now. The mmh3-turbo-clipproj4b-q4tp.cmf model will likely be a huge hit on budget, high-VRAM non-NVIDIA cards (like AMD or Intel).
Noticed from the logs that NVIDIA GPU usage isn't fully saturated yet, so there's still room to squeeze out more performance.

Overall, great work—this is huge for non-CUDA users!

q4tp video

On Mac mini 24gb, using mmh3-turbo-clipproj4b-q4tp.cmf, with 1 frame (to save time) i get this bad quality. anyways to mix in more loras to fix faces, quality etc?

second question - how to add start/end frames that mmh3 allows for?

🚀 Generating video: a_corgi_in_a_chef_hat_flipping.avi
📝 Prompt: A corgi in a chef hat flipping a pancake, sizzling sounds and a cheerful bark.
📐 Resolution: 512x256
🎞️ Frames: 39 (~1.6 seconds)
⚙️ Steps: 4
2026-08-14T13:58:03.827493Z INFO cortiq_core::format: Opened CMF v2: minimax-h3-av | 2086 tensors | 0 masks | vocab embedded | 14130.0 MB
2026-08-14T13:58:03.913457Z INFO cortiq_engine::tokenizer: Tokenizer loaded: 151669 vocab, 151387 merges, 26 added, eos=Some(151643)
encode 0/1 (0.1s)
encode 1/1 (1.0s)
2026-08-14T13:58:04.927438Z INFO cortiq_engine::gpu_metal: Metal GPU path: on (Apple M4)
denoise 1/4 (31.6s)
denoise 2/4 (61.9s)
denoise 3/4 (92.4s)
denoise 4/4 (122.7s)
video vae 0/1 (122.7s)
video vae 1/1 (164.3s)
audio vae 0/1 (164.3s)
audio vae 1/1 (173.8s)
2026-08-14T14:00:57.596742Z INFO cortiq_engine::videogen: stages: prepare 0.1s · text encode 0.9s · denoise 121.8s · video vae 41.5s · audio vae 9.5s
a_corgi_in_a_chef_hat_flipping.avi: 512x256, 39 frames at 24 fps (1.62s), 1.62s of 32000 Hz stereo, 2.4 MB in 173.9s
a_corgi_in_a_chef_hat_flipping.wav: stereo PCM

✅ Finished! Raw video saved to a_corgi_in_a_chef_hat_flipping.avi (took 173.9 seconds)

@hololabs — three levers, in the order of what they'll do for that clip:

1. Resolution. The trained short edge is 768 — at 512×256 you're asking the model to paint faces with a third of the detail it learned. Try --width 768 --height 384 (multiples of 32); denoise time scales with pixels, so roughly 2.3× your current two minutes.
2. Steps. The merged Turbo LoRA targets 4; 6–8 still buys a little.
3. The encoder. clipproj4b approximates the 32B encoder's conditioning with a 4B model and a learned projection. Composition and motion survive the approximation; fine detail like faces is exactly where it shows. That's the trade that makes the file fit a 24 GB machine.

On LoRAs: not at runtime. A .cmf is a baked artifact — the 4-step Turbo LoRA itself was merged at pack time, which is why the file is the turbo model. The path for a face/quality LoRA today is to merge it into the source checkpoint (ComfyUI or diffusers) and pack that. Runtime LoRA mixing isn't planned for the inference path: one file with no sidecar weights is the point of the format.

Your start/end frame question is answered in #2 — short version: the flags exist (--first-frame/--last-frame), but you need an mmh3-turbo-fl2va-* file for them; the clipproj file doesn't carry the vision tower that path needs.

0.5.95 is out, and the number that moved is the eight-bit one. RTX 5090, MiniMax-H3, 22 frames, four steps, alternating runs:

At 512×288 four-bit is still ahead on denoise (21.5 s against 29.6 s); at 768×448 the order reverses and eight-bit is 1.9× faster overall, with its VAE decoding in 11.7 s against 38.5 s. int8 now unpacks each weight into an f16 plane once and sends both halves of the FFN to the matrix units; the four-bit path still unpacks inside its FFN kernel, per tile of activations, which costs more the wider the panel. That is the next thing to fix, and it belongs to q4tp.

Also in this release: --stream-chunk renders a clip in chunks against the RAVEN streaming adapter (mmh3-raven-streaming-q4tp.cmf is up), and a 321-frame clip that previously died at a 2 GiB storage binding now renders — 354 s chunked, 404 s whole.

0.5.96 — and this one is for your card, not for the Macs.

You asked (twice, and rightly) for the internals. Here they are, because they are also the answer to why this release exists: an evening of profiling a four-bit render found that three quarters of a fused FFN call was host time, not GPU time. RTX 5090, 768×448, per call:

DiT  shape 5376x14336, b=2447 : host 126.2 ms | card 41.4 ms
VAE  shape 2048x8192,  b=1797 : host 100.2 ms | card 12.6 ms

The single biggest piece of that host time was scanning the activation panel for max|x| — 13.2 M floats, one core, before anything is submitted — to compute the f16 scale. The card already computes exactly that scale for operands it holds itself, and the kernel already reads it from a buffer. Moving the scan there, two runs each way:

Output byte-identical across all four runs. CMF_FFN_HOST_SCAN=1 puts the old arm back if you want to see it on your own hardware — and on a 4650G I would expect this to matter more than it did on the pod's 27-core box, because it is exactly the kind of work a slower CPU serializes the whole render behind.

Two things that did not work, so you do not spend a weekend on them: caching the unpacked f16 plane across calls (eight alternating runs, one fast outlier, spread inside a single arm 10%), and lowering the four-bit plane gate (CMF_Q4TP_PLANE_MIN=8 — worse at both sizes).

On your question about building a CMF from scratch — the format and the codecs are written up rather than left to be reverse-engineered:

• FORMATS.md — every codec in the container: q4tp (4.17 bpw, a 5-bit rung ladder per group), q2tp, q8_row, q8_2f (two-field int8, w = q·row[o]·col[i]), q1t.
• cmf.md — the container itself: header, tensor table, hashes, what cortiq verify checks.
• The packer is cortiq animate-pack and the general converter cortiq convert; both take safetensors and --quant. Source: https://github.com/infosave2007/cmf

On the VAE specifically, since that is what you asked: this container packs the 3D VAE in the same codec as the rest rather than keeping it f16 — that is why the file is 14.5 GB and not 20 GB. It is worth knowing that the VAE is the component most sensitive to it, so it is the one to check first with cortiq verify and a side-by-side decode when you build your own.