Download docs/audio2sph_l.md from IN03X/SAID: direct link, hf CLI and curl.
- Browser
- Download file 13.7 kB
-
https://huggingface.co/IN03X/SAID/resolve/main/docs/audio2sph_l.md
- Command line
-
hf download hf://IN03X/SAID/docs/audio2sph_l.md
-
curl -L -o audio2sph_l.md https://huggingface.co/IN03X/SAID/resolve/main/docs/audio2sph_l.md
Audio2Sph-L and SAID-L
2026-11-11 update — extension to the paper models.
Audio2Sph-L is an additional model developed after the paper experiments. The paper's Audio2Sph, SAID-PaSST and SAID-AudioMAE remain unchanged.
The main changes are an additional 13-class output head alongside the Any head and an increase in input duration from 2 seconds to 4 seconds. Audio2Sph-L code and pretrained weights are available. SAID-L has training code but no pretrained full-model weights or reported performance results.
Model differences
| Audio2Sph with panoramic decoder | Audio2Sph-L | |
|---|---|---|
| Input | Four-channel audio, 48 kHz, 2 seconds | Four-channel audio, 48 kHz, 4 seconds |
| Spherical features | 16 channels on a 45 × 90 grid | 128 channels on a 45 × 90 grid |
| Feature formation | Separate attention at three encoder scales, followed by summation | Joint multi-scale attention with eight spatiotemporal refinement blocks |
| Output decoder | Shallow spatial decoder | Separate spatiotemporal residual decoders for Any and classes |
| Output maps | One class-agnostic Any map at 180 × 360 | One Any map and 13 DCASE class maps at 45 × 90 |
The L model uses a 256-channel internal spherical state before projecting to the 128-channel exported features. Its Any and class decoders have six and eight residual blocks, respectively. It predicts class maps directly from audio, without a PaSST or AudioMAE Class Feature Encoder.
Each class map represents the union of sources of that class. It is not a list of separately identified source instances. The Any map represents activity across all classes.
Training supervision
Audio2Sph-L uses the original multi-loss training objective:
- Class maps: weighted binary cross-entropy and Dice on active class maps.
- Any map: weighted binary cross-entropy and Dice on all frames.
- Wrong-class penalty and class-margin penalty, each with weight 0.25.
For binary cross-entropy, pixels with positive soft targets have weight 30; other pixels have weight 1. Each term is normalized by its total pixel weight. The class-margin parameter is 0.25. The Any target is the maximum of the 13 class targets at each pixel.
For a four-second training clip, the network produces 41 temporal positions.
SourceBank and prepared-scene training align logits to the 40 target frames by linear interpolation
with align_corners=False, before applying the losses.
The historical L pretraining data include static, extended and moving sources, same-class crossing sources, and measured room responses. This differs from the original two-second static-scene recipe.
Relationship to SAID
Audio2Sph-L is a standalone model. Its feature dimensions differ from the paper Audio2Sph, so its checkpoint is not a drop-in initialization for the published SAID models. SAID-L connects its 128-channel spherical features directly to Sph2Imaging, whose input convolution accepts 128 rather than 16 channels. The instance decoder, Class Feature Encoder (PaSST or AudioMAE), and source-instance losses follow SAID. The pretrained Any and class-map heads are retained for checkpoint compatibility but are frozen and skipped in SAID-L; they do not replace the instance decoder.
SAID-L model and training code are included. No pretrained SAID-L checkpoint or performance results are provided.
Train Audio2Sph-L with SourceBank
The default training route uses SourceBank and four-second simulated scenes.
Install pip install -e '.[render]', download the source datasets following
Data, and set the sourcebank and renderer paths and dataset authorizations in
configs/data_sourcebank_l.yaml. Then run:
said prepare --config configs/training/audio2sph_l.yaml
said train --config configs/training/audio2sph_l.yaml
prepare builds the SourceBank index, independent source-identity train/validation
partitions, Eigenmike responses and the TAU response subset. train renders four-second audio and
13-class spatial targets online; no private CSV or prepared NPZ is required.
The default starts from random weights and saves to runs/audio2sph_l/.
Set load_audio2sph_ckpt to a local L checkpoint to initialize from weights.
The supplied learning rates (audio2sph_learning_rate: 1e-6 and
learning_rate: 1e-5) and 100,000-step budget are starting settings, not a
validated from-scratch reproduction of the historical 543,000-step model.
Loading weights initializes a new optimizer; --resume restores a saved
training state, including its optimizer and step.
The four-second renderer includes static point sources, extended sources, measured-room point sources, moving sources and same-class crossings. Their sampling probabilities are 26.67%, 26.67%, 26.67%, 15% and 5%, respectively. It also applies source activity, source gains, room backgrounds and non-target interference. Targets contain the target sources only; same-class maps use a pixelwise maximum.
Additional inputs for renderer:
tau_root: extracted TAU Spatial Room Impulse Response Database, containingTAU-SRIR_DB/andTAU-SNoise_DB/.audioset_ontology: AudioSet ontology.json, used to exclude target classes from FSD50K interference.prepared_root: a writable directory for generated assets. Keep it on a volume with sufficient free space.training_use_authorized: set totrueafter reviewing the TAU and source-dataset terms.
The supplied source pool uses the four downloadable SourceBank corpora described in Data. It does not reconstruct the historical checkpoint's exact 500k-scene pool, which also included ASA2 stems and locally indexed AudioSet audio.
Resources and acceptance
Use batch size 1 initially. The four-second SourceBank routes have passed
short CUDA training, validation and checkpoint-saving checks on an RTX 4090
(24 GB), with two data workers. L forward computation uses BF16 on CUDA;
matching and losses use FP32. The CLI enables expandable CUDA allocations
unless PYTORCH_CUDA_ALLOC_CONF is already set. More workers increase CPU/RAM
usage, not GPU batch size; lower training.num_workers if resources are shared.
In our preparation check, generated SourceBank/TAU assets occupied about 1.3 GiB, in addition to the downloaded corpora. Reserve several GiB for these assets and additional space for retained model and optimizer checkpoints. Online rendering varies by scene: same-class crossings and moving sources can take longer than static sources. The progress bar measures completed training steps, including data generation.
These checks establish runnable training, not convergence or reproduction of the historical checkpoint's scores. CPU training is also supported.
Optional DCASE training / fine-tuning
Download DCASE audio and labels following Data, then set
dcase_recordings.root in configs/data_said_l.yaml:
said train --config configs/training/dcase_audio2sph_l.yaml
This optional entry point loads runs/audio2sph_l/latest.ema.ckpt from the
SourceBank training above. Set load_audio2sph_ckpt to another local L
checkpoint if needed; setting it to null explicitly starts from random weights.
The loader builds the 13 class targets automatically by taking the pixelwise
maximum of valid source maps within each class. No CSV or NPZ conversion is
needed. DCASE's 40 official frames use the first 40 predictions of each
four-second clip; the endpoint prediction is excluded.
Use the same configuration for inference to retain this frame alignment:
said infer recording.wav --config configs/training/dcase_audio2sph_l.yaml --checkpoint runs/audio2sph_l_dcase/latest.ema.ckpt
The multi-term losses are retained. This DCASE training entry point is not a reproduction of the released Audio2Sph-L checkpoint's mixed synthetic-data pretraining described above. The five-family renderer is included; the historical expanded source collection is not reconstructed by the public recipe.
Train SAID-L with SourceBank
After the Audio2Sph-L training above, the default SAID-L configuration loads
runs/audio2sph_l/latest.ema.ckpt and shares configs/data_sourcebank_l.yaml.
Run:
said train --config configs/training/said_l.yaml
For another Audio2Sph-L checkpoint, set load_audio2sph_ckpt in said_l.yaml.
For optional DCASE fine-tuning, use configs/training/dcase_said_l.yaml and
set configs/data_said_l.yaml. To fine-tune a complete SAID-L model, set
load_said_ckpt and clear both component initialization fields.
This trains a new Sph2Imaging decoder on four-second clips, initialized with
Audio2Sph-L and the frozen PaSST Class Feature Encoder from the paper checkpoint.
The latter downloads automatically. To use AudioMAE, change
class_feature_encoder to audiomae and load_class_feature_encoder_ckpt to
../../checkpoints/said_audiomae.ckpt in the same training configuration.
These are starting settings; long-training SAID-L performance has not been evaluated.
The pretrained Audio2Sph-L encoder uses audio2sph_learning_rate; the new
decoder uses learning_rate. The Class Feature Encoder remains frozen.
The SourceBank route supplies 40 source-map frames at 45 × 90. SAID-L
supervises its first 40 output frames and uses bilinear 180 × 360 targets
for the refinement loss. Standalone Audio2Sph-L retains its historical
41-to-40 prediction interpolation and multi-term class/Any losses.
Training saves weights and resumable state under runs/said_l/.
Use the saved weights for inference:
said infer recording.wav --config configs/training/said_l.yaml --checkpoint runs/said_l/latest.ema.ckpt
SAID-L returns labeled source-instance maps in the same JSON format as SAID.
Use said train --config configs/training/said_l.yaml --resume PATH to restore
the complete training state. To start a new run from your SAID-L weights,
set load_said_ckpt to their path and set both component checkpoint fields
to null.
Inference and demos
To use weights from your own training, run:
said infer recording.wav --config configs/training/audio2sph_l.yaml --checkpoint runs/audio2sph_l/latest.ema.ckpt
The following commands automatically download the released pretrained
checkpoint to checkpoints/audio2sph_l.ckpt on first use:
said demo --model audio2sph_l
said infer recording.wav --model audio2sph_l
Input channel selection is the same as for SAID: the complete Eigenmike
recording or capsules [6, 10, 26, 22] in that order, using 1-based indices.
Long recordings are processed in four-second windows. The final window is
zero-padded; saved frames cover only the original recording duration.
The default JSON contains a dense class_maps[13,45,90] array and an
any_map[45,90] array per frame, at 10 frames per second. Class names and
axis names are included in the file. These are class-union maps, so this
format is distinct from SAID's source-instance DCASE JSON.
Rows use colatitude from 0° (up) to 180° (down), with 45 evenly spaced positions. Columns use azimuth from 0° to 360°, with 90 evenly spaced positions. Both axes include their endpoints, matching the training targets; the first and last columns therefore address the same direction.
Add --format npz for compressed arrays or --visual for a video preview.
NPZ chunks contain class_maps[T,13,45,90] and any_maps[T,45,90], with
frame offsets in manifest.json. Visualization enlarges maps to 180 × 360
for display; saved predictions retain the native 45 × 90 resolution.
Fine-tuning with prepared scenes
Install the training dependencies described in Training.
Set prepared_scenes.root in configs/data_l.yaml, then run:
said train --config configs/training/prepared_audio2sph_l.yaml
The dataset directory contains train/*.npz and validation/*.npz.
Each file represents one four-second example and contains exactly:
| Array | Shape | Meaning |
|---|---|---|
audio |
[4,192000] |
Float32 waveform, 48 kHz, capsule order above |
class_maps |
[40,13,45,90] |
Float32 class-union targets in [0,1], at 10 fps |
Class order follows Data. Targets include frame activity; sources
of the same class are combined by a pixelwise maximum. The Any target is
derived during training. Use separate scenes for training and validation.
The supplied configuration loads checkpoints/audio2sph_l.ckpt and uses
the multi-loss supervision described above.
This interface supports fine-tuning on prepared scenes. The original L training used a separate four-second scene-generation pipeline; the paper's two-second Online Scene Generation recipe does not reproduce that dataset. The default SourceBank entry point above provides online four-second generation.
Release status
The selected checkpoint is step 543,000 of the original multi-loss training run. The public SourceBank routes have passed short training, validation and checkpoint-saving checks for both L models. The checkpoint is available on Hugging Face. Existing paper checkpoints and commands are unchanged.
The L source pools extend the paper data with ASA2 stems, AudioSet-indexed recordings, TAU room responses/backgrounds, and additional FSD50K clips. Source recordings are not included in this repository. Their terms remain separate from the software license. The weights are provided for non-commercial research under the same model-weight license as the paper checkpoints. See the checkpoint card for provenance and applicable notices.