Domofon-XML-0.8b

A small, fast document-to-XML converter. Given raw document text, it wraps the content into semantic XML tags (<document>, <section>, <heading>, <paragraph>, <quote>, <list>, <aside>, ...) without changing the text itself. Built for large-scale pretraining-data pipelines where frontier-grade XML structure markup is needed at high throughput.

This is a full fine-tune of Qwen3.5-0.8B-Base on domofon/Document-XML-100k (~118k raw-text → XML pairs, English + Russian). The hybrid linear-attention (Gated DeltaNet) architecture makes long-document inference significantly faster than a pure-attention model of the same size.

Usage

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "DomofonResearch/Domofon-XML-0.8b"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="cuda")

raw_text = "Your raw document text goes here."

prompt = f"<|im_start|>user\n{raw_text}<|im_end|>\n<|im_start|>assistant\n"
ids = tok(prompt, return_tensors="pt").to("cuda")
out = model.generate(
    **ids,
    max_new_tokens=4096,
    do_sample=False,
    eos_token_id=tok.convert_tokens_to_ids("<|im_end|>"),
    pad_token_id=tok.pad_token_id,
)
xml = tok.decode(out[0, ids["input_ids"].shape[1]:], skip_special_tokens=True)
print(xml)

Notes:

• Greedy decoding (do_sample=False) is recommended — the task is deterministic.
• Trained on sequences up to 4096 tokens (input + output). For longer documents, split them into chunks before conversion.
• The model is text-only (no vision tower), so it loads fast and stays small.

Inference speed

Metric	Throughput
Prefill (prompt processing, 8k-token doc)	~110,000 tok/s
Decode, single stream	~44 tok/s
Decode, batch of 8 (aggregate)	~311 tok/s

Measured on 1 x NVIDIA H100 80GB (SXM), bf16, greedy decoding, HF transformers with the torch fallback kernels. A serving stack with fused recurrent kernels (e.g. vLLM) is significantly faster on decode; the hybrid linear-attention architecture keeps long-document prefill cheap.

Generation examples

Russian recipe blog post (input: raw text, 675 output tokens):

<document>   <paragraph>Местные яблоки уже созрели и пора печь пироги. Я хотела
сначала написать рецепт пирожков, но потом, как-то незаметно для себя,
переключила внимание на яблочный пирог. ...</paragraph>   ... </document>

English biography (input: raw text, 437 output tokens):

<document>   <section>     <paragraph>Born to a Jewish father and a Catholic
mother, Otto grew up in a city well known for its musical tradition. The younger
of two children, Otto began studying the piano at age 10. ...</paragraph>     ...
</section> </document>

On held-out samples from the verified split the model reproduces the reference XML structure and preserves the source text verbatim (token-level content is copied, only markup is added).

Training details

• Base model: Qwen/Qwen3.5-0.8B-Base (text weights only, vision tower dropped)
• Dataset: domofon/Document-XML-100k (splits good + verified, 117,983 pairs; samples over 4096 tokens filtered out)
• Format: chat-style prompt (<|im_start|>user raw text, <|im_start|>assistant XML), loss on the completion only, sequences packed to 4096 tokens
• Full fine-tune, bf16, 2 epochs, lr 1e-5 (cosine, 3% warmup), global batch 32 sequences × 4096 tokens (~131k tokens/step)
• Hardware: 1 × NVIDIA H100 80GB (SXM)

Limitations

• Output schema follows the training data (English/Russian web documents); other languages or exotic schemas may degrade.
• Very long documents must be chunked; the model does not see beyond 4096 tokens.
• XML validity should be verified downstream for strict pipelines.