baguettotron-vision-vqa

Baguettotron-VLM is an open, fully-reproducible, multilingual Vision-Language Model in the sub-1B parameter class. It extends PleIAs/Baguettotron — a 321M text-only reasoning SLM — with visual capabilities via the InternViT-300M-448px-V2.5 vision encoder and a lightweight MLP projector, for a total of ~628M parameters. It inherits six European languages (EN, FR, DE, ES, IT, PL) from the Baguettotron backbone.

Two checkpoints are published:

baguettotron-internvit-alignment — the projector-only warmup. It describes images, and nothing more.
baguettotron-vision-vqa — instruction-tuned on top of it. It goes past plain description and follows visual instructions, so prefer it for a richer chat experience.

Apache 2.0. Both live in the Baguettotron-VLM collection. Source: github.com/andreagemelli/baguettotron-vlm · Write-up: andreagemelli.me/posts/baguettotron-vlm

Instruction-tuned checkpoint. Full fine-tuning of the projector + LLM on The Cauldron (47 subsets) mixed with 10% PleIAs/SYNTH text-only data. It follows visual instructions and answers VQA directly, without <think> traces. It is at its best on short factual questions — multiple choice, counting, "what is in the picture" — and weakest on long open description.

Architecture

Image (448×448)
  → InternViT-300M-448px-V2.5 (304M, frozen) → 1024 tokens × 1024d
  → Pixel unshuffle (factor=2)                → 256 tokens × 4096d
  → MLP projector (2-layer, ~2.7M)            → 256 tokens × 576d
  → Interleave with text tokens
  → Baguettotron (321M, Llama arch, 80L, h=576)

Total: ~628M parameters

Usage

pip install "transformers>=4.56,<5" torch pillow timm einops accelerate

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
from PIL import Image

model = AutoModelForImageTextToText.from_pretrained(
    "andreagemelli/baguettotron-vision-vqa",
    trust_remote_code=True,
    dtype=torch.bfloat16,
    device_map="auto",  # also tested on Apple Silicon (mps) and CPU
)
processor = AutoProcessor.from_pretrained(
    "andreagemelli/baguettotron-vision-vqa",
    trust_remote_code=True,
)

image = Image.open("photo.jpg").convert("RGB")
inputs = processor(
    messages=[{"role": "user", "content": "<image>\nDescribe the image concisely."}],
    image=image,
)
inputs = {k: v.to(model.device) for k, v in inputs.items() if v is not None}

print(model.chat(**inputs))

Example output

Greedy, prompt Describe the image concisely. — verbatim output. Images from COCO val2017.

	output
	`The image is a picture of two cats sitting on the couch. The cat has its head down and legs are raised.`
	`The image shows a bear sitting on the grass.`
	`A sign that says stop.`
	`The red bus is on the road.`

Chat template

Trained with the assistant turn always starting with a closing </think> token ("no-thinking" mode) followed by the answer, because The Cauldron carries no reasoning traces. The processor pre-fills that </think>\n prefix after <|im_start|>assistant\n when add_generation_prompt=True (the default), so you pass the user message and the model completes the answer.

For the raw prompt without the auto-prefix, pass add_generation_prompt=False to the processor.

Limitations

Resolution ceiling. One 448×448 crop → 256 visual tokens puts document text at roughly 2–4 px/char. OCR, charts and documents are out of reach by architecture, not by budget. Neither published checkpoint reads text in an image.
Hallucinations, especially on fine-grained or text-heavy questions. Asking for more than a short answer it starts inventing some funny stuff - so be aware when using it thay may produce some hallucinated content.
Would benefit from more instruction tuning. The visual instruction budget is small relative to the size of the data mixture, and it shows on long or compositional prompts.
Multilingual capability is inherited, not verified. The backbone covers six languages; the VLM was never evaluated on non-English benchmarks.

Tested against transformers 4.57. Newer major versions may need adjustments.

Contributions and suggestions are very welcome — issues, PRs, and ideas for better data mixes, training recipes, or evaluation setups are all appreciated. Open an issue or PR on the GitHub repo.

Citation

If you use or extend Baguettotron-VLM in your research, please cite it:

@misc{gemelli2026baguettotronvlm,
  title        = {Baguettotron-VLM: An Open, Reproducible, Multilingual Sub-1B Vision-Language Model},
  author       = {Gemelli, Andrea},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/andreagemelli/baguettotron-vision-vqa}},
  note         = {Code: \url{https://github.com/andreagemelli/baguettotron-vlm}}
}