---
pipeline_tag: image-to-text
tags:
- openvision2
- image-captioning
- vision-encoder
library_name: open_clip
---

# OpenVision2 · L/14 @224 — vision encoder + caption decoder

This repo now contains the **full OpenVision2 generative model**: the **vision encoder**
(originally released here) **plus the caption text decoder** it was jointly trained with.
Together they map an image -> a descriptive caption. The encoder files are unchanged;
only the decoder (and this card) were added.

## Files
| file | role |
|---|---|
| `open_clip_pytorch_model.bin`, `open_clip_config.json` | L/14 vision encoder (open_clip format) |
| `caption_decoder.safetensors` | caption text decoder weights |
| `text_decoder_config.json` | decoder architecture config |
| `modeling_openvision2_decoder.py` | standalone PyTorch decoder + `generate()` |
| `bert_base_vocab_bos_eos.txt` | tokenizer vocab (`[PAD]=0 [bos]=1 [eos]=2`) |
| `caption_example.py` | end-to-end image -> caption demo |

## Decoder architecture
A **concat / prefix-LM** autoregressive transformer (not a CoCa cross-attention decoder):
ViT patch tokens are linearly projected and **prepended as a bidirectional prefix**, and
text is generated **causally** while attending to all image tokens.

`12 layers · width 768 · 12 heads · mlp 3072 · vocab 32000 ·
pre-LN · gelu(tanh) · LayerNorm eps 1e-6 · no positional embedding on the text stream.`
It consumes the encoder's pre-final-norm patch tokens (open_clip `output_tokens=True`).

## Usage
Needs the patched `open_clip` providing `create_vision_encoder_and_transforms`
(https://github.com/UCSC-VLAA/OpenVision), plus `torch`, `safetensors`, `pillow`.
See `caption_example.py`:

```bash
python caption_example.py --image your.jpg
```

## Notes
- Encoder and decoder are a **matched pair** exported from the same training checkpoint.
- Captions are LLaVA-style **dense** descriptions (multi-sentence, detailed).