vector-institute/open-pmc-18m-clip

Open-PMC-18M: A High-Fidelity Large Scale Medical Dataset for Multimodal Representation Learning

🎉 Accepted at the International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI) 2026 🎉

---

A biomedical CLIP model trained on ~18M image–text pairs (Open-PMC-18M) from PubMed Central. It uses the BiomedCLIP architecture — a ViT-B/16 image encoder and a PubMedBERT text encoder — projecting images and text into a shared 512-d space for zero-shot classification and image↔text retrieval.

• Image encoder: ViT-B/16 (timm vit_base_patch16_224), 224×224 · ~86M params
• Text encoder: PubMedBERT (microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract), 256 tokens · ~110M params
• Embedding dim: 512
• Repo: https://github.com/VectorInstitute/pmc-data-extraction

Install

pip install open_clip_torch

Load the encoders

The model loads in one line — this builds both encoders and loads the trained weights:

from open_clip import create_model_from_pretrained, get_tokenizer

model, preprocess = create_model_from_pretrained("hf-hub:vector-institute/open-pmc-18m-clip")
tokenizer = get_tokenizer("hf-hub:vector-institute/open-pmc-18m-clip")

image_encoder = model.visual   # ViT-B/16  → image → 512-d
text_encoder  = model.text     # PubMedBERT → text  → 512-d

Encode images and text

import torch
from PIL import Image

image = preprocess(Image.open("figure.jpg")).unsqueeze(0)          # (1, 3, 224, 224)
text  = tokenizer(["a chest x-ray", "a brain MRI"], context_length=256)

with torch.no_grad():
    image_emb = model.encode_image(image, normalize=True)          # (1, 512)
    text_emb  = model.encode_text(text,  normalize=True)           # (2, 512)

similarity = image_emb @ text_emb.T                                # cosine similarity

Zero-shot classification

labels = ["chest x-ray", "brain MRI", "histopathology slide"]
text = tokenizer(["this is a photo of " + l for l in labels], context_length=256)

with torch.no_grad():
    image_emb = model.encode_image(image, normalize=True)
    text_emb  = model.encode_text(text,  normalize=True)
    probs = (model.logit_scale.exp() * image_emb @ text_emb.T).softmax(dim=-1)

print({l: round(p, 4) for l, p in zip(labels, probs[0].tolist())})

Evaluation

The pmc-data-extraction repo provides the full mmlearn benchmark suite (zero-shot retrieval on Quilt-1M / MIMIC-CXR / DeepEyeNet, and classification). It reads the raw .pt checkpoint directly:

CKPT=/path/to/open_pmc_18m_clip.pt
mmlearn_run 'hydra.searchpath=[pkg://openpmcvl.experiment.configs]' \
    +experiment=biomedclip_localckpt_retrieval job_type=eval \
    +datasets@datasets.test.quilt=Quilt datasets.test.quilt.split=val \
    +datasets/transforms@datasets.test.quilt.transform=biomedclip_vision_transform \
    task.encoders.text.checkpoint_path="$CKPT" \
    task.encoders.rgb.checkpoint_path="$CKPT"

See scripts/eval/ for ready-made retrieval and classification scripts.

Files

File	Description
open_clip_pytorch_model.bin	Inference weights — enables the one-line hf-hub: load above.
open_clip_config.json	OpenCLIP architecture + preprocessing config.
open_pmc_18m_clip.pt	Full training checkpoint (weights + optimizer + epoch/step/samples_seen), for resuming training or the eval pipeline.

Limitations

Research artifact — not a medical device. Trained on published figure–caption pairs, which differ from raw clinical images and may carry publication and demographic biases. Do not use for clinical decisions.

Citation

@article{baghbanzadeh2025open,
  title={Open-pmc-18m: A high-fidelity large scale medical dataset for multimodal representation learning},
  author={Baghbanzadeh, Negin and Islam, Mohammed Saidul and Ashkezari, Sajad and Dolatabadi, Elham and Afkanpour, Arash},
  journal={arXiv preprint arXiv:2506.02738},
  year={2025}
}