opsis-v1-nano-g3

opsis-v1-nano-g3 is a local visual-ingestion co-processor for retrieval systems. It takes one image or document crop and returns tagged discovery text or a Markdown table, so visual content can be indexed beside its source. The published FP32 ONNX graphs run on CPU without a hosted vision API.

Results at a glance

Measured on 600 visually reviewed cases with greedy FP32 ONNX inference on the release CPU:

Metric	Result
Image-kind accuracy	99.5%
Valid Markdown tables	98.8%
Exact table matches	3.0%
Median CPU latency	3.635 s
Peak process memory	2.4 GiB

The full evaluation below gives the remaining metrics and benchmark limits.

Native V1 Output Modes

Mode	Native output	Intended use
description	<description>concise factual text</description>	Ordinary images, charts, and diagrams that need searchable RAG metadata.
table	<table>GitHub-flavored Markdown table</table>	Table images that need recoverable rows and cells rather than prose.

Charts stay in description mode and should include visible labels, values, and trends. Diagrams stay in description mode and should include visible nodes and relationships.

Output Contract

The raw Hugging Face model output is one tagged string. It is not unrestricted assistant prose. Decode it by wrapper:

Wrapper	Parsed kind	Decoding
<description>...</description>	description	Strip the wrapper and retain concise factual text.
<table>...</table>	table	Strip the wrapper and retain the complete Markdown table.

A caller can normalize either result into one object:

{
  "kind": "description",
  "text": "A line chart shows quarterly revenue rising from $12M in Q1 to $18M in Q4.",
  "latency_seconds": 3.64,
  "model_id": "yafitzdev/opsis-v1-nano-g3",
  "prompt_version": "rag-image-v3"
}

The model does not return bounding boxes, OCR confidence, source coordinates, PDF text blocks, or verified numeric facts. Preserve the source image or page crop beside generated text when exact visual evidence matters.

Intended Use

Use this model when a RAG or retrieval system needs local visual signals for:

• indexing ordinary images with concise discovery descriptions,
• converting compact table images into Markdown,
• recording visible chart labels, values, and trends,
• recording labeled diagram nodes and relationships,
• enriching visual regions routed out of otherwise native-text PDFs,
• keeping document ingestion local on CPU-only systems.

This model is not intended to replace reliable native table extraction, parse full scanned pages as a layout engine, verify high-stakes measurements, or replace source review when exact cells and labels matter.

Input Format

The model accepts one raster image per inference. For direct Transformers use, pair the image with the v1 parser prompt:

Parse this image for RAG search. If it is a table, output only the complete
table as Markdown. Otherwise output only a concise factual description. Include
meaningful visible text. For a diagram, name its labels and relationships. For
a chart, mention its labels, values, and trend. Do not speculate or add a heading.

For PDFs, extract reliable native text normally and send only table, image, chart, or diagram regions to Opsis. Crop quality and readable resolution have a direct effect on output quality.

Quick Start

import torch
from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor

MODEL_ID = "yafitzdev/opsis-v1-nano-g3"
PROMPT = """Parse this image for RAG search. If it is a table, output only the complete
table as Markdown. Otherwise output only a concise factual description. Include meaningful
visible text. For a diagram, name its labels and relationships. For a chart, mention its labels,
values, and trend. Do not speculate or add a heading."""

processor = AutoProcessor.from_pretrained(
    MODEL_ID,
    size={"longest_edge": 1024},
)
model = AutoModelForImageTextToText.from_pretrained(
    MODEL_ID,
    dtype=torch.float32,
).eval()
image = Image.open("image.png").convert("RGB")

messages = [{
    "role": "user",
    "content": [
        {"type": "image"},
        {"type": "text", "text": PROMPT},
    ],
}]
prompt_text = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=False,
)
inputs = processor(text=prompt_text, images=[image], return_tensors="pt")

with torch.no_grad():
    output_ids = model.generate(
        **inputs,
        do_sample=False,
        max_new_tokens=768,
        repetition_penalty=1.05,
    )

generated = output_ids[:, inputs["input_ids"].shape[-1]:]
text = processor.batch_decode(generated, skip_special_tokens=True)[0]
print(text)

CPU ONNX

The model repository includes the split FP32 ONNX graphs used in CPU evaluation:

• onnx/vision_encoder.onnx
• onnx/embed_tokens.onnx
• onnx/decoder_model_merged.onnx

The split graphs are provided for ONNX Runtime integrations. The Transformers quick start above runs on CPU directly; this model repository does not publish a standalone ONNX command-line runner.

Mixed INT8 reduced routing and Markdown validity in evaluation and is not included in this repository.

Evaluation

The gold benchmark has 600 visually reviewed, source-unique cases with no training overlap: tables, charts, diagrams, ordinary images, PDF-derived crops, and hard cases. The checkpoint was measured with greedy FP32 ONNX inference at a 1024-pixel longest edge on the release CPU environment.

Metric	G3 result
image-kind accuracy	0.9950
valid Markdown tables	0.9879
exact tables	0.0303
correct table shape	0.4242
mean table cell precision	0.4886
mean table cell recall	0.4406
mean table cell F1	0.4486
chart required-term recall	0.9183
diagram required-term recall	0.7008
diagram directed-relation recall	0.5481
ordinary-image required-term recall	0.3612
median CPU latency	3.635 s
p95 CPU latency	11.531 s
peak process RSS	2,402 MiB

Image-kind routing and Markdown formatting are strong on this benchmark. Exact cell transcription remains unreliable. The benchmark measures a fixed set of source images and one CPU configuration; it does not establish performance on arbitrary documents or hardware.

Training Data

Training bucket	Examples	Validation examples	Role
clean tables	25,000	250	Compact table image to Markdown.
clean charts	12,500	125	Chart labels, values, and trends.
clean diagrams	12,500	125	Diagram nodes and relationships.
clean ordinary images	25,000	250	Ordinary-image discovery descriptions.
PDF-style visual crops	25,000	250	Document-domain versions balanced across all four content types.
Total	100,000	1,000	Source-held-out balanced visual parsing.

After distributing the PDF bucket, the effective mix is 31,250 tables, 18,750 charts, 18,750 diagrams, and 31,250 ordinary images. The corrected chart curriculum contains 12,500 real ChartQA examples and 6,250 synthetic examples. The manifest has 96,875 unique image paths; 3,125 real-chart examples are deterministically rehearsed to preserve the exact balanced 100,000-example run.

Sources include PubTabNet, ChartQA, AI2D, DOCCI, and locally generated charts and diagrams. Training used LoRA rank 8 with alpha 16 for one epoch at a 1024-pixel longest edge and selected checkpoint 6,000 by validation loss. It updated 692,736 of 257,177,664 parameters (0.269%), restricted to vision attention and the vision-to-text connector; the text decoder remained frozen. G3 was fine-tuned from the earlier, private Opsis G2 checkpoint. The merged weights in this repository load independently for inference.

The Opsis V1 training-data provenance record pins the corrected train and validation manifest hashes, source archive hashes, curator code hashes, and a fingerprint of all 97,875 local image files across those splits. Its viewer contains five source-archive metadata rows. It does not provide the training examples: the mixture includes AI2D data whose bundled license forbids redistribution.

Artifacts

This repository contains:

• model.safetensors: standalone merged Transformers checkpoint,
• onnx/vision_encoder.onnx: FP32 vision encoder and connector,
• onnx/embed_tokens.onnx: FP32 token embedding graph,
• onnx/decoder_model_merged.onnx: FP32 autoregressive decoder,
• tokenizer, processor, generation, and model configuration files,
• rag_image_parser_config.json: Opsis release and training metadata,
• SHA256SUMS: checksums for the packaged model and ONNX graphs.

Limitations

1. Exact table recovery remains low. Better shape accuracy does not mean reliable exact cell transcription; exact table accuracy is 3.0%.
2. Generated labels and values can be wrong. Descriptions and cells are RAG discovery metadata, not verified evidence.
3. English-centric mixed-source training. Multilingual behavior is not established, and some source pools use deterministic augmentation.
4. Crop quality matters. Tiny text, dense pages, poor scans, and incorrect visual-region routing reduce performance.
5. CPU latency varies by image complexity. Dense tables and long outputs can be substantially slower than the reported median.

License

Mixed-source research preview. The training mixture includes research-restricted AI2D data and sources with separate attribution, redistribution, or underlying- image terms. This package must not be represented as commercially cleared. Review the training-data provenance record and satisfy every source obligation before commercial use.