Qwen3-VL-D2R-Item-Reader-4B-Instruct

A LoRA fine-tune of Qwen3-VL-4B-Instruct that transcribes a single Diablo II: Resurrected (D2R) item tooltip from a cropped screenshot into one structured JSON object.

• Base model: Qwen/Qwen3-VL-4B-Instruct
• Method: LoRA (vision + language layers) via Unsloth + TRL
• Task: image-text-to-text → strict JSON (constrained decoding)
• Languages: English tooltips

Intended use

Transcribing a single, cropped D2R item tooltip (English) into structured fields. It is a narrow, in-domain transcription model, not a general-purpose VLM and not a chat model.

Out of scope: general OCR, other games, non-tooltip images, open-ended Q&A.

Input format

The model expects one tightly cropped tooltip, preprocessed to a fixed 784×504 canvas (aspect-preserving fit, black padding). This matches the training distribution; other sizes/aspect ratios degrade accuracy. Decode greedily (temperature = 0) and constrain the output to the JSON schema below.

User turn: Extract the item from this tooltip.

System prompt (verbatim):

Transcribe the single Diablo II: Resurrected item tooltip in the image into one JSON object. Read it line by line, top to bottom. Output only the JSON.

Field order matches the tooltip top-to-bottom. Shape: {"name": string, "baseType": string|null, "category": "weapon"|"armor"|"shield"|"offhand"|"jewel"|"ring"|"amulet"|"charm"|"gem"|"rune"|"material"|"potion"|"consumable", "itemType": string|null, "quality": "crude"|"normal"|"superior", "rarity": "low"|"normal"|"magic"|"rare"|"set"|"unique"|"crafted", "runes": string|null, "damage": {"oneHand": {"min": integer, "max": integer}|null, "twoHand": {...}|null, "throw": {...}|null, "smite": {...}|null}|null, "defense": integer|null, "chanceToBlock": integer|null, "durability": {"current": integer, "max": integer}|null, "classRestriction": string|null, "requiredDexterity": integer|null, "requiredStrength": integer|null, "requiredLevel": integer|null, "attackSpeed": string|null, "mods": string[], "ethereal": boolean, "sockets": integer}

Read EVERY digit — `+202%` is not `+2%`, `550` is not `55`. Never guess, complete, round, reorder, or fix spelling.

Fields:
- `name`: the item title (line 1). For a runeword, line 1 only — never append the base type.
- `baseType`: the base-type line shown under the name (e.g. Ring, Great Helm, Phase Blade, Monarch). If the type is only baked into the name (no separate line), `baseType` = null.
- `category`: the broad kind. weapon; shield (a blocking shield); offhand (a non-shield off-hand like a necromancer head / grimoire); armor (everything else worn); jewel/ring/amulet/charm; gem; rune (a loose rune); material (Essence/Shard/Statue/Token/Key/quest); potion; consumable (scroll/tome).
- `itemType`: the finer slot / weapon class (e.g. Sword, Axe, Helm, Boots, Shield, Javelin, Bow, Orb). null for non-equipment.
- `classRestriction`: the character class an item is limited to (Amazon/Sorceress/Necromancer/Paladin/Barbarian/Druid/Assassin/Warlock), shown as an "(X Only)" line — RED if the viewing character is a different class, WHITE if it matches; either way set the class name; else null.
- `quality`: the base item tier, read from the NAME TEXT only — crude (name marked Cracked/Damaged/Low Quality/Crude), superior (name starts with "Superior"), else normal. Ignore the name COLOR here; default "normal" when no crude/superior word is present.
- `rarity`: the NAME COLOR tier — low (gray, also ethereal/socketed-normal), normal (white), magic (blue), rare (yellow), set (green), unique (gold/tan), crafted (orange). Runewords show gold → unique. Materials/runes are usually orange → crafted. Read the actual name color carefully (gold vs yellow vs orange differ); always set it.
- `ethereal`: true if the tooltip shows "Ethereal (Cannot Be Repaired)".
- `sockets`: the N in "Socketed (N)", else 0.
- `defense`: the number in "Defense: N", else null.
- `chanceToBlock`: the number in "Chance to Block: N%", else null.
- `durability`: from "Durability: X of Y" → {"current": X, "max": Y}, else null.
- `damage`: from the white damage lines → set the matching key, others null. "Throw Damage: A to B" → throw; "One-Hand Damage" → oneHand; "Two-Hand Damage" → twoHand; "Smite Damage" → smite (paladin shields). An item may have more than one (a javelin has throw AND oneHand). null if no damage line.
- `attackSpeed`: the verbatim "<Class> Class - <Speed> Attack Speed" line (e.g. "Javelin Class - Very Fast Attack Speed"), else null.
- `runes`: for a runeword (a line in apostrophes like `'TalThul'`), the rune string without apostrophes; baseType = the base item line. Otherwise null.
- `requiredLevel`/`requiredStrength`/`requiredDexterity`: from the "Required X: N" lines, else null.
- `mods`: every REMAINING stat line, in order, verbatim (keep `+`, `-`, `%`, ranges like `15-45`, parentheses). These are the magical properties only (procs, auras, `+X to Skills`, resistances, `Adds A-B Fire Damage`, …). Do NOT put into mods anything already captured above (name, baseType, itemType, classRestriction line, quality, ethereal/socketed line, defense/chanceToBlock/durability/damage, attackSpeed line, the Required-* lines) or UI hints ("Keep in Inventory to Gain Bonus", "Ctrl + Left Click ...", "Right Click to Use").

Output schema

The model is trained for constrained / structured decoding against the following JSON schema (response_format → json_schema, strict: true). Field order mirrors the in-game top-to-bottom reading order.

{
  "type": "object",
  "properties": {
    "name": {"type": "string"},
    "baseType": {"type": ["string", "null"]},
    "category": {
      "type": "string",
      "enum": [
        "weapon", "armor", "shield", "offhand", "jewel", "ring", "amulet",
        "charm", "gem", "rune", "material", "potion", "consumable"
      ]
    },
    "itemType": {"type": ["string", "null"]},
    "quality": {
      "type": "string",
      "enum": ["crude", "normal", "superior"]
    },
    "rarity": {
      "type": "string",
      "enum": ["low", "normal", "magic", "rare", "set", "unique", "crafted"]
    },
    "runes": {"type": ["string", "null"]},
    "damage": {
      "type": ["object", "null"],
      "properties": {
        "oneHand": {"$ref": "#/$defs/range"},
        "twoHand": {"$ref": "#/$defs/range"},
        "throw": {"$ref": "#/$defs/range"},
        "smite": {"$ref": "#/$defs/range"}
      },
      "required": ["oneHand", "twoHand", "throw", "smite"],
      "additionalProperties": false
    },
    "defense": {"type": ["integer", "null"]},
    "chanceToBlock": {"type": ["integer", "null"]},
    "durability": {
      "type": ["object", "null"],
      "properties": {
        "current": {"type": "integer"},
        "max": {"type": "integer"}
      },
      "required": ["current", "max"],
      "additionalProperties": false
    },
    "classRestriction": {"type": ["string", "null"]},
    "requiredDexterity": {"type": ["integer", "null"]},
    "requiredStrength": {"type": ["integer", "null"]},
    "requiredLevel": {"type": ["integer", "null"]},
    "attackSpeed": {"type": ["string", "null"]},
    "mods": {"type": "array", "items": {"type": "string"}},
    "ethereal": {"type": "boolean"},
    "sockets": {"type": "integer"}
  },
  "required": [
    "name", "baseType", "category", "itemType", "quality", "rarity", "runes",
    "damage", "defense", "chanceToBlock", "durability", "classRestriction",
    "requiredDexterity", "requiredStrength", "requiredLevel", "attackSpeed",
    "mods", "ethereal", "sockets"
  ],
  "additionalProperties": false,
  "$defs": {
    "range": {
      "type": ["object", "null"],
      "properties": {
        "min": {"type": "integer"},
        "max": {"type": "integer"}
      },
      "required": ["min", "max"],
      "additionalProperties": false
    }
  }
}

Note on rarity: rarity is essentially a function of the item-name color, which VLMs read unreliably (gold vs. yellow vs. orange are very close in hue). The model predicts it best-effort; for reliable results, treat it as a hint and prefer a pixel-based (HSV) color read of the name line.

Example outputs

Unique ring

{
  "name": "Raven Frost",
  "baseType": "Ring",
  "category": "ring",
  "itemType": null,
  "quality": "normal",
  "rarity": "unique",
  "runes": null,
  "damage": null,
  "defense": null,
  "chanceToBlock": null,
  "durability": null,
  "classRestriction": null,
  "requiredDexterity": null,
  "requiredStrength": null,
  "requiredLevel": 45,
  "attackSpeed": null,
  "mods": ["+221 to Attack Rating", "Adds 15-45 Cold Damage", "+18 to Dexterity", "+40 to Mana", "Cold Absorb 20%", "Cannot Be Frozen"],
  "ethereal": false,
  "sockets": 0
}

Runeword (base item + rune sequence)

{
  "name": "Insight",
  "baseType": "Scythe",
  "category": "weapon",
  "itemType": "Polearm",
  "quality": "normal",
  "rarity": "unique",
  "runes": "RalTirTalSol",
  "damage": {"oneHand": null, "twoHand": {"min": 18, "max": 40}, "throw": null, "smite": null},
  "defense": null,
  "chanceToBlock": null,
  "durability": {"current": 65, "max": 65},
  "classRestriction": null,
  "requiredDexterity": null,
  "requiredStrength": 87,
  "requiredLevel": 27,
  "attackSpeed": null,
  "mods": ["Level 15 Meditation Aura When Equipped", "+35% Faster Cast Rate", "+247% Enhanced Damage", "+9 to Minimum Damage", "213% Bonus to Attack Rating", "Adds 5-30 Fire Damage", "+75 Poison Damage Over 5 Seconds", "+5 to Critical Strike", "+5 to All Attributes", "+2 to Mana After Each Kill", "23% Better Chance of Getting Magic Items"],
  "ethereal": false,
  "sockets": 4
}

Magic charm (procedural name)

{
  "name": "Harpoonist's Grand Charm of Sustenance",
  "baseType": "Grand Charm",
  "category": "charm",
  "itemType": null,
  "quality": "normal",
  "rarity": "magic",
  "runes": null,
  "damage": null,
  "defense": null,
  "chanceToBlock": null,
  "durability": null,
  "classRestriction": null,
  "requiredDexterity": null,
  "requiredStrength": null,
  "requiredLevel": 42,
  "attackSpeed": null,
  "mods": ["+2 to Javelin and Spear Skills (Amazon Only)", "+38 to Life"],
  "ethereal": false,
  "sockets": 0
}

Training data

• Synthetic tooltips (primary): procedurally rendered from D2R game-data tables (item bases, affixes, uniques, sets, runewords) and composited over real stash/inventory backgrounds. Rendering is calibrated to the in-game tooltip (font, colors, line spacing, panel, border, separator) to minimize the synthetic→real gap.
• Real captures (small, verified): a handful of hand-verified in-game tooltip crops mixed in (oversampled) as a domain anchor.

Each training example is: system prompt + user turn + the 784×504 tooltip image, with the target JSON as the assistant turn.

Training procedure

• Framework: Unsloth + TRL SFTTrainer, LoRA on vision and language layers.
• Precision: bf16 with gradient checkpointing.
• Key hyperparameters: LoRA r=16 / alpha=16, learning rate 2e-4, linear schedule, effective batch via gradient accumulation, ~1 epoch over the synthetic set.
• Hardware: AMD Radeon RX 9070 (gfx1201) under WSL2 + ROCm.

How to run

Use any OpenAI-compatible local server (e.g. llama.cpp / LM Studio) with the GGUF build (model + vision mmproj):

1. Crop the tooltip and fit it to a 784×504 canvas (aspect-preserving, black pad).
2. Send the image with the system prompt above and temperature = 0, constraining the response to the JSON schema.

Limitations

• Domain-locked: only English D2R tooltips, cropped and canvas-fit as above.
• Color/rarity: see the rarity note — prefer a pixel-based rarity source.
• Tiny digits / very long tooltips can still be misread; verify critical values.
• No formal benchmark; evaluated qualitatively on held-out real captures.

License

Derived/LoRA weights are released under Apache-2.0, inheriting the base model's license (verify the exact terms of Qwen/Qwen3-VL-4B-Instruct before redistribution). Diablo II: Resurrected and all related assets/trademarks belong to Blizzard Entertainment; this is an unofficial, fan-made model with no affiliation or endorsement.

Acknowledgements

• Qwen team for Qwen3-VL.
• Unsloth for the fine-tuning stack.
• The blizzhackers d2data project for the game-data tables used to generate the synthetic training data.