eruku_korean

Korean styled handwriting line-image generation, fine-tuned from blowing-up-groundhogs/eruku (autoregressive styled handwriting generation, arXiv 2510.23240). Give it a style reference line and a Korean/English string; it draws that string in that style.

Training code and experiment log: https://github.com/HERIUN/Eruku_korean_finetuning

This revision requires the paired Korean VAE. config.vae_name_or_path points at HERIUN/emuru_vae_korean. The VAE was fine-tuned with encoder+decoder unfrozen, so its latent space moved and the T5 decoder here was re-adapted on top of it. Swapping in the original emuru_vae produces garbage.

Quick start

from transformers import AutoModel
from PIL import Image

model = AutoModel.from_pretrained("HERIUN/eruku_korean", trust_remote_code=True).eval().cuda()

img = model.generate_handwriting(
    style_image=Image.open("style_line.png"),   # one line of handwriting
    style_text="조용히 책을 읽었다.",              # transcription of the style line (optional, but helps a lot)
    gen_text="한국어 손글씨 생성",                 # what to write
    cfg_scale=1.5,
)
img.save("out.png")

• style_text — passing the style line's transcription markedly improves fidelity.
• cfg_scale — 1.0 is the most faithful on the eval battery; 1.5-2.0 pushes style strength.
• bbox_crop=True (default) crops the style image to its ink bounding box. Real scans carry page margins that blow up the VAE latent std and make generation run away or come back blank; this closes that gap. Pass bbox_crop=False for crops that already fill the frame.
• Preprocessing is handled for you and matches training: the style image is converted to RGB, resized to height 64, and normalised to [-1, 1] before it reaches the VAE. Width is not padded to a multiple of 8 — training never padded either, so do not add it.
• Decoding: the style, image and EOG latents are decoded by the VAE together and the style columns are cropped off afterwards; the SOG latent is left out. Training only puts the latent loss on image positions, so the SOG latent is never trained and decoding it smears the whole line (worst with an empty style_text). The EOG column often still carries the last stroke, so it is kept.

Results

CER measured with a Korean HTR reader on n=300 coherent sentences, cfg=1.0, seen fonts, fixed eval set (eval/htr_cer.py --coherent). The mean is dominated by a few runaway lines, so the median and the runaway rate (lines with CER > 1, i.e. generation that never stops) are the numbers to read.

	mean	median	runaway
VAE ceiling (encode -> decode of the target render)	0.031	0.000	0.3%
previous release (step 30000, 83 Korean fonts, 941-line corpus)	0.120	0.026	1.3%
this model (step 45000, 12,951 fonts, 50,941-line corpus)	0.069	0.000	0.3%
short strings (1-3 words), previous -> this	0.280 -> 0.142	0.000	4.2% -> 1.1%

English (n=300): 0.034 -> 0.036, median 0.000 for both — English is kept. Runaway on this set is down to the VAE ceiling: scaling the natural-sentence corpus 35x taught the model where a line ends.

Before / after

Same target text through the original English pretrained Eruku (with the original VAE) and through this release. Columns: style reference | target rendered in that font | before | after.

Korean, with Korean-font style references. The baseline has seen neither Hangul glyphs nor Korean-font styles, so it returns nothing usable:

English, with Latin style references that are in distribution for both models — English is kept, not traded away for Korean, and strokes come out crisper (the Korean VAE is a better decoder for Latin too):

How it was trained

English pretrained -> Korean (--english-frac 0.15 to hold English) -> VAE swap (--reset-vae) + re-adaptation with short-text emphasis (30000 steps), then 15000 more steps on 12,951 fonts and a 50,941-line natural-sentence corpus (45000 optimizer steps total). Data is synthesized online: fonts act as writers, text is sampled from the corpus, rendered and augmented per sample, so every sample is unique. Backbone: T5-large + ByT5 byte tokenizer (byte-level input handles Hangul without a Korean vocab), frozen VAE.

Limitations

• Short strings are still the weak spot (mean CER 0.142 vs 0.069 overall) — fewer latent columns means less context for the autoregressive decoder.
• On unseen fonts the VAE, not this model, is now the bottleneck: the remaining error sits at the VAE reconstruction ceiling.
• The VAE snaps ink density to crisp black, so it does not preserve faint/pencil strokes (prototype collapse; see docs/VAE_ROBUSTNESS.md in the repo).
• Long English lines are weaker than the original English model — Korean is the target here.

License: Apache-2.0, inherited from the base model.