Non-Latin scripts are where most models fall apart, and IMG.LY has international print and social customers. One CJK sign is enough to separate models that handle multi-byte glyphs from those that mangle them (full multilingual coverage is a planned deep-dive).
Suite pilot-0 · identical prompt and default parameters for every model ·
seeds 1111, 2222, 3333 (or repeated samples for seedless APIs)
Results
Click any image to enlarge. Checkerboard backing shows true
transparency; a rendered checkerboard pattern inside the image is a
model faking it.
FLUX.2
FLUX.2 [pro]
FLUX.2 [dev] Turbo
Gemini 2.5 Flash Image
GPT Image 1.5
GPT Image 2
GPT Image 2.5 Flare
GPT Image 2.5 Sunburst
Grok Imagine Image 2.0
Ideogram 3.0
Luma Photon
Muse Image
Nano Banana 2
Nano Banana 2 Lite
Nano Banana Pro
Qwen-Image
Qwen Image 3.0
Recraft V3
Recraft V4.1
Seedream 4.5
Seedream 5.0 Lite
Seedream 5.0 Pro
Stable Diffusion 1.5
What the models actually wrote
Expected text vs the rendered result, character by character.
Transcribed by the Claude vision judge in the pilot; the OCR tier
automates this at full-suite scale.
Expected
東京ラーメン
Rendered
FLUX.2 · seed 1111 東 京 ラ ー メ ン 100% match · 5/5
The rendered text matches the required string 東京ラーメン exactly, with all six characters correct and clearly legible. There are no misspellings or broken glyphs.
FLUX.2 · seed 2222 東 京 ラ ー メ ン 100% match · 5/5
The rendered text exactly matches the required '東京ラーメン' with correct, well-formed characters and no invented glyphs. The neon is bright and highly legible against the dark background.
FLUX.2 · seed 3333 東 京 ラ ー メ ン 100% match · 5/5
The text renders exactly as required with all six characters (東京ラーメン) correct and clearly legible in glowing red neon. No spelling errors or malformed glyphs are present.
FLUX.2 [pro] · seed 1111 東 京 ラ ー メ ン 100% match · 5/5
The rendered neon text matches the required string 東京ラーメン exactly, with correct kanji and katakana. The characters are sharp, well-formed, and highly legible against the dark storefront.
FLUX.2 [pro] · seed 2222 東 京 ラ ー メ ン 100% match · 5/5
The neon sign renders exactly '東京ラーメン' with correct, legible glyphs. A secondary noren curtain also reads 'ラーメン', consistent with the required text.
FLUX.2 [pro] · seed 3333 東 京 ラ ー メ ン 100% match · 5/5
The main neon sign renders exactly 東京ラーメン with accurate, legible glyphs. Background signage shows other partial characters but the required text is correct and clearly readable.
FLUX.2 [dev] Turbo · seed 1111 東 京 ラ ー メ ン 100% match · 5/5
The rendered text matches the required '東京ラーメン' exactly, with all characters correct, legible, and glowing in red neon. No misspellings or broken glyphs are present.
FLUX.2 [dev] Turbo · seed 2222 東 京 ラ ー メ ン 100% match · 5/5
The rendered text matches the required 東京ラーメン exactly, with all six characters correctly formed and clearly legible in the neon sign. There are no spelling errors or broken glyphs.
FLUX.2 [dev] Turbo · seed 3333 東 京 ラ ー メ ン 100% match · 5/5
The rendered text exactly matches the required 東京ラーメン with correct characters and no invented glyphs. The neon lettering is clear and legible against the dark background.
Gemini 2.5 Flash Image · sample 1 東 京 ラ ー メ ン 100% match · 5/5
The rendered text matches the required 東京ラーメン exactly. The characters are crisp, correctly formed, and legible in the neon sign. No spelling errors or broken glyphs are present.
Gemini 2.5 Flash Image · sample 2 東 京 ラ ー メ ン 100% match · 5/5
The rendered text exactly matches the required 東京ラーメン in glowing blue neon. The kanji and katakana are correct and legible with no spelling errors or broken characters.
Gemini 2.5 Flash Image · sample 3 東 京 ラ ー メ ン 100% match · 5/5
The sign renders exactly the required text 東京ラーメン with the kanji in blue and the katakana ラーメン in red neon. All characters are correct, legible, and properly formed.
GPT Image 1.5 · sample 1 東 京 ラ ー メ ン 100% match · 5/5
The main neon sign renders exactly 東京ラーメン with correct, legible characters. The noren curtain also shows ラーメン, consistent with the theme. No misspellings or invented glyphs in the primary sign.
GPT Image 1.5 · sample 2 東 京 ラ ー メ ン 100% match · 5/5
The neon sign renders the required text 東京ラーメン exactly and legibly. The lantern also reads ラーメン and the noren curtain shows ラーメン, all correct. No misspellings or broken glyphs.
GPT Image 1.5 · sample 3 東 京 ラ ー メ ン 100% match · 5/5
The rendered text matches the required 東京ラーメン exactly, with correct kanji and katakana. The characters are sharp, well-lit, and fully legible in neon form.
GPT Image 2 · sample 1 東 京 ラ ー メ ン 100% match · 5/5
The main neon sign renders 東京ラーメン clearly and correctly. Secondary signage (ラーメン, 営業中, 東京ラーメン, ホテル/カラオケ, etc.) is also present and mostly legible, but the primary required text is rendered exactly and accurately.
GPT Image 2 · sample 2 東 京 ラ ー メ ン 100% match · 5/5
The neon sign renders exactly 東京ラーメン with clean, correct kanji and katakana. Surrounding elements add more legible Japanese text: a red lantern reads ラーメン, a standing sign reads 営業中, the noren curtain reads ラーメン, side signs read ホテル, カラオケ, BAR, and a menu board reads 東京ラーメン with price list. The primary required text is accurate and legible.
GPT Image 2 · sample 3 東 京 ラ ー メ ン 100% match · 5/5
The main neon sign clearly reads 東京ラーメン with correct, legible characters. Supporting signage (lantern, banners, side sign) also reads ラーメン and related Japanese text, all consistent with the target.
GPT Image 2.5 Flare · sample 1 東 京 ラ ー メ ン 100% match · 5/5
The primary neon sign renders exactly 東京ラーメン with clean, accurate CJK characters. Additional legible Japanese text appears elsewhere (menu board with ラーメン, 味噌ラーメン, 塩ラーメン, つけ麺, 餃子; a lantern reading ラーメン; 営業中; カラオケ; 居酒屋), but the required target sign is correct.
GPT Image 2.5 Flare · sample 2 東 京 ラ ー メ ン 100% match · 5/5
The primary neon sign renders exactly 東京ラーメン, correct and legible. Secondary elements repeat 東京ラーメン (lantern, noren curtain, sidewalk sign) and menu text (醤油ラーメン, 味噌ラーメン, 塩ラーメン, こだわりの味), all accurate. Transcribing the required sign text only.
GPT Image 2.5 Flare · sample 3 東 京 ラ ー メ ン 100% match · 5/5
The main neon sign clearly and correctly renders 東京ラーメン. Additional accurate Japanese text appears elsewhere (ラーメン on the lantern and noren, menu text, 東京ラーメン on the signboard), all well-formed, but the required target text is rendered exactly.
GPT Image 2.5 Sunburst · sample 1 東 京 ラ ー メ ン 100% match · 5/5
The main neon sign renders 東京ラーメン exactly and legibly. Additional correct Japanese text appears elsewhere (lantern reading ラーメン, noren and signs), but the required text is rendered correctly.
GPT Image 2.5 Sunburst · sample 2 東 京 ラ ー メ ン 100% match · 5/5
The primary neon sign renders exactly 東京ラーメン with correct, legible characters. The same text also appears correctly on the red noren curtain and the hanging lantern (ラーメン). Additional secondary Japanese text appears elsewhere (夢を食べよう, 営業中, こだわりの一杯) but the required string is rendered exactly and clearly.
GPT Image 2.5 Sunburst · sample 3 東 京 ラ ー メ ン 100% match · 5/5
The primary neon sign clearly and accurately renders 東京ラーメン. The same text also appears correctly on the red noren curtain, the vertical red lantern (ラーメン), and other secondary signage. The required text is rendered exactly and legibly.
Grok Imagine Image 2.0 · sample 1 東 京 ラ ー メ ン 100% match · 5/5
The main neon sign legibly renders exactly 東京ラーメン (red kanji 東京 followed by blue katakana ラーメン). The characters are accurate and well-formed. Smaller decorative text on the red noren curtain below is not clearly legible but is not the required text.
Grok Imagine Image 2.0 · sample 2 東 京 ラ ー メ ン 100% match · 5/5
The main neon sign clearly and correctly renders 東京ラーメン. The kanji 東京 and katakana ラーメン are all accurate and legible. Background signage and the noren/lantern contain small blurry decorative characters that are not clearly readable, but the required text is rendered perfectly.
Grok Imagine Image 2.0 · sample 3 東 京 ラ ー メ ン 100% match · 5/5
The neon storefront sign clearly and accurately renders 東京ラーメン. Each kanji and katakana character is correctly formed and legible.
Ideogram 3.0 · seed 1111 東 京 ラ ー メ ン 100% match · 5/5
The rendered text matches the required 東京ラーメン exactly, with correct characters and good legibility. The neon glow does not distort the glyphs into invented forms.
Ideogram 3.0 · seed 2222 東 京 ラ ー メ ン A t a s t e o f T o k y o 100% match · 5/5
The main neon text correctly renders 東京ラーメン with accurate, legible kanji and katakana. An additional tagline 'A taste of Tokyo' appears at the bottom, which is legible and correctly spelled.
Ideogram 3.0 · seed 3333 東 京 ラ ー メ ン O P E N 83% match · 3/5
The main neon sign renders 東京ラーメン exactly and legibly across two lines, matching the required text. An additional 'OPEN' sign appears below, which was not required but does not conflict. Smaller poster text on the door is illegible gibberish but is separate from the required text.
Luma Photon · seed 1111 ヘ カ 木 杰 丈 0% match · 0/5
The required text 東京ラーメン is not rendered; the sign instead shows garbled, invented characters that only loosely resemble strokes of real glyphs. None of the intended kanji or katakana are correct or legible.
Luma Photon · seed 2222 の 展 T E H I O J U A L I A N 0% match · 0/5
The required text was 東京ラーメン, but the sign displays a jumble of distorted, fabricated characters plus meaningless Latin text like 'TEHIOJUALIAN'. Only fragments resembling kanji/kana appear, and none spell the intended phrase.
Luma Photon · seed 3333 ち ち ち 0% match · 0/5
The signs show distorted, pseudo-Japanese shapes that do not spell 東京ラーメン; the top red sign reads something like repeated 'ちちち' style glyphs, and the rest are illegible invented marks. The required text is not legibly rendered.
Muse Image · sample 1 東 京 ラ ー メ ン 100% match · 5/5
The main neon sign clearly and correctly renders 東京ラーメン. A hanging noren curtain also shows ラーメン vertically, consistent and legible. The required text is rendered exactly and accurately.
Muse Image · sample 2 東 京 ラ ー メ ン 100% match · 5/5
The main neon sign clearly and correctly renders 東京ラーメン. Additional accurate Japanese text appears on the noren curtain (ラーメン) and menu signs (醤油ラーメン, 味噌ラーメン, 餃子, 円 prices), but the required target text is rendered exactly and legibly.
Muse Image · sample 3 東 京 ラ ー メ ン 100% match · 5/5
The main pink neon sign renders 東京ラーメン exactly and legibly. Secondary noren fabric also shows ラーメン, consistent with the required text.
Nano Banana 2 · sample 1 東 京 ラ ー メ ン 100% match · 5/5
The primary neon sign renders 東京ラーメン exactly and legibly with correct glyphs. Surrounding signage includes additional Japanese text (e.g. 営業中, 東京ラーメン repeated on side panels) but the required text is accurate.
Nano Banana 2 · sample 2 東 京 ラ ー メ ン 100% match · 5/5
The primary neon sign renders 東京ラーメン exactly as required, with clean, correct characters. The text is repeated legibly elsewhere in the scene (noren, signboard), all matching the target.
Nano Banana 2 · sample 3 東 京 ラ ー メ ン 100% match · 5/5
The neon sign and the noren both render 東京ラーメン exactly and legibly, matching the required text with correct kanji and katakana. Surrounding signage shows other plausible Japanese text, but the target phrase is accurate.
Nano Banana 2 Lite · sample 1 東 京 ラ ー メ ン 100% match · 5/5
The primary neon sign renders 東京ラーメン exactly and legibly, matching the required text. Additional supporting text like 'RAMEN', 'OPEN', and 博多豚骨 also appears elsewhere but the core required string is correct.
Nano Banana 2 Lite · sample 2 東 京 ラ ー メ ン R A M E N T O K Y O 100% match · 5/5
The primary neon sign renders 東京ラーメン exactly and legibly, matching the required text precisely. An additional 'RAMEN TOKYO' subtitle appears below but does not detract from the correct rendering. Other background signage contains additional Japanese characters unrelated to the requirement.
Nano Banana 2 Lite · sample 3 東 京 ラ ー メ ン R A M E N I Z A K A Y A 100% match · 5/5
The main neon sign renders 東京ラーメン exactly and legibly with correct characters. Additional accurate text (RAMEN IZAKAYA, ラーメン on the noren, 拉麺 on lanterns) also appears cleanly.
Nano Banana Pro · sample 1 東 京 ラ ー メ ン 100% match · 5/5
The primary neon sign renders exactly 東京ラーメン with correct, legible characters and no misspellings. A secondary vertical sign also shows 東京ラーメン consistently.
Nano Banana Pro · sample 2 東 京 ラ ー メ ン 100% match · 5/5
The large neon storefront sign reads 東京ラーメン as required, matching the target text. The characters are mostly legible though the katakana portion is slightly warped, and secondary signs elsewhere show broken or nonsensical glyphs.
Nano Banana Pro · sample 3 東 京 ラ ー メ ン 100% match · 5/5
The rendered text matches the required 東京ラーメン exactly, with all six characters correct and legible. There is also a smaller vertical 東京ラーメン sign visible on the side pillar, consistent and correct.
Qwen-Image · seed 1111 東 京 ラ ー メ ン 100% match · 5/5
The primary neon sign renders exactly 東京ラーメン with correct spelling and clean, legible glyphs. Smaller secondary signs in the storefront contain garbled or invented characters, but the required text is fully accurate.
Qwen-Image · seed 2222 東 京 ラ ー メ ン 100% match · 5/5
The primary neon sign renders the required text exactly as 東京ラーメン with correct, legible kanji and katakana. A smaller superscript ラーメーン appears above but does not detract from the main correct rendering.
Qwen-Image · seed 3333 東 京 ラ ー メ ン 100% match · 5/5
The main neon sign renders 東京ラーメン exactly as required, with the characters legible and correctly formed. Secondary signage in the windows is illegible but not required.
Qwen Image 3.0 · seed 1111 東 京 ラ ー メ ン 100% match · 5/5
The sign legibly renders 東京ラーメン exactly as required. The kanji 東京 appear in red neon and the katakana ラーメン in white neon; all characters are correct and readable with no invented or broken glyphs.
Qwen Image 3.0 · seed 2222 東 京 ラ ー メ ン 100% match · 5/5
The main neon sign clearly renders 東京ラーメン exactly as required. A red paper lantern and a side menu board carry additional smaller Japanese text, but the primary required text is accurate and legible.
Qwen Image 3.0 · seed 3333 東 京 ラ ー メ ン 100% match · 5/5
The primary neon sign renders 東京ラーメン exactly and legibly. Additional accurate Japanese text appears elsewhere: RAMEN and らーめん on side signage, 神田町 on a small sign, OPEN, TRASH, and 営業時間 11:00-23:00 on a window notice.
Recraft V3 · seed 1111 T O K Y O R A M E N 0% match · 0/5
The image shows the English romanization 'TOKYO' and 'RAMEN' in red neon rather than the required Japanese '東京ラーメン'. The intended Japanese characters are entirely absent.
Recraft V3 · seed 2222 T O K Y O R A M E N 0% match · 0/5
The main sign renders 'TOKYO RAMEN' in Roman characters instead of the required '東京ラーメン'. A faint vertical sign contains some illegible pseudo-glyphs that do not spell the correct Japanese text.
Recraft V3 · seed 3333 T O K Y O R A M E N 0% match · 0/5
The visible text is 'TOKYO RAMEN' in Latin script, which is the romanized equivalent but not the required Japanese characters 東京ラーメン. The prompt explicitly demanded the Japanese glyphs, which are entirely absent.
Recraft V4.1 · sample 1 東 京 ラ ー メ ン 100% match · 5/5
The neon sign legibly renders exactly 東京ラーメン. The characters are well-formed and match the required text precisely.
Recraft V4.1 · sample 2 東 京 ラ ー メ ン 100% match · 5/5
The sign clearly renders 東京ラーメン with accurate, legible glyphs matching the required text exactly.
Recraft V4.1 · sample 3 東 京 ラ ー メ ン 100% match · 5/5
The neon sign legibly renders 東京ラーメン. The kanji 東京 are crisp and accurate; the katakana ラーメン is present and readable, with only mild neon-styling distortion on the final characters.
Seedream 4.5 · seed 1111 東 京 ラ ー メ ン 100% match · 5/5
The text renders exactly as required, '東京ラーメン', and appears correctly twice in the image. The characters are legible and accurately formed in neon script.
Seedream 4.5 · seed 2222 東 京 ラ ト メ ン 83% match · 3/5
The required text was 東京ラーメン, but the rendered sign shows 東京 correctly followed by distorted katakana approximating ラトメン, missing the prolonged sound mark (ー) and using an incorrect character. The kanji are legible but the katakana is inaccurate.
Seedream 4.5 · seed 3333 東 京 ラ ー メ ン 100% match · 5/5
The rendered text matches the required 東京ラーメン exactly, with the characters clear and correctly formed in neon. There are no misspellings or broken glyphs; the kanji and katakana are all accurate and legible.
Seedream 5.0 Lite · seed 1111 東 京 ラ ー メ ン 100% match · 5/5
The sign renders exactly 東京ラーメン with all six characters correct and legible. The neon glow is crisp and there are no misspellings or broken glyphs.
Seedream 5.0 Lite · seed 2222 東 京 ラ ー メ ン 100% match · 5/5
The text renders exactly the required '東京ラーメン' with correct, legible characters. There are no misspellings or invented glyphs, and it is clearly displayed in the neon sign.
Seedream 5.0 Lite · seed 3333 東 京 ラ ー メ ン 100% match · 5/5
The main neon sign renders 東京ラーメン exactly and correctly with all characters legible. The reflection in the puddle shows a mirrored, distorted version of the same text, which is expected. No spelling errors are present.
Seedream 5.0 Pro · seed 1111 東 京 ラ ー メ ン 100% match · 5/5
The neon sign legibly renders the exact required text 東京ラーメン in red neon tubing. All characters are correctly shaped and clearly readable.
Seedream 5.0 Pro · seed 2222 東 京 ラ ー メ ン 100% match · 5/5
The neon sign clearly and legibly renders 東京ラーメン exactly as required, matching the target string with correct kanji and katakana.
Seedream 5.0 Pro · seed 3333 東 京 ラ ー メ ン 100% match · 5/5
The sign clearly renders 東京ラーメン (Tokyo Ramen). All six characters are accurate and legible: 東京 in kanji and ラーメン in katakana. The reflection on the wet ground mirrors the same text.
Stable Diffusion 1.5 · seed 1111 L C O N 0% match · 0/5
The visible neon renders Latin letters resembling 'LCON' plus several distorted, non-standard glyphs, rather than the required 東京ラーメン. The correct Japanese characters do not appear at all.
Stable Diffusion 1.5 · seed 2222 N O P R E O D E 0% match · 0/5
The required text was 東京ラーメン, but the sign instead shows distorted Latin-like characters reading approximately 'NOP REODE' along with illegible vertical glyphs on the right. None of the correct Japanese characters are legibly present.
Stable Diffusion 1.5 · seed 3333
no legible text
0% match · 0/5
No legible or correct text is present; the red glowing marks are distorted, invented glyphs that do not spell 東京ラーメン or any real characters.
Inspect the letterforms
FLUX.2
FLUX.2 [pro]
FLUX.2 [dev] Turbo
Gemini 2.5 Flash Image
GPT Image 1.5
GPT Image 2
GPT Image 2.5 Flare
GPT Image 2.5 Sunburst
Grok Imagine Image 2.0
Ideogram 3.0
Luma Photon
Muse Image
Nano Banana 2
Nano Banana 2 Lite
Nano Banana Pro
Qwen-Image
Qwen Image 3.0
Recraft V3
Recraft V4.1
Seedream 4.5
Seedream 5.0 Lite
Seedream 5.0 Pro
Stable Diffusion 1.5
Move the cursor over any image: all panels magnify the same spot at 3×
so letterforms and fine detail compare directly.
Run-to-run consistency
Same model, same prompt, different seed or sample. Click a card to
flip between the two generations: the less it moves, the safer the
model is for templated, variable-data production.
Scores
Mean per model across this prompt's runs. Auto-measured criteria plus an
interim Claude-VLM tier for prompt adherence and composition; any
criterion still awaiting review is labeled pending, never zeroed.
We re-run the identical 37-prompt suite on every model release. Get the
scores, and what changed in the rankings, in your inbox.
You're on the list.
We'll email you when new benchmark results ship.
Frequently asked questions
Image generation is stochastic: the same prompt produces different images on every run, so a single sample measures luck, not ability. Every model runs each benchmark three times with fixed seeds (1111, 2222, 3333), or three unseeded samples where the API accepts no seed. Scores average all three samples, and the consistency benchmarks measure the variation itself.
Yes. Every model receives the same prompt text with default parameters and no per-model tuning, so differences in output reflect the model, not prompt engineering. The suite is versioned and prompts are append-only, which keeps historical scores comparable.
The results on this page are scored on this exact prompt, so the strongest model for it is easy to spot; this benchmark sits in the Typography & Text Rendering category. Scores are per model version and reflect this benchmark only. For a ranking across every benchmark see the model rankings, and to weight the numbers by a specific job see the use-case pages.
Each image is scored 0 to 5 per criterion. Measured criteria (resolution, latency, cost, transparency, color accuracy) are computed automatically; quality criteria (text accuracy, prompt adherence, composition) are judged by an automated Claude vision tier against each prompt’s checklist. A model’s score in any scope averages its runs per criterion first, then gives each criterion one equal vote, so a criterion measured on every prompt cannot outweigh one measured on a few. Blind expert-panel review has not run yet; the dataset is pilot-0.
Model output is a starting point, not a finished asset. Production work usually needs background removal, exact brand colors or editable text, none of which generation guarantees on every run. The IMG.LY AI Editor gives your users those controls to refine any model’s output to production quality.
The suite re-runs on notable model releases so the rankings stay current. The results shown are the pilot-0 dataset, scored by measured criteria plus an automated Claude vision tier, with blind expert review planned. Prompts are versioned and append-only, so scores stay comparable across runs.
![FLUX.2 [pro]: CJK Neon Sign, magnified detail](https://storage.googleapis.com/imgly-ai-benchmarks/t05-cjk-sign--flux-2-pro--s1111.webp)
![FLUX.2 [dev] Turbo: CJK Neon Sign, magnified detail](https://storage.googleapis.com/imgly-ai-benchmarks/t05-cjk-sign--flux-2-turbo--s1111.webp)
