VisEditBench: A Benchmark for Vector-Format Diagram Editing with Visual Instructions

Akito Taneguchi1,*, Haruto Yoshida1, Jun Suzuki1,3,4, Itsumi Saito1,2,3,*
1Tohoku University 2Institute of Science Tokyo 3RIKEN 4NII LLMC
NeurIPS 2026 · Evaluations & Datasets Track
*Indicates Equal Contribution
Paper arXiv Code Dataset

Paper links are coming soon.

Text-only instruction versus visual instruction for diagram editing: in the visual setting, the user circles nodes B and S and writes 'Change the node color to red' directly on the diagram.

Instead of describing an edit in verbose text, a visual instruction draws it directly on the diagram. VisEditBench asks a model to read these annotations and apply them to the diagram's TikZ source code.

694
human-curated samples
5
edit types
1–3
edit operations per instruction
18
annotators
32
models evaluated

Abstract

We introduce a new task and benchmark dataset, VisEditBench, for understanding visual instructions in vector-format diagram editing. While recent advances have made progress in generating vector graphics from text or images, editing such diagrams remains relatively underexplored. Moreover, existing approaches predominantly rely on textual instructions, which are often unintuitive for specifying edits in structured visual content, particularly when users need to identify specific elements or indicate their spatial positions in a diagram. We propose visual instructions as a more intuitive interface for diagram editing, and construct a dataset of 694 samples with human-curated annotations overlaid on TikZ diagrams. The dataset covers multiple edit types as well as instructions with multiple editing operations. We evaluate 32 recent state-of-the-art multimodal models and observe consistent performance degradation for edits involving global layout or diagram structure, compared to local operations such as element deletion or modification. Furthermore, instructions that include multiple editing operations tend to result in lower performance than single-edit cases. Even recent models struggle to achieve consistently accurate edits, highlighting the inherent difficulty of the task. VisEditBench provides a valuable benchmark for advancing research on visually grounded diagram editing and structured visual understanding.

Task: Diagram Editing with Visual Instructions

Given an original diagram image Dorig, its TikZ code Corig, and a visual instruction image Ivis that overlays annotations such as arrows, sketches, and handwritten notes on Dorig, the model must produce modified TikZ code Cedit whose rendered diagram Dedit faithfully reflects the intended edits.

Visual instructions let users point at the exact element to change, or sketch where it should go, without writing a long description. To follow them, a model has to work out which element is targeted, what spatial relationship is intended, and where that element lives in the source code.

Input
  • Original diagram image Dorig
  • Original TikZ code Corig
  • Visual instruction image Ivis
Output
  • Edited TikZ code Cedit
  • Rendered edited diagram Dedit

Dataset Construction

Two-step construction pipeline: Step 1 uses MLLMs to generate textual editing instructions and edited diagrams from TikZ diagrams; Step 2 has annotators draw visual instructions on the original diagram while referencing the edited diagram.

Real-world diagram editing data is rarely recorded, so we build VisEditBench from existing TikZ diagrams in the Stack Exchange subset of DaTikZ-v2.

  1. Step 1: textual instructions and edited diagrams. Gemini 3 Pro Preview generates a textual editing instruction for each diagram, given a target edit type and a required number of edit operations (1–3). Several proprietary LLMs (Gemini 3 Pro Preview, GPT-5.2, GPT-5-mini, and Claude Sonnet 4.5) then apply the instruction to the TikZ code with minimal changes. Annotators checked 2,819 generated examples, and 2,148 (76.2%) met all quality criteria.
  2. Step 2: visual instructions. Annotators look at each original/edited diagram pair and draw a visual instruction on the original diagram in Excalidraw, either freehand (pen) or with built-in shapes (tool). The final dataset contains 694 triplets (Dorig, Ivis, Dedit), annotated by 18 annotators.
Donut charts: edit operations are 34% one edit, 34% two edits, 32% three edits; edit types are ADD 20%, REMOVE 18%, UPDATE 23%, LAYOUT 15%, STRUCTURE 24%.
Edit type Scope #
ADD Local 140
REMOVE Local 124
UPDATE Local 160
LAYOUT Global 103
STRUCTURE Global 167
Total 694

UPDATE has ten fine-grained subtypes (e.g., text, line, and fill properties). 1 / 2 / 3 edit operations: 236 / 234 / 224 samples. Pen / tool annotations: 406 / 288 samples.

Visual Instruction Examples

Examples of visual instructions for each edit type, with one to three edit operations per instruction.

ADD visual instruction with one edit operation
1 edit operation
ADD visual instruction with two edit operations
2 edit operations
ADD visual instruction with three edit operations
3 edit operations
REMOVE visual instruction with one edit operation
1 edit operation
REMOVE visual instruction with two edit operations
2 edit operations
REMOVE visual instruction with three edit operations
3 edit operations
UPDATE visual instruction with one edit operation
1 edit operation
UPDATE visual instruction with two edit operations
2 edit operations
UPDATE visual instruction with three edit operations
3 edit operations
LAYOUT visual instruction with one edit operation
1 edit operation
LAYOUT visual instruction with two edit operations
2 edit operations
LAYOUT visual instruction with three edit operations
3 edit operations
STRUCTURE visual instruction with one edit operation
1 edit operation
STRUCTURE visual instruction with two edit operations
2 edit operations
STRUCTURE visual instruction with three edit operations
3 edit operations

Evaluation

The final visual result matters most, so image-level evaluation is the primary measure and code-level evaluation is supplementary. Each image-level aspect is rated on a 5-point scale; if compilation fails, all three aspects get the minimum score of 1.

Image-level primary

IA — Instruction Adherence
How well the edited diagram follows the instruction.
CP — Content Preservation
Whether it avoids unintended changes beyond the instruction.
DR — Diagram Readability
Whether it stays readable, e.g., no overlapping elements or text.
FSR — Full-score Rate
Share of samples that score 5 on all of IA, CP, and DR.

Code-level supplementary

CFR — Compile Failure Rate
Share of generated code that fails to render.
IA / CP — on code
Instruction adherence and content preservation, judged from the diff of Corig, Cedit, and Cgen.
OES — Overall Edit Score
A holistic score combining code-level IA and CP.

MLLM-as-a-judge agrees with humans. Five Amazon Mechanical Turk workers rated each of 300 sampled outputs. Among the candidate judges, Gemini 3.1 Pro agreed most closely with human scores (Spearman's ρ = 0.843 / 0.813 / 0.793 for IA / CP / DR), so it serves as the judge for all main results. Inter-annotator agreement (Krippendorff's α) was 0.80, 0.67, and 0.65 for IA, CP, and DR, respectively.

Main Results

Results for 29 code-editing models on VisEditBench, judged by Gemini 3.1 Pro. Click a column header to sort. best overall best open-source

Image-level Code-level
Model IA ↑ CP ↑ DR ↑ FSR ↑ CFR ↓ IA ↑ CP ↑ OES ↑
Closed-source models
GPT-5.4 4.329 4.036 4.542 47.26 0.72 3.751 3.679 3.390
GPT-5.4-mini 3.063 3.362 4.241 18.16 1.73 2.575 3.059 2.298
GPT-5.4-nano 1.856 2.977 3.915 4.611 3.17 1.579 2.767 1.439
Gemini 3.1 Pro Preview 4.945 4.797 4.945 83.57 0.00 4.490 4.614 4.329
Gemini 3 Flash Preview 4.643 4.398 4.624 71.76 0.86 4.347 4.133 3.984
Gemini 3.1 Flash Lite Preview 4.408 3.916 4.503 44.81 1.01 3.816 3.442 3.273
Claude Opus 4.6 4.382 4.438 4.797 59.94 0.58 3.780 4.190 3.638
Claude Sonnet 4.5 4.176 4.105 4.731 42.36 0.58 3.516 3.834 3.280
Claude Haiku 4.5 3.219 3.617 4.318 25.22 2.16 2.895 3.526 2.686
Open-source models
Qwen3.5-4B 2.532 2.641 3.318 12.25 9.22 2.141 2.349 1.859
Qwen3.5-4B Thinking 3.380 3.075 3.624 28.09 9.22 2.932 2.934 2.570
Qwen3.5-9B 2.713 2.763 3.555 14.12 7.78 2.248 2.514 1.941
Qwen3.5-9B Thinking 3.912 3.513 4.045 38.04 4.47 3.458 3.461 3.102
Qwen3.5-35B-A3B 3.630 3.653 4.324 31.12 1.15 3.020 3.405 2.742
Qwen3.5-35B-A3B Thinking 4.403 4.046 4.460 53.89 1.73 3.860 3.847 3.503
Qwen3.5-397B-A17B 3.829 4.195 4.591 38.04 0.87 3.333 4.039 3.159
Qwen3.5-397B-A17B Thinking 4.679 4.451 4.725 66.28 0.72 4.140 4.206 3.892
Qwen3-VL-4B-Instruct 2.468 3.732 4.389 12.82 3.03 2.078 3.625 1.977
Qwen3-VL-4B-Thinking 3.673 3.406 4.033 29.97 4.03 3.102 3.305 2.805
Qwen3-VL-8B-Instruct 2.918 3.656 4.422 17.86 2.16 2.416 3.409 2.284
Qwen3-VL-8B-Thinking 3.690 3.406 4.076 30.84 2.16 3.235 3.375 2.924
Qwen3-VL-30B-A3B-Instruct 2.660 4.251 4.586 18.73 1.44 2.304 4.075 2.258
Qwen3-VL-30B-A3B-Thinking 4.105 3.867 4.441 42.07 1.44 3.481 3.722 3.220
Qwen3-VL-235B-A22B-Instruct 3.824 4.023 4.558 36.02 2.31 3.282 3.829 3.086
Qwen3-VL-235B-A22B-Thinking 3.572 3.414 3.925 35.73 1.59 3.615 3.595 3.203
Gemma-4-E4B-IT 2.545 2.421 3.344 6.05 4.90 2.058 2.186 1.637
Gemma-4-31B-IT 3.976 4.039 4.389 43.52 1.59 3.627 3.854 3.352
GLM-4.5V 3.875 3.710 4.290 38.33 3.75 3.336 3.553 3.078
GLM-4.6V 3.951 3.973 4.477 40.49 1.59 3.352 3.733 3.143
  • The task is hard even for frontier models. Gemini 3.1 Pro leads with 83.6 FSR, but GPT-5.4 (47.3) and Claude Opus 4.6 (59.9) leave substantial room for improvement.
  • Performance drops sharply with model size across the GPT, Claude, and Qwen3.5 families.
  • Open models are catching up. Qwen3.5-397B-A17B Thinking reaches 66.3 FSR, competitive with closed-source models, and the Qwen3.5 series improves substantially over Qwen3-VL.
  • Avoiding unintended edits is harder than following the instruction. CP is often lower than IA, and thinking raises IA while sometimes lowering CP.

Analysis

Global edits are harder than local ones

Heatmap of FSR by edit type and model; UPDATE and REMOVE scores are generally higher than LAYOUT and STRUCTURE.

Local edits such as REMOVE and UPDATE score high for many models, while LAYOUT and STRUCTURE edits are much harder, especially for weaker models. Failures there come from misreading the target, its spatial relationships, or the intended scope of the edit, and from failing to ground them in the right part of the source code.

More operations, lower scores

FSR by number of edit operations (1, 2, 3) for each model; most models decline as operations increase.

FSR generally falls as the number of edit operations grows. Gemini 3.1 Pro and Gemini 3 Flash change little.

The source code matters most

FSR under different input modalities; removing the original TikZ code causes the largest drop.

Removing the original TikZ code Corig causes the largest drop. Inferring and editing TikZ from the image alone remains difficult.

Visual instructions are harder to follow than text

FSR with textual versus visual instructions; textual is higher for every model, with smaller gaps for stronger models.

Textual instructions beat visual ones for every model, though the gap narrows for stronger models. The model has to infer the target and scope from the annotations itself.

Editing code beats editing pixels

FSR of code-based editing versus direct image editing for each model series; code-based is higher.

Code-based editing outperforms direct image editing within each model series. Gemini 3 Pro Image reaches 60.8 FSR, versus 12.0 for GPT Image 1.5 and 0.1 for Qwen-Image-Edit.

Verbalizing the visual instruction helps

FSR under four prompting conditions across 16 model configurations: visual only, visual then verbalized text, gold text, and visual plus text.

Having the model first describe the visual instruction in text (Ivis → Îtext) improves FSR over Ivis alone in 15 of 16 model configurations. Gold textual instructions still do better overall.

Qualitative Examples

Successful and failed model edits for each edit type.

Success
Original diagram
Original
Visual instruction
Visual instruction
Model output
Model output
Failure
Original diagram
Original
Visual instruction
Visual instruction
Model output
Model output
Success
Original diagram
Original
Visual instruction
Visual instruction
Model output
Model output
Failure
Original diagram
Original
Visual instruction
Visual instruction
Model output
Model output
Success
Original diagram
Original
Visual instruction
Visual instruction
Model output
Model output
Failure
Original diagram
Original
Visual instruction
Visual instruction
Model output
Model output
Success
Original diagram
Original
Visual instruction
Visual instruction
Model output
Model output
Failure
Original diagram
Original
Visual instruction
Visual instruction
Model output
Model output
Success
Original diagram
Original
Visual instruction
Visual instruction
Model output
Model output
Failure
Original diagram
Original
Visual instruction
Visual instruction
Model output
Model output

BibTeX

@inproceedings{taneguchi2026viseditbench,
  title={{VisEditBench}: A Benchmark for Vector-Format Diagram Editing with Visual Instructions},
  author={Taneguchi, Akito and Yoshida, Haruto and Suzuki, Jun and Saito, Itsumi},
  booktitle={Advances in Neural Information Processing Systems (NeurIPS), Evaluations and Datasets Track},
  year={2026}
}

Acknowledgements

This work was supported by JST PRESTO Grant Number JPMJFR242S, JST BOOST Grant Number JPMJBY24E3, and JSPS KAKENHI Grant Number 24K20829. We thank BAOBAB Inc. for their contributions to the manual validation of generated editing instructions and edited diagrams, as well as the annotation of visual instructions. We would also like to express our sincere gratitude to the members of the Tohoku NLP Group for their invaluable advice and support throughout this research. This study was carried out using the TSUBAME4.0 supercomputer at Institute of Science Tokyo.