Do vision-language models (VLMs) share our ability to communicate visually? We share preliminary findings, starting from a surprising observation: simple visual symbols like a red heart can strongly influence model behavior on seemingly unrelated tasks, such as steering models towards recursive solutions on simple coding problems. Notably, this steering effect is largely implicit, as models' direct interpretations and thinking traces about visual symbols diverge substantially from their observed behavior. In visual communication between models, this leads to a generation--interpretation gap: while models tend to send shape primitives that they (and humans) find intuitive, these visual cues often fail to steer other models. In further communication games involving abstract concepts and stories, models can creatively use visuals to transmit meaning to one another, although some communications remain non-intuitive to humans. These findings reveal that VLMs might be capable of rich, creative, and visually-grounded communication, but their interpretation and production of visual meaning diverges from human conventions in surprising ways. Together this work highlights both the promise and challenges of visual communication between humans and VLMs.
simple visual symbols like a red heart can strongly influence model behavior on seemingly unrelated tasks, such as steering models towards recursive solutions on simple coding problems
selected symbols shift coding choices, with small per-condition samples and prompt-dependent effects limiting generality
models' direct interpretations and thinking traces about visual symbols diverge substantially from their observed behavior
unvalidated trace classification and symbol-level attribution limit inference about implicit influence
VLMs generate creative visualizations to communicate abstract concepts and stories, but do not always interpret their own visualizations in the intended way.
controlled games demonstrate communication successes and failures, with same-model sender–receiver testing limiting transfer
Models' visual meaning spaces are only partially aligned with human interpretations.
seven participants and outcome-conditioned visualization selection provide only exploratory evidence of human–model alignment
Beyond coding, the same 288 colored shapes reliably steer both writing style and responses to binary preference questions.
binary preferences were tested with five selected symbols rather than the claimed 288
Derived from the full evaluation — not a separate score.
Strengths
Token-matched captions, Gaussian noise, natural images, and direct text instructions test competing explanations for coding-strategy shifts. Perturbations further probe the role of shape geometry and orientation.
↳ Section 3.2; Appendices B.2.2–B.2.3
A 288-symbol assay is linked to sender–receiver games and human interpretation, bringing behavioral influence and visual communication into one empirical framework.
↳ Sections 3–5
The related-work section distinguishes simple-symbol steering from adversarial manipulation, visual distractors, visual reasoning, and emergent communication. The discussion explicitly treats deployment implications as prospective.
↳ Sections 2 and 6
Limitations
Section 3.3 attributes both writing-style and binary-preference steering to the same 288 shapes, but Appendix B.3.1 lists only five preference symbols. The preference evidence therefore has substantially narrower coverage than the Results statement.
↳ Section 3.3; Appendix B.3.1, Binary preference choices
The regex classifier is not validated like the separate code-strategy judge. Its assignment of symbol-level steering effects to individual traces limits conclusions about whether models recognize an image's influence.
↳ Section 3.4; Appendix B.8
Headline symbol effects are selected from 288 conditions with 20 generations each and no held-out rerun. Prompt wording and image placement substantially change steering, limiting extrapolation from the strongest examples.
↳ Section 3.1; Appendix B.2.4
The coding experiments combine balanced generation sampling with caption, noise, natural-image, and perturbation controls, supporting an empirical contribution beyond isolated visual-prompt examples. The strongest symbol effects are selected from a broad exploratory search, and Appendix B.2.4 shows that their magnitude and identity depend on prompt wording and placement. The implicit-influence interpretation receives less support because Appendix B.8's trace classifier lacks independent validation and assigns aggregate symbol effects to individual traces. Section 3.3 also overstates binary-preference coverage as 288 symbols when the methods list five, while the human comparison remains exploratory with seven participants.
Nabu’s assessment, alongside the field’s view.
Are you an author of this paper?
Sound3.5
Confidence mediumThe study connects a 288-symbol behavioral assay with visual controls, communication games, and a human comparison, extending the cited steering and emergent-communication literature. The advance is empirical characterization rather than an identified mechanism or a general account of visual semantics.
Balanced sampling, orthogonal controls, perturbations, and code-judge validation support the steering experiments, while post-selection, an unvalidated trace classifier, and the small human study limit execution. The merit-based score includes those methodological limitations; the five-versus-288 reporting-scope contradiction is separately surfaced through the Red reliability flag.
The progression from steering to communication to human interpretation is readily followed. Central claims about implicit influence exceed what the trace analysis establishes, and reporting-scope and aggregation ambiguities impede precise interpretation.
Related work distinguishes visual prompting, adversarial steering, faithfulness, and emergent communication, while the discussion explicitly limits deployment and human-population extrapolation. Prompt dependence and other complicating appendix findings receive less emphasis in the main interpretation.
Concerns4 of 4 checks
Ethics approval and informed consent are declared. No contradictory registration, disclosure, or availability claim is established in the supplied text.
Ethics approval: declared
56 references in manuscript 41 of 54 checkable references found in an index 2 are books, websites or datasets — counted, but not index-checkable
References not verified automatically are resolved manually within 24 hours.
No retraction or expression of concern found in Crossref (includes Retraction Watch).
Sources: Crossref (incl. Retraction Watch) ✓
Where this paper’s evidence sits on the path from initial observation to real-world use.
The evidence remains at the Mechanistic readiness stop: controlled synthetic experiments characterize behavior and boundary conditions. No mitigation or naturalistic agent workflow is validated.
AI-generated, human-governed. Something look off? Contact us to request a review.
Keep reading
Other papers Nabu has evaluated blind, starting with the same field.
Andrew Sellergren, Sahar Kazemzadeh, Fereshteh Mahvar, et al. · 2026 · Medical and Health Sciences - Biomedical research
For out-of-distribution tasks, MedGemma achieves improvements of 2.6–10% in medical image question answering, 15.5–18.1% in chest X-ray finding classification and 10.8% in agentic evaluations compared with the base models.
TentativeKeren, Calderon, Yehudai, et al. · 2026 · Computer and information sciences
models nearly saturating τ²-Bench suffer severe performance drops on our tasks
TentativeFoundation · 2026 · Computer and information sciences
Our testing shows humans can solve 100% of the environments, in contrast to frontier AI systems which, as of March 2026, score below 1%.
Tentative