Learning to Look

A Foundation for Visual Understanding

¹Aalto University ²ELLIS Institute Finland ³University of Oulu

Code BibTeX

Overview

Vision-language models can describe objects that are not present in an image. We ask whether a model knows where it is looking before interpreting what it sees.

We introduce Look, a task in which the model identifies the region of an image available to it. We construct examples by cropping images and retaining the crop’s position information from the full image. The crop coordinates provide supervision without semantic annotation.

Existing VLMs struggle to localize the visible region. Fine-tuning on Look improves localization and visual-input awareness. We also introduce VisABench to measure whether a model’s answers depend on the presence of visual evidence.

The LOOK task

We resize the image to the model’s patch grid, then sample a rectangular crop aligned to complete visual tokens. The model receives only the crop pixels, with positional information inherited from the full image. It predicts the crop’s bounding box in full-image coordinates.

A crop is sampled on the visual-token grid. Its full-image positional information is retained, and its coordinates become the training target. — The crop supplies both the input and its label. Pixels outside the crop do not enter the vision encoder. Resetting the positions to the crop’s own grid provides a control condition.

Crop example

Move the crop across the photograph. The model receives the selected pixels and predicts their location within the full image.

Sheep grazing in a field. — Full image, with the selected region outlined

Crop supplied to the model

Horizontal position

Coordinates are normalized to 0–1000, relative to the full image.

VisABench

VisABench asks the same object-presence questions under three conditions: the correct image, no visual evidence, and a different image. In the no-evidence condition, visual features are withheld while the input sequence layout is preserved.

Is there a sheep in the image?

No visual evidence

Visual features withheld

Expected answer: No

A dog lying in the grass beside a red frisbee. — Expected answer: No

Fine-tuning on Look reduces hallucinated object presence when evidence is absent.

VisABench results · 10,752 items
Model	No image		Real image		Different image		Gap ↑
Model	Yes (%) ↓	p(Yes) ↓	Yes (%) ↑	p(Yes) ↑	Yes (%) ↓	p(Yes) ↓	Gap ↑
Qwen3-VL-8B-Instruct	2.5	0.041	93.3	0.927	15.6	0.158	90.7
SFT	31.2	0.409	94.9	0.915	17.0	0.196	63.7
LocPos	5.7	0.272	92.6	0.890	15.0	0.170	86.9
Ours (1:10)	2.3	0.202	94.1	0.910	15.7	0.183	91.8
Ours (2:10)	4.9	0.241	93.9	0.911	16.0	0.188	89.0
Ours (10:10)	0.8	0.101	94.1	0.909	15.1	0.175	93.2
Qwen2.5-VL-7B^†	—	—	90.0	0.868	12.5	0.129	—
LLaVA-1.5-7B^†	—	—	95.3	0.775	30.2	0.355	—
Gemma-4-12B	42.9	0.434	88.2	0.880	12.2	0.123	45.4

Gap is the real-image Yes rate minus the no-image Yes rate, in percentage points. † No-image results are omitted for models that produce neither Yes nor No without visual input.

Quick demo

Run two notebooks to inspect visual awareness and compare models on VisABench samples.

Visual-awareness probes
Compare pixel masks, patch-embedding masks, and crops with reset or restored positions.
VisABench model comparison
Test Qwen3-VL-8B-Instruct, SFT, LOOK (10:10), Qwen2.5-VL, LLaVA, and Gemma on the same samples.

git clone https://github.com/wenshuaizhao/look.git
cd look
python -m pip install -r requirements.txt -r requirements-notebooks.txt
python -m pip install -e vendor/ms-swift
python -m ipykernel install --user --name look --display-name 'Python (LOOK)'
jupyter lab notebooks

Inference uses a GPU. Set the SFT and LOOK adapter paths in the second notebook; see the setup instructions.

Citation

@misc{zhao2026learningtolook,
  title={Learning to Look: A Foundation for Visual Understanding},
  author={Wenshuai Zhao and Yihao Wang and Benjamin Robson and Juho Kannala and Arno Solin},
  year={2026},
  url={https://github.com/wenshuaizhao/look}
}