Noru FlowResearch
Navigation menu
All researchSeeing

The same context, in a better order

In VisualWebArena's tested GPT-4V baseline, adding screenshots raised success from 7.3% to 15.1%; Set-of-Marks reached 16.4%. The result supports structured visual context, but Set-of-Marks is not a Noru feature.

On this page
  1. Pixels, labels, and order are different interventions
  2. Order is not neutral
  3. Hard benchmarks define the boundary
  4. The Noru decision
  5. Sources
External VisualWebArena results are separated from Noru’s actual evidence contract.

Two payloads can contain related pixels and words yet be easier or harder to use. Structure affects which evidence a model notices, how it connects text to an interface, and whether important context is buried. The individual techniques need to stay distinct: adding pixels is not the same as labeling elements, and neither is the same as ordering the payload.

Pixels, labels, and order are different interventions

VisualWebArena evaluated a GPT-4V agent on realistic, visually grounded web tasks11VisualWebArena: Evaluating Multimodal Agents on Realistic Visually Grounded Web Tasks Koh et al., 2024:

Context given to the agentSuccess rate
Text only (accessibility tree)7.3%
+ screenshot15.1%
+ labeled elements (Set-of-Marks)16.4%
Human88.7%

Adding screenshots more than doubled the tested baseline. Adding Set-of-Marks labels lifted it again. Set-of-Marks is an external technique that overlays visual references on interactive elements; Noru does not add those labels. The absolute scores belong to the tested model, agent, and dataset.

Order is not neutral

Tan et al. found large performance swings when they changed the position of information in multimodal inputs22Order Matters: Exploring Order Sensitivity in Multimodal Large Language Models Tan et al., 2024. Their experiments concern context position, not OCR reading order. The connection to Noru is a design analogy: evidence should arrive with a deliberate structure, and that structure should be described precisely.

Hard benchmarks define the boundary

MuirBench tests multi-image reasoning; ScreenSpot-Pro tests precise GUI grounding in professional applications33MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding Wang et al., 202444ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use Li et al., 2025. Both show that visual reasoning remains difficult in their tested settings. Neither measures Noru, nor do they show that ordered OCR alone solves the problem.

The Noru decision

Noru currently contributes a fresh image plus recognized text in reading order. It keeps the capture identity, dimensions, and provenance attached so the evidence remains inspectable. It does not add Set-of-Marks labels, and it does not turn an external benchmark into a product-performance claim.

Structure is part of the payload, not decoration. The cited work explains the design direction; it does not measure a Noru performance gain.

Sources

  1. VisualWebArena: Evaluating Multimodal Agents on Realistic Visually Grounded Web Tasks, Koh et al., 2024
  2. Order Matters: Exploring Order Sensitivity in Multimodal Large Language Models, Tan et al., 2024
  3. MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding, Wang et al., 2024
  4. ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use, Li et al., 2025

macOS 13+ · Apple Silicon

Noru for Mac.

Download free

v0.3.1 · Signed & notarized by Apple

Installation guide