Streaming vision-language evaluation

A benchmark for models that know when to speak.

SVI-Bench measures proactive triggering, correct silence, latency, response quality, and long-horizon memory on the same live video timeline.

Explore results View code

Open workflow Auditable evidence Resumable evaluation

SVI stream · decide · respond

D1Trigger

D2Silence

D3Latency

D4Content

D5Memory

75items

9scenarios

4.21 hsource video

272expert anchors

375recordings

The benchmark

Interaction is more than answering correctly.

A streaming assistant observes continuously. It must detect a relevant event, decide whether a response is warranted, answer within the required window, and preserve context over time. SVI-Bench makes each failure mode visible.

Dataset statistics, representative tasks, and the five SVI-Bench evaluation dimensions — **SVI-Bench at a glance.** Seventy-five items span autonomous interaction, real-time response, and delegation with memory; videos range from 5 seconds to 24.9 minutes.

A Autonomous interaction

32 items

A1 Monitoring & alerting 12
A2 App guidance 6
A3 Live commentary 9
A4 Vision-driven interaction 5

B Real-time response

33 items

B1 Real-time translation 11
B2 Real-time counting 12
B3 Time awareness 10

C Delegation & memory

10 items

C1 Long-horizon visual memory 8
C2 Agent delegation 2

Anchored rubric

Five dimensions. One observable timeline.

Select a dimension to see what the Judge measures.

Evaluation question

Does the system speak when a target event occurs?

Match each annotated event with one valid response—without misses, premature fires, or duplicate triggering.

Full creditExactly one valid trigger

Coverage55 items

Evaluation protocol

Fix what the system sees. Preserve how the system works.

Every configuration receives the same local MP4, task prompt, query schedule, foreground frame schedule, recording frontend, and scoring rule. Native streaming sessions, memory, and serving paths remain system-specific.

Local MP4 upload Fresh session per item 1 fps foreground schedule One in-flight inference Busy frames skipped No RTSP conversion

Common source video and recording frontend routed to native streaming systems — The protocol standardizes the observable task—not the internal architecture of each deployed system.

Reward-hacking-resistant aggregation

Triggering and silence only earn credit together.

Independent D1 and D2 averages reward degenerate behavior. A mostly silent system can appear excellent on silence correctness; a constantly talking system can do the reverse. SVI-Bench couples D1 and D2 inside each item before aggregation.

Per-item interaction component

a_i = min(D1_i, D2_i)

When both dimensions apply; a lone D1 or D2 is retained.

Example when all five dimensions apply

q_i = min(D1, D2) + D3 + D4 + D54

Only applicable components enter the denominator.

Official leaderboard score

S = 100 / 75 · Σ q_i

Every item has equal weight; no optional importance weights.

≠

Not a minimum of global means.

The official score is not min(mean(D1), mean(D2)) and not a five-dimension mean. Coupling happens per item, so success on one item cannot compensate for the opposite failure on another.

Three-level item scoring, item-level D1 D2 coupling, human pilot, and automatic Judge selection — Grades are anchored to observable times, counts, or information points. Human labels select the Judge but never enter formal Judge inputs.

375-recording evaluation

Current systems still struggle to interact in time.

Five deployed configurations were evaluated on all 75 items. The Overall score is the sole ranking metric; category, dimension, and latency-removed views are diagnostic.

Official leaderboard

Overall score

Item-level D1/D2 coupling · higher is better

Overall is the only leaderboard score. Values are computed before rounding.

53.61

Highest Overall

JoyAI-VL-Interaction leads the strongest comparison configuration by 28.17 points.

29.09

Best raw D1

No evaluated system exceeds 30 on proactive triggering, exposing a shared weakness.

17.27 / 91.84

Doubao D1 / D2

The large split shows why silence correctness cannot be rewarded independently.

Official results

Item-coupled leaderboard

75 items per system

Rank	Deployed configuration	Overall S ↑	S without D3 ↑
1	JoyAI-VL-Interaction	53.61	39.67
2	MOSS-VL-Realtime	25.44	26.33
3	MiniCPM-O-4.5-9B	21.33	20.44
4	Doubao Seed 2.1 Pro	21.22	29.44
5	Mage-VL	19.50	21.78

S without D3 preserves item-level D1/D2 coupling before removing latency. It is a sensitivity analysis, not a second leaderboard.

Explore scenario-level results 9 categories

System	A1	A2	A3	A4	B1	B2	B3	C1	C2
JoyAI-VL-Interaction	65.28	41.67	44.44	46.67	36.36	48.61	67.08	68.75	75.00
Doubao Seed 2.1 Pro	45.83	22.22	12.96	40.00	0.00	12.50	30.83	16.67	0.00
Mage-VL	18.06	5.56	20.37	50.00	6.06	22.22	37.92	8.33	0.00
MOSS-VL-Realtime	37.50	8.33	37.04	50.00	0.00	22.22	40.83	18.75	0.00
MiniCPM-O-4.5-9B	23.61	33.33	22.22	6.67	18.18	18.06	36.67	12.50	0.00

Human-calibrated Judge

Selected on expert agreement, then frozen.

A 20-item pilot covering all nine scenarios provides 71 dimension-level expert labels. The selected five-stage Judge is applied unchanged to the full benchmark.

0.0986MAE ↓

on the 1 / 0.5 / 0 scale

83.10%Exact match ↑

59 of 71 labels

97.18%Within one level ↑

69 of 71 labels

0.7803Quadratic κ ↑

human–Judge agreement

Scope: the pilot contains JoyAI-VL-Interaction recordings. Cross-system human annotation remains necessary to quantify Judge generalization across response styles and latency regimes.

InteractFlow

From an annotated item to an auditable score.

Recording and judging are deliberately decoupled. Valid traces can be inspected or rejudged without rerunning the evaluated model; invalid traces return to bounded recording recovery.

Seven-stage InteractFlow recording, validation, evidence, judging, and aggregation pipeline — One workflow binds task identity, source media, system identity, evidence provenance, Judge stages, and final aggregation.

01 Annotate→ 02 Synchronize→ 03 Record→ 04 Validate→ 05 Build evidence→ 06 Judge ×5→ 07 Aggregate

Reproduce with one entrypoint

Configure once. Validate locally. Resume safely.

The campaign supervisor records all enabled systems, verifies every accepted trace, constructs label-free Judge manifests, and runs all five judging stages. Completed hash-compatible artifacts survive interruption.

✓ 375 validated recordings ✓ 1,875 successful stage predictions ✓ Five complete model summaries ✓ Blindness and provenance audits

SVI-Bench · bash

$ git clone https://github.com/YidanHAI/VL-Interaction-interactflow.git
$ cd VL-Interaction-interactflow

# Install the lightweight orchestration environment
$ python3 -m venv .venv
$ source .venv/bin/activate
$ pip install -r requirements.txt
$ npm ci

# Configure private endpoints and credentials
$ cp .env.example .env
$ chmod 600 .env

# Validate, start, and monitor
$ bash scripts/run_all.sh validate
$ bash scripts/run_all.sh start
$ bash scripts/run_all.sh status

Resources

Use the benchmark, inspect the protocol, or automate the run.

</>

Code

Recording, Judge, audits, adapters, and tests.

↗ ▦

Data guide

Workbook, MP4 layout, manifests, and outputs.

↗ ◇

Codex Skill

Configure, run, monitor, resume, and audit.

↗

Citation

Building on SVI-Bench?

Please cite the accompanying manuscript and link to this repository.

@misc{lin2026svibench,
  title  = {SVI-Bench: A Benchmark for Streaming Vision-Language
            Interaction with Latency, Silence, and Long-Horizon Memory},
  author = {Jianan Lin and Yidan Huang and Xiaoyi Qiao and Dongyi Lv and
            Yicheng Wang and Shi Suo and Shiying Fan and Yifei Li and
            Dingyu Yao and Junhao Zhou and Chuanyu Qin and Chenxu Yang and
            Qingyi Si and Nan Duan and Jiaqi Wang},
  year   = {2026},
  url    = {https://github.com/YidanHAI/VL-Interaction-interactflow}
}