TeaBench

Single request per item at temperature 0; the answer must be exactly one option key. Anything else is a format failure and counts as wrong.

Every item is graded deterministically against a frozen answer key by exact key match; there is no LLM judge anywhere in this benchmark.

Eighteen paired items check that answers follow real evidence changes and resist injected instructions or irrelevant narrative; a pair counts only when both members are correct.

95% descriptive item-sampling intervals from a fixed-seed bootstrap over the 188 items; they describe sensitivity to item selection, not run-to-run variation (grading itself is deterministic).

Median wall time from request to final answer, one request at a time, on one workstation GPU (RTX PRO 6000 Blackwell). The decision-native models read their answer straight off the model's logits and generate no answer text.

Everything here runs locally on that single GPU with weights resident in memory: there are no per-token API fees to compare. What serving costs is the memory a model needs and the wait it causes. Energy use and hardware amortization are not measured and are not published as costs.

Two cohorts share the headline score: the 88 everyday items carried over from release v1 and the 100 compositional stress items added in v2. Items that every model passes add no separating evidence, so check both cohorts before trusting an ordering.

The corpus is private and freshly authored, but the knowledge-control group uses public tea facts. Ordinal rubric items require the exact rubric level; adjacent levels count wrong. The calibration group (known-probability items) is excluded from scoring until a confidence protocol exists. A single answer per item at zero temperature measures the decision, not its stability across re-prompts.

Sanitized aggregates only. The private corpus and per-item answers are not published.

One request per question: one image and four lettered options, with the correct answer's letter position permuted across items.

Every item is exact-match against an answer key: the chosen letter must match, and an answer that does not follow the required format counts wrong. No judges are involved at any step.

Answer keys come from the vendors' own catalogue taxonomy and listing claims, not from tea expertise. Origin and quality claims are reported separately and never scored.

Images are public product photography from well-known tea vendors, so models may have seen these catalogues during training. Scores describe decisioning on familiar catalogues, not sealed generalization. This is the deliberate inverse of the Tea Decisions board above, which runs on a private corpus the models have not seen.

The interval is a 95% Wilson score interval over the 155 scored items.

All models ran locally on one RTX PRO 6000 GPU, one question at a time. Jev-Omni answers with its own decision head; Qwen3-VL 30B answers in prompted chat mode, and its text-only row shows how much of its score comes from actually reading the image.

Version 0: 155 items across two model families and one control condition. Vendor taxonomy is marketing language, so a correct answer means agreement with the catalogue, not tea truth.

Sanitized aggregates only: item counts and scores, nothing from the corpus or the answer keys.