assay — an open System One decision model, measured against the real Jev

01Head-to-head, identical rows

Metric is per-family balanced accuracy (mean per-class recall, so majority-class guessing scores at chance), macro-averaged across families — families are never pooled. The evaluator iterates gold labels: a missing or errored prediction counts as a failure. n = 100 rows per family.

assay-4B (ours) real Jev 1.13 laya (encoder, 421M) untrained pointer-head

In-distribution — trained task families, held-out splits

Family	Type	untrained	laya	assay-4B	real Jev
ag_news	choice-4	0.245	0.956	0.927	0.873
boolq	noul	0.502	0.791	0.843	0.979
mnli	choice-3	0.348	0.873	0.860	0.834
sst5	score-5	0.200	0.346	0.539	0.558
Macro		0.324	0.741	0.792	0.811

Out-of-domain — task families never seen in training

Family	Type	untrained	laya	assay-4B	real Jev
emotion	choice-6	0.167	0.548	0.630	0.655
imdb	noul	0.960	1.000	0.860	0.940
yelp	noul	0.500	0.500	0.981	1.000
Macro		0.542	0.683	0.823	0.865

Three findings. (1) The reproduction works — training lifted the architecture from 0.324 (random head) to 0.792, and past the separate zero-training LM-head bar of 0.503 / 0.717. (2) It does not collapse out of domain: 0.823 on unseen tasks, the same ~4 pp gap to Jev as in-distribution — the failure mode other reproductions warn about did not occur. (3) Ordinal is unsolved by everyone, including Jev: sst5 tops out at 0.558 (Jev), 0.539 (assay), 0.346 (laya). That is the open research gap, not an implementation bug.

02The near-Jev landscape

Every open reproduction of TypeSafe's Jev, and the axes that actually separate them. The load-bearing difference is the readout: only a pointer head over per-option hidden states is faithfully listwise — where adding an irrelevant option can legitimately shift the odds ratio between two existing ones, the behaviour Jev's IIA experiment demonstrated.

Project	Backbone	Readout	Training	Listwise-faithful	Multiling.
Jev (TypeSafe)	frontier, likely sparse MoE	slot / pointer, prefill-only	RLCD (proper scoring)	— reference	EN
kev (jaredpalmer)	Qwen3 0.5–8B dense	pointer + block-causal	LoRA, frozen suites, autoresearch	yes	no
assay (ours)	Qwen3-4B → 3.8	pointer + block-causal	LoRA r16, lr 5e-5	yes	no
Nimble (Bespoke)	Qwen3.5-9B dense	LM-head candidate logits	LoRA + contrastive curation	partial	no
SemIf (TheoLeeCJ)	Qwen3.5-4B frozen	LM-head letter logits	none (zero-shot)	partial	no
laya (NandhaKishorM)	ModernBERT 421M / mmBERT	encoder + router	RLCD (proper scoring)	↔ encoder branch	100+

Which is the best open implementation?

No single winner — it depends on the axis, and saying so is more useful than crowning one.

most faithful + rigorous

kev

The only project that reproduces Jev's listwise architecture and wraps it in real research discipline — frozen checksummed suites, a locked test read once, a provenance ring, autoresearch, a whole size family. The reference open Jev.

most practical

laya

Fastest (~26 ms/row), tiniest (421M), pip install laya, 100+ languages with a per-request router. Best choice for shipping multilingual routing today — at the cost of ordinal quality (sst5 0.346) and some OOD instability (yelp 0.500).

best data engine

Nimble

Plan → pair → necessity-gate → label-by-code contrastive curation: the LLM never produces a final label, Python computes it from audited rules over verified facts. The single most worth-copying component in the field.

03How assay is built

Fuse the best of each, then add what nobody has built.

Architecture (from kev, proven): block-causal segment mask + per-branch position restart + a pointer head scoring each option's closing-delimiter hidden state against the decision token. Questions are mutually invisible; question order is irrelevant; K is free at request time.
Mechanism-gated, and mutation-verified: packed-vs-separate agreement and a hostile-sibling isolation probe both hold below 2e-3. Replacing the block-causal mask with a plain causal mask pushes the isolation delta to 0.2655 — the gate really does go red, so the guarantee is enforced rather than decorative.
Data: Nimble-style necessity-gated contrastive curation for natural language, plus kev-style programmatic policy rule-trees and none-of-the-above minimal pairs for logic.
New — independent-family verifier: generator = model A, fact-checker = model B from a different family, breaking the "same model generates and verifies → correlated errors" ceiling that Nimble flags in its own manifests.
New — cumulative-threshold ordinal head: predict K−1 monotone thresholds → CDF → level probabilities, instead of a flat softmax. This is kev's own top hypothesis for the ordinal gap, which it never built, and our measurements show ordinal is exactly where everyone stalls.
Objective: move from plain cross-entropy toward a proper scoring rule (Brier / log loss on outcomes) — that is what produces calibration, Jev's actual product — and report binned ECE, not just accuracy.
Capacity path (Qwen3.8): capacity dominates out of domain — we measured +22 pp from 0.6B to 4B with no training at all. The architecture is base-agnostic (it strips the LM head and reads the last hidden state), so scaling is a --base change plus a pinned revision. If Qwen3.8 ships a sparse-MoE variant it is also the most Jev-faithful choice: a prefill-only decision model is compute-bound, exactly what sparse routing saves, without MoE's usual decode-time penalty.

Honest caveats. n = 100 rows per family, so confidence intervals are wide — this is a first result, not a publication number. Training and evaluation use disjoint splits of the same sources (in-distribution transfer); the out-of-domain suite shares task families with training, so it is "easy OOD" — hard OOD (date arithmetic, multi-hop policy composition, paraphrase) is the next probe and is where the Jev gap is expected to widen. Single seed, no hyperparameter search, no temperature calibration yet. assay runs bf16; Jev numbers come from its live API.