Horizon RunMap - experimental evidence
Reviewed 2026-09-16. Values below are generated from the existing public editions, without new model execution. Machine-readable claims.
Separate experiments
| Experiment | Population | Evidence / validation / publication / reproduction |
|---|---|---|
| Residency and executors | 13 configurations; 39 workers; 468 measured responses | E3 / valid / public / not_attempted |
| Historical BF16/Horizon comparison | 120 paired prompts | E3 / valid / public / not_attempted |
| IFEval | 541 prompts; 834 instructions | E1 standalone_benchmark / valid / public / not_attempted |
These are separate populations with separate scorers and questions. Public admission, evidence maturity, verification and independent reproduction are distinct.
Performance contract and results
PERF-REAL-v1; 12 prompts repeated in 3 blocks; 36 measured responses of 128 tokens per configuration; greedy selection, EOS suppressed; 64-token warmup excluded. Canonical initial placement, evolving LRU through fixed prompts; concurrency 1.
Pooled decode = sum(N - 1) / sum(internal decode seconds); it excludes first-token wait. Overall output rate uses sum(N) / sum(internal response runtime). TTFT and response time are arithmetic request means. VRAM is a device-wide whole-worker driver peak including preparation, warmup and shared desktop use; RSS is a separate whole-worker process peak. H2D is a sum over measured requests, not a rate. Isolated prefill is UNKNOWN; OLMoE HTTP observation is UNSUPPORTED. Block spread is a descriptive range, not a confidence interval.
| Configuration / source | Executor | Decode token/s | Whole-worker VRAM GiB |
|---|---|---|---|
| OLMoE 0924 c30/64 | PRODUCT-core scalar | 23.00 | 4.11 |
| OLMoE 0924 c40/64 | PRODUCT-core scalar | 27.69 | 4.62 |
| OLMoE 0924 c50/64 | PRODUCT-core scalar | 31.50 | 5.15 |
| OLMoE 0924 c60/64 | PRODUCT-core scalar | 35.01 | 5.67 |
| OLMoE 0924 c64/64 | Grouped T3 | 61.26 | 5.99 |
| OLMoE 0125 c30/64 | PRODUCT-core scalar | 23.04 | 4.15 |
| OLMoE 0125 c40/64 | PRODUCT-core scalar | 27.12 | 4.62 |
| OLMoE 0125 c50/64 | PRODUCT-core scalar | 31.22 | 5.15 |
| OLMoE 0125 c60/64 | PRODUCT-core scalar | 35.44 | 5.67 |
| OLMoE 0125 c64/64 | Grouped T3 | 62.07 | 6.01 |
| Qwen c20/60 | top4-fused-v1 | 15.22 | 7.11 |
| Qwen c30/60 | top4-fused-v1 | 17.17 | 8.08 |
| Qwen c39/60 | top4-fused-v1 | 19.95 | 8.99 |
Capacities are per routed layer. OLMoE c30-c60 use PRODUCT-core scalar; c64 uses Grouped T3, so c30-to-c64 or c60-to-c64 is not a residency-only ablation. Qwen c20/c30/c39 use top4-fused-v1. Keep model revisions and executors attached to every value; no cross-runtime ranking is measured.
Performance manifest · Contract/report · Verification method · Verifier
Historical behavioral comparison
Under 120 paired structured prompts, BF16 produced 29/120 structurally valid responses and Horizon produced 28/120. There were 8 paired structural regressions, 7 improvements, 19 observable-behavior parity cases and 86 directionless divergences. Structural conformity is not factual-answer correctness. The original campaign's publication sanitizer failure is retained in provenance; the CPU-only successor recovers the original responses without changing the frozen scorer.
Paired manifest · Contract · Summary · Audit method · ZIP
IFEval 541
Model: allenai/OLMoE-1B-7B-0924-Instruct, revision 7f1c97f440f06ce36705e4f2b843edb5925f4498. The complete declared population was scored using frozen evaluator inputs. All 467 EOS and 74 length-limit endings remain included.
| Metric | Numerator / denominator | Percent |
|---|---|---|
| Prompt strict | 217 / 541 | 40.11% |
| Prompt loose | 247 / 541 | 45.66% |
| Instruction strict | 433 / 834 | 51.92% |
| Instruction loose | 474 / 834 | 56.83% |
The original frozen contract keeps its historical internal_only field; separate publication metadata records later public admission as E1 standalone_benchmark. These instruction-following scores do not establish matched BF16 preservation. External model-card numbers are contextual, non-controlled references; one reference does not specify the IFEval variant.
IFEval manifest · Frozen contract · Public admission · All responses · Audit method · ZIP
Verification and reproduction
The performance verifier checks hashes and aggregate-copy consistency; that release does not include the complete raw-run bundle. Behavioral and IFEval verifiers recompute their declared scores from retained responses on CPU. None reruns the private inference runtime. Independent reproduction is not_attempted for all three editions. UNKNOWN denotes an absent observation; UNSUPPORTED denotes unavailable observation capability.