Horizon RunMap - results and evidence
Current overview · 2026-10-01. The RTX 5070 campaign records execution, memory and speed across 13 configurations. The published IFEval edition evaluates all 541 prompts and retains every response; prompt-level loose accuracy is 45.66%. The public app completed the full IFEval process on another computer with an RTX 4060. Public app · Machine-readable claims.
Public app and external execution
The downloadable desktop app gives an external user a guided path to prepare the pinned public model and run IFEval. Its complete execution on the other RTX 4060 computer is a project milestone. The campaign's memory and speed numbers below belong to the RTX 5070; the published IFEval score belongs to its own retained-response edition. Project presentation · Download page.
Separate experiments
| Experiment | Population | Evidence / validation / publication / reproduction |
|---|---|---|
| Residency and executors | 13 configurations; 39 workers; 468 measured responses | E3 / valid / public / not_attempted |
| Historical BF16/Horizon comparison | 120 paired prompts | E3 / valid / public / not_attempted |
| IFEval | 541 prompts; 834 instructions | E1 standalone_benchmark / valid / public / not_attempted |
The three editions have separate populations, scorers and questions. Their original manifests and responses remain available for inspection.
Performance contract and results
PERF-REAL-v1; 12 prompts repeated in 3 blocks; 36 measured responses of 128 tokens per configuration; greedy selection, EOS suppressed; 64-token warmup excluded. Canonical initial placement, evolving LRU through fixed prompts; concurrency 1.
Pooled decode = sum(N - 1) / sum(internal decode seconds); it excludes first-token wait. Overall output rate uses sum(N) / sum(internal response runtime). TTFT and response time are arithmetic request means. VRAM is a device-wide whole-worker driver peak including preparation, warmup and shared desktop use; RSS is a separate whole-worker process peak. H2D is a sum over measured requests, not a rate. Isolated prefill is UNKNOWN; OLMoE HTTP observation is UNSUPPORTED. Block spread is a descriptive range, not a confidence interval.
| Configuration / source | Executor | Decode token/s | Whole-worker VRAM GiB |
|---|---|---|---|
| OLMoE 0924 c30/64 | PRODUCT-core scalar | 23.00 | 4.11 |
| OLMoE 0924 c40/64 | PRODUCT-core scalar | 27.69 | 4.62 |
| OLMoE 0924 c50/64 | PRODUCT-core scalar | 31.50 | 5.15 |
| OLMoE 0924 c60/64 | PRODUCT-core scalar | 35.01 | 5.67 |
| OLMoE 0924 c64/64 | Grouped T3 | 61.26 | 5.99 |
| OLMoE 0125 c30/64 | PRODUCT-core scalar | 23.04 | 4.15 |
| OLMoE 0125 c40/64 | PRODUCT-core scalar | 27.12 | 4.62 |
| OLMoE 0125 c50/64 | PRODUCT-core scalar | 31.22 | 5.15 |
| OLMoE 0125 c60/64 | PRODUCT-core scalar | 35.44 | 5.67 |
| OLMoE 0125 c64/64 | Grouped T3 | 62.07 | 6.01 |
| Qwen c20/60 | top4-fused-v1 | 15.22 | 7.11 |
| Qwen c30/60 | top4-fused-v1 | 17.17 | 8.08 |
| Qwen c39/60 | top4-fused-v1 | 19.95 | 8.99 |
Capacities are per routed layer. Compare OLMoE c30-c60 within the PRODUCT-core scalar family; c64 is a separate Grouped T3 executor result. Qwen c20/c30/c39 use top4-fused-v1. Keep model revisions and executors attached to every value; a runtime-to-runtime ranking calls for a matched comparison.
Performance manifest · Contract/report · Verification method · Verifier
Historical behavioral comparison
Under 120 paired structured prompts, BF16 produced 29/120 structurally valid responses and Horizon produced 28/120. There were 8 paired structural regressions, 7 improvements, 19 observable-behavior parity cases and 86 directionless divergences. Structural conformity is not factual-answer correctness. The original campaign's publication sanitizer failure is retained in provenance; the CPU-only successor recovers the original responses without changing the frozen scorer.
Paired manifest · Contract · Summary · Audit method · ZIP
IFEval 541
Model: allenai/OLMoE-1B-7B-0924-Instruct, revision 7f1c97f440f06ce36705e4f2b843edb5925f4498. The complete declared population was scored using frozen evaluator inputs. All 467 EOS and 74 length-limit endings remain included.
| Metric | Numerator / denominator | Percent |
|---|---|---|
| Prompt strict | 217 / 541 | 40.11% |
| Prompt loose | 247 / 541 | 45.66% |
| Instruction strict | 433 / 834 | 51.92% |
| Instruction loose | 474 / 834 | 56.83% |
The complete IFEval population and all scored responses are public. Its frozen scorer defines these instruction-following metrics. External model-card numbers provide context for readers; the original contract and publication metadata retain the formal evidence classification.
IFEval manifest · Frozen contract · Public admission · All responses · Audit method · ZIP
Verification and reproduction
The performance verifier checks hashes and aggregate-copy consistency; that release does not include the complete raw-run bundle. Behavioral and IFEval verifiers recompute their declared scores from retained responses on CPU. None reruns the private inference runtime. Independent reproduction is not_attempted for all three editions. UNKNOWN denotes an absent observation; UNSUPPORTED denotes unavailable observation capability.