# Horizon RunMap - experimental evidence

Reviewed 2026-09-16. Values below are generated from the existing public editions, without new model execution. [Machine-readable claims](/claims.json).

## Separate experiments

| Experiment | Population | Evidence / validation / publication / reproduction |
| --- | --- | --- |
| Residency and executors | 13 configurations; 39 workers; 468 measured responses | E3 / valid / public / not_attempted |
| Historical BF16/Horizon comparison | 120 paired prompts | E3 / valid / public / not_attempted |
| IFEval | 541 prompts; 834 instructions | E1 standalone_benchmark / valid / public / not_attempted |

These are separate populations with separate scorers and questions. Public admission, evidence maturity, verification and independent reproduction are distinct.

## Performance contract and results

PERF-REAL-v1; 12 prompts repeated in 3 blocks; 36 measured responses of 128 tokens per configuration; greedy selection, EOS suppressed; 64-token warmup excluded. Canonical initial placement, evolving LRU through fixed prompts; concurrency 1.

Pooled decode = sum(N - 1) / sum(internal decode seconds); it excludes first-token wait. Overall output rate uses sum(N) / sum(internal response runtime). TTFT and response time are arithmetic request means. VRAM is a device-wide whole-worker driver peak including preparation, warmup and shared desktop use; RSS is a separate whole-worker process peak. H2D is a sum over measured requests, not a rate. Isolated prefill is UNKNOWN; OLMoE HTTP observation is UNSUPPORTED. Block spread is a descriptive range, not a confidence interval.

| Configuration / source | Executor | Decode token/s | Whole-worker VRAM GiB |
| --- | --- | --- | --- |
| [OLMoE 0924 c30/64](/evidence/releases/technical-preview-residency39-v3/olmoe-0924-instruct-c30-residency39.json) | PRODUCT-core scalar | 23.00 | 4.11 |
| [OLMoE 0924 c40/64](/evidence/releases/technical-preview-residency39-v3/olmoe-0924-instruct-c40-residency39.json) | PRODUCT-core scalar | 27.69 | 4.62 |
| [OLMoE 0924 c50/64](/evidence/releases/technical-preview-residency39-v3/olmoe-0924-instruct-c50-residency39.json) | PRODUCT-core scalar | 31.50 | 5.15 |
| [OLMoE 0924 c60/64](/evidence/releases/technical-preview-residency39-v3/olmoe-0924-instruct-c60-residency39.json) | PRODUCT-core scalar | 35.01 | 5.67 |
| [OLMoE 0924 c64/64](/evidence/releases/technical-preview-residency39-v3/olmoe-0924-instruct-c64-residency39.json) | Grouped T3 | 61.26 | 5.99 |
| [OLMoE 0125 c30/64](/evidence/releases/technical-preview-residency39-v3/olmoe-0125-instruct-c30-residency39.json) | PRODUCT-core scalar | 23.04 | 4.15 |
| [OLMoE 0125 c40/64](/evidence/releases/technical-preview-residency39-v3/olmoe-0125-instruct-c40-residency39.json) | PRODUCT-core scalar | 27.12 | 4.62 |
| [OLMoE 0125 c50/64](/evidence/releases/technical-preview-residency39-v3/olmoe-0125-instruct-c50-residency39.json) | PRODUCT-core scalar | 31.22 | 5.15 |
| [OLMoE 0125 c60/64](/evidence/releases/technical-preview-residency39-v3/olmoe-0125-instruct-c60-residency39.json) | PRODUCT-core scalar | 35.44 | 5.67 |
| [OLMoE 0125 c64/64](/evidence/releases/technical-preview-residency39-v3/olmoe-0125-instruct-c64-residency39.json) | Grouped T3 | 62.07 | 6.01 |
| [Qwen c20/60](/evidence/releases/technical-preview-residency39-v3/qwen15-moe-chat-c20-residency39.json) | top4-fused-v1 | 15.22 | 7.11 |
| [Qwen c30/60](/evidence/releases/technical-preview-residency39-v3/qwen15-moe-chat-c30-residency39.json) | top4-fused-v1 | 17.17 | 8.08 |
| [Qwen c39/60](/evidence/releases/technical-preview-residency39-v3/qwen15-moe-chat-c39-residency39.json) | top4-fused-v1 | 19.95 | 8.99 |

Capacities are per routed layer. OLMoE c30-c60 use PRODUCT-core scalar; c64 uses Grouped T3, so c30-to-c64 or c60-to-c64 is not a residency-only ablation. Qwen c20/c30/c39 use top4-fused-v1. Keep model revisions and executors attached to every value; no cross-runtime ranking is measured.

[Performance manifest](/evidence/releases/technical-preview-residency39-v3/manifest.json) · [Contract/report](/evidence/releases/technical-preview-residency39-v3/residency-report.md) · [Verification method](/evidence/releases/technical-preview-residency39-v3/audit-method-v1.md) · [Verifier](/evidence/releases/technical-preview-residency39-v3/verify-evidence-edition.mjs)

## Historical behavioral comparison

Under 120 paired structured prompts, BF16 produced 29/120 structurally valid responses and Horizon produced 28/120. There were 8 paired structural regressions, 7 improvements, 19 observable-behavior parity cases and 86 directionless divergences. Structural conformity is not factual-answer correctness. The original campaign's publication sanitizer failure is retained in provenance; the CPU-only successor recovers the original responses without changing the frozen scorer.

[Paired manifest](/evidence/behavioral/olmoe-0924-fidelity-v2/manifest.json) · [Contract](/evidence/behavioral/olmoe-0924-fidelity-v2/contract.json) · [Summary](/evidence/behavioral/olmoe-0924-fidelity-v2/summary.json) · [Audit method](/evidence/behavioral/olmoe-0924-fidelity-v2/audit-method.md) · [ZIP](/evidence/behavioral/olmoe-0924-fidelity-v2.zip)

## IFEval 541

Model: allenai/OLMoE-1B-7B-0924-Instruct, revision 7f1c97f440f06ce36705e4f2b843edb5925f4498. The complete declared population was scored using frozen evaluator inputs. All 467 EOS and 74 length-limit endings remain included.

| Metric | Numerator / denominator | Percent |
| --- | --- | --- |
| Prompt strict | 217 / 541 | 40.11% |
| Prompt loose | 247 / 541 | 45.66% |
| Instruction strict | 433 / 834 | 51.92% |
| Instruction loose | 474 / 834 | 56.83% |

The original frozen contract keeps its historical internal_only field; separate publication metadata records later public admission as E1 standalone_benchmark. These instruction-following scores do not establish matched BF16 preservation. External model-card numbers are contextual, non-controlled references; one reference does not specify the IFEval variant.

[IFEval manifest](/evidence/ifeval/olmoe-0924-ifeval541-v1/manifest.json) · [Frozen contract](/evidence/ifeval/olmoe-0924-ifeval541-v1/observations/contract.json) · [Public admission](/evidence/ifeval/olmoe-0924-ifeval541-v1/publication.json) · [All responses](/evidence/ifeval/olmoe-0924-ifeval541-v1/explore-en.html) · [Audit method](/evidence/ifeval/olmoe-0924-ifeval541-v1/audit-method.md) · [ZIP](/evidence/ifeval/olmoe-0924-ifeval541-v1.zip)

## Verification and reproduction

The performance verifier checks hashes and aggregate-copy consistency; that release does not include the complete raw-run bundle. Behavioral and IFEval verifiers recompute their declared scores from retained responses on CPU. None reruns the private inference runtime. Independent reproduction is not_attempted for all three editions. UNKNOWN denotes an absent observation; UNSUPPORTED denotes unavailable observation capability.