# Horizon RunMap: OLMoE 0924 IFEval-541

The full 541-prompt IFEval population was completed by Horizon c64 full residency,
Grouped T3, with INT4 routed expert sources and FP16 activations. Scores:

| Metric | Passed / evaluated | Accuracy |
| --- | --- | --- |
| Prompt strict | 217 / 541 | 40.11% |
| Prompt loose | 247 / 541 | 45.66% |
| Instruction strict | 433 / 834 | 51.92% |
| Instruction loose | 474 / 834 | 56.83% |

All 541 responses count, including 74 that reached the 1,280-token limit.
467 ended at EOS. Greedy decoding; no input truncation; no automatic retries.
11 closed sessions; 287,363 generated tokens. Recorded per-request fallback,
cache misses/fills and expert H2D operations/bytes are all zero.

## What this result contributes

This records instruction following across a complete external benchmark for
Horizon’s compact-expert execution. The original OLMoE 0924 model card reports
48.1 as Loose Acc; the later 0125 model card lists 45.29 for the 0924 checkpoint
as IFEval without naming the variant. These give published context, with
different or incompletely specified evaluation contracts. They are not paired
baseline runs. This experiment measures instruction following; it does not
grade factual correctness or establish BF16 quality equivalence.

## Recalculate the scores

Download and extract the full ZIP. Inspect the Python files before running them.
Use Python 3.13 and install the pinned CPU dependencies once:

```sh
python -m venv ../ifeval-verify-env
# Activate this environment using your platform's normal command.
python -m pip install -r requirements.txt
python -B verify-ifeval.py
```

Create the virtual environment outside the extracted edition directory so its
files do not change the edition inventory. Dependency installation needs access
to a package registry (or an existing local wheel cache). Verification itself
uses local files; no model, GPU, runtime source, or network download is needed.
The dataset, minimal pinned upstream evaluator, licenses and NLTK data resource
are included. Python random and langdetect seeds are both 0, as in the verified
scoring pass. Raw generations were not changed or rerun during packaging.

The verifier checks the public manifest, source identity, complete key order,
runtime counters and cleanup, then recomputes every Boolean score and all four
metrics from the original responses. This is independent-process recomputation
of retained outputs, not an independent reproduction of model generation.

`observations/` preserves the original seven-file deterministic bundle exactly.
Its manifest SHA-256 is
`26a7973d6d6047e2fa129284d736c78006b2755a0ee68da6bd6f6312e5332e9a`.
`publication.json` records the owner's later public admission: E1,
standalone_benchmark, validation valid, independent reproduction not attempted.
The original contract's historical internal_only field remains unchanged.

The public manifest binds the files to this edition. Its hash and the archive
hash are recorded in `/evidence/ifeval-index.json`. Hashes establish consistency
with the published inventory, not external authentication of the publisher.

## References

- https://huggingface.co/allenai/OLMoE-1B-7B-0924-Instruct#evaluation-snapshot
- https://huggingface.co/allenai/OLMoE-1B-7B-0125-Instruct#performance
- Dataset and evaluator revisions, URLs and licenses: `sources/source-manifest.json`.

The performance campaign and the historical 120-prompt BF16 comparison are
separate experiments. Their original public URLs and evidence remain available.