The workbench
The GPU does the calculations. VRAM is the memory workspace it uses.
Technical Preview
Independent research by Lucas RibeiroAn architecture for executing MoE models.
Store compactly. Use what is needed. Keep the source ready.
An AI model can contain many experts while using only some at a time. Horizon RunMap organizes how they are stored, computed and brought to the GPU, the part of the computer that performs these calculations. Explore the architecture, my engineering decisions and the recorded results.
01 / The idea
First, the idea. Then, the names.
The warehouse
RAM keeps the complete, immutable source of the expert weights.
The tools
Experts are parts of the model. Each step uses a selection of them.
Imagine a workshop: the workbench has limited room and tools stay in storage. Here, each tool represents a set of model calculations.
Some AI models contain many experts. Each step uses a selection of them, but the complete model still needs somewhere to live.
Why does memory matter? Explore the large-model context
In MoE models, a selection of experts works on each token while the complete parameter set needs storage. Compare BF16 scale with each project’s published representation.
The scale of memory
The same model. Two memory scenarios.
GLM-5.3744 billion total parameters · 40 billion active per token
BF16 comparison
16 bits · 2 bytes per parameter
- Estimated weight memory
- ≈ 1.49 TB
- Mathematical minimum · weights only
- 9 B200
- Capacity for execution planning
- 16 B200
- 2 servers with 8 GPUs
Calculated scenario reserving 20%–30% of total memory and using complete servers. Context and workload guide final sizing.
Published format
FP8 · 8 bits
- Idealized weight memory
- ≈ 744 GB
- Mathematical minimum · weights only
- 5 B200
- Execution reference
- 8 B200 · documented configuration
The vLLM recipe describes FP8 weights and FP8 KV cache for up to 1 million context tokens. Concurrency is tuned to the memory budget.
FP8 is the default checkpoint; Z.ai also publishes a separate BF16 version.
How these numbers are calculated
The comparison assumes all weights reside in GPU memory, ideally distributed across devices. Weights ≈ total parameters × bits ÷ 8. Each B200 provides a nominal 180 GB; each server groups 8 GPUs, totaling 1.44 TB. GB and TB use decimal units.
Minimum GPUs = weights ÷ 180 GB, rounded up. Servers with headroom = weights ÷ (1.44 TB × 0.80) through weights ÷ (1.44 TB × 0.70), always rounded up. The reserve is a planning assumption selected for this comparison.
Calculations use rounded published parameter counts rather than actual file sizes. Scales, mixed-precision tensors, buffers, KV cache (context memory) and parallel layout contribute to the execution budget. GPU count and response speed are separate dimensions.
For GLM-5.3, we use Z.ai’s stated 744 billion total and 40 billion active parameters. The vLLM recipe describes approximately 743/39 billion; this difference leaves the rounded GPU counts unchanged in this comparison.
B200 system capacity · NVIDIA ↗Frontier models show the scale of the memory and infrastructure challenge motivating Horizon RunMap. OLMoE and Qwen are the models used to investigate storage, transfer and execution on available hardware. Applying the architecture at larger scales requires separate validation; the rates below belong to the identified campaign.
Cost references per platform
Offers recorded on 2026-09-10, per platform or instance tier. The purchase and rental figures below have their own scope; a multi-server deployment also depends on networking and contracted services.
NVIDIA B200 (180 GB product) · recorded sources and estimates
NVIDIA B200 (180 GB product)
External context · estimates · Checked Sep 10, 2026Lambda: per GPU-hour, tier-dependent
USD 6.69–6.99 / GPU-hour
Published-offer estimate
Formula, assumptions and limits
1 GPU × USD 6.69 to 6.99/GPU-hour × 1 hour- Lambda published instance-tier range, not a market-wide minimum or a uniform single-GPU offer.
- Eight-GPU tier: USD 6.69 each; four-GPU tier: USD 6.79 each; two-GPU tier: USD 6.89 each; single-GPU tier: USD 6.99.
- Region not specified on the pricing table; applicable sales tax/VAT/GST additional. Availability was not booked.
- The low end requires an eight-GPU instance. Host resources are tier-specific; external storage, networking and operations are not priced here.
Lambda (opens in new tab)Primary source · Checked Sep 10, 2026
Runpod: per GPU-hour, on demand
USD 6.79 / GPU-hour
Published-offer estimate
Formula, assumptions and limits
1 Runpod Secure Cloud B200 × USD 6.79/GPU-hour × 1 hour- Runpod published Secure Cloud on-demand offer; not a commitment or serverless rate.
- Listed with 283 GB RAM and 28 vCPU. Region not specified in the cited guide.
- Storage, networking, tax and service availability require separate verification.
Runpod (opens in new tab)Primary source · Checked Sep 10, 2026
Lambda: hypothetical month, eight-GPU instance
USD 38,534.4
Hypothetical
Formula, assumptions and limits
8 GPUs × USD 6.69/GPU-hour × 720 hours = USD 38534.40- One Lambda eight-GPU instance for a hypothetical 30-day month of continuous rental.
- The corresponding hourly instance total is 8 × USD 6.69 = USD 53.52.
- No model-fit or deployment claim. Applicable taxes and separately billed services excluded; region not specified.
Lambda (opens in new tab)Primary source · Checked Sep 10, 2026
ITCT: one eight-GPU platform, pre-order
USD 390,000
Published-offer estimate
Formula, assumptions and limits
1 advertised eight-GPU HGX B200 platform × USD 390000/platform- ITCT Shop's own advertised pre-order offer only; not a bare-GPU price, verified invoice or complete data-center quote.
- Final bill of materials and fulfillment region are not established. No claim that CPU, RAM, storage, chassis extras or networking are included.
- External switches, cabling, electricity, cooling, operating labor, freight and taxes are not established as included.
- The vendor's engineering tables and marketing claims are not used as technical evidence; NVIDIA supplies the capacity specification.
- This is one attributable offer, not a market acquisition range or a guarantee of delivery.
ITCT Shop (opens in new tab)Primary source · Checked Sep 10, 2026
NVIDIA (opens in new tab)Primary source · Checked Sep 10, 2026
NVIDIA (opens in new tab)Primary source · Checked Sep 10, 2026
Lambda (opens in new tab)Primary source · Checked Sep 10, 2026
Runpod (opens in new tab)Primary source · Checked Sep 10, 2026
ITCT Shop (opens in new tab)Primary source · Checked Sep 10, 2026
GPU capacity, platform acquisition, rental, networking, electricity, cooling and operations are separate cost planes. This is not a complete quote, total cost of ownership or purchase recommendation.
02 / The question
Three decisions working together.
Which weights should be compressed, how should they be computed, and where should they live? Horizon addresses these decisions together. Full residency and expert offloading are operating regimes of this architecture.
I built a runtime — the software that coordinates model execution — bringing these three decisions together. Each appears below with its purpose and a path to the details.
01 / Store
Compact experts
INT4
Weights of routed experts stay compact in RAM and on the GPU. Attention, embeddings and other components remain outside this quantization.
Expert weights: INT4 · scales: FP16
Understand the formats →02 / Compute
Execute the chosen format
W4A16
Kernels are GPU computation routines. Horizon’s kernels use compact weights and convert the values needed during each operation, without retaining a complete expanded expert.
Compact weights → custom kernel + FP16 activations → output
Explore the computation path →03 / Organize
Work with copies
RAM → GPU
RAM keeps the complete source. The GPU uses copies of all experts or a subset, depending on the space reserved. The source stays ready after eviction.
Immutable source · disposable GPU copies
See hits, misses and eviction →03 / Architecture
One architecture. Two residency regimes.
Resident means stored in GPU memory. Both cases use compact INT4/qpacked expert weights, and kernels receive FP16 activations — values computed by the previous stage. Each measurement identifies its executor.
Full residency
All routed experts remain on the GPU in compact form. This regime exercises compressed-expert execution without fetching missing copies during generation.
GPU memoryPartial residency
A subset of experts stays on the GPU. A requested missing copy arrives from RAM already quantized. Eviction releases only the GPU copy; the source in RAM stays ready for future use.
- RAMComplete immutable source
- Transfer stagingBounded, on the host
- GPU memoryCached copies
Explore the technical details
In the partial-residency flow shown, the source of expert weights in RAM stays complete and immutable. A copy passes through bounded pinned host staging before transfer to the GPU. Eviction discards the GPU copy, with no transfer back or source writeback.
Residency is not activation: the cache may keep experts that this step does not use. This diagram explains organization; the campaign below measures partial and complete residency while preserving executor differences.
A copy enters the cache only when ready
Documented architecture · resident-qpacked-cache-v1 · ADR 0009. Original scope: experimental, default-off P0 Qwen. The executors measured below are described by their specific contracts.
- Immutable source
The complete qpacked set stays in process RAM, with validated identity and layout.
- Bounded staging
A temporary pinned arena prepares transfer. It becomes reusable only after completion.
- Unpublished spare
The qpacked row reaches temporary GPU storage; readers still see the previous generation.
- Atomic publication
After routed-group completion, one transition publishes the new complete cache generation.
- Use and discard
Readers use a valid generation. An evicted copy is reused only after its consumers retire; the source is not rewritten.
- Resource admission
Capacity is fixed before requests. Insufficient staging or spare capacity rejects startup; failures before publication preserve the previous generation.
Earlier profiles have a different contract: offload-qpacked materializes FP16/BF16 execution slots from a quantized source. In the qpacked cache described here, experts remain compact. The kernel converts the values needed during computation without retaining a complete expanded expert slot. That conversion does not recover information lost during quantization.
Read decisions, sources and limits (English) ↓04 / Recorded observations
The architecture in execution.
The OLMoE and Qwen campaign on an RTX 5070 records runs with full and partial residency. Its configurations show how the system operated within each memory, executor and workload contract.
Architecture in operation · Exploratory E3
Compact experts, executed with full or partial residency.
The records below identify the model, space for copies and executor. Rates describe text generation in each configuration.Full residency
OLMoE 0924
Grouped T3 executes the compact experts, with all of them on the GPU during generation.
64/64 experts per layer · Grouped T3
Execution record
Inspect this result 61.26 tokens/s
Decode after the first token
Full residency
OLMoE 0125
Grouped T3 executes the compact experts, with all of them on the GPU during generation.
64/64 experts per layer · Grouped T3
Execution record
Inspect this result 62.07 tokens/s
Decode after the first token
Partial residency
Qwen
top4-fused-v1 executes compact experts with some copies on the GPU and on-demand transfers.
39/60 experts per layer · top4-fused-v1
Execution record
Inspect this result 19.95 tokens/s
Decode after the first token
Highest measured residency for each model · RTX 5070 · 36 responses of 128 tokens per configuration. OLMoE and Qwen use different executors; each card describes its own run.
Exploratory E3
468
measured responses · campaign volume
39/39valid worker runs
13configurations
1h 50min 21scampaign duration
36 responses of 128 tokens per configuration · 12 prompts × 3 blocks · 39 warmups excluded from rates.
How the numbers were calculated
Decode divides the sum of tokens after the first by the sum of internal generation times. It is a pooled, time-weighted rate. First-token wait and response duration are request means, with internal boundaries specific to each executor.
- Decode
- 37min 06s
- Until first token
- 35min 10s
- Other stages (by subtraction)
- 38min 05s
Observed relative block spread: 0.27%–4.78%. This is observed variation, not a confidence interval. Recorded maximum temperature: 58 °C.
The consolidation recorded matching token sequences between repetitions of each configuration. This observation describes within-configuration consistency; quality evaluation and output comparisons across configurations have their own scope. Other-stage time is calculated as the remainder of controller duration.
Tokens are small pieces of text, not always whole words.
05 / Memory & speed
More room on the workbench. What changes?
Choose a model and a measurement to explore the recorded configurations. Each group identifies its executor, the computation path used in that run.
E3 exploratorySep 11, 2026
36 responses per configuration · 128 tokens per response · 3 blocks
Hardware · NVIDIA GeForce RTX 5070, as recorded in the campaign; shared Windows desktop.
Post-first-token decode · tokens/s
Pooled rate: Σ(tokens − 1) / Σ(internal decode time). Excludes first-token wait; not a mean of individual rates.
Measurements after warmup, with fixed prompts. Research scope
Resident experts per layer · discrete measurements, without interpolation or cross-model comparison.
View all 13 configurations
The table includes every configuration. On narrow screens, scroll the table sideways.
| Model / executor | Resident experts per layer | Pooled post-first-token decode (tokens/s) | Internal TTFT — mean (s) | Internal 128-token response — mean (s) | Peak VRAM¹ (GiB) | Source |
|---|---|---|---|---|---|---|
| OLMoE 0924PRODUCT-core scalar | 30/64 | 23.00 | 6.17 | 11.69 | 4.11 | JSON |
| OLMoE 0924PRODUCT-core scalar | 40/64 | 27.69 | 4.49 | 9.07 | 4.62 | JSON |
| OLMoE 0924PRODUCT-core scalar | 50/64 | 31.50 | 3.62 | 7.65 | 5.15 | JSON |
| OLMoE 0924PRODUCT-core scalar | 60/64 | 35.01 | 3.00 | 6.63 | 5.67 | JSON |
| OLMoE 0924Grouped T3 | 64/64 | 61.26 | 0.75 | 2.82 | 5.99 | JSON |
| OLMoE 0125PRODUCT-core scalar | 30/64 | 23.04 | 6.07 | 11.58 | 4.15 | JSON |
| OLMoE 0125PRODUCT-core scalar | 40/64 | 27.12 | 4.53 | 9.21 | 4.62 | JSON |
| OLMoE 0125PRODUCT-core scalar | 50/64 | 31.22 | 3.59 | 7.66 | 5.15 | JSON |
| OLMoE 0125PRODUCT-core scalar | 60/64 | 35.44 | 2.97 | 6.55 | 5.67 | JSON |
| OLMoE 0125Grouped T3 | 64/64 | 62.07 | 0.70 | 2.74 | 6.01 | JSON |
| Qwentop4-fused-v1 | 20/60 | 15.22 | 9.42 | 17.76 | 7.11 | JSON |
| Qwentop4-fused-v1 | 30/60 | 17.17 | 7.51 | 14.90 | 8.08 | JSON |
| Qwentop4-fused-v1 | 39/60 | 19.95 | 5.80 | 12.17 | 8.99 | JSON |
¹ Driver-observed VRAM over the whole worker, including preparation, warmup and shared desktop use. Not process-exclusive or decode-only VRAM.
OLMoE 0924 · 30/64 · PRODUCT-core scalar — contract and additional measurements
- Metric
- Pooled rate: sum of tokens after the first / sum of internal decode times. Excludes first-token wait, preparation and warm-up; not a mean of individual rates.
- Model / revision
allenai/OLMoE-1B-7B-0924-Instruct- Executor
- PRODUCT-core scalar
- Storage / execution
- INT4 / block 256 / FP16 scales. FP16 execution values from low-bit source; original checkpoint dtype is a separate plane.
- Peak process RAM (whole worker)
- 4.95 GiB
- RAM → GPU transfer (sum of measured responses)
- 1,108.10 GiB
- Three block rates
- 23.12 / 23.14 / 22.74 tokens/s
- Observed relative spread (not a confidence interval)
- 1.74%
- Output rate including first-token wait
- 10.95 tokens/s
- Hardware
- NVIDIA GeForce RTX 5070, as recorded in the campaign; shared Windows desktop.
- Run contract
- PERF-REAL-v1; 12 prompts repeated in 3 blocks; 36 measured responses of 128 tokens per configuration; greedy selection, EOS suppressed; 64-token warmup excluded. Canonical initial placement, evolving LRU through fixed prompts; concurrency 1.
- Run date
- Sep 11, 2026
- Model revision
7f1c97f440f06ce36705e4f2b843edb5925f4498- Original artifact (repository-relative path)
artifacts/residency-executors-39-20260911-01/consolidation-01/summary.json
Context for this observation
- Exploratory E3 on a shared desktop. Quality evaluation and E4 validation remain open.
- Driver VRAM and process RSS are whole-worker peaks including preparation and warmup; VRAM includes desktop use.
- OLMoE c60 to c64 combines changes in executor and residency; interpret the result in that joint context.
- Block spread is an observed range, not a confidence interval. Token parity concerns repetitions within each configuration only.
- TTFT and response time use internal request boundaries; isolated prefill is UNKNOWN and OLMoE HTTP observation is UNSUPPORTED.
Campaign extract with original identifiers and source text in English. The full bundle and raw observations have separate availability.
OLMoE 0924 · 40/64 · PRODUCT-core scalar — contract and additional measurements
- Metric
- Pooled rate: sum of tokens after the first / sum of internal decode times. Excludes first-token wait, preparation and warm-up; not a mean of individual rates.
- Model / revision
allenai/OLMoE-1B-7B-0924-Instruct- Executor
- PRODUCT-core scalar
- Storage / execution
- INT4 / block 256 / FP16 scales. FP16 execution values from low-bit source; original checkpoint dtype is a separate plane.
- Peak process RAM (whole worker)
- 4.95 GiB
- RAM → GPU transfer (sum of measured responses)
- 576.17 GiB
- Three block rates
- 27.76 / 27.63 / 27.69 tokens/s
- Observed relative spread (not a confidence interval)
- 0.45%
- Output rate including first-token wait
- 14.11 tokens/s
- Hardware
- NVIDIA GeForce RTX 5070, as recorded in the campaign; shared Windows desktop.
- Run contract
- PERF-REAL-v1; 12 prompts repeated in 3 blocks; 36 measured responses of 128 tokens per configuration; greedy selection, EOS suppressed; 64-token warmup excluded. Canonical initial placement, evolving LRU through fixed prompts; concurrency 1.
- Run date
- Sep 11, 2026
- Model revision
7f1c97f440f06ce36705e4f2b843edb5925f4498- Original artifact (repository-relative path)
artifacts/residency-executors-39-20260911-01/consolidation-01/summary.json
Context for this observation
- Exploratory E3 on a shared desktop. Quality evaluation and E4 validation remain open.
- Driver VRAM and process RSS are whole-worker peaks including preparation and warmup; VRAM includes desktop use.
- OLMoE c60 to c64 combines changes in executor and residency; interpret the result in that joint context.
- Block spread is an observed range, not a confidence interval. Token parity concerns repetitions within each configuration only.
- TTFT and response time use internal request boundaries; isolated prefill is UNKNOWN and OLMoE HTTP observation is UNSUPPORTED.
Campaign extract with original identifiers and source text in English. The full bundle and raw observations have separate availability.
OLMoE 0924 · 50/64 · PRODUCT-core scalar — contract and additional measurements
- Metric
- Pooled rate: sum of tokens after the first / sum of internal decode times. Excludes first-token wait, preparation and warm-up; not a mean of individual rates.
- Model / revision
allenai/OLMoE-1B-7B-0924-Instruct- Executor
- PRODUCT-core scalar
- Storage / execution
- INT4 / block 256 / FP16 scales. FP16 execution values from low-bit source; original checkpoint dtype is a separate plane.
- Peak process RAM (whole worker)
- 4.95 GiB
- RAM → GPU transfer (sum of measured responses)
- 230.88 GiB
- Three block rates
- 31.75 / 31.57 / 31.19 tokens/s
- Observed relative spread (not a confidence interval)
- 1.75%
- Output rate including first-token wait
- 16.73 tokens/s
- Hardware
- NVIDIA GeForce RTX 5070, as recorded in the campaign; shared Windows desktop.
- Run contract
- PERF-REAL-v1; 12 prompts repeated in 3 blocks; 36 measured responses of 128 tokens per configuration; greedy selection, EOS suppressed; 64-token warmup excluded. Canonical initial placement, evolving LRU through fixed prompts; concurrency 1.
- Run date
- Sep 11, 2026
- Model revision
7f1c97f440f06ce36705e4f2b843edb5925f4498- Original artifact (repository-relative path)
artifacts/residency-executors-39-20260911-01/consolidation-01/summary.json
Context for this observation
- Exploratory E3 on a shared desktop. Quality evaluation and E4 validation remain open.
- Driver VRAM and process RSS are whole-worker peaks including preparation and warmup; VRAM includes desktop use.
- OLMoE c60 to c64 combines changes in executor and residency; interpret the result in that joint context.
- Block spread is an observed range, not a confidence interval. Token parity concerns repetitions within each configuration only.
- TTFT and response time use internal request boundaries; isolated prefill is UNKNOWN and OLMoE HTTP observation is UNSUPPORTED.
Campaign extract with original identifiers and source text in English. The full bundle and raw observations have separate availability.
OLMoE 0924 · 60/64 · PRODUCT-core scalar — contract and additional measurements
- Metric
- Pooled rate: sum of tokens after the first / sum of internal decode times. Excludes first-token wait, preparation and warm-up; not a mean of individual rates.
- Model / revision
allenai/OLMoE-1B-7B-0924-Instruct- Executor
- PRODUCT-core scalar
- Storage / execution
- INT4 / block 256 / FP16 scales. FP16 execution values from low-bit source; original checkpoint dtype is a separate plane.
- Peak process RAM (whole worker)
- 4.96 GiB
- RAM → GPU transfer (sum of measured responses)
- 37.62 GiB
- Three block rates
- 35.47 / 34.72 / 34.85 tokens/s
- Observed relative spread (not a confidence interval)
- 2.14%
- Output rate including first-token wait
- 19.31 tokens/s
- Hardware
- NVIDIA GeForce RTX 5070, as recorded in the campaign; shared Windows desktop.
- Run contract
- PERF-REAL-v1; 12 prompts repeated in 3 blocks; 36 measured responses of 128 tokens per configuration; greedy selection, EOS suppressed; 64-token warmup excluded. Canonical initial placement, evolving LRU through fixed prompts; concurrency 1.
- Run date
- Sep 11, 2026
- Model revision
7f1c97f440f06ce36705e4f2b843edb5925f4498- Original artifact (repository-relative path)
artifacts/residency-executors-39-20260911-01/consolidation-01/summary.json
Context for this observation
- Exploratory E3 on a shared desktop. Quality evaluation and E4 validation remain open.
- Driver VRAM and process RSS are whole-worker peaks including preparation and warmup; VRAM includes desktop use.
- OLMoE c60 to c64 combines changes in executor and residency; interpret the result in that joint context.
- Block spread is an observed range, not a confidence interval. Token parity concerns repetitions within each configuration only.
- TTFT and response time use internal request boundaries; isolated prefill is UNKNOWN and OLMoE HTTP observation is UNSUPPORTED.
Campaign extract with original identifiers and source text in English. The full bundle and raw observations have separate availability.
OLMoE 0924 · 64/64 · Grouped T3 — contract and additional measurements
- Metric
- Pooled rate: sum of tokens after the first / sum of internal decode times. Excludes first-token wait, preparation and warm-up; not a mean of individual rates.
- Model / revision
allenai/OLMoE-1B-7B-0924-Instruct- Executor
- Grouped T3
- Storage / execution
- INT4 / block 256 / FP16 scales. FP16 execution values from low-bit source; original checkpoint dtype is a separate plane.
- Peak process RAM (whole worker)
- 4.98 GiB
- RAM → GPU transfer (sum of measured responses)
- 0.00 GiB
- Three block rates
- 59.72 / 62.65 / 61.47 tokens/s
- Observed relative spread (not a confidence interval)
- 4.78%
- Output rate including first-token wait
- 45.41 tokens/s
- Hardware
- NVIDIA GeForce RTX 5070, as recorded in the campaign; shared Windows desktop.
- Run contract
- PERF-REAL-v1; 12 prompts repeated in 3 blocks; 36 measured responses of 128 tokens per configuration; greedy selection, EOS suppressed; 64-token warmup excluded. Canonical initial placement, evolving LRU through fixed prompts; concurrency 1.
- Run date
- Sep 11, 2026
- Model revision
7f1c97f440f06ce36705e4f2b843edb5925f4498- Original artifact (repository-relative path)
artifacts/residency-executors-39-20260911-01/consolidation-01/summary.json
Context for this observation
- Exploratory E3 on a shared desktop. Quality evaluation and E4 validation remain open.
- Driver VRAM and process RSS are whole-worker peaks including preparation and warmup; VRAM includes desktop use.
- OLMoE c60 to c64 combines changes in executor and residency; interpret the result in that joint context.
- Block spread is an observed range, not a confidence interval. Token parity concerns repetitions within each configuration only.
- TTFT and response time use internal request boundaries; isolated prefill is UNKNOWN and OLMoE HTTP observation is UNSUPPORTED.
Campaign extract with original identifiers and source text in English. The full bundle and raw observations have separate availability.
OLMoE 0125 · 30/64 · PRODUCT-core scalar — contract and additional measurements
- Metric
- Pooled rate: sum of tokens after the first / sum of internal decode times. Excludes first-token wait, preparation and warm-up; not a mean of individual rates.
- Model / revision
allenai/OLMoE-1B-7B-0125-Instruct- Executor
- PRODUCT-core scalar
- Storage / execution
- INT4 / block 256 / FP16 scales. FP16 execution values from low-bit source; original checkpoint dtype is a separate plane.
- Peak process RAM (whole worker)
- 4.97 GiB
- RAM → GPU transfer (sum of measured responses)
- 1,115.87 GiB
- Three block rates
- 22.65 / 22.98 / 23.51 tokens/s
- Observed relative spread (not a confidence interval)
- 3.74%
- Output rate including first-token wait
- 11.05 tokens/s
- Hardware
- NVIDIA GeForce RTX 5070, as recorded in the campaign; shared Windows desktop.
- Run contract
- PERF-REAL-v1; 12 prompts repeated in 3 blocks; 36 measured responses of 128 tokens per configuration; greedy selection, EOS suppressed; 64-token warmup excluded. Canonical initial placement, evolving LRU through fixed prompts; concurrency 1.
- Run date
- Sep 11, 2026
- Model revision
b89a7c4bc24fb9e55ce2543c9458ce0ca5c4650e- Original artifact (repository-relative path)
artifacts/residency-executors-39-20260911-01/consolidation-01/summary.json
Context for this observation
- Exploratory E3 on a shared desktop. Quality evaluation and E4 validation remain open.
- Driver VRAM and process RSS are whole-worker peaks including preparation and warmup; VRAM includes desktop use.
- OLMoE c60 to c64 combines changes in executor and residency; interpret the result in that joint context.
- Block spread is an observed range, not a confidence interval. Token parity concerns repetitions within each configuration only.
- TTFT and response time use internal request boundaries; isolated prefill is UNKNOWN and OLMoE HTTP observation is UNSUPPORTED.
Campaign extract with original identifiers and source text in English. The full bundle and raw observations have separate availability.
OLMoE 0125 · 40/64 · PRODUCT-core scalar — contract and additional measurements
- Metric
- Pooled rate: sum of tokens after the first / sum of internal decode times. Excludes first-token wait, preparation and warm-up; not a mean of individual rates.
- Model / revision
allenai/OLMoE-1B-7B-0125-Instruct- Executor
- PRODUCT-core scalar
- Storage / execution
- INT4 / block 256 / FP16 scales. FP16 execution values from low-bit source; original checkpoint dtype is a separate plane.
- Peak process RAM (whole worker)
- 4.95 GiB
- RAM → GPU transfer (sum of measured responses)
- 580.72 GiB
- Three block rates
- 27.21 / 26.75 / 27.42 tokens/s
- Observed relative spread (not a confidence interval)
- 2.47%
- Output rate including first-token wait
- 13.90 tokens/s
- Hardware
- NVIDIA GeForce RTX 5070, as recorded in the campaign; shared Windows desktop.
- Run contract
- PERF-REAL-v1; 12 prompts repeated in 3 blocks; 36 measured responses of 128 tokens per configuration; greedy selection, EOS suppressed; 64-token warmup excluded. Canonical initial placement, evolving LRU through fixed prompts; concurrency 1.
- Run date
- Sep 11, 2026
- Model revision
b89a7c4bc24fb9e55ce2543c9458ce0ca5c4650e- Original artifact (repository-relative path)
artifacts/residency-executors-39-20260911-01/consolidation-01/summary.json
Context for this observation
- Exploratory E3 on a shared desktop. Quality evaluation and E4 validation remain open.
- Driver VRAM and process RSS are whole-worker peaks including preparation and warmup; VRAM includes desktop use.
- OLMoE c60 to c64 combines changes in executor and residency; interpret the result in that joint context.
- Block spread is an observed range, not a confidence interval. Token parity concerns repetitions within each configuration only.
- TTFT and response time use internal request boundaries; isolated prefill is UNKNOWN and OLMoE HTTP observation is UNSUPPORTED.
Campaign extract with original identifiers and source text in English. The full bundle and raw observations have separate availability.
OLMoE 0125 · 50/64 · PRODUCT-core scalar — contract and additional measurements
- Metric
- Pooled rate: sum of tokens after the first / sum of internal decode times. Excludes first-token wait, preparation and warm-up; not a mean of individual rates.
- Model / revision
allenai/OLMoE-1B-7B-0125-Instruct- Executor
- PRODUCT-core scalar
- Storage / execution
- INT4 / block 256 / FP16 scales. FP16 execution values from low-bit source; original checkpoint dtype is a separate plane.
- Peak process RAM (whole worker)
- 4.95 GiB
- RAM → GPU transfer (sum of measured responses)
- 235.35 GiB
- Three block rates
- 31.10 / 31.31 / 31.25 tokens/s
- Observed relative spread (not a confidence interval)
- 0.66%
- Output rate including first-token wait
- 16.71 tokens/s
- Hardware
- NVIDIA GeForce RTX 5070, as recorded in the campaign; shared Windows desktop.
- Run contract
- PERF-REAL-v1; 12 prompts repeated in 3 blocks; 36 measured responses of 128 tokens per configuration; greedy selection, EOS suppressed; 64-token warmup excluded. Canonical initial placement, evolving LRU through fixed prompts; concurrency 1.
- Run date
- Sep 11, 2026
- Model revision
b89a7c4bc24fb9e55ce2543c9458ce0ca5c4650e- Original artifact (repository-relative path)
artifacts/residency-executors-39-20260911-01/consolidation-01/summary.json
Context for this observation
- Exploratory E3 on a shared desktop. Quality evaluation and E4 validation remain open.
- Driver VRAM and process RSS are whole-worker peaks including preparation and warmup; VRAM includes desktop use.
- OLMoE c60 to c64 combines changes in executor and residency; interpret the result in that joint context.
- Block spread is an observed range, not a confidence interval. Token parity concerns repetitions within each configuration only.
- TTFT and response time use internal request boundaries; isolated prefill is UNKNOWN and OLMoE HTTP observation is UNSUPPORTED.
Campaign extract with original identifiers and source text in English. The full bundle and raw observations have separate availability.
OLMoE 0125 · 60/64 · PRODUCT-core scalar — contract and additional measurements
- Metric
- Pooled rate: sum of tokens after the first / sum of internal decode times. Excludes first-token wait, preparation and warm-up; not a mean of individual rates.
- Model / revision
allenai/OLMoE-1B-7B-0125-Instruct- Executor
- PRODUCT-core scalar
- Storage / execution
- INT4 / block 256 / FP16 scales. FP16 execution values from low-bit source; original checkpoint dtype is a separate plane.
- Peak process RAM (whole worker)
- 4.96 GiB
- RAM → GPU transfer (sum of measured responses)
- 40.81 GiB
- Three block rates
- 35.45 / 35.49 / 35.39 tokens/s
- Observed relative spread (not a confidence interval)
- 0.27%
- Output rate including first-token wait
- 19.54 tokens/s
- Hardware
- NVIDIA GeForce RTX 5070, as recorded in the campaign; shared Windows desktop.
- Run contract
- PERF-REAL-v1; 12 prompts repeated in 3 blocks; 36 measured responses of 128 tokens per configuration; greedy selection, EOS suppressed; 64-token warmup excluded. Canonical initial placement, evolving LRU through fixed prompts; concurrency 1.
- Run date
- Sep 11, 2026
- Model revision
b89a7c4bc24fb9e55ce2543c9458ce0ca5c4650e- Original artifact (repository-relative path)
artifacts/residency-executors-39-20260911-01/consolidation-01/summary.json
Context for this observation
- Exploratory E3 on a shared desktop. Quality evaluation and E4 validation remain open.
- Driver VRAM and process RSS are whole-worker peaks including preparation and warmup; VRAM includes desktop use.
- OLMoE c60 to c64 combines changes in executor and residency; interpret the result in that joint context.
- Block spread is an observed range, not a confidence interval. Token parity concerns repetitions within each configuration only.
- TTFT and response time use internal request boundaries; isolated prefill is UNKNOWN and OLMoE HTTP observation is UNSUPPORTED.
Campaign extract with original identifiers and source text in English. The full bundle and raw observations have separate availability.
OLMoE 0125 · 64/64 · Grouped T3 — contract and additional measurements
- Metric
- Pooled rate: sum of tokens after the first / sum of internal decode times. Excludes first-token wait, preparation and warm-up; not a mean of individual rates.
- Model / revision
allenai/OLMoE-1B-7B-0125-Instruct- Executor
- Grouped T3
- Storage / execution
- INT4 / block 256 / FP16 scales. FP16 execution values from low-bit source; original checkpoint dtype is a separate plane.
- Peak process RAM (whole worker)
- 4.96 GiB
- RAM → GPU transfer (sum of measured responses)
- 0.00 GiB
- Three block rates
- 62.69 / 61.39 / 62.15 tokens/s
- Observed relative spread (not a confidence interval)
- 2.10%
- Output rate including first-token wait
- 46.67 tokens/s
- Hardware
- NVIDIA GeForce RTX 5070, as recorded in the campaign; shared Windows desktop.
- Run contract
- PERF-REAL-v1; 12 prompts repeated in 3 blocks; 36 measured responses of 128 tokens per configuration; greedy selection, EOS suppressed; 64-token warmup excluded. Canonical initial placement, evolving LRU through fixed prompts; concurrency 1.
- Run date
- Sep 11, 2026
- Model revision
b89a7c4bc24fb9e55ce2543c9458ce0ca5c4650e- Original artifact (repository-relative path)
artifacts/residency-executors-39-20260911-01/consolidation-01/summary.json
Context for this observation
- Exploratory E3 on a shared desktop. Quality evaluation and E4 validation remain open.
- Driver VRAM and process RSS are whole-worker peaks including preparation and warmup; VRAM includes desktop use.
- OLMoE c60 to c64 combines changes in executor and residency; interpret the result in that joint context.
- Block spread is an observed range, not a confidence interval. Token parity concerns repetitions within each configuration only.
- TTFT and response time use internal request boundaries; isolated prefill is UNKNOWN and OLMoE HTTP observation is UNSUPPORTED.
Campaign extract with original identifiers and source text in English. The full bundle and raw observations have separate availability.
Qwen · 20/60 · top4-fused-v1 — contract and additional measurements
- Metric
- Pooled rate: sum of tokens after the first / sum of internal decode times. Excludes first-token wait, preparation and warm-up; not a mean of individual rates.
- Model / revision
Qwen/Qwen1.5-MoE-A2.7B-Chat- Executor
- top4-fused-v1
- Storage / execution
- INT4 / block 256 / FP16 scales. FP16 execution values from low-bit source; original checkpoint dtype is a separate plane.
- Peak process RAM (whole worker)
- 7.97 GiB
- RAM → GPU transfer (sum of measured responses)
- 2,702.75 GiB
- Three block rates
- 15.08 / 15.03 / 15.56 tokens/s
- Observed relative spread (not a confidence interval)
- 3.49%
- Output rate including first-token wait
- 7.21 tokens/s
- Hardware
- NVIDIA GeForce RTX 5070, as recorded in the campaign; shared Windows desktop.
- Run contract
- PERF-REAL-v1; 12 prompts repeated in 3 blocks; 36 measured responses of 128 tokens per configuration; greedy selection, EOS suppressed; 64-token warmup excluded. Canonical initial placement, evolving LRU through fixed prompts; concurrency 1.
- Run date
- Sep 11, 2026
- Model revision
ec052fda178e241c7c443468d2fa1db6618996be- Original artifact (repository-relative path)
artifacts/residency-executors-39-20260911-01/consolidation-01/summary.json
Context for this observation
- Exploratory E3 on a shared desktop. Quality evaluation and E4 validation remain open.
- Driver VRAM and process RSS are whole-worker peaks including preparation and warmup; VRAM includes desktop use.
- OLMoE c60 to c64 combines changes in executor and residency; interpret the result in that joint context.
- Block spread is an observed range, not a confidence interval. Token parity concerns repetitions within each configuration only.
- TTFT and response time use internal request boundaries; isolated prefill is UNKNOWN and OLMoE HTTP observation is UNSUPPORTED.
Campaign extract with original identifiers and source text in English. The full bundle and raw observations have separate availability.
Qwen · 30/60 · top4-fused-v1 — contract and additional measurements
- Metric
- Pooled rate: sum of tokens after the first / sum of internal decode times. Excludes first-token wait, preparation and warm-up; not a mean of individual rates.
- Model / revision
Qwen/Qwen1.5-MoE-A2.7B-Chat- Executor
- top4-fused-v1
- Storage / execution
- INT4 / block 256 / FP16 scales. FP16 execution values from low-bit source; original checkpoint dtype is a separate plane.
- Peak process RAM (whole worker)
- 7.99 GiB
- RAM → GPU transfer (sum of measured responses)
- 1,852.07 GiB
- Three block rates
- 17.45 / 17.02 / 17.04 tokens/s
- Observed relative spread (not a confidence interval)
- 2.51%
- Output rate including first-token wait
- 8.59 tokens/s
- Hardware
- NVIDIA GeForce RTX 5070, as recorded in the campaign; shared Windows desktop.
- Run contract
- PERF-REAL-v1; 12 prompts repeated in 3 blocks; 36 measured responses of 128 tokens per configuration; greedy selection, EOS suppressed; 64-token warmup excluded. Canonical initial placement, evolving LRU through fixed prompts; concurrency 1.
- Run date
- Sep 11, 2026
- Model revision
ec052fda178e241c7c443468d2fa1db6618996be- Original artifact (repository-relative path)
artifacts/residency-executors-39-20260911-01/consolidation-01/summary.json
Context for this observation
- Exploratory E3 on a shared desktop. Quality evaluation and E4 validation remain open.
- Driver VRAM and process RSS are whole-worker peaks including preparation and warmup; VRAM includes desktop use.
- OLMoE c60 to c64 combines changes in executor and residency; interpret the result in that joint context.
- Block spread is an observed range, not a confidence interval. Token parity concerns repetitions within each configuration only.
- TTFT and response time use internal request boundaries; isolated prefill is UNKNOWN and OLMoE HTTP observation is UNSUPPORTED.
Campaign extract with original identifiers and source text in English. The full bundle and raw observations have separate availability.
Qwen · 39/60 · top4-fused-v1 — contract and additional measurements
- Metric
- Pooled rate: sum of tokens after the first / sum of internal decode times. Excludes first-token wait, preparation and warm-up; not a mean of individual rates.
- Model / revision
Qwen/Qwen1.5-MoE-A2.7B-Chat- Executor
- top4-fused-v1
- Storage / execution
- INT4 / block 256 / FP16 scales. FP16 execution values from low-bit source; original checkpoint dtype is a separate plane.
- Peak process RAM (whole worker)
- 7.97 GiB
- RAM → GPU transfer (sum of measured responses)
- 1,172.50 GiB
- Three block rates
- 20.31 / 19.78 / 19.76 tokens/s
- Observed relative spread (not a confidence interval)
- 2.74%
- Output rate including first-token wait
- 10.52 tokens/s
- Hardware
- NVIDIA GeForce RTX 5070, as recorded in the campaign; shared Windows desktop.
- Run contract
- PERF-REAL-v1; 12 prompts repeated in 3 blocks; 36 measured responses of 128 tokens per configuration; greedy selection, EOS suppressed; 64-token warmup excluded. Canonical initial placement, evolving LRU through fixed prompts; concurrency 1.
- Run date
- Sep 11, 2026
- Model revision
ec052fda178e241c7c443468d2fa1db6618996be- Original artifact (repository-relative path)
artifacts/residency-executors-39-20260911-01/consolidation-01/summary.json
Context for this observation
- Exploratory E3 on a shared desktop. Quality evaluation and E4 validation remain open.
- Driver VRAM and process RSS are whole-worker peaks including preparation and warmup; VRAM includes desktop use.
- OLMoE c60 to c64 combines changes in executor and residency; interpret the result in that joint context.
- Block spread is an observed range, not a confidence interval. Token parity concerns repetitions within each configuration only.
- TTFT and response time use internal request boundaries; isolated prefill is UNKNOWN and OLMoE HTTP observation is UNSUPPORTED.
Campaign extract with original identifiers and source text in English. The full bundle and raw observations have separate availability.
06 / Follow a token
A small piece of text. A sequence of decisions.
Tokens are small pieces of text. The model processes them through layers; routing selects the experts involved at each step. The runtime coordinates the resources that make this execution possible.
From input to response: follow how the work is organized. The next diagram shows what happens to expert copies. See hits, misses and eviction
Read the input
Text is split into tokens and processed by the model.
Select experts
Routing chooses experts at each routed layer.
Check the cache
A GPU copy can be used when present. On a miss, it arrives from RAM through staging.
Produce the next token
Layer calculations contribute to the next output. The process repeats.
07 / Back to the workshop
The warehouse stays intact.
The warehouse holds the complete set of experts. The workbench uses copies. Here are three situations handled by the same mechanism: the cache, which keeps copies ready in GPU memory.
One complete source. Copies to work with.
In every situation, the compact experts remain stored in RAM. This is the immutable source.
Hit
Use the ready copy
RAMSource intactAlready on GPU
GPUCopy availableThe requested expert already has a ready copy on the workbench. Computation uses that copy.
The source stays intact; this lookup needs no new transfer of the expert.
Miss
Bring a copy
RAMSource intactCopy to GPU
GPUCopy receivedThe requested copy comes from RAM through transfer staging. It becomes usable when transfer finishes and the new complete cache state is published.
The source stays ready in RAM, including after the transfer.
Eviction
Free space
RAMSource intactRelease on GPU
GPUCopy releasedWhen it can safely be released, the copy leaves the workbench. Its location is reused only after the computations using that copy finish.
The source stays ready for another copy; weights do not need to return to the CPU.
08 / Precision & quality
Precision is an engineering decision, too.
In the qpacked architecture presented here, routed expert weights remain compact INT4 in both RAM and GPU memory. Kernels use FP16 activations: W4A16 describes these two formats. Other model components stay outside this INT4 quantization.
The stored format, the format used during computation and the precision of accumulation are separate decisions. Compact weights are converted inside the kernel as computation needs them; the complete expert does not occupy a persistent expanded slot.
Store
W4: routed expert weights in INT4, with FP16 scales. The GPU cache retains that compact format.
Compute
A16: FP16 activations. Kernels convert weights during computation, with FP32 accumulation stages and explicit rounding. W4A16 alone does not describe all arithmetic.
Other components
Attention, embeddings, routing and other components stay outside INT4. In this campaign, non-routed weights are materialized in FP16 from the BF16 checkpoint.
09 / Related work
Different ways to share the memory.
Horizon sits in a wider field of memory management approaches. These projects move different units of work, use the CPU differently, and target different workloads.
This map compares architectural decisions: what each project stores, transfers and computes. Comparing runtime performance requires a dedicated experiment with comparable checkpoint, precision, hardware and workload.
10 / Reading the evidence
Research with a defined scope.
This presentation brings together the architecture, engineering decisions and an OLMoE and Qwen test campaign on a shared desktop. Results have exploratory E3 status: they record memory, transfers and timing for the stated configurations.
Output quality has not yet been evaluated. Longer sequences, growth of context memory (KV cache), other models and hardware, production use and causal bottleneck identification require their own investigations. These are the limits of this campaign’s conclusions.
Questions for reading the research
How should the results be read?
The rates record how Horizon operated in the published configurations. A performance comparison with another runtime is a separate research question: it requires comparable checkpoint, precision, hardware, prompts and generation settings.
Which data explain the cache?
H2D is the volume transferred from the host to the GPU. The explorer shows the total recorded over measured responses. Hits, misses and per-miss cost require their own counters and timings; that volume alone does not determine those measurements.
What can I verify in the downloads?
The files support integrity and consistency checks across published data. Reproducing the full experiment also requires sufficient code, environment details and raw observations. The audit block describes the shared material and code availability.
The scope of this campaign
This page includes OLMoE 0924, OLMoE 0125 and Qwen. Each configuration uses 12 prompts in three blocks, with 128-token responses.
Short checks and historical series use different conditions and remain separate from this campaign. Each set of files identifies its execution conditions.
Unknown (UNKNOWN) means absent or unmeasured information. Unsupported (UNSUPPORTED) means the platform or dependency cannot provide the observation.
Download this edition’s audit guideThis edition’s guide in English.
11 / Engineering
From question to runtime: the builder’s decisions.
I conceived Horizon and develop it individually, connecting research, implementation and analysis of results. Working with limited resources means choosing what to build, how to verify each decision and where to focus the next effort.
The work builds on NeuronMap research and develops into its own runtime, with recorded provenance for reused code. The decisions below explain this engineering direction; campaign contracts identify the configurations actually measured.
Memory is a budget
Problem · A step activates a few experts, but the full set of weights must remain accessible.
Engineering decision · Distinguish storage, residency and execution; declare capacity per layer and profile.
How to interpret · This decision makes the space reserved for experts explicit in each configuration.
Technical notes · backend contracts ↓Publish only complete state
Problem · Replacing a copy in use must preserve the authority of weights and cache readers.
Engineering decision · Keep the source in RAM immutable, prepare new copies in temporary GPU locations and make the new cache state visible all at once, after transfer completion.
How to interpret · Calculations receive a complete set of copies ready for use, while the source in RAM remains intact.
Technical notes · ADR 0009 ↓Separate variables in the reading
Problem · OLMoE c60 and c64 change residency and executor together.
Engineering decision · Keep the recorded results available, separate Grouped T3 and place the contract beside the measurement.
How to interpret · The reading considers both changes together. The three block rates show recorded variation; isolating the causal effect of residency requires a different experimental design.
Selected campaign report ↓This is the research presentation: Horizon’s architecture, technical decisions and execution records.
12 / Contact
Meet the person building Horizon.
This project brings together the work I want to take into new opportunities: investigating AI systems problems, building solutions and evaluating results. I welcome technical conversations, collaboration and professional opportunities.
How to audit these data
This edition contains curated extracts from the completed campaign. You can check file integrity and consistency between each extract and the aggregate dataset.
Edition · technical-preview-residency39-v1
- Audit guide (English) ↓
- Inventory and SHA-256 ↓
- Data for all 13 configurations ↓
- Original report (English) ↓
- Offline verifier (Node.js) ↓
- Technical decisions and sources (English) ↓
Run verification on your computer
- Follow the guide to download the manifest and its 19 files into a new directory, preserving names. A PowerShell command downloads the complete edition.
- With Node.js 22.19.0, run inside that directory:
node verify-evidence-edition.mjs --directory .PASS confirms 19 file hashes and agreement of 13 aggregate copies. The checker uses only local files, without installing dependencies or running models.
Verification covers file integrity and consistency of this edition’s aggregates. Experiment reproduction and recomputation from raw observations require the full material and a separate procedure. Hashes check files against the supplied manifest; publisher authentication is a distinct check.
The runtime source code remains private by the author’s decision. This presentation shares the architecture, engineering decisions and documented results. Downloads include extracts, a report, technical notes and a verifier.
Inspect the current edition and two historical extracts ↓