Architecture comparison matrix
Checked 2026-09-16. This is a source-scoped architecture comparison, not a benchmark. Cells summarize the specific documentation linked below; they are not exhaustive statements about every version, backend or configuration. UNKNOWN means the consulted source does not settle the property, not that the project lacks it. Mutable upstream documentation may change.
| Dimension | Horizon immutable cache [H] | Eliseev / Mazur [E] | MoE-Infinity [M] | KTransformers [K] | llama.cpp placement [L] |
|---|---|---|---|---|---|
| Expert source representation | INT4 qpacked, FP16 scales | HQQ mixed low-bit experts | Format-dependent; INT4 and FP4 paths documented | CPU INT4/INT8; other formats supported | GGUF; format depends on model |
| GPU cached representation | Qpacked copies | Quantized expert storage | Path-dependent; compact expert paths documented | GPU GPTQ supported; placement-dependent | Backend/format-dependent; dynamic cache UNKNOWN |
| Expert execution | GPU W4A16, recorded executor | GPU, HQQ-based path | GPU kernels; Marlin INT4 and FP4 paths listed | CPU/GPU hybrid expert computation | Configured CPU/GPU backend |
| Persistent full expansion | No expanded expert cache in this contract | UNKNOWN | UNKNOWN across all paths | UNKNOWN across all paths | UNKNOWN across all backends |
| Source authority | Complete immutable process RAM | Host expert backing; immutable-copy contract UNKNOWN | Host/SSD backing; immutable-copy contract UNKNOWN | Heterogeneous placement; immutable-copy contract UNKNOWN | Model loading/mapping; immutable-copy contract UNKNOWN |
| Eviction writeback | Discard copy; no expert D2H writeback | UNKNOWN | UNKNOWN | UNKNOWN | Dynamic expert eviction contract UNKNOWN |
| Partial/full relationship | Shared design; measured executors differ | Cache budget is configurable | Activation-aware GPU cache | Hot/cold CPU-GPU placement | GPU layer count and tensor placement controls |
| Publication/staging semantics | Bounded staging; completion before atomic generation publication | Per-layer LRU and speculative next-layer loading; atomic generation UNKNOWN | Tracing and prefetching; atomic generation UNKNOWN | Scheduling depends on selected path; atomic generation UNKNOWN | Dynamic publication contract UNKNOWN |
Sources and scope
- H: Horizon technical source map: immutable-cache contract, its original P0 scope and campaign executor boundaries. Current architecture explanation.
- E: Eliseev and Mazur, paper v1, sections 3-4: Mixtral expert offloading, LRU, speculative loading and mixed quantization. Author implementation. This paper is an especially close architectural precedent, not a matched Horizon baseline.
- M: MoE-Infinity README: host/SSD offload, activation-aware caching, prefetching and compact execution paths. The README explicitly distinguishes the released HuggingFace-oriented implementation from the paper version. This table describes the consulted implementation documentation.
- K: KTransformers README, inference capabilities: heterogeneous expert placement, CPU AMX/AVX computation, CPU quantization and GPU GPTQ. CPU computation is part of this documented path; Horizon's documented host role is authority and transfer coordination.
- L: llama.cpp server options:
--cpu-moe,--n-cpu-moe,--gpu-layersand tensor overrides. These placement controls alone do not specify a dynamic expert cache. No unnamed fork or proposed cache is attributed to upstream llama.cpp here.
Reading the comparison
Caching, quantization and offloading overlap across these projects. Horizon's explicit ownership, representation and publication rules are useful comparison coordinates; an unresolved cell cannot establish novelty. Matching those contracts requires inspecting a specific implementation path. Comparing performance additionally requires the same model revision, hardware, workload, precision, output policy and timing boundary. No such cross-runtime experiment is included in the published Horizon population.
Technical brief · Recorded experiments
Additional prior-art audit
Checked 2026-09-16 against the linked primary sources. The matrix above is retained from the original architecture comparison. The following additions distinguish papers, released controls and proposed changes; they introduce no Horizon performance comparison.
HOBBIT
HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference combines dynamically selected lower-precision experts on misses, adaptive prefetching and an expert caching policy. It is directly relevant to the combination of quantization, offloading and caching. Horizon's documented immutable-cache profile instead describes a fixed qpacked representation and complete-generation publication; it does not claim HOBBIT's dynamic mixed-precision loading policy. The paper's reported speedups apply to its own experiments, not to Horizon.
llama.cpp: placement controls and cache proposals
The upstream server documentation exposes CPU MoE placement, a CPU MoE layer count, GPU layers and tensor overrides. Those controls should be distinguished from dynamic caching proposals.
RFC discussion #24528 describes caching hot CPU-resident experts in VRAM with hybrid GPU-hit/CPU-miss computation. Its predecessor PR #24524 was closed without merging. These are explicit architectural precedents and discussion artifacts, not evidence that their flags are supported in every upstream release. The documented hybrid miss path differs from Horizon's host-to-GPU miss transport contract. Forks and proposals require their own pinned revision and acceptance review.
vLLM: weight offload and expert cache work
The upstream offload configuration documents CPU weight offloading, parameter-name selection and layer-group prefetch controls. Weight offloading is distinct from KV-cache offloading.
Expert-cache RFC #38256 and PR #37190 describe a pinned-host expert source and a bounded GPU cache with frequency/recency eviction. The PR was open, not merged, when checked. Its documented first stage uses synchronous H2D transfers and requires eager execution; asynchronous pipeline work is separately proposed. Horizon's atomic-generation and lifetime rules provide concrete comparison questions, not evidence of novelty or superiority. A proposal in a project's repository must not be represented as an already released universal feature.
Existing comparison retained
Eliseev and Mazur remain a close precedent for quantized expert offloading, per-layer caching and speculative loading. MoE-Infinity remains relevant to activation-aware caching/prefetching; its current implementation documentation distinguishes the released code from the paper system. KTransformers remains relevant to CPU/GPU expert computation and placement. Their original source links and scoped matrix cells are retained above. UNKNOWN in that matrix is a source limit, never a claim that an upstream system lacks the property.
The wider context references on the main presentation remain available there. Published results across different hardware, models, precision, workloads and timing boundaries are not matched Horizon baselines.
Wider related work from the main presentation
The following source-scoped entries preserve the published site's existing review. Source-check dates are retained from that review; this is not a new benchmark.
FlexGen / FlexLLMGen
Weights, activations and KV cache may be placed across GPU, CPU memory and disk. Optional four-bit weight and KV-cache compression. Tensors and layers reused across batches.
Scope: Optimized for throughput-oriented batches, with a latency trade-off. The released README lists automatic policy optimizer release as future work. The curated sources do not resolve a universal CPU operator partition.
- FMInference — checked 2026-09-10.
- FlexGen authors — checked 2026-09-10.
DeepSpeed ZeRO-Inference
CPU or NVMe holds weight backing; one or a few layers occupy GPU memory temporarily. Half-precision original design; later official example supports four-bit weights. Layer weights, with partitioned fetch in multi-GPU configurations.
Scope: Transfer amortization depends on workload. Inference weight offload must not be confused with training optimizer offload.
llama.cpp
Full or split CPU/GPU placement, depending on configuration. GGUF supports multiple quantized storage formats. Configured layer and tensor boundaries; a general dynamic expert cache is not established by these options.
Scope: Memory use and behavior depend on model, backend and configuration. No universal overlap strategy or dynamic expert-cache policy is claimed.
PowerInfer
Frequently activated neurons reside on GPU; cold neurons reside on CPU. Specialized PowerInfer GGUF, with unquantized and Q4_0 options. Neuron-aware CPU/GPU split and intermediate exchanges.
Scope: Specialized sparse models and predictors are required for the reported mechanism. Ordinary checkpoints do not inherit the reported benefits. A universal CPU/GPU overlap schedule is not claimed.
- PowerInfer authors — checked 2026-09-10.
- Tiiny-AI / PowerInfer — checked 2026-09-10.
PowerInfer-2
Smartphone DRAM neuron cache backed by flash storage. FP16 and four-bit cases are disclosed. Neuron bundles and clusters with quantization-dependent flash reads.
Scope: Evaluation requires rooted devices and specific sparse models. Some comparison arms use sparsified models while others use originals; this is not an unchanged-checkpoint comparison. The smartphone mechanism is not a generic desktop GPU deployment contract.
- PowerInfer-2 authors — checked 2026-09-10.
AirLLM
Layerwise disk shards with transient accelerator residency. Uncompressed default; optional four- or eight-bit blockwise weight compression. Model layers.
Scope: Layer sharding needs additional disk capacity. Prefetch support is model-specific. A memory-fit example does not establish interactive latency. The curated summary does not establish a universal CPU arithmetic partition.
- AirLLM maintainers — checked 2026-09-10.
Mixtral offloading
Host expert weights with a bounded GPU expert cache. Selected experiments use HQQ two/three-bit experts and four-bit attention; specified other tensors retain sixteen bits. Selected experts.
Scope: Evaluation covers a narrow model and hardware population. Quantization changes checkpoint values. Cache behavior and speculative loading outcomes do not generalize automatically.
- Eliseev and Mazur — checked 2026-09-10.
- Mixtral offloading authors — checked 2026-09-10.