Every Expert Counts: ExactMoE for Memory-Efficient W4A16 Inference

Abstract:Sparse mixture-of-experts (MoE) language models reduce arithmetic by activating only a small subset of experts per token, yet deployment still requires storing and moving the full expert bank. We present ExactMoE, an inference design that applies symmetric group-128 four-bit weight quantization only to routed experts, stores those experts in kernel-native MARLIN form in pinned host memory, and executes all selected experts through a configurable GPU-resident slot cache and fused grouped MoE kernels. The router, attention, embeddings, normalization layers, and language-model head remain in BF16. "Exact" refers to complete expert availability and an unchanged top-k routing procedure: no expert is pruned, substituted, or forced to execute on the CPU. It does not imply numerical identity with the BF16 model. On OLMoE-1B-7B-0924-Instruct, evaluated on a single NVIDIA L4, a 16-slot configuration reduces peak reserved GPU memory from 14.168 to 1.836 GiB (87.04%) while retaining 81.85% of BF16 decode throughput. A fully resident 64-slot configuration reaches 31.923 tokens/s versus 21.662 tokens/s for BF16 while reserving 4.061 GiB. Across 12,450 zero-shot multiple-choice questions, ExactMoE obtains 70.3534% normalized accuracy versus 70.8996% for BF16, retaining 99.23% of the baseline accuracy. In a matched 16-token ablation, fused grouped execution is 1.97x as fast as a sequential W4 reference. These results identify a practical memory-transfer-throughput frontier for complete-expert MoE inference.

Machine Learning (cs.LG)

arXiv:2608.15383 [cs.LG]

(or arXiv:2608.15383v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2608.15383 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

exact_moe_olmoe_1b_7b_0924_instruct/config.json

exact_moe_olmoe_1b_7b_0924_instruct/dolly_manifest.jsonl

exact_moe_olmoe_1b_7b_0924_instruct/logs/architecture.log

exact_moe_olmoe_1b_7b_0924_instruct/logs/bf16.log

exact_moe_olmoe_1b_7b_0924_instruct/logs/download.log

exact_moe_olmoe_1b_7b_0924_instruct/logs/micro.log

exact_moe_olmoe_1b_7b_0924_instruct/logs/reload.log

exact_moe_olmoe_1b_7b_0924_instruct/logs/repair_parity.log

exact_moe_olmoe_1b_7b_0924_instruct/logs/report.log

exact_moe_olmoe_1b_7b_0924_instruct/logs/systems.log

exact_moe_olmoe_1b_7b_0924_instruct/logs/w4a16.log

exact_moe_olmoe_1b_7b_0924_instruct/mc_manifest.jsonl

exact_moe_olmoe_1b_7b_0924_instruct/olmoe_exactmoe_runtime.py

exact_moe_olmoe_1b_7b_0924_instruct/results/architecture_gate.json

exact_moe_olmoe_1b_7b_0924_instruct/results/artifact_audit.json

exact_moe_olmoe_1b_7b_0924_instruct/results/batch_sweep_bf16.csv

exact_moe_olmoe_1b_7b_0924_instruct/results/batch_sweep_w4a16.csv

exact_moe_olmoe_1b_7b_0924_instruct/results/bf16_quality.json

exact_moe_olmoe_1b_7b_0924_instruct/results/bf16_runtime_reference.json

exact_moe_olmoe_1b_7b_0924_instruct/results/bundle_manifest.json

exact_moe_olmoe_1b_7b_0924_instruct/results/capacity_tradeoff.png

exact_moe_olmoe_1b_7b_0924_instruct/results/comparison_summary.csv

exact_moe_olmoe_1b_7b_0924_instruct/results/cuda_graph_decode.json

exact_moe_olmoe_1b_7b_0924_instruct/results/dataset_provenance.json

exact_moe_olmoe_1b_7b_0924_instruct/results/decision.json

exact_moe_olmoe_1b_7b_0924_instruct/results/download.json

exact_moe_olmoe_1b_7b_0924_instruct/results/eval_batches_bf16.json

exact_moe_olmoe_1b_7b_0924_instruct/results/eval_batches_w4a16.json

exact_moe_olmoe_1b_7b_0924_instruct/results/export.json

exact_moe_olmoe_1b_7b_0924_instruct/results/export_gate.json

exact_moe_olmoe_1b_7b_0924_instruct/results/fix_validation_matrix.csv

exact_moe_olmoe_1b_7b_0924_instruct/results/fused_marlin_preflight.json

exact_moe_olmoe_1b_7b_0924_instruct/results/kernel_gate.json

exact_moe_olmoe_1b_7b_0924_instruct/results/kernel_tuning_candidates.json

exact_moe_olmoe_1b_7b_0924_instruct/results/kernel_unit.csv

exact_moe_olmoe_1b_7b_0924_instruct/results/marlin_tuning.json

exact_moe_olmoe_1b_7b_0924_instruct/results/mc_bf16.csv

exact_moe_olmoe_1b_7b_0924_instruct/results/mc_w4a16.csv

exact_moe_olmoe_1b_7b_0924_instruct/results/memory_bf16_cold_process.json

exact_moe_olmoe_1b_7b_0924_instruct/results/paired_quality.csv

exact_moe_olmoe_1b_7b_0924_instruct/results/paper_capacity_summary.csv

exact_moe_olmoe_1b_7b_0924_instruct/results/paper_claims.json

exact_moe_olmoe_1b_7b_0924_instruct/results/paper_quality_summary.csv

exact_moe_olmoe_1b_7b_0924_instruct/results/paper_summary.md

exact_moe_olmoe_1b_7b_0924_instruct/results/parity_keys.csv

exact_moe_olmoe_1b_7b_0924_instruct/results/parity_live_index.json

exact_moe_olmoe_1b_7b_0924_instruct/results/parity_live_repaired_index.json

exact_moe_olmoe_1b_7b_0924_instruct/results/parity_reload.json

exact_moe_olmoe_1b_7b_0924_instruct/results/parity_reloaded_index.json

exact_moe_olmoe_1b_7b_0924_instruct/results/parity_reloaded_repaired_index.json

exact_moe_olmoe_1b_7b_0924_instruct/results/parity_repair.json

exact_moe_olmoe_1b_7b_0924_instruct/results/parity_stages.csv

exact_moe_olmoe_1b_7b_0924_instruct/results/quality_comparison.png

exact_moe_olmoe_1b_7b_0924_instruct/results/reload.json

exact_moe_olmoe_1b_7b_0924_instruct/results/reload_device_integrity.json

exact_moe_olmoe_1b_7b_0924_instruct/results/routing_groups_bf16.json

exact_moe_olmoe_1b_7b_0924_instruct/results/routing_groups_w4a16.json

exact_moe_olmoe_1b_7b_0924_instruct/results/runtime_bf16.csv

exact_moe_olmoe_1b_7b_0924_instruct/results/runtime_reload.csv

exact_moe_olmoe_1b_7b_0924_instruct/results/runtime_w4a16.csv

exact_moe_olmoe_1b_7b_0924_instruct/results/serving_batch_comparison.csv

exact_moe_olmoe_1b_7b_0924_instruct/results/speed_ablation.csv

exact_moe_olmoe_1b_7b_0924_instruct/results/speed_ablation_summary.json

exact_moe_olmoe_1b_7b_0924_instruct/results/stream_load_inventory.json

exact_moe_olmoe_1b_7b_0924_instruct/results/systems_lab.json

exact_moe_olmoe_1b_7b_0924_instruct/results/w4a16_device_integrity.json

exact_moe_olmoe_1b_7b_0924_instruct/results/w4a16_forward_smoke.json

exact_moe_olmoe_1b_7b_0924_instruct/results/w4a16_load_memory.json

exact_moe_olmoe_1b_7b_0924_instruct/results/w4a16_quality.json

exact_moe_olmoe_1b_7b_0924_instruct/results/w4a16_runtime_reference.json

exact_moe_olmoe_1b_7b_0924_instruct/runtime_prompts.jsonl

exact_moe_olmoe_1b_7b_0924_instruct/wiki_manifest.jsonl

verify_results.py

(70 additional files not shown) You must enabled JavaScript to view entire file list.

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.