IAAIRAMP
RAMP: Recurrent Adaptive Mixed Precision for Efficient Prefill in Recurrent-Depth Language Models
Shows that quantization errors injected early in recurrent depth are attenuated by later steps. RAMP therefore runs early refinement in low-precision weights and activations and restores high precision for the terminal steps, speeding up long-context prefill.
Inside the method
[ Core ] × r low precision → [ Core ] × r′ high precision
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Core recurrence
- Depth control
- Fixed recurrence; precision varies with depth
- KV / state strategy
- See paper
Reading note
Preserves near-full-precision prefill quality up to 4K context, with up to 1.16× end-to-end first-prefill speedup (13.9% lower latency) on an H100.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{ramp2026,
title = {RAMP: Recurrent Adaptive Mixed Precision for Efficient Prefill in Recurrent-Depth Language Models},
author = {Ke Wan and Chen Chen},
year = {2026},
eprint = {2610.13041},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.13041}
}