IAAIDeep Equilibrium Models
Deep Equilibrium Models
Directly solves for the fixed point that a weight-tied layer would reach if iterated infinitely, using root-finding forward and implicit differentiation backward, so memory is constant in effective depth.
Inside the method
z* = f(z*, x) solved by root-finding; implicit backward
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Full-stack recurrence
- Depth control
- Iterate to equilibrium
- KV / state strategy
- Constant training memory regardless of depth
Reading note
Applied to Transformers and trellis networks, it matches or improves language modeling on WikiText-103 with up to 88% lower memory.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{deq2019,
title = {Deep Equilibrium Models},
author = {Shaojie Bai and J. Zico Kolter and Vladlen Koltun},
year = {2019},
eprint = {1909.01377},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/1909.01377}
}