IAAITimestep Encoding for Loops
On Expressive Power of Looped Transformers: Theoretical Analysis and Enhancement via Timestep Encoding
Establishes approximation rates of looped transformers for sequence-to-sequence functions via a modulus of continuity, revealing a looped-specific limitation that per-loop scaling parameters conditioned on timestep encoding can remove.
Inside the method
[ Block, scaled by timestep encoding(t) ] × T
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Full-stack recurrence
- Depth control
- Fixed loops with timestep conditioning
- KV / state strategy
- See paper
Reading note
Experiments show more loops improve performance, with further gains from timestep encoding.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{loopedtimestepencoding2024,
title = {On Expressive Power of Looped Transformers: Theoretical Analysis and Enhancement via Timestep Encoding},
author = {Kevin Xu and Issei Sato},
year = {2024},
eprint = {2410.01405},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2410.01405}
}