IAAIInside the Loop
Inside the Loop: A Mechanistic Study of Weight-Tied Transformers on Depth-Bound Algorithmic Tasks
A mechanistic study on multi-hop permutation traversal, where every intermediate hop has a known answer. Looped models advance through the hops gradually, forming a staircase that adapts to the iteration budget, and behave like iterative program learners rather than simulated deeper stacks.
Inside the method
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Core recurrence
- Depth control
- Fixed iterations; progress adapts to the budget
- KV / state strategy
- See paper
Reading note
A four-layer block iterated eight times solves the task in every seed, while an identically trained untied model of the same effective depth never does unless given a lower learning rate or intermediate supervision. Synthetic algorithmic setting.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{insidetheloop2026,
title = {Inside the Loop: A Mechanistic Study of Weight-Tied Transformers on Depth-Bound Algorithmic Tasks},
author = {Magnus Sesodia and Juil Sock and Philip Torr and Christian Schroeder de Witt},
year = {2026},
eprint = {2610.24135},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.24135}
}