IAAIInner Thinking Transformer
Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinking
Treats layer computation as implicit thinking steps: Adaptive Token Routing sends critical tokens through extra iterations, Residual Thinking Connections refine them, and Thinking Step Encoding marks each phase.
Inside the method
Token → router → [ Layer ] × rᵢ with residual thinking connections
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Adaptive depth
- Depth control
- Adaptive token routing; elastic at inference
- KV / state strategy
- See paper
Reading note
From 162M to 466M parameters, a 162M ITT reaches 96.5% of a 466M Transformer's performance and cuts training data by 43.2%, beating Transformer and loop variants on 11 benchmarks.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{innerthinkingtransformer2025,
title = {Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinking},
author = {Yilong Chen and Junyuan Shang and Zhenyu Zhang and Yanxi Xie and Jiawei Sheng and Tingwen Liu and Shuohuan Wang and Yu Sun and Hua Wu and Haifeng Wang},
year = {2025},
eprint = {2502.13842},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2502.13842}
}