IAAIPonderLM
PonderLM: Pretraining Language Models to Ponder in Continuous Space
Teaches language models to ponder in continuous space: within one token step, the model feeds back the probability-weighted sum of token embeddings and runs another forward pass, learned through ordinary self-supervised pretraining.
Inside the method
Forward → expected embedding Σ pᵢ eᵢ → Forward … → token
Simplified conceptual schematic. Consult the paper for the complete architecture.
- Recurrence family
- Full-stack recurrence
- Depth control
- Fixed pondering steps per token
- KV / state strategy
- See paper
Reading note
Works on GPT-2, Pythia and LLaMA architectures; PonderPythia-2.8B surpasses Pythia-6.9B and rivals Pythia-12B on nine benchmarks.
Sources checked 2026-10-28. This catalog does not imply independent reproduction.
Related works
Cite this work
@misc{ponderlm2025,
title = {PonderLM: Pretraining Language Models to Ponder in Continuous Space},
author = {Boyi Zeng and Shixiang Song and Siyuan Huang and Yixuan Wang and He Li and Ziwei He and Xinbing Wang and Zhiyu Li and Zhouhan Lin},
year = {2025},
eprint = {2505.20674},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2505.20674}
}