An open, community-reviewed atlas of looped Transformers, recurrent depth, adaptive computation and latent reasoning.
199 works by month of first publication, colored by recurrence family.
LoopFold: Parallelizing Recurrent Depth in Looped Transformers
Cuts the sequential latency of deep looped models by folding several recurrent depths into one parallel batch for the same prefix. A lightweight predictor guesses the inputs of later loops so they can start early, and a corrector refines them once earlier loops finish; everything is trained with the ordinary next-token loss.
StateLoop: Persistent State for Looped Transformers
Adds one persistent state per token that survives across loop iterations. A learned gate decides how much of the old state to keep and how much of the new backbone output to absorb, and the state is projected back into the stream for later iterations and the final prediction.
Cross-Loop Query Handoff: How Later Loops Redirect Two-Hop Retrieval in the Ouro Looped Language Models
A mechanistic study of how later loops in Ouro repair two-hop retrieval. Transplanting loop-3 attention queries into loop 1 rescues most questions that loop 1 got wrong, while late keys and values help far less, pointing to queries as the carrier of the later-loop correction.
LoopGR: Parameter-Efficient Scaling for Generative Recommendation via Recurrent Depth
Brings recurrent depth to generative recommendation with a hierarchical mechanism that alternates a shared generative decoder and a lightweight context updater, plus recurrent trajectory self-distillation from a high-budget teacher to a low-budget student.
Recursive CLIP: Vision-Language Distillation with Looped Transformer Blocks
Asks how much block reuse CLIP tolerates. Recursive backbones apply N blocks M times with N·M fixed at ViT-B depth, either in the vision encoder only or in both encoders, trained contrastively or by distillation from a strong ViT-B/16 teacher.
Prefix Keys and Values Matter Late: RIPE Caching for Self-Speculative Decoding in Looped Language Models
Shows that in recurrent-depth models the KV planes of late loops matter far more than early ones and converge geometrically. RIPE therefore stores only the deepest KV plane for most positions and keeps full depth only for a recent window, which suits self-speculative decoding with few-loop drafts and full-depth verification.
ScaleRLT: Exact Log-Probabilities and Fast Deferred Rollouts for Reinforcement Learning on Looped Transformers
Makes on-policy RL practical for looped language models such as Ouro. Rollouts defer and batch the deeper loops, and the learner uses a hybrid log-probability whose value replays the rollout schedule exactly while its gradient comes from a dense pass, removing the floating-point mismatch between sampler and learner.
One Model, Two Paths: Balancing Single-Pass and Looped Performance in Language Models
Adapts a pretrained model so it can run either a single pass or a looped pass that reuses its middle layers. Joint weighted losses on both outputs, plus guidance from the frozen original model, stop looped training from degrading single-pass quality.
Accordion: Elastic-Depth Inference via Fixed-Prefix Layer Looping
A mixture-of-experts model that serves five inference budgets through fixed-prefix layer looping, with the smallest budget equal to the unlooped model. Pretraining samples one budget per micro-batch, and SFT and RL with budget-matched multi-teacher distillation preserve every budget.
Do Halting Heads Really Save Compute in Looped Language Models?
Pretrains looped models up to 1.3B parameters to test learned halting heads. Heads trained on a mix of iterations predict early stops yet tokens run almost every iteration under the usual rule, and the commonly reported average-iterations metric overstates savings in batched generation.
LSTM-Style Looped Transformers with Shared KV Caches
Studies an LSTM-style gated recurrent transformer whose recurrent core also shares a single KV cache. A consistency loss late in training aligns the cache entries seen in parallel training with those produced recursively at generation time.
BeNice! Closing Looping's Inference-Memory Gap via KV-Sharing
Makes looped models KV-shareable so earlier recurrences can reuse the final recurrence's cache. Stochastic depth during pretraining induces contractive dynamics, and the BeNice! post-training step broadcasts the final KV cache into the first recurrence while preserving quality.
Agreement rather than normalisation keeps depth reusable in looped language models
Asks what keeps extra loop steps usable beyond the trained budget. Removing Ouro's loop-boundary normalisation breaks the released models without norm blow-up, yet small models trained without it extrapolate better, suggesting that agreement between the training and inference loop is what matters.
Loop to Look Better: Learning Adaptive Depth for Question-Guided Perception in Vision-Language Models
Adds adaptive recurrent perception to pretrained vision-language models by revisiting a middle-to-late window of the vision encoder with the question injected at each loop. A learned stopper sets the depth and a token-wise Mixture-of-Looks blends features from different loops.
Looped Uniform Diffusion Language Models
Applies selective looping, which repeats a few early layers inside each denoising step, to uniform diffusion language models without changing their objective or sampler.
Asymmetric Looped Transformers: Fixed Global Representations for an Iterative Decoder
Separates context building from iterative refinement: a single-pass causal encoder writes fixed per-layer global KV caches, and a weight-tied recurrent decoder attends jointly to them and to local KV from its evolving state.
PAR: Progress-Aligned Recurrence for Length Generalization in State Tracking
Progress-Aligned Recurrence makes extra iterations useful for longer state-tracking problems by supervising the prediction at iteration k to match the state after k operations and anchoring recurrent state norms to the first iteration. A learned completion readout decides when to answer.
How Much Gradient Does Truncated Backprop Lose in Looped Transformers?
Measures exactly how much weight gradient truncated backprop discards for every truncation length, across nine looped language models including Parcae and Ouro.
Internalizing Chain-of-thought into Looped Transformers
A theoretical construction of constant-bit-size looped transformers that interpolates between chain-of-thought and latent recurrence, trading generated tokens for loop iterations when simulating Turing machines.
recAMPLIFY: Large-Scale Pretraining of Recursive Protein Language Models
Adapts recursive pretraining to masked protein language models: a weight-tied core refines residue-aligned states over an adjustable number of cycles. Gated Mixture Recurrence then decodes from a probability-weighted mixture of visited states with a learned residual gate, unifying halting and gating.
Spherical Fixed-Point Reasoners: Adaptive Loop Halting on the Hypersphere
Spherical Fixed-Point Reasoners move fixed-point looped reasoning onto the unit hypersphere of normalized transformers, turning input injection, damping and halting into geodesic operations that are bounded by construction.
Truncate or Not? A Scaling Study of Looped Transformers with Truncated Backpropagation
A scaling study of truncated backpropagation through time in prelude–recurrent–coda looped transformers. Nearly 600 runs jointly vary model size, tokens and forward/backward loop counts, and a unified law expresses looping and truncation through effective parameters.
Block Recursive Transformers for Structured Parameter Sharing
Argues that how shared layers are organized matters. BRT splits the stack into contiguous blocks and recurses each block before moving to the next, an intermediate topology between per-layer repetition and looping the entire stack.
Dual-Slot: From Serial Loops to Parallel Slots in Language Models
Reorganizes serial looped computation into parallel latent and prediction slots. The prediction slot uses the previous token's deep latent state, while the latent slot prepares the state for the next step, replacing two sequential invocations with one paired call.
UpLoop: Upcycling Large Language Models with Gated Loop
A post-training recipe that upcycles a pretrained LLM into a looped one by looping selected layers locally rather than the whole stack, with hidden-wise gates on attention and MLP residuals so later loops refine instead of rewrite.
Causal Latent Workspaces for Looped Transformers
Gives a shared-depth Transformer a dedicated causal workspace: at each of four rounds, positions write gated vectors into prefix-specific latent slots and read back a content-addressed summary before the next application of the shared block, while the input embedding stays accessible.
Inside the Loop: A Mechanistic Study of Weight-Tied Transformers on Depth-Bound Algorithmic Tasks
A mechanistic study on multi-hop permutation traversal, where every intermediate hop has a known answer. Looped models advance through the hops gradually, forming a staircase that adapts to the iteration budget, and behave like iterative program learners rather than simulated deeper stacks.
Towards Looped Models Done Right, Part I: Topology, Input Injection, Recurrent-State Design
Disentangles three design axes of looped language models trained from scratch at matched parameters, logical depth and tokens: where recurrence sits, how input is injected, and how recurrent state is organized. The prelude–recurrent–coda sandwich and input injection come out ahead, favoring the Huginn-style design over Ouro.
LoopAhead: Speculating Ahead for Faster Decoding in Looped Language Models
A speculative decoding framework for looped transformers that gives different recurrent depths complementary roles: a lightweight lookahead drafter predicts several future tokens from early-loop states, deeper-loop feedback decides which draft branches to continue, and the full-depth model makes final decisions.
Retrofitting Recurrent Vision-Language Models from Pretrained Backbones
Retrofits depth recurrence into a pretrained vision-language model by pruning language-backbone layers and looping a subset of the rest, with a linear adapter merging each loop's output with earlier token representations. A three-stage recovery with distillation from the original model restores quality.
Learning to Compute Is Not Learning to Retain or Reuse in Looped Transformers
Separates three causes of overthinking in looped transformers on in-context Mealy-machine execution: failed computation, lost retention and non-reusable states. The computation itself is exact, but answers decay with extra loops unless training asks for retention.
LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocation
Finds that in a looped MoE that applies a shared block twice, the first visit can be scaled down without harm while the second cannot. LatentLoop compresses the sequence on the first visit, keeps full resolution on the second, and spends the savings on a larger hidden size.
LEV: Thinking Fast and Deep with Looped Decision Models
A decision model that answers typed multiple-option questions in one pass but can think deeper by rerunning its layers in a loop instead of generating text. It reads the options after every loop step, so one checkpoint answers at any depth, and a small stopper sets depth per question.
Test-Time Forecasting: Recurrent Depth Enables In-Context Dynamics Prediction
Links linear attention to dynamic mode decomposition to show that looped transformers can emulate propagators of dynamical systems in context, and builds a framework in which a one-step forecaster uses gradient-descent iterations as recurrent depth.
LoopREG: Representation Supervision Enables Test-Time Recurrent Depth in Diffusion Transformers
A recurrent diffusion Transformer that repeatedly applies a shared block at a state aligned to a pretrained visual encoder after the final recurrent visit, so inference can extend the same operator to additional visits.
LoOPSD: On-Policy Self-Distillation with Looped Teachers
Builds a stronger teacher from the same frozen weights by searching which layers to repeat, then distills that looped teacher's predictions on student-generated prefixes back into the student, alternating search and distillation over several rounds.
Monitoring latent reasoning in looped language models
An interpretability-based inspection protocol with five lenses applied to eight looped language models from four families, asking whether iterations actually improve capability and whether models stay stable under prompt perturbation.
Looped Reasoning Finishes Earlier Than You Think
The Quotient Recurrent Reasoner adds an exact per-token phase-rotation symmetry so that slow oscillatory motion lives along directions that do not change the reasoning-relevant quotient state, avoiding an oscillation bottleneck in dynamic-depth looped reasoning.
LoopQuant: Accurate and Efficient Post-Training Quantization for Looped Transformers
A loop-aware post-training quantization framework for looped language models. It canonicalizes activation distributions across loops while keeping weights shared, allocates activation precision per loop under a fixed bit budget, and calibrates the quantized model along its own low-bit recursive trajectory.
Learning to Extrapolate in Depth with Looped Transformers
A theoretical and empirical study of how far a looped block trained at depth K can be run deeper, using all-pairs graph connectivity. Training converges to an update governed by one effective parameter, which yields a finite extrapolation window with explicit endpoints.
Running Fewer Loops Removes Refusal Behaviour in Recurrent-Depth Language Models
Shows that running a recurrent-depth model with fewer loops than trained, a tempting deployment shortcut, erodes safety much faster than capability: harmfulness is still detected internally, but the refusal that should follow is missing.
Adaptive-Depth Is a Finishing Stage
Introduces fixed-first training: train a looped model at one fixed depth for most of the run and vary depth only in a short final phase, instead of training across depths from the start.
RAMP: Recurrent Adaptive Mixed Precision for Efficient Prefill in Recurrent-Depth Language Models
Shows that quantization errors injected early in recurrent depth are attenuated by later steps. RAMP therefore runs early refinement in low-precision weights and activations and restores high precision for the terminal steps, speeding up long-context prefill.
One Model to Loop Them All: Elastic Latent Recurrence for Language Models
A training recipe that removes the one-loop tax of looped language models, so one checkpoint at one loop matches a non-recurrent model of the same size and gains accuracy with each additional loop. It uses gated state updates with learned step size, mixed loop counts per update, and loop-index and budget conditioning.
Scaling to Tens of Thousands of Test-Time Iterations with Loop-Native Attention Residuals
Identifies latent overthinking, where noisy updates overwrite correct intermediate deductions, and introduces a loop-native residual connection that keeps a constant-memory running summary of past recurrent states using content-based weighting and learned temporal decay.
From Chain-of-Thought to Loops: Non-Autoregressive Latent Reasoning via Looped Transformers
Replaces left-to-right latent thought generation with iterative refinement of a compact latent workspace by a shared transformer. A probabilistic head then samples latent tokens in parallel to condition an autoregressive answer decoder.
One Block, Multiple Depths: Recurrent Vision Transformers with Depth-Programmed Experts
Shows a single Transformer block applied recurrently can match a full-depth vision encoder at comparable FLOPs. Depth-specific behavior comes from expressing each depth's FFN as a convex combination of a small shared expert bank, programmed by a continuous normalized-depth coordinate.
Recurrent Self-Improvement: Dynamic Cross-Loop On-Policy Distillation for Looped Language Models
Uses a looped model's own terminal loop as a compute-privileged teacher for an intermediate-loop student on student-generated rollouts. The dynamic variant keeps refreshing the teacher as shared weights update, giving recurrent self-improvement without an external teacher.
ResidualQuant: KV Cache Quantization for Looped Transformers with 2-Bit Residuals
Exploits the high similarity of KV states across loops: it stores the final-loop KV as a reference and encodes other loops as low-precision residuals with least-squares scaling, rotations and loop-wise mixed precision.
Safe at One Loop, Risky at Another: Aligning Safety Across Recurrent Depths in Looped Language Models
Evaluates jailbreak robustness of looped language models at different recurrent depths and finds that safety can differ by depth and attacks transfer across depths. SafeBridge aligns safety across depths with depth-specific control of shared layers, selective state bridging and joint supervision.
Enabling Dynamic Computation in Looped LMs
Points out that Ouro's early exit does not save compute in practice because every depth needs its own KV level. A training-free best-available KV caching strategy lets exited tokens reuse the deepest cache they have, and a fix to the early-exit prior makes tokens exit at genuinely different depths.
Hybrid Latent Attention for Looped Language Models
Shrinks the T-fold KV cache of looped models by keeping exact keys and values only for a sliding window of recent tokens and storing older tokens as compact latents that every loop's query reads directly. It is uptrained on frozen Ouro models, training only the added parameters.
Recurrent Looped Transformer
Splits the layers into a parallel causal encoder and a recurrent decoder that merges each token's encoder output with the previous token's final decoder state, so the computation path grows with sequence length at a fixed per-token cost.
Towards Looped Models Done Right, Part II: Rethinking at Fixed Points
Argues that states near fixed points make the path to them matter less, enabling truncated backprop, terminal KV sharing, faster distilled prefill and cheaper RL updates. It then improves the depth prior, learned from prediction feedback with an entropy term, and introduces orthogonal input injection.
Loopy: Low-Bit Quantization Framework for Looped Language Models
A post-training quantization framework that selects shared-core low-bit representations by their final prediction loss at the target deployment depth, because the ranking of quantization configurations changes with recurrent depth. Calibration windows are allocated progressively to promising candidates using forward passes only.
AdaLoop: Adaptive-Depth Latent Reasoning for Audio Language Models
A lightweight plug-in for audio language models: a shared transformer block iterates over the audio representation under question guidance, and a learned halting mechanism exits once the representation is ready, so perception-heavy questions get more steps.
On the Chain-of-Thought Monitorability of Looped Language Models
The first systematic evaluation of whether looped language models remain chain-of-thought monitorable, varying loop depth within a family and comparing against non-looped models matched by size, layer count or effective depth.
LAST: Looped Audio Spectrogram Transformer
The Looped Audio Spectrogram Transformer processes all tokens once and then reuses the same blocks to refine only the class token over fixed audio features, making later passes cheap.
Looping Beyond Twice: A Scalable Recipe for Looped Mixture-of-Experts
Diagnoses why looped MoE models stop improving after two loops: variance growth across iterations and expert-selection collapse. LOOM scales residual updates, re-injects the input embedding every loop, gives each loop its own router, and adds a looping residual that carries earlier outputs forward.
Decoding Looped Transformers Better for (Almost) Free
Training-free contrastive decoding that contrasts a looped model's final prediction with an earlier recurrent pass, which serves as a built-in weak model. It works in logit space with one extra output pass or in hidden-state space with no extra output cost.
The Surprising Effectiveness of Shared Memory in Looped Transformers
Pretrains looped models in which only the first recursion writes a KV cache and later recursions read it while keeping a short local window, so memory does not grow with loops. Sharing memory improves rather than hurts quality.
Rotate the Loop: Blockwise Conformality for Looped Transformers
Treats whether a loop can track state as a geometric question about the linear map applied to the residual between steps. Blockwise conformal injections split the residual into groups and rotate within each, giving each group its own timescale while keeping exact geometry.
Closing the Loop: Practical Training Recipes for Looped Language Models
Practical recipes for looped language models: a from-scratch pipeline that cuts Ouro's 7.7T-token budget to 310B tokens, controlled comparisons against dense models, and a minimal conversion of pretrained dense models using one learned input-mixing scalar and a smoothed exit loss.
Shared Weights, Selected Computations: How Looped Transformers Route What Each Loop Does
Asks whether each loop of a looped transformer repeats the same operation. On graph walks, loops can advance by varying steps, and a learned linear map on the entering hidden state can steer a frozen loop toward different transitions, with attention routing as the causal pathway.
Scaling Laws for Looped Mixture of Experts
The first scaling law that models recurrence and MoE sparsity jointly with model size and data, through a bounded, sparsity-conditional mapping from loops to effective parameters. It recovers dense and MoE laws as special cases and guides looped MoE design.
Looped Diffusion Transformer
Scales text-to-image computation by repeating shared Transformer blocks within each denoising step. Deep supervision across intermediate loops and self-modulating attention fix the weak supervision and eroding local detail that make naive looping fail.
What Limits Recursive Reasoning Models: Optimization, Architecture and Test-Time Scaling
Puts HRM, TRM and URM-style recursive reasoners into one experimental pipeline across six algorithmic domains to separate the effects of architecture, gradient propagation and training procedure. A simple recipe emerges: an intermediate gradient horizon, large physical batches and controlled recurrent-state updates, with no explicit hierarchy needed.
A Tilted Bowl Is Not a Slippery Slope: Compressing Looped Models
Tests the common claim that compressed looped models collapse because rounding error accumulates across loops. Across more than 30 models from five families, this holds only for loops that never settle; for settling loops, error shifts the fixed point instead of accumulating.
Looped Actor: Depth-Recurrent Reasoning Models for Reinforcement Learning
Brings dynamic looping to reinforcement learning: a transformer policy refines a latent toward a fixed point with a shared block and loops more in states that need planning. A complexity argument shows state-adaptive policies can need less expected computation than any fixed-runtime optimal policy.
Thinking in Depth, Speaking Directly: Recurrent Latent Reasoning for Paralinguistically Grounded Spoken Dialogue
A spoken dialogue model that reuses a decoder block for latent reasoning with acoustic grounding at every pass, trained in two stages that separate learning to reason from learning to respond so it can reply directly without chain-of-thought.
LoopVL: Recurrent Visual Intelligence
Extends looped transformers to vision-language models trained from scratch, combining Module-Loop and Model-Loop computation to iteratively update a unified vision-language state through shared modules.
Looped Transformers as Optimizers
Views the looped hidden state as a fast weight and derives loop transitions in closed form from a projection, a local objective and an optimizer update rule. This exposes mismatches in existing transitions and yields OperLoop, with explicit weight decay, adaptive step size and a delta objective.
Scheduling Recursive Reasoning in Looped Transformers
Adapts the scale of each recurrent update online. The Trajectory Adaptive Progress-Fluctuation Scheduler decomposes sensitivity of the terminal loss into persistent progress and fluctuation terms and adjusts the step size accordingly, at inference without retraining or also during training.
What Makes Recurrence Effective in Looped Language Models?
Controlled experiments on when recurrence helps, where to place it, and how to condition it, across inference budgets below, within and beyond the training horizon. It proposes channel-wise history-state injection with timestep conditioning as a cheap, more robust alternative to initial-state injection.
Shallow Queries, Mature Values: Depth-Asynchronous Self-Speculation for Looped Transformers
Observes that queries and keys reach their final-depth form earlier than values. DAS lets shallow draft queries read full-depth prefix values without extra recurrence, and DAS-Wave adds parallel refinement, progressive block growth and an independent full-depth verifier.
FlexLoop: Depth-Elastic Looped Policies for Adaptive Test-Time Computation in Deep RL
Finds that pretrained looped RL policies specialize to their full trained depth. FlexLoop post-trains them with the original RL objective plus adjacent-depth policy distillation, making shallower depths reliable and enabling state-wise adaptive inference.
How to Loop MoE: Flatten the Experts, Untie the Attention
Answers how to loop a mixture-of-experts model at fixed expert parameters and compute: flatten the experts by halving expert layers, doubling experts per layer and doubling passes, and untie attention so each pass has its own attention while experts and routers stay shared.
Not All Thinking is Created Equal: Latent Reasoning Discovers a Recurrent Search Algorithm for Depth Generalization
Compares vanilla, chain-of-thought, pause-token and two latent-reasoning variants of one backbone trained on extended multi-hop reasoning. Only the latent variants generalize to longer hops, and circuit analysis finds a sparse recurrent forward-search circuit propagating reachability across steps.
Loop Dropout: Regularizing Shared Updates in Looped Language Models
Finds that standard LoRA on looped models mostly adapts late loops. Loop Dropout randomly masks adapter applications per loop during training and rescales by inverse survival probability, so the shared update stays strong and effective at every loop.
LoopICL: Looping a single transformer block to solve tabular tasks
A tabular foundation model built from a single looped block with coupled cell and row streams refined by within-column and cross-column attention. Varying loop counts in pretraining and a learned exit gate let users trade inference cost for accuracy.
Improving Test-Time Scaling with Adaptive Looped Transformers
Studies whether looping improves test-time scaling as outputs grow longer and finds fixed-depth looping wastes iterations on tokens that do not benefit. TaH2 post-trains the backbone with an iteration decider supervised by online lookahead labels indicating whether another iteration improves the prediction.
Beyond the Training Horizon: Mechanisms and Limits of Length Generalization in Looped Transformers
A mechanistic comparison of two recurrence-training schemes for looped transformers on polynomial iteration, finite-state composition and knowledge-graph traversal, revealing distinct mechanisms and why both fail at greater depths.
MoR-MLLM: Mixture of Recursions for Efficient Multimodal Large Language Models
Applies Mixture-of-Recursions to multimodal LLMs so each token picks its recursion depth, spending more on visually or linguistically hard tokens. A three-stage tuning schedule and entropy-regularized routing loss stabilize recursive sparsity.
Quantizing Looped Transformers: Feedback Exposure and Calibration Blindness
Identifies two failure modes of post-training quantization in looped models: feedback exposure, where a quantized non-residual layer such as the loop-entry adapter perturbs the state and the error is fed back, and calibration blindness, where one-step calibration ignores directions used later in the recurrence.
FlashLoop: Fast and Memory-Efficient Looped Transformers via Lazy Updates
A training-free inference framework that removes cross-loop redundancy: later loops update only the few tokens whose states still change, attend to a sparse stable set of key columns, and store KV residuals between adjacent loops in low bit-width.
Depth through recurrence: Looped transformers for flow-matching TTS
Studies how to organize depth through recurrence in flow-matching text-to-speech by varying the amount, order and placement of weight reuse across seven layouts with the same number of block calls.
Attention Routing Stabilizes Early: Working-Set Inference for Recurrent Language Models
Finds that attention routing in recurrent-depth models stabilizes well before hidden states do. WISE uses full attention in early recurrent steps to discover a block-structured working set, then reuses that support in later steps while keeping attention weights and refinement dynamic; no training is needed.
LoopCD: Loop-wise Contrastive Decoding for Improving Reasoning in Looped Language Models
Links reasoning errors in looped language models to loop instability that produces uncertain hard tokens, and contrasts logits from earlier iterations with the final iteration at decoding time to refine them, with no training and negligible overhead.
WaveFront Decoding: Parallelized Self-Speculative Decoding for Looped Language Models
Training-free self-speculative decoding that arranges token states at different positions and depths into a diagonal wavefront, so drafting new positions at shallow depth and verifying earlier ones at full depth happen in the same batched call of the shared block.
Trading Depth for Time in Recurrent Transformers
Compares spending extra compute on temporal recurrence versus physical depth. Latent Recurrent Transformers feed each token's high-level hidden state into the next, and inserting one latent thought token between vocabulary tokens adds a refinement pass through the shared backbone.
Prediction Dynamics in Depth-Recurrent Language Models
Explains why intermediate answers of depth-recurrent models can already match the endpoint while scores keep changing, with a margin characterization that separates common translation, direction relative to the winner, and pairing of competitor updates with score gaps.
Beyond Depth Truncation: Controlled Evaluation of Depth Utilization in Recursive Language Models
Argues that the standard depth-truncation curve conflates fewer block applications, less distinct computation and an out-of-distribution readout. The Depth Control Protocol adds positive and negative controls, including running the full budget while executing only one distinct iteration, to measure whether a recurrent model really uses its depth.
Storing Is Not Remembering: LSTM-UT and Bounded Gated Memory for Looped Transformers
Shows on cellular-automaton tasks that keeping more history is not the same as making it usable: CoTFormer's full cache can pull corrected trajectories back toward failure. LSTM-UT adds a small, bounded, gated cell state to the shared block instead.
LoopSpec: Pipelined Self-Speculative Decoding for Looped Transformers
Training-free pipelined self-speculative decoding for looped transformers: drafts from early recurrent states overlap with verification of the current token, and a selective second proposal from a deeper depth improves draft accuracy while keeping decoding lossless.
Right Direction, Wrong Step: Geometric Analysis of Finite-Step Failure in Looped Transformers
Separates whether an extra loop's update direction is unhelpful or its full step simply moves too far. Along the model's own update direction, many failures are locally improving directions with harmful full displacement, analyzed with a pathwise curvature decomposition.
T-LoopFormer: Token-Level Elastic-Depth Looped Transformers with Dynamic Routing
Token-level elastic depth for looped transformers: a token-choice router lets each token pick its number of loop iterations from its hidden state, and a recursion-wise KV cache keeps a separate cache per loop for active tokens.
Thinking with Looped Flows
Trains the recurrence of a looped model with local denoising objectives at progressively decreasing noise levels with shared noise, so states learn to carry useful computation forward even though gradients cover only a few updates. Inference integrates a probability flow, and finer time grids spend more compute.
LoopVAE: Recurrent Depth Across Scales for Visual Tokenization
Reuses a scale- and loop-conditioned core within and across spatial scales of a hierarchical visual tokenizer, keeping only resolution-changing transitions independent.
RecurTrace: Adaptive Latent Reasoning with Loop-Time Memory
Improves middle-layer latent recurrence in two ways: Loop Memory Attention lets each looped layer attend to its own states from earlier iterations along the loop-time axis, and a halting head trained with oracle labels decides when extra depth still reduces loss.
Flow Reasoning Models: Turning Flows Into Efficient Recurrent Reasoners
Flow Reasoning Models turn a non-causal flow denoiser into a recurrent reasoner by feeding its clean-solution prediction back as a conditioning state. Fixed-Point Forcing trains the model on states produced by its own recurrent rollouts, enabling deeper iterative refinement while retaining the standard flow-matching objective.
Diffusion as a Training Curriculum for Timestep-Free Iterative Reasoning
Diffusion model with recurrent memory that does more than state-based self-correction
Looped Transformers under the Jacobian Lens: Does the Global Workspace Survive Recurrence?
Extends the Jacobian lens to iterated architectures to test whether a global-workspace-like band of verbalisable, causally potent representations survives recurrence, comparing Ouro-2.6B, Huginn-0125 and an untied Qwen baseline.
SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers
Repeats the middle half of a sparse MoE Transformer's layers twice while closely matching per-token FLOPs, total non-embedding parameters and KV cache to an unlooped baseline. Budget-matched scaling experiments show improved loss and downstream performance from reusing the shared middle layers.
Dissecting Hierarchical Reasoning Models: A Mechanistic Study
A mechanistic study of the Hierarchical Reasoning Model on Sudoku, Maze and ARC-AGI-2 using baselines with and without recurrence, state interventions, probes and sparse autoencoders.
Dynamical phase selection controls compute scaling in looped transformers
Shows that looped networks with the same architecture, objective and accuracy can land in different dynamical phases depending on initialization, and that the bifurcation type of each phase, such as a saddle-node fold or a Neimark–Sacker transition, sets how test-time compute scales.
Allocating Recurrent Compute in Looped Language Models
Asks what should loop. Using Iterative Transport Rank to measure the new cross-position influence each application adds, it repeats each Gated DeltaNet mixer while applying the dense FFN only once.
Think Shallow, Solve Deep: Controlling Recurrent Dynamics for Reliable Test-Time Depth
Shows that a trained recurrent operator's finite-time dynamical regime, estimated as settling, marginal or drifting, predicts whether extra test-time iterations help, preserve or hurt answers, and gives a sufficient condition for depth safety.
Looped Language Models Improve Compositional Tool Calling
Evaluates native and retrofitted looped language models on compositional tool use, where models must coordinate several API calls and keep dependencies, comparing looped and non-looped models under matched fine-tuning while varying recurrent depth at inference.
Gated Recurrent Transformers: Expressive Depth through Recurrent Modulation
A prelude–core–coda recurrent-depth transformer whose shared core update is modulated by a lightweight projection and an elementwise update gate conditioned on the hidden state, the prelude output and per-step noise, letting a few layers specialize across recurrences.
Mimir: A Sub-2B Hierarchical Reasoning Model Trained on Permissible Data Punches Above Its Weight Class
A 1.78B language model built on the HRM-Text architecture, whose layers form a low-level and a high-level stack applied repeatedly in an interleaved schedule, trained from scratch only on permissibly licensed data.
Depth-adaptive Inference of Looped Language Models via Continuous Depth Batching
The first efficient serving method for depth-adaptive looped models: new batches are formed between loop steps, looped and non-looped parts are scheduled separately, looped KV caches are managed, and early exits are predicted so batches can be prepared asynchronously.
Think Deep, Speak Once: Relit, A Recursive Latent Implicit Transformer Framework
Augments a frozen TinyLlama-1.1B with a small trainable recursive block that iteratively refines a latent thought before answering, aiming to combine TRM-style recursive reasoning with the semantics of a pretrained language model.
LoopMTP: A looped transformer guided by latent multi-token prediction
Links looping to multi-token prediction: a model that loops T times should anticipate T future tokens, so the hidden state at loop t is softly aligned with the embedding of the token t steps ahead, with a gate preserving useful information across iterations.
Retrofitting Recurrent Depth into a Pretrained Language Model: Installation, Extrapolation, Transfer, and Retention at Two Parameter Budgets
Retrofits Qwen2.5-0.5B-Instruct into prelude, weight-tied recurrent block and coda with an identity-preserving one-loop path and a re-entry bridge, and shows it learns a reusable iterative latent procedure, either as a 6M adapter over frozen weights or a 180M full block.
Looped Transformers with Source-Centered State Evolution
Source-Centered State Evolution reconciles input conditioning with shared recurrence: inputs enter through a learned anchor and initial deviation, zero deviation maps to zero, and the anchor is an exact fixed point, removing the forcing bias of additive input injection.
Nanbeige4.2-3B: Unlocking Agentic Capabilities in a Compact Model
A released language model that applies recurrent computation across a shared decoder stack.
When Does Recurrence Become an Algorithm? Convergence Selection in Weight-Tied Looped Transformers
Studies when a weight-tied looped transformer implements a real algorithm on group word problems. Training installs a linear computation frontier whose speed is set by the training contract, and weight tying, not expressivity, makes the model choose a serial algorithm over a parallel scan.
Looped Latent Attention: Cross-Loop KV Compression for Looped Transformers
Shows the loop-indexed KV cache of looped models traces a short low-rank trajectory per token, layer and head. Looped Latent Attention is a post-training codec that stores compact K/V latents and reconstructs loop-specific K/V only when attention reads them.
T^2MLR: Transformer with Temporal Middle-Layer Recurrence
Temporal Middle-Layer Recurrence fuses a cached middle-layer representation from the previous token into an earlier layer at the current position, letting intermediate computation persist across decoding steps with little overhead and without looping all layers.
DeepLoop: Depth Scaling for Looped Transformers
Revisits residual scaling for repeated parameter use, focusing on stable training of deeper looped networks.
Per-Token Fixed-Point Convergence in Depth-Recurrent Transformers
Measures what a depth-recurrent model trained with randomized recursion counts computes per token: states converge to per-token fixed points at very different speeds, and a training-free rule that halts each token once its output stabilizes beats a learned router.
Energy-guided Recursive Model
Energy-guided Recursive Model (ERM) augments recurrent reasoning with an explicit energy-based mechanism for selecting among candidate trajectories. It constructs task-structured Hopfield energies over valid local or global patterns, using them to rank recurrent outputs and to support energy-guided sampling such as parallel tempering.
CHASE: Cache-Hole-Adapted Skip Exit for Looped State-Space Language Models
Extends looping to state-space language models with Looped Mamba and looped hybrid Mamba-Transformer blocks, and adapts Ouro-style exit gates. Because early exits leave the recurrent SSM state without deeper updates, a cache-hole adaptation aligns continued training with skipped-state inference.
Adaptive Depth in Looped Transformers: Diagnosing Learned Halting Gates and Trajectory Readouts
Separates the two jobs of a learned halting distribution, choosing which state to read and weighting per-depth losses, on synthetic tasks and Ouro checkpoints. Fixed-prior depth supervision yields difficulty-aware trajectories, and simple post-hoc confidence readouts often match learned gates.
Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers
Trains parallel latent blocks with explicit reasoning-step supervision to compress the thought phase.
Stabilizing Extrapolation in Looped Transformers via Learned Stochastic Stopping
Traces the brittle length generalization of looped transformers to a spurious correlation between sequence length and loop count, and shows that randomizing loop counts in training, or learning a stochastic schedule with RL-Halting, stabilizes predictions across inference-time loop counts.
Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers
Studies stable deep recurrence and fixed-point convergence as a stopping signal for structured reasoning.
LoopCoder-v2: Only Loop Once for Efficient Test-Time Computation Scaling
Trains a family of 7B Parallel Loop Transformer coders with different loop counts from scratch on 18T tokens to study loop-count selection as a gain–cost trade-off between extra refinement and the positional mismatch introduced by cross-loop position offsets.
On the Residual Scaling of Looped Transformers: Stability and Transferability
Analyzes how weight sharing correlates residual updates across loop iterations and derives a 1/N residual scale for a block repeated N times. Factoring loop count from unique-layer depth improves training stability and supports learning-rate transfer across loop counts.
Dense Supervision Is Not Enough: The Readout Blind Spot in Looped Language Models
Shows that dense per-loop cross-entropy only controls state variables visible to the readout. Scale-invariant readouts such as RMSNorm hide radial scale from the loss while pre-norm residual recurrence keeps growing it, so early exits work but recurrent scale goes unchecked.
Skip a Layer or Loop It? Learning Program-of-Layers in LLMs
Finds that pretrained LLM layers can be skipped or repeated to form a per-input program-of-layers without training, and learns a lightweight network that predicts such execution programs for each input.
LoopMoE: Unifying Iterative Computation with Mixture-of-Experts for Language Modeling
A looped mixture-of-experts language model with IterAdaLN, a modulation conditioned on iteration index and per-token state to break weight-sharing symmetry, and a capacity-balancing strategy that restores the attention-to-FFN active-parameter ratio of tuned non-looped models.
Déjà View: Looping Transformers for Multi-View 3D Reconstruction
The Loop transformer in 3D
Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior
Reuses a high-level hidden state from the previous token as recurrent memory for the next token, adding a cross-token, cross-layer pathway while keeping standard attention, KV cache and one forward per token. Interleaved parallel training avoids sequential unrolling during pretraining.
Stabilizing Recurrent Dynamics for Test-Time Scalable Latent Reasoning in Looped Language Models
Addresses the peak-then-collapse behavior of looped language models under deeper recurrence by constraining latent states to approach asymptotically stable fixed points, using efficient Jacobian spectral-radius regularization with random loop sampling.
Looped Diffusion Language Models
Shows that selectively looping the early-middle layers of masked diffusion language models improves both training efficiency and final quality, and that varying the loop count at inference gives flexible compute scaling.
LT2: Linear-Time Looped Transformers
Combines looping with linear, sparse and hybrid token mixers to explore more efficient repeated computation.
Training-Free Looped Transformers
Loops a contiguous mid-stack block of a frozen checkpoint at inference with no training. Viewing a pre-norm block as a forward Euler step, it replaces one large update with smaller damped sub-steps, since naive block repetition usually hurts.
Equilibrium Reasoners: Learning Attractors Enables Scalable Reasoning
Equilibrium reasoners proposes a new view of recurrence as learning stable attractors as opposed to fixed point convergence. The new framework allows for more sophisticated test time scaling, and ultimately leads to stronger performance compared to the base TRM model.
Solve the Loop: Attractor Models for Language and Reasoning
A backbone proposes output embeddings and an attractor module refines them by solving for a fixed point, with gradients from implicit differentiation, so training memory stays constant in effective depth and iterations stop at convergence.
Simply Stabilizing the Loop via Fully Looped Transformer
Traces training instability of deep looping to gradient oscillation and residual explosion, and fixes them with two parameter-free changes: distributing inter-loop signals across all layers, and attention injection that reuses the existing attention block.
LoopUS: Recasting Pretrained LLMs into Looped Latent Refinement Models
Looped Depth Up-Scaling converts a pretrained LLM into an encoder, a looped reasoning block and a decoder, using representation-guided block decomposition, an input-dependent selective gate against drift, random deep supervision, and a confidence head for early exit.
Lattice Deduction Transformers
The lattice deduction transformer is a looped transformer that projects the token space through a lattice structure to perform efficient combinatorial reasoning.
Sparse Layers are Critical to Scaling Looped Language Models
Compares dense and MoE transformers with and without looping and finds that looped MoE models scale better than the standard baseline while dense looped models do not, because different experts fire on each pass through the shared layers.
LoopQ: Quantization for Looped Language Models
A loop-aware post-training quantization framework that keeps one shared quantized backbone while adding loop-aware activation scaling, selective transformations, cross-loop state alignment and trajectory-aware calibration to limit recursive error.
Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Models
The Memory-Efficient Looped Transformer keeps one KV cache per layer shared across reasoning loops and updated by a learnable gate, decoupling reasoning depth from memory. It is obtained from Ouro by chunk-wise two-phase post-training with interpolation and attention-aligned distillation.
Hyperloop Transformers
Organizes a looped Transformer into begin, middle and end blocks, loops only the middle block, and adds hyper-connections after each loop to expand the residual stream into matrix-valued streams at minimal extra cost.
Universal Transformers Need Memory: Depth-State Trade-offs in Adaptive Recursive Reasoning
Studies learned memory tokens as a scratchpad for a single-block Universal Transformer with ACT on Sudoku-Extreme, finding them necessary and showing that memory size and ponder depth substitute for each other at fixed accuracy.
How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models
Measures how much one recurrence is worth in equivalent unique parameters via a joint scaling law fitted on an iso-depth sweep over recurrence counts 1–8, yielding a recurrence-equivalence exponent φ = 0.46.
One Step Forward and K Steps Back: Better Reasoning with Denoising Recursion Models
Corrupts target solutions with noise like diffusion models but trains a looped refiner to undo the corruption over multiple recursive steps rather than one, giving a curriculum of intermediate states that better matches test-time refinement.
Stability and Generalization in Looped Transformers
A fixed-point framework that analyzes looped architectures along reachability, input-dependence and geometry, proving that looped networks without recall cannot achieve strong input dependence, while recall with outer normalization gives reachable, smooth, trainable fixed points.
Parcae: Scaling Laws For Stable Looped Language Models
Uses stable state injection to examine the scaling of looped Transformers with recurrence and training data.
A Mechanistic Analysis of Looped Reasoning Language Models
Analyzes latent states of looped reasoning language models and finds that each layer in the cycle often converges to its own fixed point, so the recurrent block follows a consistent cyclic trajectory and repeats feedforward-like stages of inference at every iteration.
ELT: Elastic Looped Transformers for Visual Generation
Elastic Looped Transformers for image and video generation use weight-shared blocks and Intra-Loop Self Distillation, distilling intermediate-loop students from the maximum-loop teacher in one training step, to obtain a family of any-time models from a single run.
Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformers
Studies implicit multi-hop reasoning in recurrent-depth transformers trained from scratch, covering systematic generalization over never-composed facts and depth extrapolation from up to 5-hop training to deeper compositions.
Universal YOCO for Efficient Depth Scaling
Combines the YOCO decoder-decoder design with recursion: a Universal Self-Decoder iterates with shared parameters, but only over shallow efficient-attention layers, while YOCO's constant global KV cache and linear prefill stay intact.
Thinking Deeper, Not Longer: Memory-Efficient Test-Time Reasoning with Depth-Recurrent Transformers for Compositional Generalization
A depth-recurrent Transformer that reasons by iterating a shared block instead of generating tokens, stabilized for 20+ steps by a final-output-only thinking objective, LayerScale initialization and an identity-biased gate.
LoopRPT: Reinforcement Pre-Training for Looped Language Models
Reinforcement pre-training for looped language models that reframes next-token prediction as next-token reasoning and assigns rewards directly to latent steps using an EMA teacher reference and noisy latent rollouts.
SCORE: Replacing Layer Stacking with Contractive Recurrent Depth
Replaces layer stacking with a single shared block applied iteratively through a contractive, ODE-inspired update h ← (1 − dt) · h + dt · F(h), using fixed discrete steps and ordinary backpropagation.
Adaptive Loops and Memory in Transformers: Think Harder or Know More?
Combines adaptive per-layer looping, where each block learns to iterate with a halting mechanism, with gated memory banks that add learned storage, to test whether models should think harder or know more.
SpiralFormer: Looped Transformers Can Learn Hierarchical Dependencies via Multi-Resolution Recursion
Explores recurrent computation at different sequence resolutions to change the cost of repeated processing.
LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation
Conditions shared computation on time and step size to support elastic inference budgets.
Prioritize the Process, Not Just the Outcome: Rewarding Latent Thought Trajectories Improves Reasoning in Looped Language Models
Reward Latent Thought Trajectories distributes RL reward across the full latent reasoning trajectory of a looped model instead of only the final latent state, as a drop-in replacement for GRPO.
Looping Back to Move Forward: Recursive Transformers for Efficient and Flexible Large Multimodal Models
A recursive Transformer for large multimodal models with a Recursive Connector that fuses intermediate hidden states with modality-specific projections across steps, and a Monotonic Recursion Loss that makes quality improve with recursion depth.
Recurrent-Depth VLA: Implicit Test-Time Compute Scaling of Vision-Language-Action Models via Latent Iterative Reasoning
Recurrent-Depth VLA gives vision-language-action models a weight-tied recurrent action head that refines a latent for any number of steps at constant memory, trained with truncated BPTT and stopped adaptively when the latent converges.
LoopViT: Scaling Visual ARC with Looped Transformers
A recursive vision architecture for ARC-AGI that iterates a weight-tied hybrid block of local convolutions and global attention as a latent chain of thought, with a parameter-free dynamic exit when predictive entropy shows the state has settled.
Depth-Recurrent Attention Mixtures: Giving Latent Reasoning the Attention it Deserves
Depth-recurrent attention mixtures combine sequence attention, attention along depth and sparse expert attention, relieving the fixed hidden-size bottleneck of many-step latent reasoning and decoupling scaling dimensions in depth-recurrent models.
Think-at-Hard: Selective Latent Iterations to Improve Reasoning Language Models
Allocates extra latent iterations to selected tokens, using a decider and duo-causal attention.
Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence
Converts pretrained non-recurrent language models such as TinyLlama, OLMo-2 and Llama-3.2 into depth-recurrent prelude–core–coda models, using a curriculum that gradually increases recurrences during training.
Scaling Latent Reasoning via Looped Language Models
Revisits hidden states with a shared decoder stack over multiple latent computation passes.
Parallel Loop Transformer for Efficient Test-Time Computation Scaling
Removes the sequential latency of looping with Cross-Loop Parallelism, which computes different loops for different tokens in the same pass, and controls memory by sharing the first loop's KV cache with later loops combined through gated sliding-window attention.
Efficient Parallel Samplers for Recurrent-Depth Models and Their Connection to Diffusion Language Models
Connects recurrent-depth models to diffusion language models and builds a diffusion-forcing sampler that decodes a new token every forward pass while earlier tokens' latents keep being refined in parallel through recurrence.
What Makes Looped Transformers Perform Better Than Non-Recursive Ones
Explains why looped transformers beat non-recursive ones on complex reasoning through loss-landscape geometry: recursion biases training toward steep V-shaped river valleys, where valley hopping speeds convergence along the river. This motivates SHIFT, a staged progressive training schedule.
MeSH: Memory-as-State-Highways for Recursive Transformers
Memory-as-State-Highways externalizes state management of recursive transformers into an explicit memory buffer with lightweight routers, addressing undifferentiated computation across iterations and overload of a single hidden state.
Encode, Think, Decode: Scaling test-time reasoning with recursive latent thoughts
Mid-trains a base model to iterate over a small subset of reasoning-relevant layers, chosen from interpretability evidence, while keeping architecture, parameter count, hyperparameters and data unchanged.
Less is More: Recursive Reasoning with Tiny Networks
The tiny recursive model is a looped transformer architecture with an intentionally simple architecture, yet yields strong generalization compared to previous models.
Two-Scale Latent Dynamics for Recurrent-Depth Transformers
Describes recurrent-depth iterates as small refinements within a looped block and larger drift across blocks; loop steps grow smaller and more mutually orthogonal over training. This motivates an early exit based on the second-order difference of step size.
Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation
Routes tokens through different numbers of shared recursions and studies recursion-aware caching strategies.
Latent Chain-of-Thought? Decoding the Depth-Recurrent Transformer
Probes Huginn-3.5B on arithmetic with the Logit Lens and Coda Lens to see whether interpretable latent chain-of-thought emerges across recurrent blocks.
Hierarchical Reasoning Model
A brain-inspired recurrent architecture with two interdependent modules: a slow high-level module for abstract planning and a fast low-level module for detailed computation, solving sequential reasoning tasks in one forward pass without intermediate supervision.
PonderLM: Pretraining Language Models to Ponder in Continuous Space
Teaches language models to ponder in continuous space: within one token step, the model feeds back the probability-weighted sum of token embeddings and runs another forward pass, learned through ordinary self-supervised pretraining.
To CoT or To Loop? A Formal Comparison Between Chain-of-Thought and Looped Transformers
A formal comparison of chain-of-thought and looped transformers: looped models efficiently simulate parallel deterministic computation over DAGs, while stochastic CoT excels at approximate inference for self-reducible compositional problems.
Reasoning with Latent Thoughts: On the Power of Looped Transformers
Argues that many reasoning problems need depth rather than parameters: a k-layer transformer looped L times nearly matches a kL-layer model on synthetic reasoning, backed by theory that iterative algorithms map onto looped models.
Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinking
Treats layer computation as implicit thinking steps: Adaptive Token Routing sends critical tokens through extra iterations, Residual Thinking Connections refine them, and Thinking Step Encoding marks each phase.
Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
A recurrent core refines hidden states between a prelude and a coda, scaling latent computation at inference time.
Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA
Converts pretrained Transformers into recursive models, relaxing sharing with depth-specific low-rank adaptations.
On Expressive Power of Looped Transformers: Theoretical Analysis and Enhancement via Timestep Encoding
Establishes approximation rates of looped transformers for sequence-to-sequence functions via a modulus of continuity, revealing a looped-specific limitation that per-loop scaling parameters conditioned on timestep encoding can remove.
Looped Transformers for Length Generalization
Shows that looped transformers with an adaptive number of steps generalize to unseen input lengths on tasks with known iterative solutions built from repeated RASP-L operations, using a dedicated training algorithm.
MoEUT: Mixture-of-Experts Universal Transformers
Combines the Universal Transformer with mixture-of-experts layers to study shared depth and sparse capacity.
Looped Transformers are Better at Learning Learning Algorithms
Studies how looping encourages Transformers to learn iterative learning algorithms in context.
CoTFormer: A Chain-of-Thought Driven Architecture with Budget-Adaptive Computation Cost at Inference
Observes that chain-of-thought resembles reapplying a model several times, except that later passes attend to earlier passes' tokens. CoTFormer mimics this at the token level and pairs it with token-wise variable depth chosen by a compute-adaptive router.
Deep Equilibrium Object Detection
Deep Equilibrium Model
Looped Transformers as Programmable Computers
Programs a fixed-weight transformer and places it in a loop so the input sequence acts as a punchcard of instructions and memory, emulating a small instruction-set computer with a 13-layer looped encoder.
Sliced Recursive Transformer
This paper presents a neat yet effective recursive operation on transformers that can improve parameter utilization without involving additional parameters. This is achieved by sharing weights across the depth of transformer networks. The proposed method can obtain a substantial gain (~2%) simply using a naive recursive operation, requires no special or sophisticated knowledge for designing principles of networks, and introduces minimal computational overhead to the training procedure. To reduce the additional computation caused by recursive operation while maintaining the superior accuracy, we propose an approximation method through multiple sliced group self-attentions across recursive layers, which can reduce the cost consumption by 10~30% with minimal performance loss.
PonderNet: Learning to Ponder
Learns end-to-end how many computational steps to take, modeling halting as a probabilistic process and trading off prediction accuracy, computational cost and generalization.
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Reduces BERT's parameters with factorized embeddings and cross-layer parameter sharing, so every layer reuses the same weights, plus an inter-sentence coherence loss.
Deep Equilibrium Models
Directly solves for the fixed point that a weight-tied layer would reach if iterated infinitely, using root-finding forward and implicit differentiation backward, so memory is constant in effective depth.
Universal Transformers
Shares a transformation across depth, with adaptive computation time allowing each position to halt independently.
Adaptive Computation Time for Recurrent Neural Networks
The original algorithm that lets a recurrent network learn how many steps to take per input with a deterministic, differentiable halting mechanism and a ponder cost.
No works match these filters. Try fewer topics or clear the search.
No works match these filters. Try fewer topics or clear the search.
Pick up to three works with the Compare box in Explore.
Nothing selected yet. Tick “Compare” on up to three works in Explore to see their recurrence, depth control and KV strategy here.
IAAI