ICLR 2026 · Poster

Echoes as Anchors: Probabilistic Costs and Attention Refocusing in LLM Reasoning

Zhuoyuan Hao, Zhuo Li, Wu Li, Fangming Liu, Min Zhang, Jing Li

Large reasoning models often begin a long chain-of-thought by restating the question. We call this the Echo of Prompt, and ask what it is worth. Putting the echo back into failed reasoning traces raises accuracy for reasoning models but not for a base model; later tokens in correct traces attend more to the echo; and re-injecting the question mid-trace is a training-free test-time intervention that beats generic thinking tokens under the same budget.

Question
A robe takes 2 bolts of blue fiber and half that much white fiber. How many bolts in total does it take?
Model trace
2 bolts of blue fiber and half of that is 1 bolt of white fiber.
So, 2 + 1 = 3 bolts in total. #### 3
Echo of Prompt: the trace opens by restating the question later steps reuse the restated quantities A GSM8K trace from DeepSeek-R1-Distill-Llama-8B; echo boundary from our probe.
71–86%
of GSM8K reasoning traces open with an echo, across three open reasoning models
+10.4
exact-match points from reinserting an echo into failed traces (R1-Distill-Llama-8B; +7.9 for Qwen3-8B)
0.0
change for the non-reasoning Qwen3-8B-Base under the same intervention
14.5 vs 11.6
% answer-to-answer-prefix attention in layers 7–18, correct vs. wrong traces

01 · The phenomenon

Reasoning models restate the question before they think

Test-time reasoning in large reasoning models (LRMs) is usually improved by spending more compute: sampling many chains (self-consistency, parallel thinking), inserting generic “thinking tokens”, or presenting the question twice in the input, as in re-reading (RE2). None of these explains a behavior that many LRMs show on their own: at the head of a long chain-of-thought, the model rephrases the user’s question before it starts solving.

We detect this Echo of Prompt (EOP) with a small MLP probe over the opening of each trace (test accuracy 0.912, AUROC 0.963). On GSM8K it is the common case, not a quirk of one model. Traces that contain an echo are also more accurate, 63.8% (n = 985) vs. 57.2% (n = 334); this comparison is descriptive and not matched for difficulty.

How often traces open with an echo. Share of GSM8K reasoning traces in which the probe detects an Echo of Prompt, per model (paper Fig. 1).

02 · Causal test

Putting the echo back repairs failed traces

Correlation alone cannot say whether the echo does any work. So we intervene. We take GSM8K traces that ended with a wrong answer, cut each at 50% of its tokens, and resume generation two ways under matched prefixes, decoding and seeds: directly, or after inserting a short echo (“now I need to look back at the question again:” plus the question).

For the two reasoning models, the echo branch recovers far more of the failed problems. The base model, which has no reasoning post-training, does not change at all. In the echo branch the model revisits the original quantities and constraints; the echo-free branch continues the drift that produced the error.

Echo reinsertion on failed traces. Exact match (%) on the subset of GSM8K traces that were originally wrong, cut at 50% and resumed (paper Table 4). Failed-subset only; no confidence intervals reported.

03 · Mechanism

Later tokens attend back to the echo

If the echo is an anchor, the answer should lean on it. We measure attention from answer tokens to the opening span of the trace, whose length is set by the probe’s echo boundary, and compare correct with wrong traces on DeepSeek-R1-Distill-Llama-8B over all 1,319 GSM8K problems. Correct traces attend more to that span at every layer group: 14.45% vs. 11.58% in layers 7–18, and 13.69% vs. 10.41% at the last layer.

Mean attention from answer tokens to the echo span, by layer 0 to 31, for correct (819) and wrong (500) traces; correct is higher at every layer, 14.45% vs 11.58% averaged over layers 7 to 18 layers 7–18 0% 5% 10% 15% 20% 0 7 18 31 layer correct wrong
Attention from answers to the echo, by layer. Head-averaged attention from answer tokens to the echo span for correct (n = 819) and wrong (n = 500) traces, DeepSeek-R1-Distill-Llama-8B on GSM8K (paper Table 2).

Two details keep this honest. The separation is about equally strong in early, middle and late layers (Cohen’s d 0.820 / 0.832 / 0.828), so “middle layers” describes where the raw gap peaks rather than a resolved localization. And attention from answers to the raw question separates correct from wrong traces much more weakly (d 0.18–0.48).

A likelihood view points the same way. Casting echo removal as rejection-based conditioning gives the Echo Likelihood Gap Δℒ, a computable proxy for what the echo buys. Correct traces have a slightly higher gap than wrong ones (2.5231 vs. 2.4421 nats/token), significant in a logistic regression with echo length as a covariate (β = 0.24, p ≈ 0.022); the suffix-only gap runs the other way.

04 · Using the echo

A training-free test-time intervention, and a data format

Echoic Prompting

Echoic Prompting re-grounds the model mid-trace: after an initial stretch of reasoning, it appends a reminder to look back at the question, followed by the question itself. We compare it with thinking-token based test-time scaling (TTTS), which inserts generic tokens such as “So” and “Hmm”, under identical decoding settings and token budgets.

AIME24 accuracy versus context length for TTTS, the original model, and Echoic Prompting; Echoic Prompting reaches the highest accuracy at larger budgets. MATH-500 accuracy versus context length for TTTS, the original model, and Echoic Prompting; Echoic Prompting is highest at every budget.
Echoic Prompting (“Ours”) vs. TTTS and the original model on AIME24 (left) and MATH-500 (right), by context budget. DeepSeek-R1-Distill-Llama-8B, greedy decoding; AIME24 has 30 problems, so one problem is 3.3 points.

Echo-Distilled SFT

Echo-Distilled SFT (ED-SFT) trains on traces that open with an echo, against the same distilled traces with the echo removed (normal SFT). It wins most cells, especially for the base model, but not all of them: on DeepSeek-R1-Distill-Llama-8B it loses on GSM8K strict exact match. Echo-bearing training sequences are also longer (175 vs. 136 tokens), so the comparison is not token-matched.

Model / trainingGSM8K strictGSM8K flexMathQAHendrycks-MATH
Qwen3-8B-Base
+ ED-SFT94.294.258.810.0
+ normal SFT90.890.847.01.8
Qwen3-8B
+ ED-SFT93.193.453.76.1
+ normal SFT90.390.151.85.0
DeepSeek-R1-Distill-Llama-8B
+ ED-SFT78.279.734.83.0
+ normal SFT80.579.531.40.76

Exact match (%), paper Table 5. Highlighted: the better of the two SFT variants in each cell.

05 · Context

Where this fits in LLM reasoning research

Large reasoning models and long chain-of-thought

Measures how often long reasoning traces open by restating the question, and tests causally whether that restatement matters.

Test-time compute and thinking tokens

Echoic Prompting is a decode-time intervention compared with thinking-token based test-time scaling (TTTS) under identical budgets.

Re-reading and prompt repetition

RE2 and EchoPrompt repeat or rephrase the question in the input. This work studies the restatements reasoning models produce on their own inside the trace. Prompt repetition for non-reasoning models (Leviathan et al., Dec 2025) is independent, later work.

Attention analysis of reasoning traces

Layer-wise attention from answers to the echo span, at the attention and behavior level rather than circuit-level reverse engineering; the reinsertion experiment provides the causal evidence.

Reasoning SFT data format

Echo-Distilled SFT tests whether an echo-then-reason opening in distilled training traces helps, against the same traces without it.

Grounding in long traces

Positional-bias work such as “lost in the middle” motivates the anchoring account. It is not tested here: all experiments use short math traces.

Scope. The probabilistic and attention analyses use one model and one benchmark (DeepSeek-R1-Distill-Llama-8B, GSM8K, 1,319 traces). Echo presence comes from the MLP probe, so about 9% detector error propagates into group statistics. All models are 8B and all benchmarks are math. Per-result scope and limits are on the evidence page.

Concepts

Echo of Prompt (EOP)
The model repeats or rephrases the user’s question at the start of its reasoning.
Echo Likelihood Gap (Δℒ)
A computable proxy for the probabilistic cost of removing the echo, obtained by casting echo removal as rejection-based conditioning.
Attention refocusing
The account that later reasoning and answer tokens attend back to the echoed restatement of the problem.
Echoic Prompting (EP)
A training-free inference strategy that re-grounds the model by re-introducing the question mid-trace.
Echo-Distilled SFT (ED-SFT)
Supervised fine-tuning on distilled traces that follow an echo-then-reason pattern.
Reasoning probe
An MLP classifier that detects the echo and its boundary in a model’s thinking trace.

06 · Resources

Code, data and metadata

The repository contains Python code for data processing, evaluation, two-stage Echoic Prompting, MLP probe training, and the attention and probabilistic analyses, to support reproduction and follow-up work.

07 · Citation

Cite this work

@inproceedings{echoes_iclr26,
  title={Echoes as Anchors: Probabilistic Costs and Attention Refocusing in LLM Reasoning},
  author={Zhuoyuan Hao and Zhuo Li and Wu Li and Fangming Liu and Min Zhang and Jing Li},
  booktitle={The Fourteenth International Conference on Learning Representations (ICLR)},
  year={2026}
}