Echoes as Anchors: Probabilistic Costs and Attention Refocusing in LLM Reasoning
Large reasoning models often begin a long chain-of-thought by restating the question. We call this the Echo of Prompt, and ask what it is worth. Putting the echo back into failed reasoning traces raises accuracy for reasoning models but not for a base model; later tokens in correct traces attend more to the echo; and re-injecting the question mid-trace is a training-free test-time intervention that beats generic thinking tokens under the same budget.
So, 2 + 1 = 3 bolts in total. #### 3
01 · The phenomenon
Reasoning models restate the question before they think
Test-time reasoning in large reasoning models (LRMs) is usually improved by spending more compute: sampling many chains (self-consistency, parallel thinking), inserting generic “thinking tokens”, or presenting the question twice in the input, as in re-reading (RE2). None of these explains a behavior that many LRMs show on their own: at the head of a long chain-of-thought, the model rephrases the user’s question before it starts solving.
We detect this Echo of Prompt (EOP) with a small MLP probe over the opening of each trace (test accuracy 0.912, AUROC 0.963). On GSM8K it is the common case, not a quirk of one model. Traces that contain an echo are also more accurate, 63.8% (n = 985) vs. 57.2% (n = 334); this comparison is descriptive and not matched for difficulty.
02 · Causal test
Putting the echo back repairs failed traces
Correlation alone cannot say whether the echo does any work. So we intervene. We take GSM8K traces that ended with a wrong answer, cut each at 50% of its tokens, and resume generation two ways under matched prefixes, decoding and seeds: directly, or after inserting a short echo (“now I need to look back at the question again:” plus the question).
For the two reasoning models, the echo branch recovers far more of the failed problems. The base model, which has no reasoning post-training, does not change at all. In the echo branch the model revisits the original quantities and constraints; the echo-free branch continues the drift that produced the error.
03 · Mechanism
Later tokens attend back to the echo
If the echo is an anchor, the answer should lean on it. We measure attention from answer tokens to the opening span of the trace, whose length is set by the probe’s echo boundary, and compare correct with wrong traces on DeepSeek-R1-Distill-Llama-8B over all 1,319 GSM8K problems. Correct traces attend more to that span at every layer group: 14.45% vs. 11.58% in layers 7–18, and 13.69% vs. 10.41% at the last layer.
Two details keep this honest. The separation is about equally strong in early, middle and late layers (Cohen’s d 0.820 / 0.832 / 0.828), so “middle layers” describes where the raw gap peaks rather than a resolved localization. And attention from answers to the raw question separates correct from wrong traces much more weakly (d 0.18–0.48).
A likelihood view points the same way. Casting echo removal as rejection-based conditioning gives the Echo Likelihood Gap Δℒ, a computable proxy for what the echo buys. Correct traces have a slightly higher gap than wrong ones (2.5231 vs. 2.4421 nats/token), significant in a logistic regression with echo length as a covariate (β = 0.24, p ≈ 0.022); the suffix-only gap runs the other way.
04 · Using the echo
A training-free test-time intervention, and a data format
Echoic Prompting
Echoic Prompting re-grounds the model mid-trace: after an initial stretch of reasoning, it appends a reminder to look back at the question, followed by the question itself. We compare it with thinking-token based test-time scaling (TTTS), which inserts generic tokens such as “So” and “Hmm”, under identical decoding settings and token budgets.
Echo-Distilled SFT
Echo-Distilled SFT (ED-SFT) trains on traces that open with an echo, against the same distilled traces with the echo removed (normal SFT). It wins most cells, especially for the base model, but not all of them: on DeepSeek-R1-Distill-Llama-8B it loses on GSM8K strict exact match. Echo-bearing training sequences are also longer (175 vs. 136 tokens), so the comparison is not token-matched.
| Model / training | GSM8K strict | GSM8K flex | MathQA | Hendrycks-MATH |
|---|---|---|---|---|
| Qwen3-8B-Base | ||||
| + ED-SFT | 94.2 | 94.2 | 58.8 | 10.0 |
| + normal SFT | 90.8 | 90.8 | 47.0 | 1.8 |
| Qwen3-8B | ||||
| + ED-SFT | 93.1 | 93.4 | 53.7 | 6.1 |
| + normal SFT | 90.3 | 90.1 | 51.8 | 5.0 |
| DeepSeek-R1-Distill-Llama-8B | ||||
| + ED-SFT | 78.2 | 79.7 | 34.8 | 3.0 |
| + normal SFT | 80.5 | 79.5 | 31.4 | 0.76 |
Exact match (%), paper Table 5. Highlighted: the better of the two SFT variants in each cell.
05 · Context
Where this fits in LLM reasoning research
Large reasoning models and long chain-of-thought
Measures how often long reasoning traces open by restating the question, and tests causally whether that restatement matters.
Test-time compute and thinking tokens
Echoic Prompting is a decode-time intervention compared with thinking-token based test-time scaling (TTTS) under identical budgets.
Re-reading and prompt repetition
RE2 and EchoPrompt repeat or rephrase the question in the input. This work studies the restatements reasoning models produce on their own inside the trace. Prompt repetition for non-reasoning models (Leviathan et al., Dec 2025) is independent, later work.
Attention analysis of reasoning traces
Layer-wise attention from answers to the echo span, at the attention and behavior level rather than circuit-level reverse engineering; the reinsertion experiment provides the causal evidence.
Reasoning SFT data format
Echo-Distilled SFT tests whether an echo-then-reason opening in distilled training traces helps, against the same traces without it.
Grounding in long traces
Positional-bias work such as “lost in the middle” motivates the anchoring account. It is not tested here: all experiments use short math traces.
Scope. The probabilistic and attention analyses use one model and one benchmark (DeepSeek-R1-Distill-Llama-8B, GSM8K, 1,319 traces). Echo presence comes from the MLP probe, so about 9% detector error propagates into group statistics. All models are 8B and all benchmarks are math. Per-result scope and limits are on the evidence page.
Concepts
- Echo of Prompt (EOP)
- The model repeats or rephrases the user’s question at the start of its reasoning.
- Echo Likelihood Gap (Δℒ)
- A computable proxy for the probabilistic cost of removing the echo, obtained by casting echo removal as rejection-based conditioning.
- Attention refocusing
- The account that later reasoning and answer tokens attend back to the echoed restatement of the problem.
- Echoic Prompting (EP)
- A training-free inference strategy that re-grounds the model by re-introducing the question mid-trace.
- Echo-Distilled SFT (ED-SFT)
- Supervised fine-tuning on distilled traces that follow an echo-then-reason pattern.
- Reasoning probe
- An MLP classifier that detects the echo and its boundary in a model’s thinking trace.
06 · Resources
Code, data and metadata
The repository contains Python code for data processing, evaluation, two-stage Echoic Prompting, MLP probe training, and the attention and probabilistic analyses, to support reproduction and follow-up work.
07 · Citation
Cite this work
@inproceedings{echoes_iclr26,
title={Echoes as Anchors: Probabilistic Costs and Attention Refocusing in LLM Reasoning},
author={Zhuoyuan Hao and Zhuo Li and Wu Li and Fangming Liu and Min Zhang and Jing Li},
booktitle={The Fourteenth International Conference on Learning Representations (ICLR)},
year={2026}
}