Read this one with its authorship in view. It is a single-author arXiv preprint by Benjamin Tannenbaum of Aiso, a company selling AI-visibility tracking.
It has not been peer reviewed. The self-interest runs one way and belongs next to the result.
Aiso sells session-level conversation tracking, which is the product this finding argues buyers need.
We are carrying it because it measures a mechanism rather than ranking who is winning. It also repeatedly declines claims it could have made.
The question is narrow. Holding the last thing a user typed constant, how much do the earlier turns change the answer?
The experiment is paired at the conversation level across 180 English multi-turn conversations. Half come from Aiso's own governed corpus and half from the public PRISM dataset.
The final user message, the answering model and the output format stay fixed. Only the representation of the preceding turns changes.
Three answers are generated per case. One comes from the full role-labelled conversation and one from the final message alone.
The third comes from the final message plus a reconstruction of the earlier turns capped at 160 words.
One design detail carries a lot of weight. The reconstruction model never sees the final message.
So it cannot write a convenient summary backwards from the endpoint.
A separate judge model then scores all three under randomized labels. A difference counts only if it could change what the user does.
Four kinds of change qualify. Two are a different recommendation or factual bottom line, and different compliance with an explicit constraint.
The other two are a different requested deliverable, and different clarification or refusal behaviour. Wording, tone, organisation and amount of detail are excluded by rule.
Removing the conversation changed the answer materially in 44.7% of cases, weighted to the eligible cohorts. The 95% confidence interval runs from 33.8% to 56.1%, and the unweighted paired rate is 53.3%.
Full-conversation answers also scored better. The weighted mean was 3.80 against 3.31 on a 0 to 4 request-satisfaction scale.
The direction is not universal, and the paper says so. Full context scored higher in 36.2% of weighted cases and lower in 6.0%, with the rest tied.
The category breakdown is the part to read closely, with one caution about how it is reported.
The results table lists a changed recommendation or conclusion at 26.7% (17.0% to 36.9%). Changed clarification or refusal behaviour is at 14.4%, and changed constraint compliance at 10.6%.
The table presents those in the same column as the 44.7%, as rates over all cases. The body text describes them as shares of the differing pairs.
Read them as the paper's own category breakdown rather than as three independent rates. Note the paper's own line that the categories can co-occur.
The compressed-prefix condition produced the most useful result here. A 160-word summary of the earlier turns closed almost the whole satisfaction gap, to 0.01 points.
It still left 30.8% of answers materially different from the full-conversation answer.
So a state summary can support an equally good answer while changing which reasonable answer the model picks. Average quality and answer identity are not the same measurement.
The paper draws the consequence for this industry itself: "For a visibility tracker, that can mean a different cited brand."
There is also no safe subset of self-contained prompts. Cases were sorted by how context-dependent the final message looked.
Isolation still changed 31.0% of the least dependent answers. The middle band saw 56.1% and the highest 48.4%.
The companion paper from ten days earlier established the other half of this. It covered 670 commercial and 7,463 public conversations.
In 50.3% and 44.8% of conversations, an explicit constraint, alternative, correction or evidence requirement appeared before the final message. It did not appear in the final message itself.
The final prompt reproduced the full set of detected request dimensions in only about 26%.
The limits are large enough that this is direction about a measurement error. It is not a rate to quote as a population figure.
One answer model was requested, gpt-5.4-mini, with gpt-5.5 as judge. Both ran through a research command-line runner rather than a native chat product.
There was one generation per condition.
The judge is automated rather than human. The paper names that as its own most important replication target.
Medical, legal, financial and other high-stakes conversations were excluded from the sample entirely.
Half the sample is Aiso's own corpus, which cannot be redistributed. The weighting leans the pooled estimate towards the public half.
The commercial half showed a far larger effect, 68.5% against PRISM's 35.4%.
The scope is preceding turns inside one conversation. It does not test persistent memory across separate sessions.
The paper is explicit that its data cannot observe such a system.
Here is the practical reading for anyone buying this kind of data. A tool reporting whether assistants mention you is measuring endpoint strings unless it tells you otherwise.
The paper asks that such prompt sets be labeled as endpoint-message evaluations. Its conclusion puts it plainly: "Replaying endpoint strings alone measures a different task."
Read it beside the IAB framework published the same day. Its decision-grade test asks about query volume and prompt-type coverage.
This paper is about what a query is in the first place.
None of it belongs in the measurement table above, whose rows count real crawling and real referrals.
