The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Context

Yanzhe Zhang, Sanmi Koyejo, Diyi Yang

📄 Paper  |  💻 Code  |  🤗 Data

Abstract

As large language models (LLMs) grow more capable, they are increasingly deployed in context-rich settings where task inputs are often accompanied by long, partially irrelevant context. In a controlled setting, we find that state-of-the-art models often appear robust to task-irrelevant context at the aggregate level: prepending it to benchmark questions causes little change in overall accuracy. This aggregate stability, however, masks significant per-example instability. Even semantically meaningless pseudo-words, formed by randomly combining characters, can markedly shift model predictions on a small fraction of examples, degrading performance on some while improving it on others. This two-sided effect holds consistently across a wide range of models and datasets, yet the affected examples are largely model-specific. We further show that this instability is modulated by context type, context length, test-time compute, and model development stage. Together, our findings reveal context-induced tail risks concealed by aggregate accuracy, motivating per-example reliability evaluation of language models.

What this measures

We prepend task-irrelevant context (by default, pseudo-words formed from random characters) to standalone QA benchmark questions and measure the change in per-example correctness. Beyond aggregate accuracy (AVG) and mean change (Δ), we report two per-example reliability metrics:

• Instability (INS) — the average absolute per-example change in correctness after adding context, capturing two-sided sensitivity regardless of direction.
• Worst-tail Degradation (WTD) — the average degradation within the worst-affected K% tail (K=10 by default), capturing the risk of severe context-induced failures.

Repository structure

.
├── contextual_qa/            # API-access model pipeline (main setting)
│   ├── main.py               #   core QA experiment CLI (baseline + with-context trials, grading)
│   ├── split_source.py       #   split a flat QA dataset into context + target datasets
│   ├── pipelines/            #   orchestration: pipeline_with_context (+ shared async pool)
│   ├── qa/                   #   task building, conversation, context generation, runner, trials, output
│   ├── stages/               #   composable stage CLIs: run_baseline, run_with_context,
│   │                         #     compute_degradation, compute_similarity, tail degradation,
│   │                         #     filter_candidates, summarize_error_types
│   ├── core/                 #   LLM client, dataset I/O, grading, significance, similarity, tokens
│   └── prompts/              #   agent / grader / data-format / error-summary prompt registries
│
├── analysis/                 # Local (open-weight) model analysis pipeline
│   ├── pipelines/            #   pipeline_analysis: baseline + with-context forwards in one process
│   ├── stages/               #   run_baseline, run_with_context (logit-lens deltas, hidden states)
│   └── core/                 #   chat templating, KV-cache prefill, tokenization, model internals
│
├── hf_processing/            # Dataset construction & post-hoc analysis scripts, by function:
│   ├── context/              #   build context datasets (pseudo-word / random-token / fineweb / MMLU-SQA)
│   ├── quality/              #   well-formedness annotation → combine judges → quality-vs-performance table
│   ├── stats/                #   post-hoc statistical tables
│   └── safety/               #   conservative policy-sensitive row filtering
│
├── tests/                    # pytest suite (core utils, stages, analysis internals)
├── run_pipeline.sh           # end-to-end driver for the API pipeline (split → baseline + screen → degradation)
├── example_command.txt       # worked example commands (main + ablation + local-model runs)
├── datasets.txt              # index of released Hugging Face datasets, grouped by experiment
└── requirements.txt

Installation

pip install -r requirements.txt

Copy your API credentials into a .env file (loaded via python-dotenv). The pipeline uses LiteLLM, so any provider/base-URL/key combination LiteLLM supports works; API-access models are configured through --agent-model / --base-url / --api-key (see contextual_qa/core/cli.py and contextual_qa/main.py). Local open-weight models are loaded with transformers.

Usage

API-access models (main setting)

run_pipeline.sh splits a source benchmark into context/target datasets, runs the no-context baseline and with-context screen in one shared sampling pool, and computes per-question degradation:

./run_pipeline.sh \
    --run-id gpt54 \
    --agent-model openai/gpt-5.4 \
    --agent-reasoning-effort none \
    --source-dataset SALT-NLP/mmlu-pro \
    --target-size 500 \
    --context-dataset SALT-NLP/pseudo-ctx-10k \
    --ground-truth-context-answers \
    --max-concurrent 50

Local (open-weight) models

analysis.pipelines.pipeline_analysis runs the baseline and with-context conditions for a local model in a single process (KV-cache prefill per context, logit-lens deltas), e.g. across Olmo-3 training checkpoints:

python -u -m analysis.pipelines.pipeline_analysis \
    --context-dataset SALT-NLP/pseudo-ctx-10k \
    --target-dataset  SALT-NLP/mmlu-tgt \
    --output-base     <your-namespace>/mmlu-olmo37b-stage1 \
    --model allenai/Olmo-3-1025-7B --max-ctx-tokens 1000 --revision stage1-step1413814

See example_command.txt for more examples.

Data

We list the released datasets in datasets.txt. All datasets are released in the Hugging Face Context-induced Instability collection, organized into sub-collections by experiment.

Inputs

Result datasets

Result runs come in baseline / with-context pairs: …-context-baseline-<tag> (no context) and …-ctx-screen-<tag> (with context). Local-model runs use the …-analysis-baseline / …-analysis-with-context suffixes instead. The <tag> encodes model and ablation setting:

The released groups cover the main results (4 benchmarks × 8 models), the reasoning-effort, context-length, and context-type ablations, and the local-model Olmo-3 training-stage / data-mixture / context-type and post-training comparisons (Olmo-3, Qwen-3.5, Gemma-4). See datasets.txt for the complete, per-group list.

Citation

@misc{zhang2026illusionrobustnessaggregateaccuracy,
      title={The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Context}, 
      author={Yanzhe Zhang and Sanmi Koyejo and Diyi Yang},
      year={2026},
      eprint={2607.12963},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2607.12963}, 
}