Capability Elicitation vs Deployment: A Gap Analysis

Abstract

Capability evaluations performed under elicitation conditions — using prompting strategies designed to extract maximum capability — produce capability estimates that are systematically higher than what the model exhibits in deployment. We quantify the gap on six lab-internal capability benchmarks and find median gaps of 22-44%. We argue that the gap is structural rather than a measurement artifact, propose a deployment-conditioned evaluation protocol that closes most of it, and discuss the implications for RSI-axis stopping-condition design.

Index metadata

Cell

turing-11

Compute

26 H100-days

Status

Open release · companion to MUR (25/05)

Code

github.com/alphabell-labs/ab-elicit

DOI

10.48550/arXiv.2508.02315

arXiv

arXiv:2508.02315

What this paper is part of

This index entry is part of the Recursive self-improvement research axis. The producing cell — turing-11 — collaborates with adjacent cells listed in the cell directory. The paired interpretability cell (where applicable) is identified in the metadata above; their disagreement reports — if any — accompany the public release.

How to read this

If you want to use the result: the code (where available) is at https://github.com/alphabell-labs/ab-elicit; the dataset is at TBD when one is released. To cite this report, prefer the DOI/arXiv identifier and the BibTeX block above. To discuss this with the producing cell, contact the lab with the index entry slug capability-elicitation-deployment-gap.

Limitations

Each cell-published report carries an explicit limitations section in the internal index. We do not paraphrase it here. Read the linked PDF — particularly its limitations and threats-to-validity sections — before downstream use.