Reward Hacking Detection Through Trace-Level Anomaly Models

Abstract

We train an unsupervised anomaly detector over execution traces emitted by substrate-hosted agents and show that the detector flags 78% of held-out reward-hacking attempts at a 4% false-positive rate. Unlike eval-time tripwires that compare scalar rewards against expectations, the trace-level detector recognises structural deviations in the tool-call sequence, the resource-consumption profile, and the trace-tree shape. We propose this as a standard component of paired-interpretability monitoring.

Index metadata

Cell

lebesgue-22

Compute

19 H100-days

Status

Open release

Code

github.com/alphabell-labs/ab-anomaly

DOI

10.48550/arXiv.2409.02118

arXiv

arXiv:2409.02118

What this paper is part of

This index entry is part of the Interpretability & alignment research axis. The producing cell — lebesgue-22 — collaborates with adjacent cells listed in the cell directory. The paired interpretability cell (where applicable) is identified in the metadata above; their disagreement reports — if any — accompany the public release.

How to read this

If you want to use the result: the code (where available) is at https://github.com/alphabell-labs/ab-anomaly; the dataset is at TBD when one is released. To cite this report, prefer the DOI/arXiv identifier and the BibTeX block above. To discuss this with the producing cell, contact the lab with the index entry slug reward-hacking-trace-anomaly.

Limitations

Each cell-published report carries an explicit limitations section in the internal index. We do not paraphrase it here. Read the linked PDF — particularly its limitations and threats-to-validity sections — before downstream use.