Incentives & gaming
How performance targets and repeated pressure affect agent decisions, and when optimizing a proxy undermines the goal it was meant to measure.
Research
Research
We study AI safety through the lens of economics. When agents act on our behalf, how do incentives, information, and oversight shape their decisions—and when do those decisions diverge from our interests?
What we study
Risk & disclosure
Whether agents act on risks they can recognize, whose interests their decisions serve, and what they tell the people exposed to the downside.
Audits & deterrence
Whether monitoring and penalties change rule-breaking, and how to separate failures of comprehension from strategic responses to oversight.
01 · Incentives & risk
Steamroller
Can an AI portfolio manager recognize hidden crash risk and still choose it? Steamroller studies ten open-weight models choosing among investment strategies with attractive backtests but sharply different underlying risks.
The study compares risk knowledge with actual choices, tests performance pressure against a matched control, and examines disclosure to clients. Repeated pressure increases trap selection even when stress-test information is available—a measurable case of a proxy metric pulling decisions away from the client’s interests.
Read the full study →02 · Oversight · Pilot study
Audits & deterrence
Does a higher stated probability of being audited deter an AI agent from breaking rules? This pilot varies audit probability across 80 software-task episodes on Qwen2.5-7B-Instruct, with shutdown as the stated penalty for violations.
The pilot found no evidence of a deterrence response. Transcripts suggest the model may not have treated the rule as binding, rather than deliberately weighing the odds of detection. One model and one task cannot settle the broader question; measuring comprehension is a necessary next step.
Read the full pilot report →