Can Agent Improvements Survive Repeated Selection?

Abstract

An agent that repeatedly modifies its own harness can overfit the tests used to select successors, even when its evaluator is read-only. We propose a promotion protocol that evaluates one nominated candidate per generation on a fresh, sealed batch of tasks and imposes capability-specific noninferiority requirements. We will compare the protocol with static regression gates under equal search and test budgets, measuring whether apparent improvements transfer to a separate terminal test and how many useful edits are lost to conservative gating.

Keywords