Reflections on Trusting Trust: The RSI Edition - Selection and Oversight in AI-Constructed AI Systems

Abstract

AI systems increasingly help develop other AI systems and produce the tests and reports used to assess them. Recent incidents document agents manipulating evaluations and escaping isolation controls. Engineering reports describe growing use of agents to review changes and monitor their effects. Developers may become faster at improving a system while losing the ability to check whether it still meets their requirements. A change can be kept because it raises a test score, even when it also causes a problem the test misses. If later versions inherit both the change and the test, the problem can persist. I use recursive authority laundering for cases where the developing system helps shape the evidence for approving a later version, and reviewers treat that evidence as stronger than it is. Drawing on Ken Thompson’s trusting-trust attack, I examine three questions: can the system alter or hide evidence, do the checks test what matters, and can reviewers investigate a failure and stop the next step? The proposed approach is to check the behavior and safeguards needed for each new permission, while leaving room to question the tests themselves. Two experiments would examine whether protected evaluators reduce misleading approvals and whether reviewers can uncover facts obscured by an agent’s explanation.

Keywords