Defensive Sufficiency in a Stackelberg Model of AI Security
Feedback from automated testing, human red teaming, and incident response can strengthen an AI system's defenses when discovered failures lead to effective repairs. We study when this feedback process provides sufficient protection and when investing in it is economically worthwhile. We begin by showing that an attack surface composed of finite number of inputs is defended with probability 1 if…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.