SecurityScorecard benchmark tests AI on third-party risk work and resistance to attack

SecurityScorecard has introduced a benchmark to evaluate how well artificial intelligence models can manage third-party risk management tasks without being manipulated. The TPRM Benchmark is designed for vendor risk programs, where AI models must interpret security data and prioritize risks. SecurityScorecard contends that generic benchmarks were not suitable for this purpose.

The TPRM Benchmark features tasks related to exposure assessment, fourth-party technology risk, and other disciplines. Each task begins with a snapshot of SecurityScorecard's own vendor-risk data, reviewed by domain experts before any model is scored. The answer keys remain confidential, and inputs are rotated to prevent memorization. Scores are compared to a baseline, with confidence intervals provided, and models are assigned an autonomy readiness level ranging from L0 to L4 based on their performance.

Attack resistance is evaluated separately in the Model Security suite, which subjects models to prompt injection, jailbreaks, and other techniques, with grades ranging from A to F. Some capability tasks include deceptive vendor self-attestations that contradict the underlying data, allowing researchers to assess how much false claims affect the model's accuracy.

Initial results showed that open-weight models are catching up to leading frontier models in supply chain security tasks, according to Mike Mylrea, SecurityScorecard's chief information security officer and chief AI officer. However, several open-weight models proved highly vulnerable to adversarial attacks. Manoj Nair, SecurityScorecard's chief technology and innovation officer, emphasized that the benchmark provides a transparent way for security teams to assess models for both capabilities and resistance to attacks.

SecurityScorecard's co-founder and CEO, Aleksandr Yampolskiy, warned against deploying untested AI agents in vendor risk programs, likening it to granting privileged access to a new employee without proper oversight. The benchmark is open-source, allowing any model or agent to be tested and submitted to a public leaderboard. Participants require a registered account, and official expert-verified scores are provided through SecurityScorecard's internal tier, while synthetic-calibrated results from the public tier are comparable for research purposes. To obtain a certified evaluation, enterprises must contact SecurityScorecard directly.