From SiliconANGLE · Duncan Riley · · 1m 20s
SecurityScorecard tests AI on third-party risk
AI models face third-party risk tests in SecurityScorecard's new benchmark.
In brief
AI models face third-party risk tests in SecurityScorecard's new benchmark. SecurityScorecard released an open benchmark to test AI models' ability to handle third-party risk without being turned against companies. Originally reported by SiliconANGLE.
SecurityScorecard launches TPRM Benchmark
SecurityScorecard Inc. released an open benchmark to test whether artificial intelligence models can handle third-party risk management work without being turned against the companies using them.
TPRM Benchmark for vendor risk programs
The benchmark is built for vendor risk programs where models interpret security telemetry and decide which exposures matter.
Model capability tasks
Tasks cover disciplines like exposure triage and fourth-party technology risk, starting with SecurityScorecard's vendor-risk telemetry.
Attack resistance testing
The Model Security suite tests models with prompt injection, jailbreaks and other techniques, grading from A to F.
Deceptive vendor self-attestations
Some tasks include deceptive vendor self-attestations that contradict underlying telemetry to test model accuracy.
Performance varies by model type
Early runs showed open-weight models are closing the gap with frontier models on supply chain security tasks.
Open-source framework
SecurityScorecard has open-sourced the framework for participants to run their own models and submit results to a public leaderboard.
Model security ratings
Models get autonomy readiness levels from L0 to L4 based on performance across repeated trials with confidence intervals.
The essence
SecurityScorecard released an open benchmark to test AI models' ability to handle third-party risk without being turned against companies.
