Red Teaming AI Systems for Security Validation

Abstract

There is a growing trend in which artificial intelligence (AI) is embedded in safety-critical, regulatory-driven, or high impact domains such as healthcare, finance, public infrastructure, enterprise automation. The larger these systems become, however, the more apparent the risks such systems pose from adversarial exploitation, model manipulability, and the unsafe integration of tools. AI models are different from traditional software in that they exhibit probabilistic and context-dependency nature and hence exhibit unique vulnerabilities including prompt injection, jailbreaks, data poisoning, model extraction, and unsafe autonomous decision-making. These threats require special testing and assurance methodologies, beyond traditional penetration tests.

Red teaming is becoming an important technique to evaluate AI systems under real world adversarial settings. It’s the systematic stress-testing of models to identify holes before bad actors can exploit them. Nevertheless, contemporary red team approaches are fragmented with little shared taxonomies, common benchmarks or integration with governance. To fill in these gaps, this paper presents COMPASS-RT, a model-agnostic and deployment-agnostic framework for AI red teaming.

The framework is comprised of six pillars: (i) risk-based scoping, consistent with the NIST AI Risk Management Framework and ISO/IEC 42001; (ii) threat modelling using MITRE ATLAS and the OWASP Top-10 adapted for LLM applications; (iii) hybrid adversarial testing, combining human expertise with automated, LLM-driven attack generation; (iv) benchmark-based validation, using standardized corpora such as AdvBench, HarmBench, and JailbreakBench; (v) governance integration to ensure that findings map to risk registers, mitigation workflows, and regulatory compliance under regimes like the EU AI Act; and (vi) continuous validations, providing sustained measurement and regression testing across model updates.

This paper makes a threefold contribution: it unifies existing best practices from adversarial AI testing, operationalizes best practices in governance, and does so in a way which incorporates reporting templates that organizations can use for audit-ready assurance. By incorporating COMPASS-RT into enterprise security and compliance programs, enterprises can establish defensible processes for demonstrating the robustness of AI systems, eroding the efficacy of attacks and accelerating response. Risk Management: You cannot eliminate risk completely, but disciplined and repeatable red teaming greatly strengthens security posture and confidence in AI deployments