AI Security & Safety

One seat. Two halves.Attack surface, and harm without an attacker.

Security is what you defend when someone pushes on the model or the agent. Safety is what you own when the system hurts people with no attacker in the loop. Same CSA spine. Different question.

Defend the model and the agent →Trace harm to an owner →

Every harm has an ownerEU AI Act · High risk

Ethical requirements it triggers

Fairness & bias testingHuman oversightTransparency & explainabilityContestability

AICM control domains that hold the fix

MDSDSPGRCHRS

3 harm classes · 7 safety control families · NIST AI 100-1

Harm to People: Harms to individuals and groups — rights, safety, access. The class with the widest ethical-requirement pipeline.

The security half

What an attacker can do. What you own next.

Start with the OWASP Top 10 for LLM applications (mapped to CSA AICM domains — our judgment, not OWASP’s). Then walk the threats in plain English. The curriculum below is the AI Security Engineer’s primary control set.

OWASP LLM Top 10

OWASP Top 10 for Large Language Model Applications · v2 (2025) ↗

Each risk maps to the CSA AI Controls Matrix (AICM) domains where the controls live. That pairing is our judgment, not OWASP’s.

LLM01
Prompt Injection
Adversarial input that manipulates the model into ignoring its instructions or exposing data. The single most consequential class of LLM attacks today.
Controls live in Model Security · Application & Interface Security
LLM02
Sensitive Information Disclosure
Models leaking PII, secrets, or proprietary data through outputs — directly or via inference from prompts.
Controls live in Data Security and Privacy Lifecycle Management · Model Security
LLM03
Supply Chain
Risks from pre-trained models, datasets, and plugins sourced outside the org — the AI version of dependency risk.
Controls live in Supply Chain Management, Transparency, and Accountability · Change Control and Configuration Management
LLM04
Data and Model Poisoning
Compromised training/fine-tuning data or model artifacts that introduce bias, backdoors, or degraded performance.
Controls live in Model Security · Data Security and Privacy Lifecycle Management
LLM05
Improper Output Handling
Downstream systems trusting LLM output as code, SQL, shell, or URL without validation — classic injection patterns reborn.
Controls live in Application & Interface Security · Model Security
LLM06
Excessive Agency
Granting an LLM/agent tool access or permissions far beyond what its task requires. The cousin of over-privileged service accounts.
Controls live in Identity & Access Management · Model Security
LLM07
System Prompt Leakage
Internal system prompts becoming externally visible — leaking implementation logic and bypassing guardrails.
Controls live in Model Security · Application & Interface Security
LLM08
Vector and Embedding Weaknesses
Misuse of embeddings/vector stores: information leakage across tenants, embedding inversion attacks, retrieval poisoning.
Controls live in Data Security and Privacy Lifecycle Management · Model Security
LLM09
Misinformation
Plausible but false output presented as authoritative — hallucinations operationalized into systems of record.
Controls live in Model Security · Governance, Risk and Compliance
LLM10
Unbounded Consumption
Resource exhaustion, model denial-of-service, runaway cost. Where AI security meets FinOps.
Controls live in Model Security · Threat & Vulnerability Management

See these risks in plain English on the threat map →

MITRE ATLAS shows up as technique cites on the threat map and on skills, not as a second scored crosswalk. We do not invent an ATLAS×AICM matrix here.

Your curriculum · 26 controls

5 skill groups on the CSA controls this role owns primarily: secure the model, attack it first, contain the agent, guard the runtime, defend the training data.

Secure the model
8 controls
Attack it before they do
5 controls
Contain the agent
5 controls
Guard the runtime
4 controls
Defend the training data
4 controls

Role one-pager · control by control →Open the threat map →Prove a fitted guard →Self-check these skills →

Skill self-check · nothing saved

Preview the prompts this seat owns, then score them on four concrete rungs. Self-assessed, nothing saved.

Secure the training pipeline end to end. Could you secure a training pipeline so data, code, and checkpoints can't be tampered with unseen?
Scan model artifacts before they deploy. Could you scan a model artifact for tampering and embedded threats before it reaches production?
Document models so others can trust them. Could you produce model documentation that tells a reviewer what it is, what it's for, and what it must not do?
Set what model documentation must contain. Could you define the documentation every model must ship with before anyone builds on it?
Validate model documentation against reality. Could you check a model's documentation against its actual behaviour and catch where they differ?

Run the scored mastery check →

The safety half

When the system does harm with no attacker.

Look up any AI harm and see who owns it, the controls that hold the fix, and the EU AI Act risk tier it lands at. Then learn the three skills the map keeps asking for.

The three skills this map keeps asking for

Evaluate how a model behaves

You tested code against a spec. A model has no spec, only behaviour: the skill is building the evaluation that catches what it does under pressure, before your users do.

Where evaluation gates the pipeline →

Fit a guardrail, and prove it

An off-the-shelf guard is a claim. The skill is refitting it to your own product and measuring what it catches before and after, on your data.

A real guard, refitted and measured →

Trace a harm to its owner

When the system hurts someone, who answers for it? The skill is reading a harm back to the accountable owner, the control that holds the fix, and the regulatory tier it lands at.

The harm map, below →

The harm classes · NIST AI 100-1

Three classes of harm, named the way regulators name them.

The National Institute of Standards and Technology (NIST) AI Risk Management Framework groups AI harms into three top-level classes. We anchor the safety lens to those classes so the controls we fit map cleanly to ethical requirements, AI Controls Matrix (AICM) domains, and the EU AI Act risk tier each class typically lands at.

Harm to People

Harms experienced by individuals, groups, communities, or society — civil liberties, rights, physical and psychological safety, economic opportunity, equal access. Includes group harms (discrimination, exclusion) and societal harms (erosion of democratic processes, harm to vulnerable populations).

Discriminatory model output (group harm)
Manipulated content that targets a person psychologically
Denial of access to credit, housing, employment
Erosion of an information ecosystem (societal harm)

Fairness & bias testingHuman oversightTransparency & explainabilityContestability

AICM domains: MDS · DSP · GRC · HRS

EU AI Act tier: High risk

Harm to an Organization

Harms to an organization — operational disruption, reputational damage, security and financial loss — caused by AI systems behaving unexpectedly, leaking data, or being weaponised against the organization itself.

Brand and trust damage from an AI-driven incident
Data exfiltration through prompt injection
Operational disruption when an AI agent acts outside its mandate
Compliance failure traced to an unaccountable AI output

Human oversightTransparency & explainability

AICM domains: MDS · AIS · LOG · IAM · SEF

EU AI Act tier: Limited risk

Harm to an Ecosystem

Harms to the interconnected systems on which society depends — the global financial system, supply chains, critical infrastructure, the natural environment. Includes Environmental harm: AI training and inference are energy-intensive, and at scale they contribute to carbon and water footprint, e-waste, and resource depletion.

Critical-infrastructure disruption from chained AI agents
Environmental footprint of large-model training and inference
Supply-chain compromise via a poisoned upstream model
Systemic risk in a market when many actors use the same model

Human oversightTransparency & explainability

AICM domains: STA · BCR · CCC · TVM

EU AI Act tier: High risk

Harm class	EU AI Act tier	Primary requirements
Harm to People	High risk	Fairness & bias testing · Human oversight · Transparency & explainability · Contestability
Harm to an Organization	Limited risk	Human oversight · Transparency & explainability
Harm to an Ecosystem	High risk	Human oversight · Transparency & explainability

The safety control spine

Seven families, harm-anchored.

A lens over the existing 18 AICM domains, not a new partition. Each family names where in the control stack a harm class is mitigated, the AICM domain its controls land in, and the standards hook a Chief Information Security Officer (CISO) is asked to evidence.

Output & Decision Safety

Block harmful or unsafe model outputs — hate, violence, self-harm, CSAM, illegal advice, hallucinations in high-impact decisions.

AICM MDS

EU AI Act: Art. 5 (prohibited practices)
NIST AI RMF: MEASURE-2.6 / MEASURE-2.7

Bias & Fairness

Detect and mitigate discriminatory or skewed model behaviour against groups defined by protected attributes.

AICM MDSAICM GRC

EU AI Act: Art. 10 (data governance)
NIST AI RMF: MEASURE-2.11

Transparency & Explainability

Make model behaviour legible to operators, users, and regulators — model cards, attribution, AI-interaction disclosure.

AICM GRC

EU AI Act: Art. 13 (transparency) · Art. 50 (AI disclosure)
NIST AI RMF: GOVERN-1.2 / MEASURE-2.8

Human Oversight & Contestability

Keep humans in the loop on high-impact decisions and give affected people a route to challenge an AI outcome.

AICM GRC

EU AI Act: Art. 14 (human oversight) · Art. 22-link (automated-decision rights)
NIST AI RMF: MANAGE-2.3

Agentic & Action Safety

Bound agent autonomy and tool scopes; gate irreversible actions; detect cascading or feedback-loop behaviour. The autonomy frontier — ties to the AI Cyber Maturity Model (AI-CMM) ladder (our model).

AICM MDSAICM IAM

NIST AI RMF: MANAGE-2.4
OWASP LLM Top 10 (2025): LLM06 — Excessive Agency

Safety Evaluation & Benchmarking

Pre-deploy safety assurance — run a benchmark suite + harm-focused red-team and gate releases on the result.

AICM TVMAICM MDS

NIST AI RMF: MEASURE-2.5 / 2.7 / 2.9

Runtime Safety Monitoring

Detect harm over time — behavioural and safety drift, guardrail efficacy in production, harm-incident response.

AICM LOGAICM BCR

NIST AI RMF: MANAGE-4.1