13.1 AI Risk Governance and Impact Assessment: First Decide Whether to Deploy

AI risk isn’t a single score tied to a model’s performance. It emerges from the interplay of purpose, data, model design, human-machine roles, organizational incentives, and the relationships between stakeholders. The same text classifier used to organize personal notes and to screen job applicants creates vastly different risks in each context.

Learning Objectives

• Define the AI system boundary by starting from decisions and paths to harm;
• Establish clear inventories of roles, purposes, data sources, and third-party dependencies;
• Organize ongoing risk management using the Govern–Map–Measure–Manage framework;
• Distinguish between model capability, system impact, and legal classification;
• Maintain audit evidence for go/no-go decisions, changes, and deactivation.

1. Map the Whole System

An AI service must include at least the following flow:

Data source → Cleaning/annotation → Training or external model API
                         ↓
User/environment → Prompt/RAG/Tools → Model output → Human or automated process → Real-world consequences
                         ↑                    ↓
                  Logs, feedback, monitoring ← Appeals, corrections, events

System inventory should record:

• Owner, provider, deployer, operator, and impacted parties;
• Intended use, prohibited uses, and deployment regions;
• Inputs, outputs, data subjects, sensitive data, and retention periods;
• Models, checkpoints, APIs, prompts, retrievers, indexes, and tools;
• Third-party agreements, subprocessors, service regions, and exit strategies;
• Level of automation, human review requirements, appeal processes, and emergency shutdowns;
• Key performance metrics, known limitations, and last review date.

Changing model versions, altering use cases, integrating new tools, or expanding to new user groups can all shift risk levels, old approvals cannot be reused.

2. Start with Decision and Harm Paths

First, express the purpose as a verifiable statement:

Who, under what conditions, based on what inputs, makes a decision using outputs, and who is affected by it.

For a hypothetical service-allocation system, expand the harm path:

A service team’s old devices miss task reports
→ Training data treats "not recorded" as "not completed"
→ System predicts low performance for that team
→ Auditor treats the score as an objective fact
→ Resources for the team are reduced
→ Missed reporting and low performance create a feedback loop

For each path, annotate the affected parties, severity, triggering conditions, reversibility, duration, difficulty of detection, and existing controls. A risk matrix can help prioritize, but it must never reduce rights, dignity, or group-level harm to a single seemingly objective score.

Also compare the consequences of not deploying the system with alternative non-AI solutions. If rules, manual sampling, or improved data collection are already sufficient, complex models may not deliver net benefit.

3. NIST AI RMF is an organizational framework, not a compliance certificate

NIST AI RMF 1.0 organizes risk management through four core functions:

• Govern: Establish policies, responsibilities, risk tolerance, and organizational capabilities;
• Map: Understand use cases, environments, stakeholders, and impacts;
• Measure: Select assessments, metrics, and uncertainty analyses;
• Manage: Prioritize, mitigate, accept, transfer, or avoid risks, and continuously monitor them.

The Govern function spans the entire lifecycle, while Map–Measure–Manage cycles iteratively as systems evolve. It is a voluntary, industry-agnostic framework (not a one-time checklist) and does not certify compliance with any specific legal jurisdiction. Generative AI practitioners may also refer to the supplemental guidance in NIST AI 600-1 on risks such as confabulation, data privacy, information integrity, and information security.

4. Risk Grading Must Be Tied to Context

Tiered risk classifications should be established based on specific use cases, for example:

These are internal review tiers, not legal classifications or a substitute for the AI Act’s definitions.

Level	Scenario	Default Requirements
Low	Personal assistance with no external consequences	Basic security, privacy, and quality testing
Medium	Impacts business processes but involves human review	Impact assessment, monitoring, override capability, change gate controls
High	High-impact decisions such as employment, education, healthcare, credit, or public services	Independent review, rigorous validation, audit logging, appeal mechanisms, and deactivation procedures
Prohibited	Uses deemed unacceptable by the organization or legally forbidden	No development, no procurement, no workarounds or bypass deployments

The presence of "human-in-the-loop" does not automatically reduce risk. Reviewers may over-rely on automated recommendations, lack time to verify inputs, or be restricted to predefined system options. It is essential to measure override rates, review quality, response times, and ensure that personnel have access to the necessary contextual information.

For example, seed a review exercise with known missing-record errors. Measure whether reviewers detect and correct them before a deadline, not just whether they click “approve.” A low override rate could mean accurate recommendations or ineffective review; it needs a reference check.

5. Legal Classification Must Be Verified by Role, Region, and Date

The EU AI Act presents a regulatory framework based on purpose and role: it distinguishes between prohibited practices, specific high-risk systems, transparency obligations, and general-purpose AI models. The applicability of these obligations depends on the role of the entity (such as provider, deployer, or importer) and whether the system's outputs are used within the European Union.

As checked on 5 September 2026, the general application date remains 2 August 2026, but Regulation (EU) 2026/1744 amended the high-risk timetable. Chapter III, Sections 1–3, apart from Article 6(5), apply from 2 December 2027 for Article 6(2)/Annex III systems and 2 August 2028 for Article 6(1)/Annex I systems. Other provisions and transition rules differ. Record the relevant article, role, system category, and check date; the Commission overview helps locate the rules, while the applicable legal text determines the obligation.

A compliance checklist is not a complete ethical assessment. Laws may specify only minimum requirements, while issues such as purpose drift, power imbalances, or irreversible harm may still constitute valid "no-go" reasons.

6. Bring Control to Owners and Evidence

Every high-priority risk must include:

risk owner
preventive control
detection / metric
threshold and alert
response playbook
residual risk and approver
evidence location
review date

For example, a risk like "outdated equipment causing group-level underreporting" cannot be addressed simply by stating "improve fairness." Controls might include fixing data collection, ensuring data completeness in display, preventing automatic resource cuts based on incomplete records, segmenting monitoring, enabling manual review, and allowing user appeals. Each control must have test results and a clearly assigned owner.

A model card can describe model training and known limitations; a system card or impact assessment must also cover specific workflows, tools, users, and downstream consequences. These two documents are not interchangeable.

7. Go / No-Go Is Not a Meeting Slogan

A go/no-go decision must be grounded in evidence that answers the following questions:

• Is the intended use case necessary, proportionate, and supported by a comparable baseline?
• Have stakeholders and domain experts been involved? How are disagreements documented?
• Do critical data and third-party sources have provenance, licensing, and deletion paths?
• Have performance, fairness, security, privacy, and robustness been validated across critical use cases?
• Are manual reviews, notifications, explanations, corrections, and appeals accessible and functional?
• Do high-severity failure scenarios include blocking mechanisms, rollback procedures, and incident response protocols?
• Who accepts residual risk, and on what basis?
• Do unmonitored, significant risks trigger scope reduction or deployment cancellation?

Approval must be explicitly tied to a version and a defined expiration date. Any extension of approved use cases, expiration of reviews, or critical dependency changes must trigger a renewed assessment.

8. Deployment Changes Future Data

System outputs influence future labeling: individuals predicted as high-risk receive more scrutiny, generating more recorded violations; under-resourced areas see reduced investment, leading to actual performance declines. These feedback loops mean that offline historical data no longer accurately reflects the environment after deployment.

Monitoring should extend beyond model quality to include usage rates, overrides, appeals, resource allocation, outcomes by affected demographic groups, data drift, anomalous behavior, and third-party changes. When significant harm occurs, the system must be capable of being disabled or rolled back to a non-model process, rather than documented as a theoretical option alone.

Common Misconceptions

• Model risk equals model accuracy: Risk is determined by use case and decision consequences, rather than accuracy metrics alone.
• Open-source models require no governance: Deployers remain responsible for their data, tools, and how results are used.
• A human click means human oversight: Additional validation of capabilities, timing, permissions, and resistance to automation bias is still required.
• Using a framework means compliance: Frameworks, standards, legal requirements, and internal ethics reviews each have distinct responsibilities.
• Onboarding approval is a one-time event: Changes in use case, model, or environment invalidate prior approval and require reevaluation.

Exercise

1. Draw the system boundary and list the roles for service-team resource recommendations.
2. Write three harm paths that contain feedback loops.
3. Use the Govern–Map–Measure–Manage framework to organize control over the same project.
4. Design a metric to validate the effectiveness of manual review processes.
5. Define reapproval triggers based on changes to model versions, use cases, and tools.

Summary

The starting point of AI governance isn't labeling models as "trustworthy", it starts by identifying their purposes, roles, decision-making processes, and potential consequences. Organizations first define their risk tolerance, then map potential harms, measure controls, and manage residual risk. Legal obligations, technical frameworks, and ethical judgments each operate within their own boundaries.

The next lesson uses the reporting example to calculate group metrics and examine what those metrics cannot decide.

References

• NIST, AI Risk Management Framework 1.0
• NIST, Generative AI Profile, NIST AI 600-1
• European Union, Regulation (EU) 2024/1689
• European Commission, AI Act implementation overview