AI governance and regulation (EU AI Act)
What Is the EU AI Act?
The Four Risk Categories
The Act divides AI into four tiers:
┌─────────────────────────┐
│ UNACCEPTABLE RISK → PROHIBITED │
│ (Social scoring, real-time biometric │
│ surveillance in public, manipulation) │
└────────────────────────┘
↓
┌────────────────────────────────┐
│ HIGH RISK → STRICT REQUIREMENTS │
│ (Critical infrastructure, employment, │
│ education, law enforcement, medical) │
└────────────────────────┘
↓
┌─────────────────────────────┐
│ LIMITED RISK → TRANSPARENCY ONLY │
│ (Chatbots, deepfakes, emotion recog.) │
└────────────────────────┘
↓
┌────────────────────────────────┐
│ MINIMAL RISK → NO OBLIGATIONS │
│ (AI-enabled games, spam filters) │
└────────────────────────┘
WHY this structure?
- Unacceptable risk: Systems that fundamentally violate human rights get banned outright (no gradual phase-in).
- High risk: Systems affecting safety/fundamental rights needorous testing, documentation, human oversight.
- Limited risk: Transparency lets users make informed choices ("This text was AI-generated").
- Minimal risk: Most AI applications—no reason to burden innovation.
High-Risk AI Requirements: The Core of Compliance
Example 1: Hiring AI System (High-Risk)
Scenario: A company uses an AI to screen job applications.
Why it's high-risk: Employment is a fundamental right; biased AI can discriminate.
Compliance steps:
-
Risk Management:
- What we do: Identify risks (gender bias, age bias, accessibility issues).
- Why this step?: You can't mitigate risks you haven't identified. Forces proactive thinking.
-
Data Governance:
- What we do: Ensure training data represents diverse demographics; test for bias across protected attributes.
- Why this step?: AI learns patterns from data. If training data overepresents certain groups, the model will too.
- Check: Run fairness metrics (demographic parity, equalized odds) on validation set.
-
Technical Documentation:
- What we do: Document model architecture, training data sources, performance metrics, known limitations.
- Why this step?: Auditability. If a rejected candidate complains, regulators need to understand how decisions were made.
-
Human Oversight:
- What we do: HR staff review AI recommendations before final decisions; can override.
- Why this step?: AI catches patterns humans miss, but humans catch context AI misses. Complementary strengths.
-
Transparency:
- What we do: Inform applicants that AI is used; provide explanation of decision factors.
- Why this step?: Applicants have a right to know and to contest automated decisions.
Prohibited AI Systems: The Red Lines
Derivation: These prohibitions derive from EU Charter of Fundamental Rights:
- Social scoring → violates dignity (Art. 1), freedom of thought (Art. 10)
- Mass biometric surveillance → violates privacy (Art. 7, 8)
- Manipulation → violates autonomy and dignity
Limited Risk: The Transparency Tier
General-Purpose AI (GPAI) and Foundation Models
The Act includes special provisions for general-purpose AI (GPAI) like GPT, DALL-E, LaMA—models not designed for a specific use case.
Example chain:
- OpenAI trains GPT-5 → GPAI provider obligations (document training data, test for systemic risks)
- Hospital buys GPT-5 to build a diagnostic chatbot → High-risk AI obligations (clinical validation, human oversight, etc.)
If the diagnostic chatbot fails, both OpenAI and the hospital may be liable (depending on where the failure originated).
Enforcement and Penalties
Why the EU AI Act Matters Globally
Practical Compliance Workflow
For a company deploying a high-risk AI system:
Step 1: Risk Assessment
↓
Is it high-risk? (Check Annex III of Act)
├─ No → Transparency requirements only (if limited risk)
└─ Yes↓
Step 2: Data Governance
↓
Audit training data for quality, bias
↓
Step 3: Technical Documentation
↓
Document architecture, training process, performance
↓
Step 4: Testing & Validation
↓
Test on representative data; compute fairness/accuracy metrics
↓
Step 5: Human Oversight Design
↓
Define human review points, override mechanisms
↓
Step 6: Conformity Assessment
↓
Self-assessment or third-party audit (depends on system type)
↓
Step 7: CE Marking & Registration
↓
Register in EU database; afix CE mark to product
↓
Step 8: Post-Market Monitoring
↓
Log incidents, retrain as needed, report serious issues
Time estimate: 6-18 months for full compliance (depending on system complexity).
Cost estimate: €100k–€5M (legal, technical, audit costs).
Recall Feynman Explanation for a12-Year-Old
Imagine you invent a robot that helps teachers grade homework. It's really smart and saves a ton of time! But what if the robot accidentally gives bad grades to kids from certain neighborhoods because it learned from biased past data? That's not fair, right?
The EU AI Act is like a rulebook for robots and AI. It says:
- Dangerous robots are banned (like robots that spy on everyone or trick people into doing things).
- Important robots have to be extra careful (like robots grading homework or helping doctors). They need to be tested to make sure they're fair, and human always checks their work.
- Chatbots have to be honest ("Hi, I'm a robot!" instead of pretending to be human).
- Simple robots can do whatever (like a robot that plays chess or filters spam).
Why do we need this? Because AI is getting super powerful, and if we don't have rules, someone could build a robot that makes unfair decisions (like rejecting people for jobs because of their name or skin color) or spies on people without them knowing. The rules make sure AI helps people without hurting them.
It's like how we have rules for driving cars: you need a license, your car needs seatbelts and airbags, and you can't drive on the sidewalk. Same idea—rules keep powerful things safe!
Connections
- 6.4.1-Value-alignmentproblem: The AI Act is a governance solution to alignment. Where technical alignment ensures AI does what we want, regulation ensures what we want is humane.
- 6.4.2-Reward-hacking: High-risk requirements (human oversight, logging) help detect when deployed AI "game" metrics in harmful ways.
- 6.4.3-Interpretability-and-explainability: Explability is mandated for high-risk systems under "transparency" requirements—directly relevant.
- 6.4.4-Robustness-and-adversarial-examples: Robustness testing is a compliance requirement. Act incentivizes research on adversarial defenses.
- 6.4.5-AI-safety-research: The Act funds AI safety research; certification bodies need standards, which safety research provides.
- 6.4.12-Long-term-existential-risks: The Act addresses near-term harms (bias, privacy). Long-term risks (AGI alignment) are out of scope, but GPAI systemic risk provisions are a step.
#flashcards/ai-ml
What is the core principle of the EU AI Act's regulatory approach? :: Risk-based approach—AI systems are classified by risk level (unacceptable, high, limited, minimal), with stricter requirements for higher-risk systems.
What are the four risk categories in the EU AI Act?
Give two examples of prohibited (unacceptable risk) AI systems under the EU AI Act.
What are the seven core requirements for high-risk AI systems?
Why is a hiring AI system classified as high-risk?
What must limited-risk AI systems (like chatbots) do under the EU AI Act?
What is the penalty for deploying prohibited AI systems?
What are GPAI (general-purpose AI) obligations under the Act?
What is the "Brussels Effect" in the context of the EU AI Act?
Why does the EU AI Act require human oversight for high-risk systems?
What is the difference between banned real-time biometric surveillance and allowed post-event analysis? :: Real-time scanning in public creates mass surveillance and chilling effects; post-event targeted analysis (e.g., reviewing footage after a specific crime with oversight) has investigative purpose with judicial safeguards.
Why does the Act mandate data governance for high-risk AI?
Concept Map
Hinglish (regional understanding)
Intuition Hinglish mein samjho
EU AI Act basicallyek regulatory framework hai jo AI systems ko unke risk level ke basis pe classify karta hai. Socho na, agar koi AI system tumhare job application ko reject karde sirf isliye kyunki tumhara naam kuch specific pattern follow nahi karta, ya koi medical AI galat diagnosis dede—yeh serious problems hain. Isliye EU ne kaha ki chalo ek rulebook bana lete hain jaise gadi chalane ke liye driving license chahiye, waise hi AI ke liye bhi rules chahiye.
Act mein chaar categories hain: Unacceptable (matlab banned—jaise government social scoring ya public mein mass surveillance), High-risk (jaise hiring AI, medical diagnosis—inke liye strict compliance chahiye with human oversight aur bias testing), Limited-risk (jaise chatbots—bas transparency chahiye ki "main AI hoon"), aur Minimal-risk (jaise spam filters—koi restriction nahi). High-risk wale systems ke liye company ko data governance, technical documentation, human oversight, sab kuch maintain karna padta hai. Agar compliance nahi ki toh penalties bahut heavy hain—€35 million tak ya company ke global revenue ka 7%, jo bhi zyada ho.
Yeh regulation sirf Europe tak limited nahi hai. Jaise GDPR ne globally privacy standards set kiye, waise hi AI Act bhi global benchmark ban raha hai. US aur China ki companies bhi isko follow kar rahi hain kyunki EU market access chahiye unko, aur do alag versions banana costly hai. So agar tum AI field mein kaam kar rahe ho, chahe India mein ya US mein, yeh Act tumhe affect karega. Samajhne ki baat yeh hai ki regulation innovation ko kill nahi karta—balki trust build karta hai.Agar log AI pe bharosa nahi karenge, toh adoption hi nahi hoga. Proactive regulation long-term mein better hai reactive backlash se.