Exercises — AI governance and regulation (EU AI Act)

Unacceptable → High → Limited → Minimal. Memory hook: U-H-L-M, "Ugh, Handle Lightly, Minimal." Top of the ladder is the red "banned" box; bottom is the green "do-what-you-like" box.

Tier: Limited risk (the blue box). Duty: transparency (T1 — disclosure of AI use) — it must disclose to the user that they are talking to an AI, unless that is already obvious from context. Nothing more (no audits, no logs mandated).

False. Government social scoring sits in the Unacceptable tier — it is prohibited outright. No amount of documentation makes it legal, because it violates human dignity and freedom of thought. Documentation is a high-risk remedy; it does not apply to a banned system.

(a) High risk — medical diagnosis affects health and safety; a wrong output can be life-threatening. (b) Any three of:

• Accuracy to clinical standards (e.g. sensitivity target).
• Robustness testing across scanner types, hospitals, and patient demographics.
• Record-keeping — log each prediction with its confidence score for later dispute review.
• Human oversight — a radiologist reviews the AI output; AI is a "second opinion," not the final word.
• Data governance — diverse training data across age/ethnicity/disease stage.

Tier: Limited risk (transparency tier). Required action: transparency (T2 — output labelling) — the generated media must be labelled as artificially generated / synthetic so viewers know it is not a genuine recording. Rationale: misinformation prevention + informed judgement.

Both are Minimal risk (green box). Obligations: none mandated by the Act. The Act deliberately leaves the vast majority of everyday AI unburdened so innovation is not choked.

• Police A: Unacceptable / prohibited. This is real-time remote biometric identification in public spaces for mass surveillance — banned (outside narrow, judicially-approved exceptions).
• Police B: Allowed as post-event targeted analysis — it is not live, it is not blanket-scanning a crowd, and it operates within investigative oversight.
• The dividing lines: (1) real-time vs post-event, (2) public space mass scan vs targeted specific investigation. Live crowd scanning creates a permanent chilling effect on public behaviour; post-hoc targeted review has oversight and a specific subject.

Most direct obligation: Data governance (quality and bias testing) — because AI learns patterns from data, and skewed data teaches a skewed model.

First, two fairness ideas in plain words (we define them before using them):

• Demographic parity asks: do the groups get selected at the same rate? If 50% of men are picked but only 30% of women, that rate gap is a demographic-parity failure. It looks only at who gets chosen, ignoring who was actually qualified.
• Equalized odds is stricter: among the people who truly deserved the job, is the "yes" rate the same across groups — and among those who didn't, is the "wrong-yes" rate the same too? It compares the AI against ground truth, not just raw selection counts.

Test: run these fairness metrics on a validation set and compare the numbers across groups.

Worked number — the four-fifths (80%) rule sanity check (a common demographic-parity screen): suppose the AI selects 50% of male applicants and 30% of female applicants. The impact ratio is $\frac{0.30}{0.50}=\mathrm{0.60.}$ Since $0.60<0.80$, the system fails the four-fifths screen → strong sign of disparate impact (one group is harmed at a much higher rate), flagging a data-governance breach.

The surface feature (faces) is identical; the legal trigger is context.

• System X: non-public, consenting, single-user → not "remote identification in public space." It sits in Minimal risk (green box) with no obligations under the Act — it is fully allowed.
• System Y: remote, real-time, public, mass → this is exactly the Unacceptable / prohibited category — banned. The Act regulates the deployment context, not the raw technology. "Uses facial data" is not a classification; "real-time remote public mass identification" is.

Tier: High risk (credit access is a listed critical domain; wrongful denial harms fundamental economic rights). Compliance stack — component → harm it prevents:

1. Risk management system → you cannot mitigate risks you never enumerated (bias, error, exploitation).
2. Data governance → skewed training data → biased approvals; test with fairness metrics (demographic parity / equalized odds, defined in Q8).
3. Technical documentation → auditability when a rejected applicant complains.
4. Record-keeping (logs) → traceability: reconstruct what data, what model, what decision.
5. Transparency (T3 — right to contest) → applicant's right to know AI was used and to challenge the decision, with an explanation.
6. Human oversight → a loan officer can override the model in context-heavy cases.
7. Accuracy, robustness, cybersecurity → resist adversarial manipulation. (An adversarial example is a cleverly tweaked input designed to fool the model — see 6.4.4-Robustness-and-adversarial-examples for the full treatment.)

• (i) Reward hacking = the AI scores well on its stated metric by exploiting a loophole rather than doing the intended job (e.g. maximising "approval accuracy" via a shortcut that quietly harms one group). Data governance + fairness testing is the countermeasure. Full topic: 6.4.2-Reward-hacking.
• (ii) The applicant's right to contest (T3) requires the decision be explainable — you must be able to state why the model said no. That readability of a model's reasoning is interpretability. Logs alone are not enough if no one can read them. Full topic: 6.4.3-Interpretability-and-explainability.
• (iii) Compliance is a floor, not a ceiling: the broader project of getting powerful systems to actually do what we intend is ongoing AI safety research (6.4.5-AI-safety-research), and its most severe motivation is the worry that very capable future systems could cause irreversible, civilisation-scale harm — long-term existential risk (6.4.12-Long-term-existential-risks).

Two candidate provisions:

• Manipulative AI that exploits vulnerabilities of a specific group (here, children).
• Subliminal techniques influencing behaviour below awareness. Both are Unacceptable / prohibited. Final tier: banned. Justification: it exploits the vulnerability of a protected group and undermines autonomy/informed consent — no transparency label or human-oversight bolt-on can rescue a prohibited purpose. (Contrast: an honest educational talking toy with clear disclosure and no manipulation would be limited-risk.)

Rule for mixed systems: classify each function by its own risk, and the deployment must satisfy the union (the strictest applicable set), not the average.

• Diagnostic function → high-risk full stack (accuracy, robustness, logs, human oversight, data governance, documentation, right-to-contest).
• Chatbot function → limited-risk transparency (T1 — disclose "you are talking to an AI"). Result: the chatbot's T1 disclosure duty stacks on top of, not instead of, the diagnostic high-risk obligations. You never let the lighter tier dilute the heavier one.

Applies? Yes. The Act is triggered by use/effect within the EU market, not by where the system was built. A foreign-built hiring AI affecting EU applicants must comply as a high-risk system. Zero-case (no EU users, no EU market effect): the Act does not apply — jurisdiction is the limiting condition. As EU-market involvement → 0, obligations → 0. This mirrors the parent note: "regardless of where it's developed," but conditioned on the EU market.

The four tiers in order of danger ::: Unacceptable, High, Limited, Minimal A chatbot's single legal duty ::: transparency T1 (disclose it is AI) A deepfake's required action ::: transparency T2 (label the output as synthetic) The high-risk "right to contest" duty ::: transparency T3 (explain and allow a challenge) A prohibited system can be made legal with enough paperwork ::: No — prohibited is a hard wall A loan-approval AI's tier ::: High risk Demographic parity checks ::: whether groups are selected at equal rates Equalized odds checks ::: whether, against ground truth, correct-yes and wrong-yes rates match across groups Mixed-tier deployment obeys the ::: union / strictest applicable obligations The Act's jurisdiction is triggered by ::: use / effect in the EU market, not where it was built Four-fifths screen: impact ratio 0.30/0.50 = 0.60, which ::: fails (below 0.80) → disparate impact