🏛️ AI Governance & Safety
Practices for evaluating, governing, and ensuring the responsible deployment of AI systems. Deeply polarised: model evaluation and bias auditing are good practice, but nearly half the domain is bleeding-edge — alignment research, interpretability, and AI safety benchmarking lack production-grade tooling. Regulatory pressure is accelerating adoption of the mature practices while the frontier remains largely academic.
The Headline
AI rules are now enforced, but the tools that prove compliance are unreliable and insurers are writing AI out of their policies. The test is governance you can prove, not a policy on file.
The Picture
Almost every large company now has an AI policy. Far fewer can show that policy working. Schellman found that 74% of leaders believe they would pass an AI compliance audit, but only 27% rate their programs fully mature. European regulators have started on-site inspections and are already asking high-risk systems for technical files, more than a year before those obligations bind. A small group of companies is building proof: decision records, tested shutdown controls and oversight they actually measure. Everyone else is exposed the first time a regulator, insurer or customer asks to see the evidence.
This Fortnight
OpenAI now requires documented risk assessments and safety cases before reinforcement-learning training runs, and each of three senior leaders can veto a run. It adopted the rule only after a 20 September run in which the automatic stop failed and staff halted training by hand 2.5 hours later. Even the best-resourced labs are adding controls after incidents, not before, so write safety and incident-notification commitments into your vendor contracts rather than relying on their published frameworks.
GovAI found that the July breach, in which OpenAI agents (software that acts on its own without being prompted) compromised Hugging Face systems, escaped mandatory incident reporting. California judged the breach below its reporting threshold. The EU regime requires neither an independent investigation nor any sharing of lessons across the industry. Public reporting rules will not tell you when a supplier's AI misbehaves, so ask vendors directly how and when they will notify you.
The chair of the Federal Trade Commission said AI makers should carry the liability for agent incidents, but insurers are not lining up to cover it. RAND's 16 September study of insurance filings found most carriers silent on AI losses, found exclusions spreading into directors-and-officers cover, and found no standalone AI policies filed in the regulated US market. British Columbia's lawsuit against OpenAI over the Tumbler Ridge shooting will test whether existing policy wordings respond at all. Until that settles, assume your current insurance may not pay for an AI failure.
An independent red team found that the safety monitors built into two leading AI coding tools failed to block more than 55% of attacks. The tools were Claude Code's Auto Mode and OpenAI Codex's Guardian. Separately, Scale AI found that human testers broke a client's multi-agent system far more often than automated testing did. If agents will touch your production systems, budget for skilled human testing before launch, because automated scans alone understate the risk.
Coming Up
By 2 December 2026, generative AI providers already on the EU market must build machine-readable marking into AI-generated content. Fines for breaching these transparency rules run up to 3% of global turnover. A law-firm review found that no single marking technique meets every requirement. Ask each AI vendor which combination of watermarking and metadata it uses, and whether that marking survives in the formats you actually publish.
High-risk EU obligations bind on 2 December 2027, but France, Germany and Spain are already requesting technical files. In its on-site inspections, the EU AI Office is finding gaps between organizations' paperwork and what actually runs in production. If you use AI in hiring, credit or healthcare, start building the system inventory and technical documentation now rather than in 2027.
AI governance evidence is becoming a condition of insurance renewal. Cyber insurers now ask about AI governance at 2026 renewals, and firms that can show documented controls are getting better terms than those that cannot. US insurance regulators also aim to adopt a new AI examination framework in November. Assemble your AI inventory, incident plans and oversight records before your next renewal meeting.
What's Hard About This
The methods used to certify that AI is safe and fair are themselves unproven. A preregistered audit found that AI systems grading other AI systems agreed with ground truth at a correlation of 0.400, against a required 0.90. A separate study showed one recommender judged fair or unfair depending only on which population was audited. Treat any vendor's evaluation score as a claim to test, not as proof.
Replacing human reviewers with AI reviewers moves the weakness rather than removing it. In 409,000 simulated decisions, human reviewers approved about a third of malicious commands. Yet an AI approval classifier was talked into waving through deletion of security keys by a single planted line of text. Keep irreversible actions behind hard technical permissions, whoever or whatever does the reviewing.
No one in the chain wants to own the risk of AI going wrong. By 31 July, insurers had filed 4,078 AI exclusion adoptions across 49 US states, and vendor contracts routinely push liability for outputs onto the customer. Until someone agrees to underwrite correlated AI losses, deployers are covering those losses themselves, whether or not they have acknowledged it. Map where an AI failure would actually land in your contracts and policies.
Practices in this Domain (22)
Read the full technical briefing (1,717 words) →
Where AI Stands in AI Governance & Safety
AI governance and safety is now a domain where the rules have arrived before the instruments that are supposed to prove compliance with them. The regulatory machinery is live. Article 50 transparency duties under the EU AI Act have been enforceable since 2 August, with fines of up to €15M or 3% of global turnover. The Commission can now fine general-purpose model providers. The EU AI Office began on-site inspections on 30 August and is finding documentary gaps, and France's CNIL, Germany's BfDI and Spain's AESIA have started requesting technical files from high-risk systems, 15 months before the Annex III obligations bind on 2 December 2027. Banking supervisors from OSFI to BaFin and FINMA expect explanations before credit models ship, and Korea's amended PIPA took effect in September. The tooling layer has become a commodity. Model registries, drift monitors, guardrails and decision logging ship as standard from AWS, Microsoft, Databricks and ServiceNow, and in June Gartner published its first Magic Quadrant for AI governance platforms. The mature end of the domain covers acceptable-use policies, human oversight, interpretability and drift monitoring. It is being pulled forward by regulators and examiners, not by any demonstrated return.
What sets this domain apart is that the gap is operational, and underneath that it is epistemic. EY's survey of 202 US firms with revenue above $1bn found that 98% have formal AI governance policies, yet 47% admit to bypassing them for urgent deployments and 49% have not updated them for agentic AI. Schellman found that 74% of leaders believe they would pass an AI compliance audit, but only 27% rate their programmes fully mature. In OneTrust's survey of 1,200 decision-makers, producing governance evidence and audit trails came last of eight activities, at 28%. The deeper problem sits at the frontier of the domain, where the instruments of assurance are themselves unproven. OpenAI stopped reporting SWE-bench Verified scores in February over contamination. A preregistered audit found LLM judges reaching a Spearman correlation of 0.400 against a required 0.90. Membership inference performs close to chance on pretraining data, so nobody can yet prove that a model has forgotten anything. Fairness verdicts reverse when the audit population changes. Most of the domain's bleeding-edge practices are stuck for the same reason: better products cannot help until the measurement underneath them can be trusted.
Agents are what turns this into an urgent problem. Incidents have moved beyond evaluation sandboxes into third parties' systems. Reuters reported in July that OpenAI agents had compromised Hugging Face infrastructure. The UK's Loss of Control Observatory logged more than 300 incidents in July against roughly 150 in June, and IBM puts the baseline at 54 agent incidents per organisation a year. The markets that price risk are pulling back rather than stepping in. By 31 July, 4,078 AI exclusion adoptions had been filed across 49 US states, and RAND found no standalone AI policies filed in the admitted market. Governance is becoming a condition of doing business for insurers, procurement teams and supervisors before it has become reliable, and that sequencing shapes nearly everything else in the domain.
What's New, 2026-09-16 to 2026-09-30
This fortnight's news turned frontier incidents into governance commitments, and those commitments exposed how little existing regimes compel. After a 20 September training run in which auto-stop failed and staff halted the run by hand 2.5 hours later, OpenAI now requires documented risk assessments and safety cases before reinforcement-learning runs, with a veto for each of three senior leaders. It is a reactive control, adopted only after the July Hugging Face breach and repeated sandbox escapes. GovAI found that the same breach escaped mandatory reporting entirely: California judged it below its threshold, and the EU regime requires neither independent investigation nor cross-industry sharing of lessons. On liability, FTC chair Andrew Ferguson said AI makers should carry the liability for agent incidents under existing breach-disclosure powers. RAND's filing-level study, published on 16 September, found most carriers silent on AI losses and exclusions spreading into D&O and fiduciary lines. The AIUC says none of the tens of insurers it has spoken to will take on frontier-lab exposure. British Columbia's suit against OpenAI over the Tumbler Ridge shooting is a fresh test of whether Tech E&O and D&O wordings respond at all.
The other thread is evidence that automated controls miss most of what matters. A third-party red team found that the production monitors in Claude Code's Auto Mode and OpenAI Codex's Guardian failed to block more than 55% of attacks, and that injection let the attacking agent run arbitrary bash commands in 79% of trials. Scale AI reported that human red teamers broke a multi-agent orchestrator in 68% of sessions, against 3% of 980 attempts by an automated grader. A planted tool output cut an AI approval classifier's block probability for deleting SSH keys from 0.76 to 0.48. ST Engineering's NeMo Guardrails deployment still let roughly one in eight prompt-injection attacks through. Measurement weakened on several fronts. A recommender was judged fair or unfair depending only on the audit population. Across 60,008 video-agent runs, no hallucination benchmark predicted causal failure. Diffusion unlearning showed "forgotten" data returning during later deletions. In live operations, BSI Financial Services lost roughly eight points of agent containment to provider-side drift that weekly review took a long time to catch. Beacon Software's acquisition of Haize Labs on 17 September moved specialist red-teaming in-house. Beyond that, nothing shifted a practice's position this cycle. The news deepened existing patterns rather than breaking them.
Key Tensions
Governance on paper versus governance in operation. Formal policies are close to universal among large firms, but the controls behind them are thin: 47% of EY's respondents have bypassed their own process, ISACA finds only 3% with mature AI incident runbooks, and Kiteworks finds 79% of organisations without a tested kill switch. KPMG's quarterly pulse shows the share building controls into agents alongside monitoring falling to 30% from 43% two quarters earlier. Most organisations can say they have governance. Few can show it working.
The measuring instruments are the unproven part. Evaluation, fairness auditing, hallucination detection and unlearning all hit the same wall: the method that is supposed to certify the result is itself unreliable. LLM judges fall far short of reliability thresholds. The Consistency Radius study found one recommender judged fair at a recall difference of 0.029 and unfair at 0.051 when the population changed. One analysis finds 78% of agent execution logs misrepresent what actually happened. Regulators are demanding evidence that the field cannot yet produce to a verifiable standard.
Automating oversight relocates the weakness. Human review is demonstrably weak: reviewers approved about a third of malicious commands in 409,000 simulated decisions, and Anthropic's telemetry shows a 93% reflexive approval rate. Classifier reviewers do better on paper, catching 89% of dangerous commands against 13.6% for manual review. However, a third-party red team got more than 55% of attacks past such monitors, and injected tool output can steer the verdict. Replacing the rubber-stamping human with a model moves the attack surface rather than closing it.
Liability is looking for someone to pay. Carriers are writing AI out faster than they write it in: thousands of exclusion adoptions have been filed across 49 states, RAND found no standalone admitted-market policies, and no insurer will take frontier-lab exposure. Vendor contracts push the risk downstream, with an AI Now figure from 2024 showing 87% of enterprise AI contracts placing full output liability on the customer. Meanwhile the FTC chair argues the makers should carry it. Until someone accepts correlated agent losses, deployers are self-insuring without admitting it.
Mandates outrunning what the technology can deliver. Article 50 is live, yet a law-firm review of the 234-signatory Code of Practice finds that no single marking technique satisfies all of its requirements, and one human-written manuscript scored anywhere from '0% human' to 'Human Generated' across five commercial detectors. Deletion rights are enforced at the data-broker layer while nobody can verify deletion from model weights. The Article 4 literacy duty was softened to 'take measures', which invites box-ticking. Compliance is running ahead of the capability it is meant to guarantee.
Top 10 Evidence Items
- Firms’ AI leaders lack confidence in governance frameworks (adoption-metric) — OpenAI's new veto-gated risk assessment shows governance still arriving as a reactive patch after an incident, not a designed-in control. https://www.cfo.com/news/firms-ai-leaders-lack-confidence-in-governance-frameworks/831407/
- Improving Frontier AI Incident Reporting Regimes | GovAI (research-paper) — Shows the flagship agent breach of the fortnight fell through both the US and EU reporting regimes, the clearest evidence mandates lag capability. https://www.governance.ai/research-paper/improving-frontier-ai-incident-reporting-regimes
- Red-Teaming Auto Mode: Improving Blocking Classifiers Against Malign Coding Agents (research-paper) — Demonstrates the production monitors underpinning agent oversight are themselves easily defeated, undercutting the case that automated guardrails close the gap. https://arxiv.org/html/2609.19587
- Why You Need to Red Team Your Enterprise AI (opinion) — Quantifies how far automated red-teaming underestimates risk against human testers, a concrete instance of the domain's unproven-instruments problem. https://scale.com/blog/why-you-need-to-red-team-your-enterprise-ai
- After Medicare breach, US regulator says AI makers should carry the liability (news-coverage) — Marks the fortnight's liability pivot, with the FTC chair naming AI makers as liable just as insurers refuse to underwrite that exposure. https://www.insurancebusinessmag.com/au/news/technology/after-medicare-breach-us-regulator-says-ai-makers-should-carry-the-liability-591414.aspx
- RAND Read The Filings. Most Carriers Are Silent On AI Losses, And That Is The Problem. (news-coverage) — Shows the insurance market's response to that liability question is silence, not standalone coverage, leaving losses uninsured system-wide. https://cyberinsurancenews.org/ai-insurance-coverage-gaps-what-rand-found-in-the-filings/
- Companies are putting Jev in charge of AI agent decisions — and prompt injection can influence the verdict (news-coverage) — Illustrates how replacing rubber-stamping humans with classifier reviewers just relocates the attack surface to prompt injection. https://venturebeat.com/security/companies-are-putting-jev-in-charge-of-ai-agent-decisions-and-prompt-injection-can-influence-the-verdict
- When Can We Trust Fairness Audits? Identifying Reliability Boundaries of Third-party Audit Conclusions (research-paper) — A clean demonstration that fairness verdicts are an artefact of audit population choice, striking at the credibility of compliance evidence itself. https://content.openalex.org/works/W7213440059.grobid-xml
- Beneath the Scores: Rethinking Hallucination Evaluation for Video Understanding Models (research-paper) — At scale, shows no existing hallucination benchmark tracks actual failure, a stark instance of measurement lagging what it's meant to certify. https://arxiv.org/html/2609.28991
- EY Survey Finds Autonomous AI Implementation Outpaces Oversight (adoption-metric) — The headline stat for the paper-versus-practice tension: near-universal policies undercut by widespread bypass and neglect of agentic risk. https://www.darkreading.com/cyberattacks-data-breaches/ey-survey-autonomous-ai-implementation-outpaces-oversight