🏛️ AI Governance & Safety
Practices for evaluating, governing, and ensuring the responsible deployment of AI systems. Deeply polarised: model evaluation and bias auditing are good practice, but nearly half the domain is bleeding-edge — alignment research, interpretability, and AI safety benchmarking lack production-grade tooling. Regulatory pressure is accelerating adoption of the mature practices while the frontier remains largely academic.
The Headline
Nearly every large company has written AI rules, but few can prove they work. Financial regulators have started asking for that proof, and even a leading AI lab admits its insurance has fallen short.
The Picture
If your AI governance is mostly documents, you are in the majority. EY found that 98% of US firms with more than $1 billion in revenue have formal AI policies, yet 47% admit they have bypassed them for an urgent deployment. Even the basic map is often missing: only 20% of 550 executives surveyed by ServiceNow keep a live inventory of the AI running in their business. A small group is pulling ahead with narrow, owned use cases where a person keeps final sign-off. At Daimler Truck, AWS reports that its agents (software that acts on its own without being prompted) stop at a prepared fix for an engineer to approve, and that incidents are resolved about 88% faster. Financial supervisors are now asking the majority to show that their controls work in practice, not merely that they exist.
This Fortnight
The Financial Stability Board published 12 sound practices for AI in finance on 15 October, and the UK Financial Conduct Authority said firms must show their controls work. The practices are non-binding. They name an organization-wide AI inventory and model documentation as core records, and they call for each AI use to be risk-assessed from inception to retirement. The FCA also expects firms to tell customers how AI was used, so regulated firms should expect examiners to ask for the inventory and the evidence, not the policy.
A survey of 400 US chief information security officers found 98% have a generative AI use policy, but only 47% enforce it with tools built for AI. Another 23% rely on an honor system. Where adoption ran ahead of governance, 77% had investigated an AI-linked incident, so ask your security team which of these three groups you are in.
Anthropic's IPO filing says its insurance did not fully cover the company's $1.5 billion copyright settlement. The filing warns that Anthropic may be unable to buy cover for some AI liabilities "on commercially reasonable terms, or at all." Some specialty insurers said they would not adopt the industry's standard generative AI exclusions and would underwrite case by case instead. Before renewal, check whether your policies exclude AI, cover it explicitly, or say nothing.
New research broke every multi-layer safety filter it tested, and the same bypasses worked on 29 filters it had never seen, eight of them commercial products. Separately, Microsoft's documentation shows that its protection against attacks hidden inside documents is off by default. Guardrails (safety rules meant to stop AI doing the wrong thing) should be one layer of defense, so limit what your agents can reach and do.
A small preregistered study found that people reviewing AI output caught only about 24% of its natural errors. Deliberately seeding the work with test errors lifted that to about 31% at best. AWS has also closed its managed human-review service to new customers. If your safeguard is human-in-the-loop review (a person checks each AI output before it ships), measure what your reviewers actually catch.
Coming Up
Providers of generative AI tools already on the EU market before 2 August 2026 must mark AI-generated content in machine-readable form by 2 December 2026. No single marking technique yet meets every requirement, and a pilot study found that providers' disclosures cannot be checked independently. Ask each vendor in writing how its output is marked and how you could verify it.
EU high-risk AI obligations apply from 2 December 2027, and only one of the harmonized standards meant to make them checkable has been published. Standards bodies have missed two deadlines, the EU AI Office is thinly staffed, and several member states still have no enforcement authority. The obligations themselves have not changed, and records added after a system is built tend to fail review, so build your inventory and risk classification now rather than reading the delay as relief.
Singapore's Monetary Authority issued final AI risk guidelines on 7 October, with governance expectations from 7 October 2027 and lifecycle controls by 7 October 2028. The guidelines cover all financial institutions and all forms of AI, and they require inventories plus board and management accountability. Firms with Singapore operations should assign a board owner and start the inventory this year.
What's Hard About This
You cannot govern AI you cannot see, and agents are multiplying faster than inventories. One large regulated asset manager scanned its systems and found about 13,000 agents, of which it considered only about 1,000 valid. Discovery tools help, but their blocking is narrow: Microsoft's new page detects several desktop agents and can block only one, and only on managed Windows devices.
The numbers used to judge AI are less reliable than they look. OpenAI stopped reporting a leading coding benchmark after an audit found substantial errors in 59.4% of its hard problems, and automated judges consistently agree with human reviewers less than humans agree with each other. Choose vendors using tests built on your own work, not public leaderboards.
Nobody in the chain wants to hold the risk, so by default it lands on you. A RAND study counted 112 AI exclusions in admitted-market insurance filings against seven endorsements adding cover. On 26 October the chair of the US House Oversight Committee opened an investigation into a Delaware proposal that would shield the funders of AI agents from liability for harm the agents cause. Negotiate liability and notice-of-change terms in AI contracts rather than accepting the vendor's standard paper.
Practices in this Domain (22)
Read the full technical briefing (1,812 words) →
Where AI Stands in AI Governance & Safety
AI governance has become a domain in which almost everything can be bought and very little can be proved. Model registries, guardrails, drift monitors, agent audit logging, policy managers and red-teaming harnesses now ship as standard features from AWS, Microsoft, Google and Databricks, as well as from a crowd of specialists. Those specialists are being absorbed. Dynatrace completed its $915M acquisition of Arize at the start of October. Lakera, promptfoo, ProtectAI, CalypsoAI and Prompt Security now sit inside larger security suites. The controls these products are meant to support remain thin. ZK Research surveyed 400 US CISOs and found that 98% have a generative AI acceptable use policy, but only 47% enforce it with AI-aware tools and 23% rely on an honour system. EY found the same 98% with formal policies among $1B-plus US firms, and 47% admitting they had bypassed governance for an urgent deployment. A DigiCert survey fielded in May found that only 64% of organisations had inventoried their AI systems. ServiceNow found that only 20% of 550 executives keep a live inventory. Futurum found that only 45% of 839 decision makers log agent actions as a control. The pattern holds across the domain: the documents exist, but the day-to-day enforcement behind them does not.
What sets governance apart from the neighbouring AI domains is that its output is evidence, and evidence is where it is weakest. OpenAI stopped reporting SWE-bench Verified after an audit found substantial errors in 59.4% of hard problems. A review of 259 studies found that none of 17 prominent verified agent benchmarks reported jointly controlling for contamination, nondeterminism and cost. An audit of 900 human-labelled question–answer pairs found that every automated label source, seven LLM judges among them, agreed with a human annotator less than two humans agreed with each other. SHAP appears in 39.1% of the titles or abstracts of 550 credit-risk papers, yet only 1.3–1.5% evaluate explanation quality. kagent describes its own prompt traces as best-effort, not a compliance log. Evaluators from METR and Redwood Research estimate that about 10% of eligible internal Codex traffic went unmonitored between May and July. No unlearning method yet gives a guarantee a regulator would accept that a model has forgotten specific data. When the instruments fail, buyers fall back on outside attestation. An ETR flash study found that 70% of respondents would be more willing to deploy with third-party evaluation in place. California has legislated an evaluator registry, while one of its own legislators warns that evaluators contract with the firms they assess.
Momentum is real where the scope is narrow and someone owns the outcome. Uber's Persona Guardrail lifted accuracy from 85.7% to 95.9% and cut the false-approved rate from 25.0% to 4.7%, though by Uber's own account. At Daimler Truck, AWS reports that self-healing agents stop at a prepared pull request for engineer approval, with incident resolution about 88% faster. Financial supervisors are turning practice like this into expectation, through the Financial Stability Board, the UK FCA and Singapore's MAS, whose final AI risk guidelines set deadlines in October 2027 and October 2028. Progress stalls wherever governance depends on general-purpose judgement at scale. In 409,000 simulated decisions, human reviewers approved about a third of malicious commands. Kiteworks found that 79% of organisations lack a tested kill switch. RAND counted 112 AI exclusions in admitted-market insurance filings against seven endorsements. The EU regime that drove much of this is itself running late: its AI Office has about 125 staff, only one harmonised standard has been published, and Annex III high-risk obligations do not apply until 2 December 2027.
What's New, 2026-10-14 to 2026-10-28
The biggest move this fortnight came from supervisors. On 15 October the Financial Stability Board published 12 non-binding sound practices for financial institutions. They call for systematic assessment of the materiality and risk of AI use cases, at inception and thereafter. They also name an organisation-wide AI inventory and model or system cards as core documentation, and eight of the practices cover the lifecycle from inception to retirement. The UK FCA's review of AI in financial services described firms deciding where AI should abstain or escalate to a qualified person with authority to intervene. It expects firms to tell consumers how they used AI and its limits, and to show that their controls work in practice. In Europe, the gap between the law and the machinery to apply it became concrete. A European Parliamentary Research Service briefing records that CEN and CENELEC missed two deadlines for high-risk standards, and only one harmonised standard, for quality management, had been published by October. The AI Office has sent more than 30 requests for information with about 125 staff, and CNN reports that several member states still have no enforcement authority. In insurance, Anthropic's S-1 said its cover did not fully pay for the $1.5bn Bartz settlement, and that it may be unable to buy cover for some AI liabilities "on commercially reasonable terms, or at all". Excess and surplus executives at the Wholesale & Specialty Insurance Association said they would not adopt ISO's generative AI exclusions. On 26 October, House Oversight chair James Comer opened an investigation into Delaware's proposed AI Company, which would shield funders from harm caused by an AI agent.
Research kept dismantling both defences and measurements. The BRANCH tree-search attack achieved 100% success against six multi-scanner guardrail systems, and its bypasses transferred without further work to 29 unseen guardrails, eight of them commercial. Memory injection persistently compromised ChatGPT, Claude and Gemini across fresh sessions. Microsoft's documentation shows that document-attack detection in Azure Prompt Shields is off by default. GEMMU found that multimodal unlearning benchmarks overstate forgetting, and TRACE reconstructed removed text from model snapshots across six unlearning methods. NAME-Bench showed all twelve vision-language hallucination detectors tested wrongly flagging non-assertive mentions at rates of 14–80%. AI2's FairStress found frontier models over-correcting: Claude Opus 5 and GPT-5.5 picked the minority-coded candidate 60–65% of the time when neither candidate was stronger. A preregistered error-injection study found reviewers catching only about 24% of natural AI errors, about 31% at best under heavy injection, and AWS closed its Augmented AI human-review service to new customers. Contrary Research's account of Llama 4 Maverick, which placed second on Arena in April 2025 but 32nd when the public model was retested after 27 private variants, added to the case against leaderboard-led procurement. Credentials kept multiplying, with ISACA's AI governance certification in beta for early 2027 and CompTIA's AgentOps+ due in Q2 2027. Allianz's 130,000 completions of a one-hour course measure attendance, not competence.
Key Tensions
Policies on paper, enforcement elsewhere. Near-universal policy coverage sits on top of patchy enforcement: 98% of CISOs have a generative AI acceptable use policy, but only 47% enforce it with AI-aware tooling. Discovery tools only partly close the gap. Microsoft's Shadow AI page detects several desktop agents but blocks only OpenClaw, and only on Intune-managed Windows devices. Salt Security found that just 12.5% of organisations can consistently trace an agent's activity from instruction to outcome.
The measuring instruments are unreliable. Benchmarks, LLM judges, attribution methods and unlearning metrics all fail under independent audit. Rankings for LIME, KernelSHAP and Occlusion shift with the classifier, the baseline and the compute budget. Open-MMUnlearning found no metric reliable on both faithfulness and robustness. Fairness verdicts flip with the audit population. Governance therefore rests on numbers that cannot bear the weight regulators and buyers put on them.
Human review is degrading under volume. Reviewers in 409,000 simulated decisions approved about a third of malicious commands, and Anthropic telemetry shows a 93% reflexive approval rate among Claude Code users. Vendors are handing approval to classifiers instead: Anthropic reports its auto-mode classifier catching 89% of dangerous commands, against 13.6% for manual review. Those classifiers are attackable too. A planted tool output cut one approval model's block probability from 0.76 to 0.48.
Regulation is outrunning its own infrastructure. The EU has enforcement powers but few enforcers and almost no harmonised standards, and its Article 4 literacy duty was softened to taking measures to support development. Roughly half of 157,000 firms surveyed by Eurostat cite unclear legal consequences or data protection concerns as reasons not to adopt AI. Financial supervisors such as the FSB, the FCA and MAS are filling the gap with guidance rather than statute.
Nobody wants to hold the risk. Insurers filed 112 AI exclusions against seven endorsements, and Aon estimates that more than 90% of AI exposure sits in policies that are silent on AI. Anthropic now warns investors that some AI liabilities may be uninsurable. Vendors push risk downstream as well, with an AI Now finding that 87% of enterprise AI contracts put full liability for model outputs on the customer. Delaware's proposal would shield the funders of AI agents.
Top 10 Evidence Items
- Midmarket AI gap: Small IT teams are being asked to do enterprise-sized jobs (adoption-metric) — This is the canonical statistic behind the briefing's core tension: near-universal policy with only 47% real enforcement. https://www.networkworld.com/article/4239759/midmarket-ai-gap-small-it-teams-are-being-asked-to-do-enterprise-sized-jobs.html
- The insurance bombshell hidden in Anthropic’s IPO filing (news-coverage) — Anthropic's own S-1 admitting uninsurable AI liability is the frontier-lab proof point for the 'nobody wants to hold the risk' tension. https://www.insurancebusinessmag.com/uk/news/cyber/the-insurance-bombshell-hidden-in-anthropics-ipo-filing-595095.aspx
- AI Benchmarks: Climbing the Wrong Hills | Contrary Research (industry-report) — Llama 4 Maverick's leaderboard collapse is the sharpest concrete case against trusting benchmark rankings as governance evidence. https://research.contrary.com/report/ai-benchmarks-climbing-the-wrong-hills
- BRANCH: Bypassing Multi-Scanner AI Guardrails (research-paper) — BRANCH's 100% success and transferability across 29 guardrails shows how thin the guardrail layer that AWS/Microsoft/Google sell really is. https://arxiv.org/html/2610.10742v1
- Ransomware logs and Copilot memory put AI agent records on the eDiscovery agenda (news-coverage) — The 10% unmonitored Codex traffic figure is the direct evidence that audit logging claims don't hold even inside a frontier lab. https://edrm.net/2026/10/ransomware-logs-and-copilot-memory-put-ai-agent-records-on-the-ediscovery-agenda/
- AI Act – Implementation timeline (industry-report) — One harmonised EU standard and a missed CEN/CENELEC deadline shows the regulation-outrunning-infrastructure tension in concrete terms. https://eprs.europarl.europa.eu/EPRS_ATA(2025)772906
- Staying in the Driver’s Seat: Designing Better Human Oversight for AI (news-coverage) — Reviewers catching only ~24-31% of injected errors is the empirical case that human oversight degrades exactly where it's relied on most. https://aiinstitute.hbs.edu/staying-in-the-drivers-seat-designing-better-human-oversight-for-ai/
- Shadow AI Puts Customer Data and CX at Risk as Agents Raise the Stakes (news-coverage) — Only 12.5% of organisations can trace agent activity end-to-end, the sharpest discovery-gap figure behind the shadow AI narrative. https://cxtoday.com/shadow-ai-puts-customer-data-and-cx-at-risk
- Delaware wants to let AI run entire companies: what it means for insurers (news-coverage) — Delaware's proposed liability shield, now under congressional investigation, is the clearest sign of risk being engineered away rather than managed. https://www.insurancebusinessmag.com/us/news/breaking-news/delaware-wants-to-let-ai-run-entire-companies-what-it-means-for-insurers-595079.aspx
- Explainable Artificial Intelligence in Credit Risk: A Systematic Evidence Map for Regulated Credit Decisions (research-paper) — SHAP's 39% adoption against 1.3-1.5% quality evaluation is the starkest illustration that explainability is used far more than it is validated. https://www.mdpi.com/2227-7072/14/11/292