The AI landscape doesn't move in one direction — it lurches. Some techniques leap from experiment to table stakes in a single quarter; others stall against regulatory walls, technical ceilings, or organisational inertia that no amount of hype can dislodge. Knowing which is which is the hard part. The State of Play cuts through the noise with a rigorously maintained index of AI techniques across every major business domain — classified by maturity, evidenced by real-world adoption, and updated daily so you always know where you stand relative to the field. Stop guessing. Start knowing.
A daily newsletter distilling the past two weeks of movement in a domain or two — delivered to your inbox while the index updates in the background.
Each dot marks the weighted maturity of practices within a domain — hover for a brief summary, click for more detail
AI generating API endpoints, database schemas, or data models from natural language descriptions of requirements. Includes REST/GraphQL API scaffolding and database schema design; distinct from infrastructure-as-code which targets deployment resources rather than application interfaces.
Generating API endpoints and database schemas from natural language remains a bleeding-edge capability with visible production deployment but persistent reliability constraints. The practice has matured from research (2023) through early-vendor tooling (2024) into distributed enterprise adoption (2026) with tier-1 IDE integration (GitHub Copilot PostgreSQL extension GA, Amazon Q GA), yet the fundamental bottleneck—semantic understanding of business logic and governance infrastructure—remains unresolved. Schema format compliance matured sharply: constrained decoding now achieves 99.9% (OpenAI), 99.8% (Anthropic), 99.7% (Gemini) on structured output tasks; that solved the syntax layer. But semantic hallucinations persist at 39-54% rates across models, with schema drift having minimal effect on error rates—signaling that LLMs condition poorly on schema context regardless of deployment strategy. Production deployments exist and operate at scale (Uber QueryGPT 1.2M queries/month, AutoBE 85-90% success rates with 40+ specialist agents), but succeed only through heavy guardrails: intent classification, semantic layers, domain-specific fine-tuning, and deterministic validation pipelines. The adoption-to-value gap is critical: 97% of enterprises deploying AI agents, yet 23% seeing returns; 78% report production incidents from AI-generated code; governance lags (only 20% with mature agent governance). August 2026 evidence confirms the semantic layer is non-negotiable: enterprises using dbt Labs semantic grounding lift Claude accuracy from 90.0% to 98.2% (GPT: 84.1% → 100%), yet BEAVER benchmark on production warehouse data shows 50+ point accuracy collapse (Spider 1.0: 91.2% vs. Spider 2.0 enterprise: 21.3%), confirming benchmarks overstate production viability. The bleeding-edge tension remains unresolved: the bottleneck is not model capability or schema formatting, but semantic understanding and operational infrastructure (governance, validation, schema drift management). Until those layers mature, adoption stays confined to non-critical use cases: rapid prototyping, legacy API bridging, and exploratory analytics with human-in-the-loop validation.
August 2026 evidence confirms semantic understanding and governance remain defining blocking factors. Constrained decoding matured: structured output compliance reached 99.9% (OpenAI), 99.8% (Anthropic), 99.7% (Gemini) via grammar-based token masking—solving syntax layer entirely. Yet semantic hallucinations persist at 39-54% rates (StructHallu-Drift ACL 2026, 1,200 instances); schema drift severity has minimal effect (~44% error rate across all drift levels), signaling LLMs condition poorly on schema context. Critically, benchmark contamination undermines reported metrics: ACL 2026 SPENCE framework reveals Spider exhibits strongest training leakage, with minimal contamination in newer BIRD dataset—establishing that commonly-cited 80-90% Spider accuracy significantly overstates production capability.
Production adoption scales but accuracy crashes on real data. 97% of enterprises deploying AI agents yet only 23% seeing returns; 78% report production incidents. Semantic layer benchmarking confirms the governance formula: dbt Labs 2026 study shows semantic-layer grounding lifts Claude from 90.0% to 98.2%, GPT from 84.1% to 100%—yet raw text-to-SQL without grounding caps at 84-90% accuracy on mission-critical queries. BEAVER benchmark on production warehouse workloads reveals the reality: GPT-4o collapses from 91.2% (Spider 1.0) to 21.3% (Spider 2.0 with 1000+ columns), improving to only 30% maximum even with agentic RAG optimization—50+ point accuracy gap between controlled benchmarks and enterprise schemas. Production text-to-SQL tool ecosystem matures across 8 categories (Querio, Snowflake Cortex, BigQuery, Redshift, ThoughtSpot, Looker, Hex, custom), all requiring semantic routing to reduce errors by 66%; GPT-4o achieves only 10.1% on Spider 2.0 enterprise schemas without schema-aware grounding. Research advances accelerate (PExA achieves 70.2% Spider 2.0 via parallel exploration agents; EXPO-SQL demonstrates clause-level RL rewards outperforming supervised fine-tuning), but advances address margin optimization, not the semantic bottleneck.
Critical production failure modes persist. Schema drift silent failure: stale schema context → invalid SQL without obvious errors (healthcare case: 12 of 28 microservices). Benchmark-to-production gap is systematic: Spider 1.0 accuracy inflated by training leakage; Spider 2.0 enterprise scenarios expose semantic understanding gap as primary barrier. OrderBench (July 2026) confirms semantic correctness gap: LLMs achieve 100% schema validation yet only ~80% semantic success on ordering tasks, with open models showing double-digit unsafe acceptances—demonstrating structured output constraints alone insufficient for production correctness.
Production deployment remains anchored to guardrailed use cases. Governance patterns are non-optional: deterministic validation pipelines, semantic layers (business glossaries, column descriptions), schema-aware RAG, backward-compatible evolution, governance maturity (RBAC, audit trails, SQL review). Supabase Evals benchmark (August 2026) quantifies agent-model gaps: Opus/Kimi 100% unaided on schema tasks, Sonnet 78%→100% with skills, identifying hands-on-keyboard schema building and declarative schema definitions as agent learning opportunities. Adoption-to-value gap (23% ROI) and governance lag (20% mature governance) block tier advancement despite vendor investment and academic progress.
— Supabase open-sourced evals benchmark on real infrastructure tasks including schema building, RLS, migrations; Opus/Kimi 100% unaided, Sonnet 78%→100% with skills; identifies schema generation gaps in agent systems.
— ACL 2026 peer-reviewed research quantifying training leakage in Spider, SParC, CoSQL benchmarks; shows Spider exhibits highest contamination likelihood, undermining reported accuracy claims used for tier classification.
— ACL 2026 paper introducing fine-grained clause-level RL rewards via incremental execution analysis; demonstrates advancement over supervised fine-tuning and existing RL methods on text-to-SQL benchmarks.
— Benchmark study showing GPT-4o collapse from 91.2% (Spider 1.0) to 21.3% (Spider 2.0 enterprise schemas); semantic layer provides +17-23pp accuracy gain across models, demonstrating critical infrastructure requirement for production deployment.
— Enterprise production analysis: dbt Labs benchmark shows semantic-layer grounding lifts Claude from 90.0% to 98.2%, GPT from 84.1% to 100%; raw text-to-SQL tops out at 84-90% accuracy, requiring governance layers for production.
— ACL 2026 peer-reviewed research reformulating text-to-SQL as software test coverage problem via parallel atomic SQL generation; achieves 70.2% execution accuracy on Spider 2.0, advancing agentic approaches to schema generation.
— BEAVER benchmark on production warehouse workloads: pure LLMs 0%, improved to 10% with RAG, capped at 30% maximum; shows 50+ point gap between controlled benchmarks and production data warehouse schemas.
— Production text-to-SQL tool survey comparing 8 categories; GPT-4o 10.1% on Spider 2.0 without grounding; semantic routing reduces errors 66%; documents vendor maturity in schema-aware SQL generation across Querio, Snowflake, BigQuery, Redshift.
2023-H1: Research advances in schema understanding and text-to-SQL, with foundational benchmarks (BIRD) revealing significant accuracy gaps (40% vs 92% human). Early implementations in academic (DBCopilot) and vendor (Postgres/GPT-3) projects. Deployment limited to research and proof-of-concept stages.
2023-H2: Vendor tooling and patent filings accelerate. GraphQL Editor deploys AI-powered schema generation from natural language (September 2023); Google patents schema-based NL-to-API integration (September 2023). Academic research deepens schema routing approaches for massive databases (DBCopilot arxiv, December 2023). No major production deployments; adoption remains in mockup and experimentation phases.
2024-Q1: Vendor product launches and continued academic research. Neurelo launches Cloud Data API Platform (January 2024) with AI-assisted natural language query generation. Academic research advances GraphQL query generation (IJCAI 2024) and reinforces enterprise limitations (CIDR 2024: NL2SQL "far from resolved"). Practitioner feedback highlights API design quality concerns in AI-generated code. Deployment moves into early production but limited to non-critical schema and query generation tasks.
2024-Q2: Vendor consolidation continues with Neurelo maintaining GA platform status and expanding production use for REST and GraphQL API auto-generation from database models. General AI-assisted development tools (Amazon Q) gain enterprise traction with broad productivity claims, though API/schema generation remains a subset of broader capabilities. Adoption remains constrained by accuracy limitations on complex schemas and quality concerns in AI-generated API design. No breakthrough in enterprise-grade NL-to-schema accuracy; deployment still predominantly in lower-stakes schema prototyping and query generation.
2024-Q3: Research advances in schema linking and text-to-SQL continue (E-SQL achieves 66.29% BIRD accuracy; RoSL improves recall by 25.1% for smaller 8B models). Community adoption of GraphQL remains active but schema-related challenges persist (45K StackOverflow analysis). Open-source NL-to-GraphQL tools emerge (talk-to-graphql). Critical assessments surface recurring reliability concerns: 52% error rate in AI-generated API code, security vulnerabilities, and hallucinations. Neurelo tutorials show iterative schema refinement in production tool. Overall trajectory: incremental improvements on specific benchmarks (BIRD) but no breakthrough in production adoption; accuracy remains constrained by schema complexity, and production deployment limited to non-critical schema/query generation tasks.
2024-Q4: Focused research effort on GraphQL query generation (EMNLP 2024 industry track reports ~50% accuracy on new 10,940-pair dataset from IBM/StepZen; open-source NL2GQL dataset released October 2024). Academic interest in schema generation from requirements specifications continues (November 2024 publications). Neurelo expands operational workflows with custom API endpoint deployment via natural language queries integrated into git-based version control (December 2024). Critical reliability barriers persist: the accuracy gap between LLM-generated and human-authored code remains significant. Industry consensus emerges: custom fine-tuning and domain-specific training data are essential; zero-shot generation inadequate for production schemas. No breakthrough in enterprise adoption; market remains characterized by research intensification and vendor optimization of non-critical use cases (rapid prototyping, mockups, low-stakes query generation).
2025-Q1: Research shifts toward direct schema generation from natural language (SchemaAgent multi-agent framework with 381-pair benchmark; Nixa addresses dynamic schema discovery in multi-tenant SaaS). Vendor ecosystem expands with AI App Builder entering GA schema generation market. Open-source tools mature (GQLPT+APIPT for GraphQL/REST). Developer confidence remains low despite high adoption: Q1 2025 surveys show 90% use but 3% high trust, 66% requiring substantial modifications, accuracy across tools ranges 31–65%. Critical assessment emphasizes technical debt accumulation and systemic reliability barriers. Production deployment unchanged: non-critical experimentation only, no enterprise-grade schema adoption for critical systems.
2025-Q3: GraphQL specification update (September) optimizes for AI/LLM integration with OneOf input objects and Schema Coordinates. User study (September) shows NL2SQL systems achieve 75% accuracy and 10–30% faster query completion vs. traditional SQL, but persistent user frustration with refinement cycles. Security vulnerabilities in production AI code assistants (Amazon Q Developer prompt injection/RCE, August) highlight ongoing risks. Ecosystem consolidation continues; no breakthrough in enterprise adoption. Production constraints unchanged: accuracy gaps, design quality below human baselines, security risks preclude critical system deployment.
2025-Q4: Research advances in schema-aware generation (GenLink multi-model learning achieving 67.34% BIRD accuracy, first systematic normalization-impact study). Oracle releases GA GraphQL schema generation from relational databases. Production case study demonstrates API code generation from natural language with zero-shot success. Vendor ecosystem matures with Oracle and existing platforms. However, critical practitioner analyses identify four blocking issues—schema awareness gaps, accuracy limitations, poor optimization, security risks—alongside production brittleness from schema churn. Enterprise adoption for critical systems remains negligible; deployment limited to non-critical prototyping and low-stakes query generation. Accuracy and production reliability remain below thresholds for enterprise-grade schema/API generation.
2026-Jan: Breakthrough in NL-to-SQL accuracy: BAR-SQL achieves 91.48% on BIRD benchmark, surpassing Claude 4.5 and GPT-5, indicating narrowing of the gap. Production deployments mature: IBM deploys zero-config NLQ-to-SQL at enterprise scale (98.7% success across 17K tables, 3.1s latency). AWS Amazon Q Developer reaches GA with SmugMug case study (100% productivity gain). However, critical barriers persist: LLM planning accuracy collapses to 30-49% with 300+ API endpoints, improving only with semantic metadata and declarative APIs. DevPals demonstrates legacy API bridging in production (60% integration TCO reduction, 90% error reduction). Patent disclosures (IBM, others) focus on semantic data layers and agentic guardrails to prevent hallucination in enterprise NL-to-SQL. Accuracy ceiling in January 2026 remains: zero-shot generation inadequate for heterogeneous schemas; semantic metadata, domain-specific fine-tuning, and constraint-based generation required for production reliability. NL-to-API remains limited to non-critical query generation, rapid prototyping, and legacy system integration.
2026-Feb: Vendor ecosystem expands with AWS Bedrock structured outputs (constrained decoding for schema compliance), Oracle NetSuite N/LLM embedding native schema generation in ERP, and Apollo GraphQL agent skills for automated schema design—but each vendor acknowledgement includes caveats about AI generation quality and reliability. Real-world incident documentation surfaces schema drift patterns and API brittleness (type shifts, silent field changes causing data corruption). Practitioner testing reveals stochastic LLM API failures across Anthropic, Google, and AWS for structured output tasks. Deployment barriers persist: schema evolution causes hidden coupling; zero-shot generation inadequate; LLM reliability not deterministic. Enterprise adoption for critical schemas unchanged; non-critical prototyping and legacy bridging remain primary use cases.
2026-Mar: Product ecosystem accelerates with SharpAPI, Netlify Agent Runners, and expanded Neurelo Series A funding ($5M). Real-world deployments surface: QueryLytic at B2B SaaS (schema compression, validation, multi-database support), MANTA production instances (ChemoMaker pharmacy, Manufacturing BI). Enterprise adoption metrics mature: Bank of America Erica (19.5M+ users, 100M+ requests, 30% call center reduction), Microsoft Power BI, Tableau Ask Data (63% self-service analytics increase). Constrained decoding frameworks proliferate (Guidance, Outlines, XGrammar) but JSONSchemaBench benchmark (10K schemas) reveals significant feature coverage gaps across all frameworks. Critical assessment surfaces: practitioner analysis quantifies nested JSON schema failure rates (15-25% at 3+ nesting levels); controlled research finds zero end-task success even with formal JSON schemas, indicating semantic understanding remains the bottleneck, not schema syntactic compliance. Vendor landscape confirms: production adoption accelerating for non-critical query generation and legacy API bridging, but fundamental reliability barriers persist. Schema optimization (PARSE framework) emerges as research direction, treating schema design itself as a tuning problem rather than static interface contract.
2026-Apr: Bench-to-production gap widens on multiple fronts. SQLStructEval and Omni Analytics (4,602 failed queries) confirm that 81.2% of production SQL errors are semantic rather than syntactic, and GPT-5 drops from 86% on Spider 1.0 to 29% on enterprise-scale BIRD-Interact — establishing that benchmark scores overstate real-world reliability by a wide margin. dbt Labs benchmark validates the semantic layer approach: text-to-SQL at 85-90% accuracy vs 97-100% with structured semantic layer, confirming the bottleneck is schema understanding not LLM capability. AWS production deployment (Amazon Q with PostgreSQL schema generation in database migration) and normalized schema design research (16.8% QA accuracy gain from 3NF schemas) provide positive signals for constrained use cases, while structured output analysis identifies four unresolved failure layers — semantic validity and distribution shift remain outside constrained decoding's reach. Enterprise deployment evidence expanded: Microsoft engineer documented production use of Copilot Chat for database schema generation from natural language in enterprise context; schema drift documented as a critical production failure mode — healthcare case study found 12 of 28 microservices with schema drift causing silent failures until automated validation deployed; xAI shipped structured outputs GA alongside tool-calling failure analysis identifying schema mismatches and context limitations as primary root causes. Production deployment continues anchored to low-stakes use cases; enterprise-grade NL-to-schema for critical systems remains blocked by semantic reliability gaps and schema drift brittleness.
2026-May: Governance patterns solidify and production scale evidence emerges alongside persistent semantic bottleneck. Uber's QueryGPT (1.2M queries/month) documents the production formula: 20+ iterations of intent classification, domain-specific workspace clustering, and context limiting—not better models—reduced query authoring from 10 to 3 minutes at scale. AutoBE GA ships complete backend generation (Prisma schema, OpenAPI specs, NestJS) from conversational requirements via 40+ specialized agents with 85-90% success rates and 100% compilation guarantee, establishing production viability for non-critical backend scaffolding. Bytebase synthesis confirms deterministic governance (context limiting, structured evaluation, validation layers) as the success pattern across OpenAI, Google Cloud, Vercel, and Hex production deployments. DivSkill-SQL research achieves +11.1 pts on Snowflake and +8.3 on BigQuery with 3x fewer hallucinated schema references via agentic ensemble optimization. Structured Output Benchmark quantifies core reliability challenge: LLMs produce syntactically valid JSON with semantically incorrect hallucinated values. Security analysis identifies 10 production risks (hallucinated schema, PII exposure, cost explosions) requiring deterministic validation pipelines. Semantic context (business rules, glossaries, descriptions) confirmed as the bottleneck across independent studies—near-zero accuracy without metadata enrichment. Enterprise adoption for critical systems unchanged; deployment anchored to prototyping, legacy API bridging, and exploratory analytics with human-in-loop validation.
2026-Jun: Vendor ecosystem and negative-signal research both accelerate. Microsoft released GitHub Copilot PostgreSQL extension with GA NL-to-DDL generation (@pgsql prompts generating table creation and schema modifications), confirming tier-1 IDE vendors treat schema generation as production-ready feature. SANE research validates schema-aware approach: LLMs reliably generate SQL schemas from natural language when given schema constraints and structured prompting, no fine-tuning required—establishing guardrails as the differentiator, not model scale. FastAPI production templates document model-specific challenges: Claude Sonnet excels at async patterns while ChatGPT falls back to deprecated Pydantic v1 syntax 40% of the time, requiring explicit prompt engineering. Critical reliability research documents constraint decay in agentic code generation: 30+ point drop in assertion pass rates from baseline to fully constrained production task; ceiling effect observed where agent performance collapses rather than gracefully degrade. Industry benchmark of 34 LLMs on text-to-SQL reveals persistent 20%+ error rates on complex queries from incomplete parsing, hallucinated columns, and constraint mapping failures. Agentic technical debt framework formalizes operational cost structure: probabilistic systems incur recurring stochastic tax independent of debt accumulation (tool contracts, routing logic, governance). Production deployment patterns unchanged: schema-aware approaches enable higher accuracy, governance layers prevent hallucination damage, but zero-shot generation remains inadequate for heterogeneous enterprise schemas. Enterprise adoption for critical systems remains constrained by semantic understanding bottleneck and operational complexity.
2026-Jul: Format compliance confirmed as solved; semantic understanding confirmed as unsolved. Structured output compliance via constrained decoding now reaches 99.9% (OpenAI), 99.8% (Anthropic), 99.7% (Gemini)—effectively eliminating the syntax layer as a barrier. Yet StructHallu-Drift research (ACL SURGeLLM, peer-reviewed, 1,200 instances) finds 39-54% of structured outputs contain semantic hallucinations, with schema drift severity having minimal effect (~44% error rate across all drift levels)—demonstrating that LLMs condition poorly on schema context regardless of deployment strategy. A critical silent failure mode is now formally documented: open-weight models cease tool calling entirely when structured output constraints are enabled, returning syntactically valid JSON with hallucinated content (Constraint Tax research, arxiv 2026), making agentic API/schema generation unreliable in production without explicit two-pass mitigation. Later-month evidence confirmed both research progress and continued production caution: QBridge (ACL 2026) introduced a Gold Query intermediate representation with agentic self-correction, improving results across Spider, BIRD, and robustness benchmarks, while Spider 2.0-AIFunc extended evaluation to AI-native SQL functions on Snowflake, finding proprietary models plateau at 67-70% and identifying schema grounding and AI-function parameterization as the dominant error categories. Vendor commitment to NL-to-schema deepened at platform scale: AWS shipped agentic AI for DMS Schema Conversion GA (200-object migrations in 15 minutes vs. 45 manual, 60-70% speedup on larger projects) and Oracle GA'd Select AI on Autonomous Database for NL-to-SQL, RAG, and synthetic data generation. Research on RAG-plus-constrained-decoding for OpenAPI-based API invocation confirmed constrained decoding reliably prevents format violations but trades off against retrieval completeness, while a critical practitioner analysis reiterated that BIRD/Spider benchmark scores are "demo predictors, not production forecasts"—production reliability still requires iterative agentic loops and dedicated schema/meaning/trust context layers, not baseline model capability.
2026-Aug: New peer-reviewed research (ACL 2026: SPENCE, EXPO-SQL, PExA) advanced text-to-SQL methods while also exposing training-data contamination in benchmarks like Spider, and Supabase open-sourced an agentic evals benchmark showing Sonnet needs explicit skills to match Opus/Kimi's unaided 100% on real schema tasks. Multiple independent studies (Querio, Atlan/dbt Labs, esremedia, BEAVER) converged on the same pattern: raw accuracy collapses from 90%+ on clean benchmarks to 0-30% on production enterprise schemas, with semantic-layer grounding closing most of the gap (e.g., GPT-4o 84.1%→100%, Claude 90.0%→98.2%).