Natural language data querying & semantic search
225 evidence items
AI that translates natural language questions into SQL queries and performs semantic search across structured and unstructured data. Includes text-to-SQL tools and embedding-based retrieval; distinct from enterprise RAG which retrieves from document collections rather than databases.
Overview
Natural language data querying lets people ask questions of databases in plain words, with AI writing the query or retrieving by meaning rather than keyword. It is worth caring about because the tooling is generally available and the promise of self-service analytics is real, but it sits at a leading-edge practice, steady, because accuracy falls sharply on real enterprise schemas and the failures are quiet: queries run cleanly and return plausible, wrong answers. Deployments that work do so on top of a curated semantic layer, governed definitions and a validation process built first. Until generated queries themselves hold up on unfamiliar schemas, adoption remains a bespoke engineering commitment, not a clear path.
Current Landscape
Platform vendors now ship natural-language querying as a governed, generally available service. Databricks states that its Genie One MCP for natural-language analytics across a workspace is generally available, grounded in Genie Ontology, which it calls a governed semantic layer, though it gives no accuracy figures. Snowflake recommended on 28 August 2026 that customers move from Cortex Analyst to Cortex Agents, which signals a shift from single-shot SQL generation to agentic orchestration.
Database vendors are building the capability into the engine itself. Oracle documents Select AI generating SQL from natural-language prompts inside Autonomous Database. Google documents a QueryData API in AlloyDB AI that uses context sets of templates, facets and value search queries to translate natural language into secure SQL. The same feature set pairs embedding generation with hybrid vector and full-text search. Google claims filtered vector search up to 10x faster than standard pgvector, without stating its method.
Vendor adoption claims are large but self-reported. At Snowflake World Tour Paris on 15 October 2026, Snowflake claimed that 10,000 of its 15,000 customers use its AI and agentic features daily. It also said TF1 measures campaign impact in three minutes, compared with four days previously. Earlier earnings coverage credited AI with 37% product growth at Snowflake. Many of the business-user features are still immature: Cortex Sense is in private preview and deep research in Snowflake CoWork is in preview.
Consolidation is a risk for anyone adopting a standalone tool. Meta is acquiring Sundial, whose product let business users query a data warehouse in natural language. Secondary reporting says Sundial will stop serving external customers once the acquisition is completed. No closing date, wind-down window or export path for its semantic-layer definitions has been published, although OpenAI was a named customer.
Named deployments span finance, energy and retail. S&P Global Energy made its structured data estate conversational with Databricks Genie agents and MCP. Moody's deepened its AI work with Google to expose its proprietary data. Carrefour built an internal data assistant that turns repetitive queries into instant SQL, and did so in a weekend. Inside Databricks, the marketing analytics team rolled Genie out to more than 200 business users from a baseline of about 25% of their time spent on low-value tasks. It reported no post-deployment figures.
Uber's QueryGPT shows how much curation a production system needs. Uber reports that its first version, which used vector similarity search over 7 tier-1 tables, saw declining accuracy as more tables were onboarded. The production design adds curated domain workspaces, an intent agent, a table agent that users confirm and a column-prune agent, because some schemas ran to 40-60K tokens. Uber estimates query authoring falls from about 10 minutes to about 3 minutes, on a platform handling about 1.2 million interactive queries a month. It gives no accuracy figure.
Production architecture is converging on cost, latency and access control. AWS has published guidance on cutting Text2SQL latency with parameterised query templates. It has also described a multi-tenant analytics agent that enforces row-level security. A fine-tuned 9B Qwen3.5 assistant that runs on an 8 GB laptop GPU reached 55.6% on BIRD Mini-Dev for PostgreSQL, against 36.0% for the best published baseline. That makes self-hosted small models a credible option.
Learning schema from usage is emerging as an alternative to documenting it. An industrial team's schema-free system builds a knowledge graph from historical SQL query logs. Its authors report 95.7% execution validity on the Beaver benchmark, against 60.8% for embedding-based retrieval. On an internal deployment of 240 tables, expert data engineers preferred it 70% to 10% in blind pairwise comparison, though the evaluation is the authors' own.
Iterative repair and verification are the research consensus. ACTS-SQL uses agentic, critic-oriented tree search to correct SQL. VET traces execution so results can be verified. An academic insurance-domain pipeline combining SQLite execution feedback with SQL-to-natural-language round-trip checking reported 90% correctness using Mistral across seven coordinated LLM calls. That result covers only 60 queries on a synthetic database and gives no numeric baseline.
Ambiguity in the question is now a measured failure mode in its own right. On the ARCS benchmark of 311 naturally ambiguous questions over unmodified databases, the best model reached only 51% end-to-end execution accuracy and no open-source model exceeded 27%. The authors found that frontier models from different providers write SQL equally well but disambiguate very differently. They propose structured multiple-choice clarification in place of free-form dialogue. Separate work on agentic text-to-SQL also treats resolving underspecified questions, rather than raw correctness, as the open problem.
Real users ask underspecified questions far more often than benchmarks do. In a study of 14 physicians querying MIMIC-IV, freely composed queries triggered clarification requests in 24.8% of cases, against 1.8% for predefined questions. A conversational interface raised the mean System Usability Scale score from 68.2 to 85.9, and 86% preferred it for real-world adoption. The authors conclude that benchmark-style evaluation can substantially underrepresent the ambiguity of real clinical use.
Queries spanning several databases largely fail. DSABench's 815 human-reviewed tasks require agents to find the right databases, coordinate results between them and execute the queries. Median route-plus-execution success was 6.69% to 7.98% without clarification, and the best clarified configuration reached 16.99%. Its authors conclude that retrieving plausible databases or producing correct local SQL is not enough.
Benchmark credibility itself is in question. A UIUC audit found that half of the BIRD benchmark's gold answers are wrong. Michael Stonebraker argues that LLMs score 0% on real enterprise databases. ESQ-Bench reports that NL2SQL on Oracle diverges quietly from correct results. Elastic has run its own benchmark of whether LLMs can write ES|QL, and other researchers are extending text-to-SQL to MongoDB's query language. Both show the problem now reaches well beyond SQLite-style evaluation.
Evaluation tooling is maturing, but reporting is still uneven. IBM has released Text2SQL-Eval as open source, with over twelve metrics spanning execution accuracy, SQL equivalence and LLM-as-judge scoring across SQLite, PostgreSQL, MySQL and DB2. A survey of 114 text-to-SQL studies found that only 31 report explicit token values. Its authors say inconsistent scopes limit direct comparison of end-to-end inference consumption.
Semantic-layer vendors set accuracy expectations below the marketing. Solid tells customers to scope each semantic model to 5-10 tables, to treat 85% as the minimum for anything relied on as automated, and to expect early pilots at 85-90%. It puts 97%+ within reach only after a validation and correction loop. Omni describes a coding agent that read a semantic-model warning, hand-wrote its own join anyway and reported $55.5 billion in pipeline. That is the kind of silent error that deterministic compilation is meant to prevent.
Connecting a general-purpose model straight to a warehouse fails in ways users cannot see. BI vendor Omni relays that when Checkr connected an LLM directly to Snowflake, the same question returned different, incorrect answers every time, and questions needing a join were answered from a random field. Omni, which has an interest in the outcome, argues that chat suits ad-hoc exploration but needs a governed semantic layer underneath, because models cannot enforce per-user row-level permissions.
Governance and validation remain the binding constraint on broader adoption. Practitioner guidance favours guardrails over full autonomy, and analysts warn that Snowflake Cortex Sense will ingest a BI layer as it stands, whether governed or not. Vendors cannot yet show measured accuracy on unfamiliar, multi-database or ambiguous questions. Until they can, each deployment depends on curated semantic models, benchmark question sets and access controls that the adopting team builds and maintains itself.
Tier History
Evidence (225)
— Google documents a GA in-database QueryData API that turns natural language into secure SQL using context sets, alongside embedding and hybrid search. Speed claims are vendor benchmarks with no stated method. Publication date is not shown on the page.
— Negative signal from a self-interested vendor: Checkr's LLM wired directly to Snowflake gave different, incorrect answers to the same question and skipped joins. Omni also says models can't enforce row-level permissions.
— User study with 14 physicians on MIMIC-IV: free queries needed clarification 24.8% of the time vs 1.8% for predefined ones, and SUS rose from 68.2 to 85.9 with a conversational interface.
— Independent survey of 114 text-to-SQL studies: only 31 report explicit token values, so the cost of inference can't be compared from end to end. It names semantic correctness beyond executability as a core challenge.
— Negative signal: Meta is buying Sundial, a natural-language warehouse-querying vendor, which will stop serving external customers at close. No transition timeline or semantic-layer export path has been published.
220 more · latest 2026-10-24 →
— An industrial team reports 95.7% vs 60.8% execution validity on Beaver from learning schema off query logs, and a 70% vs 10% expert preference on a 240-table internal deployment. Self-evaluated.
— Snowflake claims 10,000 of 15,000 customers use its AI features daily and that TF1 cut campaign analysis from four days to three minutes. The claims are vendor-reported, and Cortex Sense and CoWork deep research are still in preview.
— Vendor opinion that AI queries must compile through the semantic model. Its anecdote: an agent ignored a warning, hand-wrote a join and reported $55.5 billion in pipeline, an example of a silent failure.
— Databricks documents Genie One MCP as generally available for workspace-wide natural-language analytics, grounded in its governed semantic layer, Genie Ontology. No accuracy figures are given. The date comes from a search snippet.
— Academic prototype: a multi-agent insurance NL2SQL pipeline with execution-feedback repair and a semantic round-trip check reaches 90% correctness on 60 queries using Mistral. It reports no baseline figure.
— Semantic-layer vendor guidance: scope models to 5-10 tables, treat 85% as the minimum for automation, and expect pilots at 85-90% and 97%+ only after a correction loop. The admission that pilots fall short is a candid limitation.
— A LoRA-tuned 9B Qwen3.5 assistant that runs on an 8 GB laptop GPU scores 55.6% on BIRD Mini-Dev (PostgreSQL), against a best published baseline of 36.0%. Small self-hosted models are becoming viable.
— Databricks' internal rollout of Genie text-to-SQL to more than 200 marketing users, from a baseline of about 25% of team time on low-value work. The talk gives no post-deployment metrics.
— Negative signal: across 815 multi-database tasks, the best clarified route-plus-execution success is 16.99%. Finding the right databases and writing correct local SQL is not enough.
— Negative signal: on 311 ambiguous questions over real databases, the top model reaches only 51% end-to-end execution accuracy and no open-source model exceeds 27%.
— Fully on-premise pharma batch-data prototype: retrieved validated examples doubled local-model execution accuracy from 23–25% to 46–48% on 48 questions, still far short of unsupervised use.
— Independent review of Argo-Bench (235-table, 7.49-billion-row warehouse, 210 consequence-graded tasks): best of 14 models solves 34.8%, and the reviewer calls the headline scores irreproducible by design.
— Agentic text-to-SQL returns correct results while resolving only 2.6 of 3.6 annotated ambiguities on BIRD-Lite; PlanPool lifts ambiguity recall 80.8% to 87.5% and cuts silent failures 10.5% to 8.2%.
— Natural-language querying of MongoDB lags SQL: strongest model reaches 57.38% without external evidence and 70.34% with it on 3,186 requests; earlier BIRD conversions silently dropped up to 25.9% of rows.
— Elastic's own benchmark of 6,000 graded ES|QL queries on BIRD Mini-Dev: best model 59.0% correct zero-shot, syntax rarely the failure, and about half of misses run clean but return empty results.
— S&P Global Energy exposes SME-curated Genie Agents per commodity dataset as managed MCP servers under Unity Catalog; time-to-market fell from months to days. Vendor-reported, no usage figures.
— Embedding-based attribute retrieval plus template-driven NL2SQL reported in production on an unnamed enterprise CDP over millions of profiles; outcomes are qualitative and self-reported.
— Named customer case: Miro improved LLM-generated queries from <40% to >90% accuracy via governance and context engineering alone, without model changes; ranks trust signals over schema context.
— Ericsson case: semantic foundation governance saved 90,000 business hours annually across 180+ countries; GigaOm: vendor-built semantic layer 59-67% lower 3-year TCO than DIY multi-vendor approach.
— Databricks documentation operationalizes NL2SQL quality controls: 100 instructions and 200 knowledge-store snippets per agent; trusted assets return verified answers when matched exactly.
— Databricks practitioner analysis: 54.6% of failures from wrong filters, not syntax; $12.9M annual data-quality cost; mitigations require governed metadata and validation, not prompt optimization.
— Research validation: governed semantic layer achieves zero hallucinations across all categories on 100-question enterprise benchmark; replicates on real NHTSA data, supporting deterministic compilation as production reliability solution.
— Open-source model parity: RLVR fine-tuning on corrected BIRD data (Qwen3-235B) reaches 92.96% accuracy, outperforming top five open-source systems by 10-22pp; validates fine-tuning as SOTA path.
— Oracle Autonomous Database GA documentation of Select AI natural-language actions: runsql, showsql, narrate, explainsql, chat, summarize, feedback; plus embedding-vector semantic search via AI profiles.
— Snowflake Q2 FY2027: Cortex AI penetration at 9,100+ accounts (+2,000 in quarter), 34% YoY product revenue growth, AI-driven consumption accounting for 50% of sequential acceleration.
— Practitioner guide documenting production semantic model design patterns, verified query libraries, and failure modes; identifies multiple join path ambiguity and measure vagueness as primary accuracy drivers, not model capability.
— Gartner forecasts 60% of agentic analytics projects without semantic foundation will fail by 2028; identifies organizational context governance (metric definitions, lineage, business rules) as determinative factor in NLQ deployment success.
— Google Cloud GA: History-Based Optimization delivers 35% performance gains and 40% cost reduction; designed for autonomous agentic query workloads, signaling platform infrastructure maturity for NL-to-SQL at scale.
— Independent founder benchmark on BEAVER enterprise dataset: 52.1% accuracy on real data vs 30.1% with oracle hints; demonstrates text-to-SQL cannot yet operate autonomously without guardrails and human review.
— Cortex Analyst integrated into Cortex Agents framework with semantic views and verified queries preserved; demonstrates architectural shift from standalone NL-to-SQL toward multi-modal agentic orchestration.
— Systematic comparison of 17 text-to-SQL paradigms across 4 backbones: execution-feedback refinement is only universal benefit; pipeline engineering consistently outperforms frontier model upgrade on fixed budgets.
— Enterprise-first benchmark reveals 92% of execution-passing queries exhibit silent divergence; GPT-4o accuracy collapses from 79.8% (Tier 1) to 57.2% (Tier 3); establishes critical gap between execution correctness and semantic correctness.
— Moody's embeds credit data into Gemini Enterprise via Model Context Protocol (MCP) semantic layer, enabling natural language queries on proprietary financial intelligence.
— Anthropic's own published guide: frontier AI achieves only 21% accuracy on analytical questions out-of-box; reaches 95% only after data team organizes business context.
— Professional data consulting comparison: both platforms achieve 90%+ accuracy with mature semantic models but remain unreliable without semantic layer investment.
— Published research on CrewAI-based conversational BI achieving 95.3% accuracy and 4.52/5.0 quality score on 300 production enterprise test cases.
— Comprehensive analysis: o1-preview drops from 91.2% on academic benchmarks to 21.3% on real enterprise schemas; mechanism (schema linking), not model, determines production viability.
— Oracle Autonomous Database GA for NL-to-SQL with LLM integration, multi-language support, and multi-turn conversation capabilities—major vendor platform commitment.
— dbt Labs 2026 benchmark: text-to-SQL 84-90% accuracy vs semantic layer 98-100%; hybrid architectures (Genie Ontology, Cortex Sense) emerging as production standard.
— Peer-reviewed audit found 52.8% error rate in BIRD and 66.1% in Spider 2.0-Snow; re-scoring systems moved accuracy up 19pp, undermining published benchmark claims.
— Peer-reviewed framework deployed in Volcano Engine TLS achieving 53.61% accuracy on real user queries with GPT-5, a 16.84pp improvement from 36.77% baseline.
— Production Text2SQL system achieves 80% latency reduction and 50%+ token cost savings through parameterized query caching layer instead of LLM-per-query approach.
— Expert assessment documenting three non-negotiable production components: grounding in actual schema, guardrails on safe execution, and precision/verifiability—identifies why naive text-to-SQL impresses in demos but misleads in production.
— VLDB 2026 audit reveals 52.8% annotation errors in BIRD benchmark; 19% of flagged model failures were actually model improvements over flawed gold—undermines published accuracy claims and exposes validation infrastructure gap.
— Google Cloud announces autonomous query processor with History-Based Optimizations for agentic AI workloads; 35% performance gain, 40% cost reduction confirms infrastructure maturity for high-volume agent-generated queries.
— Global retailer (335K employees) deployed RAG chatbot reducing support resolution from hours to 2 minutes, serving 700-800 active users with 75% self-service sufficiency, demonstrating production at enterprise scale.
— KDD Cup 2026 benchmark evaluating data agents on 410 cross-language tasks over heterogeneous data (CSV, JSON, SQLite, PDF, video): best accuracy 66.34%, with multimodal integration and joins consistently reducing accuracy—core leading-edge limitation.
— Named nonprofit CRM platform deployed text-to-SQL with 94%→95%+ accuracy on 143-question golden set; knowledge graph retrieval + synonym tiering + ask-or-assume behavior drove improvements from data dictionary curation, not model scale.
— IBM's open-source Text2SQL-Eval (PVLDB 2026 demo) offers over twelve metrics and agentic pipelines across SQLite, PostgreSQL, MySQL and DB2. A sign that evaluation infrastructure is maturing; no outcome data.
— Stonebraker's BEAVER benchmark on real enterprise data: plain LLM 0%, LLM+retrieval+agentic 10%, even with oracle joins 30%, vs. public benchmarks 70%+—reveals structural gap between academic and production accuracy.
— Production LLM agent + semantic layer architecture for AI analytics with self-learning governance loop; addresses notation variations and undocumented business rules through semantic layer, metadata ingestion, and CI/CD-driven data steward updates.
— Amazon Science peer-reviewed publication on semi-automatic benchmark generation addressing bottleneck of scarce domain-specific datasets; 95.11% domain expert validation accuracy; addresses critical production barrier of evaluation scarcity.
— Practitioner deep-dive diagnosing naive text-to-SQL failure modes (entity ambiguity, schema staleness, retrieval failure) and proposing 3-layer agentic architecture (governed semantic layer, active validation, modular skills) grounded in Anthropic's internal approach.
— ACL 2026 Short Paper on parallel exploration for text-to-SQL achieving 70.2% execution accuracy on Spider 2.0 SOTA by executing simpler atomic SQLs in parallel before generating final query, demonstrating test-driven SQL generation pattern.
— News roundup of BEAVER benchmark research by MIT/Intel/Harvard showing text-to-SQL failure in real production: pure LLMs 0% accuracy, with RAG/prompt engineering ~10%, capping at ~30% on 1,400+ column enterprise schemas with schema rot.
— ACL 2026 peer-reviewed research on verifiable execution tracing achieving 70.93% BIRD and 37.04% Spider 2.0-lite by executing Python steps against live databases for immediate feedback, addressing hallucination via transparency.
— Snowflake GA of CoWork (natural language AI interface), Cortex Sense (semantic layer), and agent orchestration capabilities with technical demos, signaling vendor consolidation around agentic architecture and mandatory semantic layers.
— Real production deployment by Series B transportation SaaS company using Cortex Analyst with semantic layer governance, RBAC/RLS architecture, deployed to 62 customers creating new AI-powered revenue stream in two quarters.
— ACL Findings July 2026 SOTA reinforcement learning framework achieving state-of-the-art execution accuracy across six benchmarks; 7B model outperforms prior 70B-class systems via execution-correctness reward, highlighting architectural efficiency over scale.
— Independent technical analysis documenting benchmark-to-production accuracy gap (BIRD 57.4%-81%) and identifying context quality as limiting factor; states benchmarks are demo predictors, not production forecasts.
— Peer-reviewed research on grammar-constrained text-to-SQL decoding addressing enterprise deployment demands: syntactically valid outputs, per-role/schema policy compliance, provable guarantees for policy-compliant SQL generation at scale.
— Alibaba's agentic data system transforms natural language to end-to-end analytical workflows via semantic grounding and methodology codification; tested on real industrial BI workloads, demonstrating vendor-independent production maturity.
— Critical assessment of Cortex Sense private preview: accuracy improves 24.1% → 86.3% with context, but governance risks identified—agents could confidently produce wrong answers when metric definitions conflict; adoption barrier documented.
— OneSix dbt Cortex Agent Accelerator demonstrates infrastructure-as-code pattern for governed NLQ agent deployment; MCP architecture enforces RBAC at three levels (object, tool, data); production governance pattern documented.
— Turing Award winner Stonebraker reports frontier LLMs achieve 0% accuracy on actual production warehouses vs 80-85% claimed; tested on four real data warehouses; published BEAVER benchmark to counter benchmark gamesmanship.
— Production deployment of Cortex Analyst in government environment with formal AI assurance process; three-phase governance: security scoping, RBAC architecture, least-privilege enablement demonstrates regulated-sector maturity.
— Industry analysis documenting enterprise AI agent failure rates (70-95%), with text-to-SQL agents explicitly identified as widely deployed; chained workflows hit 30-50% accuracy despite 90%+ isolated task performance.
— SURGeLLM 2026 benchmark identifies join-hop depth as single measurable bottleneck (accuracy cliff: >80% at h=1, <25% at h=6); decomposition lifts GPT-4o from 46.8% to 67.3% on complex enterprise queries.
— DySQL-Bench reveals critical practice barrier: GPT-4o only 58.34% on multi-turn database interaction with state-mutation (INSERT/UPDATE/DELETE); 23.81% on strict Pass^5, exposing limitation beyond SELECT-only benchmarks.
— Reproducible benchmark comparing open-source LLMs (Qwen2.5-Coder, CodeLlama, Llama-3.x) on BIRD with ablation testing; self-correction robust across families, schema linking ineffective, cost-efficient optimization patterns emerging.
— PAR Technology (300+ restaurants) production deployment: three-layer security architecture (cryptographic signing, semantic validation, split-plane SQL) enforces row-level security despite LLM non-determinism; exemplifies shift from demo to enterprise-grade production.
— Critical analysis of Cortex Analyst architectural constraints: 1MB semantic model cap, stateless execution, 60-second timeout, no join inference, non-deterministic output; independent validation (Spider 2.0: 21.3% on enterprise vs 91.2% on curated; BEAVER: near-0%).
— Wisdom (Cisco, ConocoPhillips, ARM) ACE architecture: context drift primary failure mode (95%→65% accuracy over one month); BIRD benchmark inadequate; production requires continuous learning with user feedback, not static deployment.
— Production cost analysis: enterprise schema overhead $25K/month in tokens alone; GPT-4o 86.6%→10.1% accuracy collapse on Spider 2.0; semantic layer recovery to 98-100%; dbt Labs benchmark: semantic layer 98.2-100% vs text-to-SQL 84-90%.
— BEAVER benchmark documents critical production barrier: Claude 4.5 Sonnet 11.4%, GPT-5.2 10.8% on real enterprise data (1,400 columns); introduces 'fluent failure' concept (syntactically correct but semantically wrong SQL); identifies semantic layer and schema governance as necessary solutions.
— SIGMOD 2026: Google's proxy model approach achieves 100x cost and 100x latency reduction for semantic queries (deployed in BigQuery and AlloyDB); shifts semantic search from costly prototype to economically viable production pattern.
— BIRD benchmark leaderboard as of June 2026: dev set 77.84% EA / 65.12% EX; 200+ community submissions tracking progress; signals active development against realistic enterprise-grade benchmark with dirty data and large schemas.
— Concrete deployment ROI: PepsiCo 12x faster root-cause analysis (2025); Top-10 pharma $12M opportunity with 2,200% ROI; documents market maturity with vendor consolidation around NL query capabilities.
— Critical negative signal: rigorously documents 70-point accuracy collapse (86-91% benchmarks → 10-21% enterprise); identifies three failure modes (scale, missing semantics, no verification) with empirically validated solutions (context layers recover 70-96%).
— dbt Labs April 2026 benchmark: semantic layer (deterministic) 98.2-100% vs text-to-SQL (probabilistic) 84-90% on same 11 insurance questions; demonstrates architectural solution to silent-wrong-answer failure mode.
— ByteDance + Georgia Tech production system: Spider 2.0-Snow pass rate 61.95%→79.42%; 100% Snowflake syntax; cross-model transferability (+19.7pp on Doubao-2.0-lite); demonstrates hint-learning architecture for real deployments.
— Cortex Sense context layer benchmark: agents alone ~24% accuracy, with context ~86%; demonstrates context assembly—not model capability—as bottleneck; validates semantic layer infrastructure as critical.
— Amazon Science framework outperforms LLM self-eval by 25.78% F1; improves execution accuracy up to 20pp on deployed systems—addresses critical production failure mode of semantically incorrect SQL passing syntax validation.
— Negative evidence: production incidents (Replit, Vibe) where AI agents destroyed databases due to governance gaps; reveals adoption reality—agents deployed in production but operational controls lag, indicating immature security posture.
— CoWork GA with adoption inflection: 13,600+ weekly active accounts with 2x QoQ growth; paired with context layers achieves 5x accuracy improvement (47% baseline, 5x with Atlan context), demonstrating context-not-model as limiting factor.
— CoCo adoption: 7,100+ accounts across 50%+ of Snowflake customer base; 72.1% pass rate on ADE-Bench real-world tasks; 3x SQL accuracy improvement with context layer (145 enterprise queries, BirdBench p<2e-10).
— Cognizant A+E Global Media deployment: conversational agent reclaimed 200 hours manual effort; 2,250+ users across 30+ enterprise use cases; 1.3M AI-driven requests processed; production maturity signal.
— Systematic adversarial testing of multi-agent text-to-SQL: 373 queries reveal 30-78% detection rates and architectural blind spots where schema metadata flows uninspected; demonstrates structural security limitations in production systems.
— 6,000+ attendee summit with production deployments: Carrefour France migrated 3,000 metrics across 40 countries; Vodafone Portugal reduced metric refresh times hours→minutes; validates semantic layers as critical infrastructure for enterprise NL data querying.
— Production AML system using Cortex Analyst for structured transaction data and Cortex Search for unstructured compliance documents reduced alert investigation time from 30-90 minutes to <5 minutes via semantic search.
— Official FY27 Q1 results: 13,600+ accounts using Snowflake AI capabilities, 7,100+ using Cortex Code, Snowflake Intelligence accounts doubled QoQ; 34% YoY product revenue growth signals mainstream adoption acceleration.
— Multi-institutional agent achieving 65.4% on Spider2-Snow (realistic enterprise benchmark) and 10%+ improvement over baselines through dynamic schema exploration—demonstrates architectural shift toward iterative agentic approaches.
— JP Morgan LLM team's EnterpriseMem-Bench reveals critical maturity barrier: all five frontier models collapse to 0% execution accuracy by Turn 3 in multi-turn SQL without working memory, representing fundamental conversational limitations.
— SIGMOD 2026 workshop research quantifies fundamental robustness limitation: LLMs produce inconsistent SQL across equivalent database schemas, with model disagreement ranging 33.85%-81.6% depending on model and domain.
— Amazon research demonstrates state-of-the-art on BIRD-SQL benchmark through multi-turn RL with feedback, showing iterative reasoning and error correction substantially outperform single-pass LLM generation for complex queries.
— MIT/Intel/Harvard benchmark of real corporate query logs reveals critical adoption barrier: GPT-4o achieves 82% on academic benchmarks but collapses to 10.8% on BEAVER's real enterprise data with undocumented schemas.
— Production comparison of Snowflake Cortex Code agent effectiveness across data freshness strategies, showing conversational SQL generation quality critically depends on semantic data grounding, not just model capability.
— Google Cloud GA of Gemini-powered natural language interface to Database Center enabling conversational queries across Cloud SQL, Spanner, and Bigtable—direct production deployment evidence.
— Synthesis of production lessons from OpenAI, Google Cloud, Vercel, and Hex revealing enterprise text-to-SQL requires context pruning, rigorous validation, and governance—engineering discipline matters more than prompt optimization.
— Peer-reviewed research exposing critical evaluation bias: most text-to-SQL benchmarks only test SQLite, creating false confidence. Cross-dialect study reveals 10.1% accuracy drop to production databases (PostgreSQL, BigQuery, Snowflake).
— Production case study showing conversational AI platform for spreadsheet data querying improved from 60% to 98% success rate through agent-based Python code generation architecture replacing custom DSL.
— Independent analysis reporting Snowflake's 9,100+ weekly active Cortex AI accounts with 200% growth in AI workloads; 50% customer adoption of Cortex Code since November 2025 launch.
— Finance organizations using multi-turn NLQ for conversational financial modeling, dynamic scenario planning, and variance analysis—showing matured NLQ practice beyond simple Q&A.
— Production security assessment: text-to-SQL risks extend beyond SQL injection—generated SQL can leak permissions, violate access controls, or answer wrong questions; deterministic validation essential after LLM generation.
— Production evaluation framework enabling continuous monitoring without schema access—addresses critical gap: current evaluations require ground-truth queries and schemas, rarely satisfied in deployment.
— Tapestry (Coach/Kate Spade parent) deployed NLQ feedback analysis on AWS Bedrock, collecting 30,000 feedback pieces and achieving 10x faster AI application development with faster business decisions.
— ACL 2026 research aggregation: semantic layers boost accuracy 17-23 percentage points across frontier models (Opus 4.7, Sonnet 4.6, GPT-5.4); R³-SQL reaches 75% BIRD-dev execution accuracy.
— Technical architecture analysis: NL-BI requires four-layer design (intent parsing, semantic layer, SQL generation, validation). Vendor consolidation around semantic layers and deterministic validation as production requirements.
— Production deployment guide from Snowflake partner documenting three semantic layer architectures—modularity vs. accuracy vs. scalability tradeoffs in multi-tenant Cortex Analyst rollouts.
— ACL 2026 paper achieving 70.2% execution accuracy on Spider 2.0 via agentic parallel exploration—addressing latency-performance tradeoff in text-to-SQL; signals continued research momentum.
— Strategic analysis: context quality, not model capability, is the limiting factor for NLQ accuracy. Enterprise context typically siloed and human-designed; Cortex agents require unified, machine-readable metadata.
— Prof. Nick Koudas (UofT NLQ researcher) expert assessment: text-to-SQL achieves ~80% production accuracy; Koudas warns against non-expert user access without verification due to silent semantic failures.
— Independent benchmarking: Claude Sonnet 4 leads at 86.6% exact match on complex multi-table queries; practical guidance emphasizing query reviewability and validation for production use.
— Snowflake announces NLQ and agentic capabilities with named customers (Capita, Logitech, Telenav, United Rentals, Wolfspeed) moving to production; Skills, MCP connectors, deep research GA soon.
— Engineer documents benchmark deception: Spider uses 146 clean DBs; production has 400 tables (opaque naming). GPT-4o drops from 90%+ on synthetic to 51% on real BI; 50-point accuracy gap on schema ambiguity.
— Amazon Science addresses benchmark-to-chatbot gap with conversational dataset featuring ambiguous questions, unanswerable queries, and four-turn clarification—core production challenge unaddressed by Spider.
— AWS engineering team documents LoRA fine-tuning pattern: $0.80/month cost for 22,000 queries/month; LoRA addresses model dialect failures without persistent hosting overhead.
— Engineer documents Spider 2.0 collapse: GPT-4o from 86.6% to 10.1% on enterprise schemas. Pinterest real-world data: ~20% first-shot acceptance despite benchmarks; schema hallucination failures.
— Scale AI deployed Ana at scale: 1,900 requests/week across Finance/Ops/HR, 1.9T rows across Snowflake/dbt/Tableau, 74.9% monthly adoption growth through centralized semantic layer governance.
— Google Cloud's QueryData GA with #1 BiRD benchmark ranking and named production deployment (Hughes Network Systems, telecoms) achieving near-100% accuracy through schema ontology and context engineering.
— Critical practitioner assessment documenting silent production failures: semantic errors (fan-out traps, NULL inconsistencies, ambiguous business terms), performance failures, security vulnerabilities from Uber/Brex engineer.
— Uber's production NLQ system processing 1.2M interactive queries monthly across Operations, reducing query authoring time from ~10 minutes to ~3 minutes through evolved intent-agent + table-agent architecture across 200+ column schemas.
— Q1 FY2026: 50% of Snowflake customer base (5,200+ weekly active users) actively using Cortex AI including Cortex Analyst text-to-SQL, signaling inflection point for mainstream enterprise adoption with 27% YoY growth in $1M+ customers.
— Production finance NLQ system progressed from 40% zero-shot accuracy to 91% (schema context + few-shot) with validation layer preventing costly errors; demonstrates adoption barrier (trust/visibility) beyond model accuracy.
— Cornell empirical study (376 databases): 70% of SQL queries covered by 13% of template types, challenging assumption that LLMs are necessary and suggesting deterministic templates could provide safer, cheaper, auditable alternatives.
— Named enterprise deployments (FrankieOne 232 hrs/week saved, Northmill 30% conversion lift, Austin Capital Bank 15% retention improvement, 50% ad spend reduction) demonstrate agentic NLQ business value in complex financial data environments.
— ICLR 2026 formal verification reveals benchmark accuracy inflation: SOTA methods lose 11-14% when evaluated with semantic equivalence (not just output matching), exposing systematic overestimation of current capabilities.
— AAAI 2026 peer-reviewed research: Dream11's CriQ (250M users) deployed fine-tuned 8B-parameter text-to-SQL achieving 98.4% execution and 92.5% semantic accuracy, outperforming GPT baseline by internalizing schema into model weights.
— Berkeley Data Agent Benchmark reveals frontier LLMs (Opus-4.6 43%, Gemini-3-Pro 38%) failing not from capability but context gaps; business context and domain knowledge, not model scale, determine practical NLQ viability.
— Comparative analysis of 7 agentic analytics platforms (DataBrain, ThoughtSpot, Looker, Power BI, Tableau, AWS QuickSight, Sisense); documents cost unpredictability (consumption models) and semantic modeling complexity as adoption barriers despite platform maturity.
— Production case study with 28-table MySQL database and 18 live workflows, documenting hybrid NLQ architecture: Text-to-SQL for analytical queries, Function Calling for controlled access, Router Pattern solving failures of pure approaches.
— Academic research identifying critical dense retrieval failure mode: semantic collapse on negation (MRR 0.023), showing hybrid retrieval trade-offs between contradiction detection and recall in production biomedical QA.
— Vendor evolution: ThoughtSpot announces Spotter for Industries, industry-specific agentic analytics with semantic context, connectors (Salesforce, Zendesk, SAP), addressing gap in generic AI through deterministic reasoning.
— Peer-reviewed research (Ambrosia+ benchmark with 6,777 examples) showing 36.5% relative improvement on ambiguous queries, identifying linguistic ambiguity and unanswerability as fundamental production barriers.
— Production deployment: large QSR chain using Snowflake Cortex Analyst NLQ for sales assistant analyzing 200+ measures (same-store sales, traffic, promotion performance), demonstrating 2-3x speedup in semantic model migration.
— Production case study: 9-county San Francisco Bay Area energy efficiency agency using Snowflake Cortex AI to enable NLQ across fragmented data silos (assessment systems, rebate databases, real estate), connecting actions to outcomes.
— Critical analysis citing Gartner 2025 Hype Cycle placing NLQ at Peak of Inflated Expectations, reporting internal testing where LLMs are >80% incorrect on raw data models, arguing semantic layers are mandatory for accuracy and governance.
— ThoughtSpot GA of next-generation Analyst Studio with agentic data prep and SpotCache (unlimited analytics queries with fixed costs), linking natural language analytics to AI readiness and data quality challenges.
— Practitioner analysis demonstrating naive text-to-SQL failures on real enterprise schemas (35 tables): silent wrong results, security gaps (PII exposure), destructive SQL, and cost issues—exposing critical production barriers beyond benchmark accuracy.
— Comprehensive market analysis comparing 15+ SQL chat tools, emphasizing the demo-to-production gap: tools fail due to lack of business context, tribal knowledge, and security risks despite functional demonstrations.
— SIGMOD 2026 paper introducing DIVER system improving text-to-SQL robustness by up to 10.82% execution accuracy, explicitly addressing production deployment challenges where state-of-the-art models suffer 10%+ performance collapse without expert assistance.
— Practitioner blog documenting unexpected $530 charges and billing complexity with Amazon Quick Suite's NLQ features after promotional period ends, exposing operational cost and transparency barriers to adoption.
— Developer tutorial for building self-learning text-to-SQL agents with knowledge-based query generation and data quality handling, demonstrating practical adoption patterns for improving accuracy through query validation and learning loops.
— Critical assessment documenting RAG/NLQ failures in production (telecom provider saw increased support calls post-rollout), citing 72% enterprise search query failure rate and proposing Intent-First architecture as necessary alternative to standard semantic search.
— arXiv research advancing natural language querying for spatio-temporal databases with three-layer architecture (knowledge base, NLU, physical plan generation), demonstrating continued academic progress on specialized NLQ domain variants.
— Open-source text-to-SQL implementation tutorial using DuckDB-NSQL-7B and Hugging Face integration, signaling developer adoption patterns for combining LLM models with structured dataset access and practical query generation workflows.
— Snowflake engineering blog detailing Cortex Analyst's text-to-SQL accuracy evaluation for real-world BI scenarios, signaling vendor investment in production reliability assessment and competitive maturity.
— Analysis of CORGI and DBASQL benchmarks revealing model limitations: GPT-4o achieves ~50% accuracy on CORGI (complex business logic), exposing continued performance gaps despite dataset advances targeting business domain complexity.
— Peer-reviewed research proposing SQLHD method to detect hallucinations in LLM-based text-to-SQL without ground-truth answers, achieving F1-scores 69.36-82.76%, addressing core production reliability limitation.
— Critical analysis reveals 10x accuracy gap between academic benchmarks (85-90%) and enterprise reality (10-30%), citing GPT-4o 86% on Spider 1.0 vs. 6% on Spider 2.0, exposing persistence of production reliability as deployment blocker.
— Elasticsearch hybrid search GA combining lexical and semantic search with reciprocal rank fusion, supporting ELSER inference and OpenAI models, advancing semantic search production-readiness.
— Critical vendor assessment documenting four fundamental LLM-based text-to-SQL barriers: schema awareness gaps, hallucinations, performance issues, and security risks, citing research showing only 10-20% of AI answers accurate for business decisions.
— Odido (largest Dutch mobile operator) deployed ThoughtSpot natural language analytics, achieving €1M annual savings and enabling business users to build analytics in 90 minutes vs. days with legacy BI tools.
— ThoughtSpot reports 133% YoY platform usage growth and 52% customer adoption of Spotter AI agent, with named deployments at Chevron, Elevance, Thrive Learning (20k+ customers in 6 weeks), signaling mainstream adoption momentum.
— CORGI benchmark from Cornell and Gena AI reveals LLM performance drops on high-level business questions, with benchmark 21% more difficult than BIRD, exposing persistent limitations in complex business intelligence reasoning.
— 2025 Gartner Magic Quadrant names ThoughtSpot Leader in Analytics and BI, with analyst recognition of agentic AI architecture and superior natural language search capabilities compared to traditional BI platforms.
— Text-to-Big SQL research reveals traditional text-to-SQL metrics insufficient for big data systems, with execution cost and latency failures at scale, highlighting critical production barriers for enterprise deployments.
— Verivox (German online services marketplace) deployed ThoughtSpot Embedded for NLQ-driven analytics, achieving 70% adoption across divisions and decommissioning two legacy dashboard tools, demonstrating enterprise production viability.
— ServiceNow GA of Natural Language Query feature translating plain language to executable queries, signaling adoption of NLQ beyond cloud-native platforms into enterprise IT service management workflows.
— Amazon WWRR organization deployed integrated Amazon Q in QuickSight with Bedrock Agents for production NLQ, combining SQL code generation with visual insights via intent classification and vector search.
— Critical assessment of NLQ implementation challenges in BI tools: enterprise data integration, contextual understanding, performance, and security, emphasizing that seamless integration with semantic layer and access controls is essential for adoption.
— Practitioner analysis of text-to-SQL evaluation complexity beyond syntactic correctness, covering execution accuracy, efficiency, and logical completeness, highlighting production challenges with hundreds of tables and domain-specific terminology.
— ACM Computing Surveys systematic review analyzing LLM-based text-to-SQL research trends, techniques, datasets, and evaluation methods, documenting continued academic research momentum and open challenges.
— Domain-specific benchmark of 219 business questions on Exaone 3.5 LLM reveals persistent limitations: 93% accuracy on simple aggregation but only 4% on arithmetic reasoning and 31% on grouped ranking, exposing production reliability gaps.
— HP Inc. deployed ThoughtSpot on Snowflake for natural language search and analytics, enabling 350 users to generate 155,000 queries in 6 months and reduce partner data turnaround from days to under 24 hours.
— Amazon Q in QuickSight reached GA in embedded dashboards and consoles (April 2025), enabling generative BI with executive summaries and natural language dashboard creation across 7 AWS regions.
— Research framework integrating thinking-mode fine-tuning with policy optimization for text-to-SQL, achieving 79.14% execution accuracy on BIRD and 11-20% efficiency gains, with schema-linking errors constituting >50% of failures.
— Systematic research review of Natural Language Interfaces for Databases, categorizing techniques and benchmarks (Spider, ATIS, GeoQuery), highlighting persistent challenges: ambiguity, real-time performance, resource limitations.
— Critical assessment questioning text-to-SQL viability due to SQL dialect complexity and limitations, advocating agentic function calling as superior alternative, citing model performance gaps (GPT-4o succeeds vs. Gemini-2.0-turbo fails 54%).
— AWS reports 2024 GA of Amazon Q in QuickSight for multi-visual natural language Q&A, automatic document generation, and scenario analysis agentic features, advancing vendor product maturity.
— Comprehensive survey of LLM-based text-to-SQL systems documenting benchmarks, applications in healthcare/education/finance, and persistent challenges in domain generalization and schema complexity.
— Retail distributor deployed ThoughtSpot on Snowflake for natural language search and analytics, reducing reporting latency from hours to minutes and demonstrating production-scale real-time self-service analytics.
— AWS re:Invent 2024 session documenting embedded analytics deployments: Docebo serving 4,000+ customers, aCommerce enabling competitive analysis across 250,000 brands, demonstrating production scale beyond early adopters.
— Oracle Select AI deployment on E-Business Suite (EBS 12.2.7+) using schema XX_NLQ and APEX, demonstrating production natural language querying for enterprise ERP systems beyond cloud-native deployments.
— AWS GA of enhanced Amazon Q in QuickSight integrating natural language querying with unstructured data via 40+ enterprise connectors, signaling vendor ecosystem expansion and maturation.
— NeurIPS benchmark with 632 real-world enterprise text-to-SQL tasks from BigQuery/Snowflake showing LLMs achieve only 17% success (vs. 91.2% on Spider 1.0), exposing critical reality gap between academic and production performance.
— Practitioner implementation of LLM-based text-to-SQL achieving 89% Spider accuracy, but revealing persistent errors in column selection, grouping, and join logic, showing gap between benchmark metrics and production reliability.
— Comprehensive survey from Peking University documenting LLM-based text-to-SQL maturity, including prompt engineering, fine-tuning, and critical production challenges (privacy, complex schemas, domain knowledge).
— Cox 2M (IoT business unit of Cox Communications) deployed ThoughtSpot natural language search handling 1.5M+ IoT messages/hour and 13B+ daily rows, achieving 88% reduction in time to insights and $70k+ annual cost savings.
— Uber's production text-to-SQL case study: vector retrieval lost accuracy as more tables were added, and fixing it took curated workspaces and agents. Uber estimates authoring time falls from about 10 minutes to 3 but gives no accuracy figure.
— Docebo embedded Amazon QuickSight with natural language capabilities into learning platform serving 3,800+ customers, achieving 5x analytics adoption increase in production.
— Critical vendor assessment documenting NLQ limitations in practice: models fail business users due to ambiguity and intent mapping failures, advocating guided NLQ over pure semantic search.
— ACL 2024 paper introducing TA-SQL framework reducing hallucinations in text-to-SQL via task alignment, improving GPT-4 baseline by 21.23% on BIRD dev benchmark.
— Peer-reviewed survey covering text-to-SQL lifecycle (models, data synthesis, benchmarks, error analysis), signaling academic research maturity with LLMs driving method advancement but ongoing production reliability challenges.
— Production practitioner experience: users found semantic search inadequate for structured property queries (ignoring numeric filters, synonyms), with hybrid SQL approach yielding superior results.
— Comprehensive survey of LLM-based text-to-SQL methods analyzing advances in prompt engineering, retrieval-augmented generation, and evaluation approaches, documenting continued research momentum in method innovation.
— AWS CloudWatch GA of natural language query generation for Logs Insights and Metrics Insights, expanding NLQ from BI into observability domain, signaling ecosystem breadth and vendor acceleration.
— CBRE (world's largest commercial real estate services, 130,000 professionals) deployed natural language query capability using Amazon Bedrock, demonstrating Fortune 500 adoption and production viability at scale.
— Research reveals widespread annotation errors in BIRD and Spider benchmarks through systematic examination, indicating that benchmark-reported accuracy improvements may overstate real-world readiness and production reliability.
— BCG partnered with Scale AI on a production text-to-SQL implementation enabling business users without SQL skills to access data via natural language, demonstrating real-world deployment beyond vendor internal use.
— Amazon Q reaches general availability in QuickSight after preview at re:Invent 2023, bringing generative BI with natural language querying to all user roles, expanding vendor product maturity.
— Preprint proposing Text-to-SQL strategy combining LLMs with keyword search platform, tested in production at energy company Petrobras, addressing schema complexity and semantic ambiguity barriers.
— PVLDB 2024 peer-reviewed paper presenting NL2SQL360 evaluation framework and SuperSQL achieving 87% Spider/62.66% BIRD accuracy, but revealing critical production gaps: real-world database performance significantly lags benchmarks due to large schemas, complex operations, and linguistic variations.
— Error taxonomy and user study with 26 participants exploring interactive error-handling mechanisms for NL2SQL, revealing lack of systematic understanding of error types and effectiveness of recovery strategies in production scenarios.
— ServiceNow GA of Natural Language Query feature in enterprise platform, with Now LLM fallback and query logging, signaling adoption beyond AWS in IT service management workflows.
— Empirical study achieving 82.1% Spider accuracy with fine-tuned GPT models, but systematically categorizing seven error types (column selection, grouping, join logic, etc.), revealing persistent limitations in LLM-based text-to-SQL.
— Critical assessment by Yellowfin CEO documenting why search-based NLQ has failed market adoption due to inherent ambiguity problems, arguing vendors over-focus on semantic problems rather than analytical problems, advocating for guided interfaces instead.
— AWS year-end product update highlighting Amazon Q in QuickSight with 80+ new capabilities, achieving Gartner Challenger and Forrester Strong Performer status in 2023.
— NeurIPS 2023 BIRD benchmark with 12,751 text-to-SQL pairs across 95 large databases showing GPT-4 achieves only 54.89% accuracy vs. human 92.96%, exposing real-world production gaps.
— EMNLP 2023 industry paper presenting DataQue, a hybrid NLQ system deployed in production addressing real-world challenges like jargon, cold-start, and complex implied conditions.
— EMNLP 2023 paper proposing schema hallucination method for text-to-SQL scalability, addressing large database limitations with 17,844+ schema elements.
— Benchmark study proposing DAIL-SQL achieving 86.6% execution accuracy on Spider, advancing LLM-based text-to-SQL state-of-the-art with systematic prompt engineering analysis.
— Critical assessment highlighting limitations of NLQ solutions: poor ROI, extensive data prep requirements, limited scope, indicating persistent adoption barriers despite vendor investment.
— Critical practitioner analysis warning against over-reliance on vector search for LLM retrieval, advocating for hybrid approaches and highlighting adoption barriers in semantic search.
— AWS case showing QuickSight Q deployment in internal sales team, enabling monthly business reviews in minutes vs. hours, with practical guidance on schema definition and synonym management.
— EACL 2023 paper addressing production reliability by augmenting neural semantic parsers with search algorithms using external criteria like query execution validation.
— Independent third-party evaluation by 30+ Tableau consultants testing ThoughtSpot's natural language search, showing adoption interest tempered by usability concerns and need for training.
— AAAI 2023 text-to-SQL model achieving 80.5% exact-match accuracy on Spider dev set and 71.7% execution accuracy on Dr.Spider robustness benchmark, with 277 stars and active development.
— Research advancing LLM-based text-to-SQL through few-shot prompting, instruction fine-tuning, and synthetic data augmentation, achieving improvements on Spider and BIRD benchmarks.
— Comprehensive survey published in Foundations and Trends in Databases covering NLQ and conversational analytics, highlighting entity identification and intent interpretation as core challenges.
— EMNLP 2022 industry paper from Amazon achieving 56.8% execution accuracy on WikiTableQuestions test set with fine-grained semantic parsing, demonstrating incremental progress on complex queries.
— COLING 2022 paper showing models trained on Spider achieve 75% accuracy on Spider but drop below 20% on unseen databases, identifying severe generalization barriers for production deployment.
— AWS blog announcing QuickSight Q expansion to data lake querying, showing continued vendor product evolution with deployment guidance for self-service analytics.
— COLING 2022 survey of text-to-SQL progress, datasets, and methods, documenting field maturity and ongoing challenges in translating natural language semantics to SQL.
— Empirical study showing crowd-powered question decomposition boosts text-to-SQL pipeline accuracy from 30% to 59% (96% relative gain).
— Critical independent assessment from Yahoo engineer questioning whether semantic search delivers in practice, highlighting failures on unseen data and unmet expectations.
— Schema expansion method improving text-to-SQL parsers by up to 13.8% relative accuracy gain on domain-generalization benchmarks.
— BARC/Eckerson survey of 214 companies showing 14% cite natural language queries as driver for increased BI adoption, with 20% adoption in North America.
— AWS GA of QuickSight Q, a managed NLQ capability for BI with embedding support, signaling major vendor commitment to natural language data interfaces.
— LLM-based text-to-SQL method achieving 85.3% execution accuracy on Spider (5.4% gain over prior SOTA), establishing SOTA performance on major benchmarks.