The AI landscape doesn't move in one direction — it lurches. Some techniques leap from experiment to table stakes in a single quarter; others stall against regulatory walls, technical ceilings, or organisational inertia that no amount of hype can dislodge. Knowing which is which is the hard part. The State of Play cuts through the noise with a rigorously maintained index of AI techniques across every major business domain — classified by maturity, evidenced by real-world adoption, and updated daily so you always know where you stand relative to the field. Stop guessing. Start knowing.
A daily newsletter distilling the past two weeks of movement in a domain or two — delivered to your inbox while the index updates in the background.
Each dot marks the weighted maturity of practices within a domain — hover for a brief summary, click for more detail
AI that performs initial exploration of datasets, identifying distributions, correlations, missing values, and notable patterns. Includes automated profiling reports and insight suggestion; distinct from predictive modelling which builds models rather than exploring data.
Automated exploratory data analysis (autoEDA) has solved the profiling problem but is stalled at the insight boundary. Forward-leaning organisations embed automated dataset profiling, quality checks, and visualization into production ML pipelines—AWS Glue DataBrew, Databricks Genie, and Snowflake Data Metric Functions demonstrate operational maturity. The tooling—both enterprise and open-source—is mature and measurably accelerates analyst workflows (97% of data analysts report task acceleration; 150–300% median ROI documented). Yet most deployments remain narrowly focused on descriptive reporting and data quality checking. The defining tension is scope: automation excels at telling you what your data looks like but has not generalised to discovering novel, actionable insights that require domain judgment. Agentic approaches (autonomous query generation, multi-step reasoning, SQL code synthesis) are advancing in the vanguard—frameworks like AIDA and deployed systems at Meta, OpenAI, and Ramp demonstrate agentic EDA maturity in controlled environments; Capital One's production system reduces large-scale classification analysis from 9 months to 10 days—but broader adoption faces structural barriers. Reliability concerns persist (55% of agentic systems reach unsupported conclusions on real-world data; data analysis remains a consistent bottleneck in LLM-based agents), cost control remains elusive (96% of organisations deploying GenAI report unexpected overruns), and critically, data governance deficits prevent effective deployment: 47% of enterprises have made material decisions on inaccurate data, and 38% lack trust in automated insights. Gartner predicts 40% of agentic AI projects will be canceled by 2027 due to cost overruns and governance failures. Until organisations establish unified data governance and master cost visibility in agentic workflows, the practice will remain leading-edge but stalled.
The autoEDA ecosystem has matured into three distinct tiers: IDE-integrated, open-source profiling, and LLM-augmented enterprise platforms. PyCharm 2026.1 embeds AI-powered automated data issue detection in Jupyter notebooks; enterprise platforms—DataRobot, H2O Driverless AI, AWS SageMaker (Glue DataBrew with 250+ prebuilt transformations), Qlik Cloud, Decube, Google Meridian—embed profiling and quality checks directly into ML workflows. DataRobot's two-stage EDA is standard practice with 90% model development time reduction reported. Open-source libraries (ydata-profiling 13.4k stars, 1.57M monthly downloads; Sweetviz, DataPrep, AutoViz) serve practitioners generating comprehensive profiling reports in single-line code. Market scale: $2.2B augmented data quality market with 15% projected CAGR; open-source tooling fragmentation drives hybrid adoption (Snowflake + dbt + Metabase) over pure open-source stacks due to integration overhead.
Agentic exploratory analysis is advancing rapidly in June 2026. Databricks Genie Code (April 2026, GA) autonomously explores data, trains models, and builds pipelines with governance integration. AIDA (arxiv:2605.07202, May 2026) demonstrates agentic frameworks scaling to 200+ metrics and 100+ dimensions with Pareto-guided reinforcement learning for superior pattern discovery. Happycapy (production GA, May 2026) automates the full EDA-to-report cycle—distribution analysis, correlation matrices, statistical analysis, and report generation in 8 minutes versus 3–4 hours manual work, eliminating 6–8 hours/week of analyst time on data cleaning and profiling. DataClaw (arxiv:2605.02503, May 2026) establishes process-oriented evaluation benchmarks for agentic EDA, moving beyond final-answer metrics to assess analytical workflow rigor, signaling institutional readiness for production deployment in regulated domains. Capital One's production deployment (June 2026) demonstrates large-scale agentic EDA reducing classification analysis on 350 cloud resources from 9 months to 10 days, with systematic discovery of optimization opportunities (12 resource types drive 30–40% of savings) and detection of false-positive rates (40–60%) missed in rule-based analysis. dbt Labs confirms Meta, OpenAI, and Ramp run production agentic analytics systems at scale; Edison Scientific's autonomous analysis of 242,000 drug sensitivity records identified biomarker relationships (p = 1.7 x 10^-62). Enterprise AI analytics adoption stands at 59% (up from 33% in 2022); 97% of data analysts report AI accelerates daily tasks; 70% of financial data-processing tasks are automatable.
Adoption barriers remain structural and deepening. Despite agentic enthusiasm—Anthropic's 2026 report shows 60% of organisations cite data analysis as most impactful agentic use case and 80% report measurable financial ROI—deployment maturity stalls at the accuracy boundary and governance constraint. Peer-reviewed research identifies data analysis as a consistent bottleneck in LLM-based exploratory agents (SANA framework, June 2026); practitioners document an 86% accuracy ceiling for ad hoc exploratory questions without perfectly constructed data architectures and extensive preparation. Peer-reviewed research documents 55% of agentic data science systems reach unsupported conclusions on real-world datasets; cost overruns persist (96% of GenAI deployments, 92% of agentic workflows). More critically, data governance deficits block effective deployment: OneStream study (May 2026, 350+ executives) reveals 47% of enterprises made material decisions on inaccurate/incomplete data in past 12 months; 72% incurred $500K+ costs from bad data; 38% lack trust in automated insights. Only 19% pull majority of AI inputs from centralised systems; 61% second-guess data monthly, 11% daily. Gartner projects 40% of agentic AI initiatives will be canceled by 2027 due to cost overruns and governance failures. Production agentic systems deployed at scale (Meta, OpenAI, Ramp, Capital One) operate in controlled environments with pre-built data infrastructure. These governance gaps prevent even mature EDA platforms from delivering reliable autonomous insight discovery. Scaling agentic EDA requires solving data governance, cost visibility, and reliability calibration—not just advancing agent architecture—before advancement to good-practice tier.
— Analysis identifies data fabric quality and governance as critical blockers for agentic AI deployment; metric definitions diverge across functions, lineage breaks at join layer; Gartner predicts 40% of agentic projects canceled by 2027 due to costs and governance failures.
— Diagnostic framework for agentic exploratory question-answering on data lakes identifies data analysis as consistent bottleneck in LLM-based EDA agents, enabling component-level failure diagnosis distinct from end-to-end accuracy.
— Capital One production deployment reduced large-scale classification analysis from 9 months to 10 days on 350 AWS/Azure/GCP resources; discovered 12 resource types account for 30-40% of savings and exposed 40-60% false-positive rates in rule-based detection.
— Comprehensive adoption metrics from 24 authoritative sources: 88% of orgs use AI in business functions; 97% of data analysts report acceleration; 70% of financial tasks automatable; 150-300% median ROI; AI quality control achieves 99.5-99.9% accuracy.
— AWS Glue DataBrew (250+ prebuilt transformations) and Glue Data Quality automate profiling, anomaly detection, and format standardization; GA products signal vendor commitment and operational maturity in enterprise data profiling.
— Gartner Magic Quadrant positions augmented data quality vendors (agentic AI reshaping discovery, profiling, anomaly detection) in $2.2B market; signals leading-edge platform maturity while noting data availability/quality remain barriers to AI adoption.
— Practical R tutorial comparing skim(), DataExplorer, and SmartEDA packages demonstrates automated profiling capabilities (distributions, correlations, missing-data patterns) enabling analysts to generate full profiles in minutes rather than hours.
— Practitioner analysis of agentic AI in data science shows agents autonomously execute EDA pipelines (data retrieval, cleaning, analysis, modeling, reporting); documents production frameworks (LangGraph, AutoGen) and required skill shifts in system design and observability.
2019: AutoEDA emerges with commercial (H2O Driverless AI) and open-source (R packages, pandas-profiling) tooling. R ecosystem maturity confirmed by systematic review of 15 packages. Adoption metrics show significant downloads but focus remains on profiling and basic visualization rather than deep insight discovery.
2020: Ecosystem expands with new Python tools (Sweetviz, continued pandas-profiling adoption) and major vendor commitment (AWS SageMaker Data Wrangler GA December 2020). ACM SIGMOD paper reviews ML approaches to EDA automation. Deployment barriers emerge: tool reliability issues, integration complexity, and persistent questions about automating insight discovery beyond profiling.
2021: Competitive maturation across open-source Python libraries (AutoViz, Pandas Profiling, SweetViz, D-Tale, Dataprep) with documented performance tradeoffs. Critical academic commentary questions whether EDA/CDA distinction can survive automation. Production deployments at scale (e.g., EDF Lab preventive maintenance) show AutoML integration but reveal feature engineering limitations in open-source tools. Fundamental scope question remains: can automation extend beyond profiling to genuine insight discovery, or is the practice limited to data quality checking?
2022-H1: LLM integration emerges as new research direction (InsightPilot from HKUST and Microsoft Research) proposing to automate insight discovery via natural language prompts and production-quality insight tools. Open-source ecosystem consolidation continues: ydata-profiling secures corporate backing from YData (Feb 2022) with 50M downloads and broad enterprise adoption (FAANG, banks, insurance), expanding support for time-series and Spark workloads. Core tension persists: profiling automation is mature and widely deployed, but advancing to genuine insight discovery remains blocked by domain expertise requirement and LLM reliability concerns.
2022-H2: Commercial ecosystem expands with YData SDK launch (Nov 2022) offering automated profiling beyond open-source tools. Open-source tooling matures with continued ydata-profiling adoption but quality challenges surface (duplicate detection bugs, edge-case handling). Practitioner engagement deepens as vendors release advanced tutorials and feature expansions. AutoEDA remains bifurcated: robust profiling for data quality checking in production, but scaling challenges and LLM integration experiments are still research-stage with uncertain ROI.
2023-H1: Ecosystem consolidation continues with ydata-profiling reaching 10k GitHub stars milestone. Enterprise deployment advances: H2O Driverless AI integrates with Snowflake Snowpark for at-scale EDA without data movement. YData extends ydata-profiling with Spark support (April 2023), enabling distributed profiling. Academic validation emerges: research framework demonstrates 2x productivity gains from automated EDA in user studies. Practice remains production-focused on data quality and descriptive profiling rather than insight discovery; scope limitations persist despite wider enterprise availability.
2023-H2: No major tool releases or deployment breakthroughs documented. Ecosystem consolidation continues with ydata-profiling and H2O Driverless AI as dominant platforms; DataRobot and other vendors advance AutoML capabilities but not specifically AutoEDA. Open-source community contribution continues with incremental tool development. LLM-based insight discovery (InsightPilot prototype from early 2022) shows no evidence of production adoption or maturity by year-end. Practice remains at leading-edge with mature tooling but constrained scope—automation delivers on profiling and data quality checking but has not expanded to general-purpose insight discovery or causal inference.
2024-Q1: Observability vendors integrate LLM-powered EDA tools; Dynatrace announces Davis CoPilot for natural language data exploration in Grail. Practitioner discourse highlights persistent methodological tension: automated profiling tools improve efficiency but cannot substitute for purpose-driven, hypothesis-linked exploration. AutoEDA ecosystem remains bifurcated between mature open-source profiling libraries and AI-augmented vendor platforms, with scope limited to descriptive analysis and data quality workflows.
2024-Q2: Real-world deployments advance: Pecan AI case study demonstrates automated EDA catching data quality issues (duplicates, join errors) on 10 TB datasets; Actian launches data profiling GA in Data Platform; H2O.ai recognized as 2x Visionary in Gartner Magic Quadrants serving Fortune 500. Negative signal surfaces: critical security vulnerability (CVE-2024-37062) in ydata-profiling 4.0+ raises reliability concerns for enterprise adoption. Practical deployment optimization shows Spark profiling scalability solutions (25-minute reduction). Ecosystem maturity confirmed but adoption barriers persist around tool reliability and methodological limitations.
2024-Q3: Integration challenges emerge: practitioner reports Streamlit application crashes when using ydata-profiling, exposing tool compatibility and deployment reliability issues. Enterprise adoption patterns narrow: automated EDA remains focused on data quality checking and profiling within larger data platforms (SageMaker, Driverless AI, Snowflake integration) rather than expanding toward autonomous insight discovery. Ecosystem consolidation continues with H2O and YData as dominant platforms.
2024-Q4: LLM-powered EDA research advances with TiInsight (PingCAP production deployment) achieving 86.3% SQL accuracy; major vendors strengthen EDA capabilities (DataRobot Workbench enhancements, DagsHub RepoViz for unstructured data). R ecosystem remains mature with established packages (skimr, SmartEDA, DataExplorer). Deployment barriers persist: persistent dependency and compatibility issues (ydata-profiling in Streamlit) continue to limit production adoption. Practice remains at leading-edge—mature profiling and data quality automation widely available, but scope remains narrowly focused on descriptive analysis rather than autonomous insight discovery.
2025-Q1: Sustained ecosystem adoption confirmed: ydata-profiling maintains 1.57M monthly downloads and 13.4k GitHub stars; pandas-profiling legacy library still at 194k monthly downloads. Market data shows EDA tools market at $15 billion with 15% projected CAGR through 2033, indicating continued enterprise investment and competitive growth despite persistent tool reliability and integration challenges.
2025-Q2: Research innovation advances with QUIS system automating question generation and insight synthesis without human curation, signaling academic progress toward autonomous EDA. Real-world deployment shows continued integration into enterprise platforms (Pricefax EDA workflows for customer analysis). Tool maturity remains constrained by persistent limitations: sweetviz visualization degradation with high-cardinality datasets (100+ columns), ydata-profiling memory consumption issues with specific data patterns. Negative signals balance positive adoption, indicating practice remains at leading-edge with mature profiling capabilities but constrained scope on insight discovery and edge-case robustness.
2025-Q3: Ecosystem consolidation continues with sustained adoption: open-source EDA tools (ydata-profiling, sweetviz, Rath, great-expectations) show strong community traction via GitHub (9,469 repos tagged 'eda'). AI-augmented EDA approaches gain practitioner interest (Observable blog, vendor integration) but reliability concerns persist—LLM-based data exploration shows promise yet cannot fully substitute for domain-specific exploration. Practice remains at leading-edge: automated profiling is production-standard across enterprise platforms, but scope remains bounded to descriptive analysis and data quality checking rather than autonomous insight discovery.
2025-Q4: Sustained vendor investment in platform maturity: H2O releases v25.08.2 with enhanced Driverless AI capabilities; Google integrates automated data checks into Meridian's production MMM workflows; DataRobot achieves Gartner Peer Insights recognition with 90% reduction in model development time. Negative signal surfaces: IDC research reveals 96% of organizations deploying GenAI report unexpected cost overruns, 92% for agentic AI workflows, highlighting adoption barriers at scale. AI-augmented EDA research (QUIS, TiInsight) achieves production deployment maturity but remains specialized domain. Traditional profiling tools face persistent edge-case quality challenges (sweetviz high-cardinality visualization, ydata-profiling memory patterns, Streamlit compatibility). Practice remains at leading-edge with proven efficiency gains in model development but constrained scope on cost control, reliability, and autonomous insight discovery.
2026-Jan: LLM-powered EDA maturity advances with TiInsight arXiv preprint demonstrating production deployment at PingCAP; practitioner case study shows real-world adoption of Gemini-based automated survey coding (400 responses). Ecosystem remains focused on profiling and structured analysis; scope expansion into unstructured data exploration signals emerging capability but not yet mainstream. Leading-edge tier sustained.
2026-Feb: DataRobot and Edison Scientific document production-grade automated EDA with real-world deployments: DataRobot two-stage EDA (schema detection, quality checks, feature association) embedded in standard ML workflows; Edison Analysis autonomous EDA on 242k drug sensitivity records identifies biomarker patterns with statistical validation. Eric Ma practitioner analysis reports 5-10x speedup with AI coding agents while cautioning rigor loss—emerging best practice. ISEDA 2026 conference (Singapore, May) dedicates track to 'AI & Open Source EDA'. Market analysis confirms $4B+ market with 8% CAGR through 2033. Ecosystem bifurcation continues between mature profiling tools (ydata-profiling, Sweetviz) and AI-augmented research systems (TiInsight, Edison Analysis).
2026-Mar: Cloud-native EDA maturation confirmed: Azure Databricks GA automated data profiling with continuous metric computation across time-series, inference, and snapshot modes, enabling drift detection without manual setup. LLM-assisted production workflows emerge: DS Stream case study (Databricks) achieves 92% PII detection precision and reduces manual audit time from weeks to hours. Editorial and practitioner surveys confirm mainstream adoption of automated EDA (5-10x speedup documented across multiple tools). Agentic EDA architectures proliferate as emerging pattern for autonomous exploration. Ecosystem validation: YData Profiling tutorial updated March 2026 in Real Python; comprehensive ecosystem surveys (Analytics Insight) document mature tooling across six major libraries. Negative signals persist: GenAI-powered EDA adoption requires constant human validation per domain experts. Practice remains at leading-edge with proven cloud integration and LLM-augmented profiling, but scope remains bounded to descriptive analysis rather than autonomous insight discovery.
2026-Apr (early): IDE integration accelerates: PyCharm 2026.1 embeds AI-powered data issue detection in Jupyter notebooks, extending automated profiling beyond standalone tools into mainstream development workflows. Data catalog and governance platforms GA automated profiling as standard feature: Decube Profiler supports major data warehouses (Snowflake, Redshift, BigQuery, Databricks); Qlik Cloud enables field-level analysis without manual exploration. Enterprise AI analytics adoption reaches 59% (Gartner 2025, up from 33% in 2022), with EDA as core component. Research validates tool performance: DataPrep.EDA research demonstrates declarative interface outperforming pandas-profiling on speed and UX. Operational maturity evident: organizations embed profiling as continuous capability in pipelines (validation, deduplication, anomaly detection). Practice remains at leading-edge with broadened vendor ecosystem and IDE adoption, but scope remains bounded to descriptive analysis and data quality checking rather than autonomous insight discovery.
2026-Apr (late): Agentic analytics deployment accelerates at leading-edge orgs: Databricks Genie Agent Mode reaches Public Preview for autonomous multi-step exploratory analysis; Azure Databricks Sample Data Explorer GA with natural language-to-SQL translation. dbt Labs reports production agentic systems at Meta (scaled from prototype to company-wide in 6 months), OpenAI, and Ramp. However, reliability and cost barriers prevent broader adoption: peer-reviewed research (Sanity Checks for Agentic Data Science) documents 55% failure rate on real-world datasets with unsupported conclusions; IDC survey confirms 96% of GenAI deployments face unexpected cost overruns, 92% for agentic workflows. Practice remains at leading-edge but trend is stalled—agentic EDA advances show promise but cost control and reliability constraints block advancement to good-practice tier.
2026-May (early): Native profiling integrates deeper into major platforms: Databricks added May 2026 native data profiling directly in SQL Editor and Notebooks for automated statistical summarization on query results (null counts, distributions, ranges); Snowflake customers automate profiling at scale via native Data Metric Functions with schema drift detection; Genie Chat public preview (April 29) and Genie scheduled tasks (April 30) enable recurring automated exploratory prompts and weekly digest generation. Agentic EDA direction solidifies as leading-edge practice while traditional profiling becomes operational standard across platforms. Bifurcation widening: specialized purpose-built tools outperform generic LLMs on rigor and reproducibility (R² improvement, multicollinearity documentation, feature engineering), while enterprise telecoms cases show automated feature importance analytics eliminating 8 monthly hours of manual exploration. Trend remains stalled: profiling automation broadly adopted across major platforms, but barriers to reliable autonomous insight discovery persist despite agentic advancement.
2026-May (mid-late): Agentic EDA maturity milestone: AIDA framework (arxiv:2605.07202, May 8) demonstrates autonomous insight discovery in complex BI environments (200+ metrics, 100+ dimensions) with Pareto-guided reinforcement learning, outperforming workflow-based agents in pattern discovery depth and breadth. Happycapy production EDA platform (May 15, GA) automates full EDA-to-report cycle (8 minutes vs 3–4 hours) with concrete metrics (eliminates 6–8 hrs/week data cleaning, 3–5 hrs/week EDA scripts, reduces dash refresh from 2–3 hrs to near-real-time). DataClaw benchmark (arxiv:2605.02503, May 6) establishes process-oriented evaluation for agentic workflows beyond final-answer metrics, signaling institutional rigor for regulated-domain deployment. Jupyter ecosystem evolution (Weaver analysis) documents infrastructure consolidation enabling seamless agentic integration. However, adoption barriers deepen: OneStream study (May 7, 350+ executives) quantifies structural data governance gap—47% made material decisions on inaccurate data (past 12 months), 72% incurred $500K+ costs, 38% lack trust in automated insights. Only 19% centralize AI data inputs; 61% second-guess data monthly. These governance deficits prevent reliable EDA deployment despite tool maturity. Practice remains at leading-edge with advanced agentic frameworks and production deployments at scale, but trend stalled due to organizational data governance gaps and cost control barriers preventing broader adoption.
2026-Jun: Agentic EDA adoption metrics confirm market interest alongside persistent accuracy constraints: Anthropic 2026 State of AI report shows 60% of organisations cite data analysis and report generation as most impactful agentic use case, with 80% reporting measurable financial ROI from AI agent investments; McKinsey data documents 27% of analyst time on actual analysis versus 45% on low-value tasks, framing automation's structural case. Adoption evidence quantified: 97% of data analysts report AI accelerates tasks; 150-300% median ROI documented; 70% of financial data-processing tasks automatable. Production adoption evidence expanded: Capital One's data agent (June 2026) reduces large-scale classification analysis on 350 cloud resources from 9 months to 10 days, systematically discovering optimization opportunities (12 resource types drive 30-40% of savings) and exposing false-positive rates (40-60%) missed in rule-based detection—demonstrating agentic EDA at production scale. Microsoft deployed hierarchical agentic RAG systems in Azure AI Search with multi-agent entity resolution for multi-step exploratory analysis; LLM-based tools (Claude Code) demonstrate 8 hours/week analyst time savings on distribution analysis, quality checks, and correlation mapping workflows. AWS Glue DataBrew GA (250+ prebuilt transformations) and Gartner's 2026 Magic Quadrant for augmented data quality ($2.2B market) confirmed enterprise vendor commitment; DataCOPE framework (June 2026) demonstrated autonomous procedural skill discovery for EDA agents achieving 9.71-32.30% performance gains without labeled supervision. Research identifies data analysis as consistent bottleneck in agentic exploratory question-answering (SANA diagnostic framework, June 2026). Accuracy ceiling persists as the binding constraint: practitioner evidence documents 86% accuracy on ad hoc exploratory questions requiring perfectly constructed data architecture and extensive preparation—often equivalent to manual analysis effort. Governance barriers deepen: Gartner projects 40% of agentic AI initiatives will be canceled by 2027 due to cost overruns and governance failures. Agentic EDA remains bifurcated: proven time savings on routine profiling and documented success at leading-edge orgs, but autonomous insight discovery insufficient for production deployment without sustained human validation, and organizational data governance gaps prevent scaling despite tool maturity.