The AI landscape doesn't move in one direction — it lurches. Some techniques leap from experiment to table stakes in a single quarter; others stall against regulatory walls, technical ceilings, or organisational inertia that no amount of hype can dislodge. Knowing which is which is the hard part. The State of Play cuts through the noise with a rigorously maintained index of AI techniques across every major business domain — classified by maturity, evidenced by real-world adoption, and updated daily so you always know where you stand relative to the field. Stop guessing. Start knowing.
A daily newsletter distilling the past two weeks of movement in a domain or two — delivered to your inbox while the index updates in the background.
Each dot marks the weighted maturity of practices within a domain — hover for a brief summary, click for more detail
AI that monitors data quality, automates cleaning and transformation, and remediates issues across data pipelines. Includes anomaly detection in data flows and automated schema mapping; distinct from data catalogue management which documents rather than transforms data.
Automated data quality, cleaning, and transformation tooling has reached technical maturity — and stalled at the organisational gates. The practice encompasses AI-driven profiling, validation-as-code frameworks, and pipeline orchestration tools that monitor data flows, flag anomalies, and remediate issues before they propagate downstream. Vendors have delivered: commercial platforms, open-source validation engines, and generative AI copilots now cover the full pipeline lifecycle. Forward-leaning organisations in regulated finance and cloud-native operations have extracted clear ROI. Yet most enterprises remain stuck. Surveys consistently find data quality cited as the primary barrier to AI deployment; 2026 research shows data readiness outranking cost and talent concerns as top AI adoption challenge, with Gartner predicting 60% of AI projects will be abandoned due to inadequate data foundations. The bottleneck is not tooling but governance: unclear data ownership, siloed architectures, and a persistent skills gap mean that automation often scales bad logic faster than it fixes it. This is a leading-edge practice defined by a paradox — the technical problem is largely solved, but the human and structural preconditions for broad adoption are not.
The vendor ecosystem has stratified into three tiers: commercial visual platforms (Alteryx Designer Cloud, Google Cloud Dataprep, Fivetran Transformations), AI-assisted approaches (IBM Auto DQ, ClicData ML nodes, LLM-powered cleaning systems), and open-source validation-as-code anchored by Great Expectations, which launched GX Cloud SaaS with SOC2 certification in early 2026. Market evidence in Q1 2026 shows data preparation platform consolidation, with the DPaaS market growing 22.7% annually to $3.22B in 2026 and projected $7.36B by 2030; Alteryx processed 380M automated workflows annually (up from 260M in 2023), demonstrating enterprise-scale adoption. Forrester's Q1 2026 analyst evaluation documents a dramatic market shift: vendors now integrate AI-driven multimodal automation across profiling, classification, validation, and remediation, with observability, governance frameworks, and unified platforms (rather than fragmented point tools) emerging as competitive differentiators in early 2026. Real-world deployments showcase measurable ROI: capital markets firms automating ingestion, extraction, validation, and orchestration achieve 60% accuracy improvement, 65% manual reduction, and 523% three-year ROI with 8-month payback; Starbucks processes 1B+ data rows monthly with 95% time reduction; Bacardi reduced 40+ monthly preparation hours to minutes; FinTech enterprises achieve 75% outcome automation via governance-first frameworks. Organizational commitment is hardening: 70% of enterprises now establish Chief Data Officer roles (a 20% increase) due to renewed focus on data quality, signaling mainstreaming of dedicated governance and quality infrastructure. Yet production deployment remains challenging: a critical gap persists between analyst speed and production readiness, with governance automation, scaling constraints, and proprietary platform dependencies creating multi-week deployment bottlenecks despite desktop tool maturity. A "data paradox" has emerged in Q1 2026: 90% of enterprises report financial impact from undetected data errors and 88.6% experience operational delays, yet 68.5% report confidence in their data for critical decisions—revealing systemic underestimation of quality gaps. The gap between AI ambition and data readiness is widening: 85% of enterprises are adopting agentic AI, but only 43% report confidence in their underlying data quality; data readiness cited by 62% of executives as blocking production GenAI deployments. Architecturally, enterprises are shifting from bolting validation onto existing pipelines toward integrated platforms with embedded policy engines, governance controls, and observability—a recognition that point tools alone cannot close the gap. Emerging frameworks (Gartner, 2026) define AI-ready data as contextually proven fitness validated continuously, positioning metadata as the foundational layer; successful organizations treat governance and quality as unified accountability models with continuous operational integration rather than separate initiatives. Critical assessments identify weak data quality as an amplification mechanism: automation scales bad data at scale. Broader scaling remains blocked by non-technical factors: governance clarity, data ownership accountability, and a workforce that only 38% of leaders consider adequately skilled for AI-era data operations.
— Data extraction automation reduces field error from 1-4% (manual) to 0.1-0.5% (AI-driven), representing 5-20x improvement; decisioning platforms achieve 30-50% cost reduction with documented remediation cost baseline of $380-$1,200 per error.
— Data transformation identified as most time-consuming analytics phase; data teams spend 44% of hours cleaning/modeling/preparing. 78% of orgs now consider transformation critical or very important, up from 52% in 2022; 67% of pipeline failures originate in transformation.
— Uber deployed D3 automated data drift detection in production across critical ML pipelines; detects data incidents 5X faster than manual; 10% corruption across major US cities for 45 days would cost millions in lost revenue.
— Enterprise-scale data wrangling analysis: 54% of CIOs discover unsanctioned data prep work; version conflicts, governance gaps create compliance-audit liability; 85% report explainability delays from undocumented transformation, indicating organizational adoption barriers.
— Gartner finding: 72% of enterprise AI projects fail; seven of ten failures trace to poor data quality and missing governance, not model problems. Master data hygiene and governance as critical prerequisites.
— Deloitte survey of 3,235 business/IT leaders (24 countries) reveals data management maturity at 40% vs. tech infrastructure 43%, talent 20%; identifies data management as most critical bottleneck for enterprise AI scaling.
— Enterprise survey: 64% cite data quality as top AI risk; only 48% automate drift detection, 52% catch drift post-incident. Establishes quantified market concern with data quality governance as fundamental AI blocker.
— Enterprise case study: 200+ data workflows migrated from Alteryx, achieving $500K first-year savings and hours-to-minutes speedup. Cloud strategy misalignment and cost drivers (10-20 analysts = $50K-$100K+/year licensing) identified as adoption signals for modernized tooling.
2019: Visual data preparation tools (Trifacta, Google Cloud Dataprep, Fivetran) matured with production deployments across cloud platforms and major enterprises; Great Expectations launched as open-source validation framework; Trifacta fundraising and customer growth signaled strong market validation, though adoption barriers around data governance and bias detection persisted.
2020: Market research quantified demand: Trifacta survey showed 1/3 of AI/analytics cloud projects fail due to data quality and 75% of executives lack confidence in their data. Great Expectations v0.13.0 released with major feature improvements, establishing validation-as-code as mature open-source approach. Commercial platforms (Dataprep, Fivetran) continued embedding in cloud stacks, but adoption remained constrained by context-dependent rules and the need for continuous human oversight of data governance.
2021: Cloud vendors advanced ecosystem integration: Google Cloud Dataprep announced BigQuery pushdown optimization, and Trifacta achieved 2x productivity gains with Snowflake pushdown optimization. MIT researchers introduced PClean, a Bayesian probabilistic system for automated data cleaning at scale. IBM released Data Quality Toolkit for ML workflows. Surveys confirmed adoption momentum: 66% of organizations improved data quality through governance programs, rising to 83% for mature programs. OpenLineage and community projects demonstrated growing ecosystem maturity around validating data in production pipelines.
2022-H1: Market consolidation validated the practice: Alteryx acquired Trifacta for $400M, merging engineering and analytics capabilities. Product maturation advanced: Designer Cloud introduced schema drift detection, and Great Expectations deepened workflow orchestration with Prefect integration (75% error reduction in user deployments). Academic research formalized cleaning methodology, while industry cost analysis estimated poor data quality at $3.1 trillion annually. However, adoption barriers remained: 75% of organizations cited data quality as a significant challenge in analytics projects, indicating the practice was still not universally adopted despite proven ROI.
2022-H2: Open-source adoption accelerated: Great Expectations grew from 80,000 to 225,000 daily downloads, reaching 6.7M monthly downloads with 320 GitHub contributors. Industry recognition advanced: ThoughtWorks elevated Great Expectations to Adopt status, recommending it as production standard. Industry surveys revealed widespread data quality crisis: 40% of engineer time spent fighting data quality issues, 26% of revenue impacted by poor quality, with completeness (39%), consistency (38%), and accuracy (35%) as leading failure modes. Critical assessments identified systemic barriers: tool fragmentation, lack of data ownership clarity, and insufficient skilled personnel limiting broader deployment beyond cloud-native workflows.
2023-H1: Commercial product maturity accelerated: Alteryx rebranded Trifacta to Designer Cloud with automated cleansing features; Great Expectations advanced with community-driven features (ID/PK row identification, fluent-config datasources). Formal frameworks emerged: healthcare DQ-DO framework synthesized 227 articles identifying six data quality dimensions. Critical research gap identified: NSF-funded ICDE study (26,000+ model evaluations) found automated data cleaning more likely to worsen fairness than improve accuracy, signaling automation pitfalls without fairness safeguards. Adoption barriers persisted despite tool maturity: May 2023 survey confirmed data quality remains largest obstacle to AI success, reinforcing need for human oversight in automated pipelines.
2023-H2: Open-source platform maturity solidified: Great Expectations community analysis (September 2023) revealed most-deployed validation checks (nullness, value ranges, schema consistency), indicating production-scale adoption patterns. Organizational prioritization sustained: Monte Carlo survey (August 2023) of 350+ data leaders confirmed 40%+ ranked data quality as top 2023 priority despite GenAI pressure. Technical integration challenges surfaced: Great Expectations compatibility issues with SQLAlchemy 2.0 on MS SQL (September 2023) revealed adoption friction in leading open-source tools. Academic advances in automation methodology: Poznan University research (August 2023) demonstrated GPT-3 and ML-based approaches to product data quality verification. Economic drivers reinforced: industry analysis citing $12.9M annual cost per organization (Gartner) and Samsung Securities case study (2018 manual error, $187M loss, documented in 2023) underscored ROI but also persistent reliance on human oversight and governance discipline to prevent automation-driven fairness degradation.
2024-Q1: Alteryx expanded Data Engineering Cloud offering with 180+ data connectors and multi-cloud capabilities, while Google Cloud positioned Dataprep Premium for enterprise deployments with claimed 90% build time reduction. MuleSoft survey of 1,050 IT leaders revealed persistent barriers: 62% lack data harmonization systems for AI, 81% report data silos blocking transformation, only 28% of enterprise apps integrated. Great Expectations maintained leadership as de facto open-source standard, yet community feature requests and usability issues (expectation filtering, Six Sigma methodology gaps) surfaced limitations. Academic research articulated 29-dimension DQ assessment framework, while critical practitioner analysis questioned whether context-dependent data cleaning decisions could be meaningfully automated, maintaining healthy skepticism about automation claims despite vendor positioning.
2024-Q2: Early signals of generative AI adoption emerged with Prophecy and similar tools introducing AI-assisted data transformation workflows, positioning copilots for code generation and documentation; Great Expectations consolidated ecosystem adoption across cloud platforms and orchestration tools; vendors maintained competing positions (Alteryx's Designer Cloud, Google's Dataprep, open-source GX) though underlying challenges (data silos, integration complexity, governance clarity) persisted. Data scientists continued spending ~25% of time on cleaning and loading, indicating that tool maturity had not yet achieved significant time recovery. Industry understanding evolved to recognize data transformation as inherently contextual and judgment-dependent work requiring human expertise even with advanced tooling.
2024-Q4: Data quality pain intensified as adoption barriers persisted despite tool maturity: Precisely survey (550+ professionals) found data quality is top challenge (64%, up from 50% in 2023) and top investment priority (60%), with 77% rating data quality as average or worse. Platform maturity continued with GX cloud-backed improvements and Alteryx/Designer Cloud positioning; however, real-world deployments revealed friction (serverless Databricks compatibility issues, performance limitations >1TB, steep learning curves). Competitive consolidation evident with Databricks/Tabular acquisition and Snowflake Polaris catalog launch reshaping the governance landscape.
2025-Q1: Generative AI maturation accelerated adoption of transformation automation: Fivetran GA of Transformations with AI-assisted Quickstart models reduced preparation from weeks to hours (customer quote: 1 week to 1 hour); Lingaro case study showed 4x faster report delivery with GenAI. Adoption barriers remained structural: KPMG survey (85% of C-suite) and Informatica CDO survey (38% of 600 CDOs) identified data quality and trust as critical blockers to AI scaling. Named deployments continued in regulated sectors: Commerzbank and Crédit Agricole deployed Trifacta/Alteryx for compliance and risk reporting automation. Tool friction persisted: Great Expectations users reported validation failures on large datasets (38.6M+ rows), revealing ongoing usability challenges. The practice consolidated at leading-edge: generative AI copilots promised efficiency gains, but business adoption remained constrained by governance gaps and the contextual nature of cleaning decisions.
2025-Q2: Platform maturation widened with competing approaches: Alteryx/Designer Cloud expanded feature set (Multi Column Binning, Data Cleanse Pro Pro), ClicData introduced ML-driven transformation nodes and advanced data flow automation, and Great Expectations extended cloud integration with Redshift datasource API support, signaling ecosystem diversification. Adoption barriers proved sticky—contrary to generative AI hype, 64% of organizations (Precisely 2025, up from 50% in 2023) identified data quality as top challenge, and 31% of revenue remained at risk from quality issues. Industry analysis highlighted failure costs: Unity's 2022 ML error from inaccurate data caused 37% stock drop, underscoring ROI for automation. Enterprise commitment remained: 85% of C-suite (KPMG 2025) and 38% of CDOs (Informatica 2025) prioritized data quality for AI success. High-value sector deployments (regulatory reporting at Commerzbank, Crédit Agricole) demonstrated clear ROI in compliance automation, but scaling remained blocked by governance clarity gaps, data silos (81% of enterprises), and personnel capability constraints. The practice remained at leading-edge with maturing tooling but persistent organizational adoption barriers preventing rapid scaling.
2025-Q3: Ecosystem maturity reached production-scale but organizational adoption stalled: IBM Auto DQ (watsonx.data) claimed 80% manual effort reduction; GX achieved cloud-native maturity (Microsoft Fabric, Redshift API); Fivetran continued AI-assisted automation gains (1-hour vs. 1-week deployment baselines). However, market sentiment shifted to reflect structural barriers: only 8% of organizations (Protiviti) achieved true AI transformation with data quality as #1 blocker; only 7.6% AI-ready (TDWI) with 31% citing quality as primary obstacle; 34% reported data inadequate for transformation (Grant Thornton). Critical insight emerged: partial automation bias—73% of organizations attempting AP automation remained trapped in partial automation with quality defects causing systematic failures, revealing that naive automation scaling without governance led to negative ROI. Signaled that practice had hit the boundaries of pure technical maturity; broader scaling required solving non-technical organizational barriers: governance clarity, data ownership accountability, and personnel capability development.
2025-Q4: Ecosystem consolidation continued with vendor maturity: Great Expectations v1.8.0 released with enhanced Snowflake Key Pair Auth and Volume Expectations row conditions; Trifacta/Alteryx serving 12,000+ global clients including Fortune 100 companies. Market dynamics shifted from tool adoption to implementation barriers: practitioners published comprehensive deployment guides for GX in production (Airflow, Databricks, Spark integration patterns), indicating ecosystem stabilization. However, structural barriers showed no signs of weakening—data transformation challenges (rapid growth, security, legacy systems, talent gaps, integration complexity, quality assurance, cost management) remained sticky organizational problems with no automation silver bullet. Signaled that the practice had reached the maturity-adoption plateau: technology solved the technical problem, but human-dependent factors (governance clarity, ownership accountability, contextual judgment in cleaning decisions, organizational discipline) continued limiting scaling and advancement.
2026-Jan: Platform commercialization accelerated: Great Expectations launched GX Cloud SaaS with SOC2 compliance and collaborative governance features, marking strategic shift toward commercial offerings while maintaining OSS ecosystem. Market research showed persistent adoption barriers despite technological maturity: 64% of organizations cited data quality as top challenge (up from 50% in 2023), 77% rated data quality as average or worse, and foundational governance gaps remained primary blocker with 40.9% of leaders prioritizing improved data governance in 2026. Industry predictions emphasized DataOps becoming strategic function with quality automation as operational core for AI success. Data transformation remained contextual and judgment-dependent, with tool maturity failing to drive mass adoption. The practice remained at leading-edge plateau: technology solved technical problems, governance and organizational accountability remained limiting factors.
2026-Feb: AI-driven automation exposed cost and scalability limitations: practitioner research revealed LLM-based data cleaning approaches cost $2,250 per 100K rows, spurring shift toward hybrid architectures combining AI analysis with deterministic execution tools. Organizational barriers intensified despite vendor maturity—73% of data leaders cited quality as primary AI barrier, Gartner predicted 60% AI project abandonment due to unready data. The "Agentic AI Data Integrity Gap" became explicit: 85% of enterprises adopting Agentic AI but only 43% confident in data readiness, exposing that AI acceleration revealed foundational quality gaps that automation alone could not resolve. Architectural evolution accelerated toward policy-first platforms and integrated governance rather than point tooling, signaling recognition that integration and organizational investment, not just better tools, drive value. The practice remained at leading-edge plateau with persistent organizational barriers limiting advancement.
2026-Apr: Analyst and survey evidence reinforces the adoption paradox. Gartner's 2026 Magic Quadrant for Augmented Data Quality Solutions (13 vendors) projects 70% of organisations will adopt modern DQ solutions by 2027; a companion Gartner operating model report finds only 40% of AI prototypes reach production, with data availability cited as the primary barrier. The Cloudera Data Readiness Index (1,270+ IT leaders) documents a stark gap: 96% use AI, but 79% say data access limits AI success and only 18% have full governance in place. Forrester's Q1 2026 Wave identified unified observability and governance architectures as the competitive differentiator as platforms converge on AI-driven multimodal automation; BigQuery Data Preparation reached GA for Cloud Storage, extending hyperscaler-native AI-assisted transformation. IDC capital markets evidence confirms 523% three-year ROI and 8-month payback from end-to-end automation, while 70% of enterprises now establishing CDO roles signals organisational mainstreaming—but practitioner assessments of Great Expectations continue to surface real constraints (test maintenance overhead, dependency complexity, diagnostic delays), confirming tooling maturity still outpaces governance readiness.
2026-May: Survey and practitioner evidence reinforces the adoption paradox with quantified cost stakes. AI-driven data extraction reduced field error rates 5–20x (1–4% manual to 0.1–0.5%), and decisioning platforms deliver 30–50% cost reduction, with remediation costs benchmarked at $380–$1,200 per error. Uber published its D3 automated drift detection system deployed across critical ML pipelines, detecting data incidents 5x faster than manual processes. Deloitte's survey of 3,235 leaders across 24 countries placed data management maturity at only 40%, identifying it as the primary bottleneck for enterprise AI scaling. Gartner reinforced: 72% of enterprise AI projects fail, with seven in ten failures tracing to poor data quality rather than model problems. Industry analysis found data teams spend 44% of hours on cleaning and preparation, and 67% of pipeline failures originate in transformation—confirming the practice addresses a structurally critical problem that tooling maturity alone has not resolved.
2026-Q1: Enterprise deployment matured at select vanguard organizations: Starbucks processes 1B+ data rows monthly with 95% time reduction via automated preparation; Bacardi reduced 40+ hours monthly to minutes; North American FinTech enterprise ($1B+ revenue) achieved 75% outcome automation via AWS modular framework with three-layer quality controls. Alteryx processed 380M automated workflows annually (up from 260M in 2023), and DPaaS market grew 22.7% annually to $3.22B with forecast to $7.36B by 2030, validating enterprise adoption trajectory at scale. Yet critical production gaps emerged: governance automation, platform scalability constraints, and multi-week deployment bottlenecks revealed maturity challenges despite desktop tool adoption; a "data paradox" shows 90% report financial impact from undetected errors and 88.6% experience delays despite 68.5% reporting confidence in data quality. Data readiness emerged as #1 barrier to AI adoption, surpassing cost and talent concerns; 62% of executives cite data readiness as blocking production GenAI deployment, with Gartner forecasting 60% AI project abandonment due to inadequate data foundations. Critical assessments identify weak data quality as amplification mechanism—automation scales bad data at scale. BARC analyst research shows data quality reclaimed top priority over AI initiatives, signaling market recognition that AI acceleration exposed foundational quality gaps. The practice remained at leading-edge plateau: vendor ecosystem consolidated around quality-first platforms (Great Expectations SaaS, Alteryx/Designer Cloud, integrated governance), but organizational adoption remained blocked by governance clarity, data ownership accountability, and workforce capability constraints.