The AI landscape doesn't move in one direction — it lurches. Some techniques leap from experiment to table stakes in a single quarter; others stall against regulatory walls, technical ceilings, or organisational inertia that no amount of hype can dislodge. Knowing which is which is the hard part. The State of Play cuts through the noise with a rigorously maintained index of AI techniques across every major business domain — classified by maturity, evidenced by real-world adoption, and updated daily so you always know where you stand relative to the field. Stop guessing. Start knowing.
A daily newsletter distilling the past two weeks of movement in a domain or two — delivered to your inbox while the index updates in the background.
Each dot marks the weighted maturity of practices within a domain — hover for a brief summary, click for more detail
AI applied to streaming data for real-time pattern detection, alerting, and decision-making on live data flows. Includes stream processing with ML models and real-time anomaly detection; distinct from batch analytics which processes historical rather than live data.
Real-time streaming analytics has matured into established good practice. The discipline — applying ML models, statistical aggregations, and pattern detection to continuous data flows rather than batch windows — now rests on a stable ecosystem of GA tooling, managed cloud services, and battle-tested deployment patterns. Apache Flink and Kafka have become de facto standards; all three major cloud providers offer managed streaming services (AWS Managed Service for Apache Flink, Microsoft Fabric Real-Time Intelligence, Google Cloud Dataflow); and the industry recognizes data streaming as a formal software category. Production deployments span banking (fraud detection, real-time risk assessment), payments (sub-10ms decision latency), fintech (Toss processing 7-day frequency capping state at 68GB scale), and operational analytics (billions of events daily). Market evidence signals mainstream adoption: analysts project $146.59B market by 2030 (33% CAGR); enterprises report 764% ROI on implementations (Starbucks), 200+ hour annual savings (Arla); and deployments now address cost-optimization as much as capability. By June 2026, enterprise adoption momentum has shifted decisively toward AI-driven architectures: ISG research predicts one-third of enterprises will integrate streaming with AI and generative AI inferencing by 2028 for real-time agentic applications. Simultaneously, the market-wide cost correction has deepened—vendors (Databricks, MotherDuck) explicitly recommend micro-batching and warehouse-native ingestion for use cases where sub-second latency is not a genuine business requirement, documenting that continuous streaming overhead (70% engineering complexity, 30% infrastructure) justifies only the latency extremes. The practice has transitioned past the "whether" question to pragmatic "when"—architectural decisions now rest on clear latency/cost trade-offs rather than universal real-time. The binding constraint remains organizational: the multi-disciplinary expertise in distributed systems, state management, and streaming semantics that production deployments demand continues to limit adoption to technology-forward organizations with dedicated data engineering capability.
Vendor consolidation has solidified around Apache Flink as the stateful processing engine and Apache Kafka as the transport layer. AWS completed the shift to Managed Service for Apache Flink, Microsoft Fabric earned Forrester Wave leader recognition (Q4 2025), and vendor ecosystem shows maturation signals: Confluent ranks #1 in event streaming across AI search platforms (35% presence, 6,500+ enterprise customers, 40%+ Fortune 500); IBM acquisition of Confluent at $11B (closed March 2026) signals strategic platform consolidation. Flink 2.2.0 (December 2025) introduced ML_PREDICT and VECTOR_SEARCH—embedding LLM inference and vector similarity directly into streaming pipelines. Apache Kafka 4.2.0 (March 2026) GA introduced Share Groups (enabling per-record acknowledgement) and Streams Rebalance Protocol (faster, stable rebalances), advancing ecosystem maturity. Commercial distributions advanced with Ververica Platform achieving Forrester Leader status (100B+ events/day, <10ms latency, 40% TCO reduction versus open-source Flink). Databricks launched Lakehouse//RT (June 2026) powered by Reyden compute engine, delivering millisecond latency directly on Delta/Iceberg without separate serving layers; named customers (Cisco, Magnite) report 5-16x performance improvements. Financial institutions (Rabobank, ING Bank, Capital One, Nationwide) run real-time fraud detection on event-driven Flink; Tier-2 regional bank reduced false positives from 12,000 to 600 daily (95% reduction) with sub-45ms fraud scoring and $1.4M annual operational savings. Beyond finance, PayTech (Toss) deploys 7-day frequency capping at 68GB state, PostNL migrates IoT asset tracking to managed Flink, Intuit operates 200+ Kubernetes clusters (5B daily messages), ByteDance maintains 70,000+ Flink jobs (11M+ slots, hundreds of trillions records/day). Uber demonstrates Kappa patterns (Kafka+Spark) for multi-team latency/correctness trade-offs in dynamic pricing and published petabyte-scale Flink ingestion replacing batch (25% compute reduction, hours-to-minutes freshness across Finance/Delivery/Rider).
Market growth reflects sustained enterprise adoption acceleration with AI as primary driver, accelerating consolidation. June 2026 ISG research predicts one-third of enterprises will integrate streaming with AI/generative AI inferencing by 2028 for real-time agentic applications; three-quarters will adopt standard information architectures including streaming data by 2028. Market forecasts project $87.27B by 2032 (17.21% CAGR from 2025 baseline of $28.71B), driven by IoT adoption, real-time AI integration, cloud-native deployment, and regional expansion. Strategic consolidation accelerated: IBM acquired Confluent at $11B (closed March 2026) for platform dominance, Salesforce acquired Informatica for $8B, signaling executive investment in event-driven architecture. Customer deployments demonstrate quantified ROI when freshness drives decisions: Starbucks processes 1B+ monthly rows across 17 countries with 764% ROI; Arla saves 1,200+ manual hours annually; financial sector achieves 50K TPS fraud detection at 800ms latency (vs 3-hour batch) with 8% false positives (down from 25%), deployed in 8-12 weeks; Uber handles 10M events/day with sub-minute latency enabling real-time operator visibility (replacing 1-hour batch). However, critical ROI analysis (Logiciel, 2026) documents decision-value dependency: many organizations pay streaming infrastructure premium for decisions that work equally well on batch schedule—a hidden adoption barrier. Practitioner cost analyses document that infrastructure represents <30% of streaming system cost; the remaining 70% derives from engineering and configuration complexity — confirming that organizational maturity, not technology, is the binding constraint.
Operational friction persists, however, and SLA misalignment constrains adoption more than technology barriers. Integration barriers emerge when combining best-of-breed tools: Flink's exactly-once guarantees require two-phase commit, but ClickHouse lacks full ACID support, making native connectors impossible and forcing latency/correctness trade-offs. IBM documentation from early 2026 details Kubernetes operator edge cases (JobManager cleanup deleting HA metadata, Java cipher suite restrictions blocking SSL). Critical scaling challenges at volume: 200k TPS fraud detection requires 3.2GB state/second with pod-crash recovery, revealing why state management expertise remains gatekeeping. Economics become punitive at scale: Kinesis for transitional 100TB/day workloads costs "high five figures per month." Practitioners report checkpoint overhead, schema evolution failures, and write amplification in lakehouse architectures. Regulated sectors face headwinds: healthcare and pharma find platform speed outpaces validation frameworks (GAMP 5), creating compliance gaps. Beyond technical friction, decision-value misalignment drives adoption friction: industry analysis (Kai Waehner, July 2026) documents that most enterprises operate in low-latency or near-real-time tiers (seconds to minutes), not hard-real-time (milliseconds), yet organizational narratives assume millisecond latency requirements; SLA alignment remains undefined in many deployments, making "latency benchmarks noise" without clear use-case definitions. By June 2026, cost pressures and operational complexity have shifted vendor guidance: Databricks and MotherDuck explicitly recommend micro-batching and warehouse-native ingestion when sub-second latency is not a genuine business requirement, acknowledging continuous processing overhead (70% engineering, 30% infrastructure) is only justified at latency extremes. Simultaneously, Goldsky's replacement of Flink with Rust-based Streamling achieving 30x compute reduction and $1M+/year cost savings across 3,000+ pipelines signals negative pressure from operational overhead, though represents architectural optimization rather than practice rejection. This pragmatic boundary-setting indicates maturation from "whether to stream" to "when streaming is worth its operational cost."
— Apache Fluss graduates to TLP: production use at 6 named organizations (Alibaba, Xiaohongshu, JD.com, Ant Group, Fresha, iQiyi) handling hundreds of billions of events—signals streaming-native storage adoption and ecosystem maturity.
— Critical assessment of streaming adoption: 'actionability gap' (real-time data valueless if humans act on batch schedule), cost complexity (24/7 overhead), simpler micro-batch often sufficient—important negative signal for tier classification preventing premature promotion.
— 2+ years production experience: state explosion outage, partition scaling data loss, checkpoint duplication—measured latency (Flink 45ms, Kafka Streams 120ms, Spark 850ms) and throughput, validating production-readiness with honest operational failures documented.
— 3 years Flink production experience: case studies with metrics (e-commerce 30% inventory lift, retail 18% sales uplift, fintech $50M fraud blocked), anti-patterns (stream-batch confusion, millisecond obsession, dashboard misuse)—validates real-world ROI and common pitfalls.
— Independent engineering firm documents five critical RisingWave production failure patterns (MV lag, offset loss, OOM aggregations, backfill starvation, sink blocking) with diagnostic queries—signals operational maturity and known failure modes in production deployments.
— Uber production ad platform (UberEats) processes real-time ad impression/click events with exactly-once semantics across Flink, Kafka, Pinot with record deduplication and two-phase commit—demonstrates revenue-critical streaming at scale with zero-tolerance correctness.
— Datamagnet synthesis of ROI evidence: 44% enterprises report 5x+ ROI from streaming (Confluent 2025), lead response 21x faster within 5 minutes (MIT/InsideSales), B2B contact decay 2.1% monthly—quantifies streaming adoption ROI decision framework.
— Kafka-to-Redis streaming feature pipeline achieves sub-10ms p99 latency with 100K+ QPS throughput; demonstrates quantified business outcome: 12% conversion lift from real-time feature freshness vs. batch materialization.