The AI landscape doesn't move in one direction — it lurches. Some techniques leap from experiment to table stakes in a single quarter; others stall against regulatory walls, technical ceilings, or organisational inertia that no amount of hype can dislodge. Knowing which is which is the hard part. The State of Play cuts through the noise with a rigorously maintained index of AI techniques across every major business domain — classified by maturity, evidenced by real-world adoption, and updated daily so you always know where you stand relative to the field. Stop guessing. Start knowing.
A daily newsletter distilling the past two weeks of movement in a domain or two — delivered to your inbox while the index updates in the background.
Each dot marks the weighted maturity of practices within a domain — hover for a brief summary, click for more detail
AI applied to streaming data for real-time pattern detection, alerting, and decision-making on live data flows. Includes stream processing with ML models and real-time anomaly detection; distinct from batch analytics which processes historical rather than live data.
Real-time streaming analytics has matured into established good practice. The discipline — applying ML models, statistical aggregations, and pattern detection to continuous data flows rather than batch windows — now rests on a stable ecosystem of GA tooling, managed cloud services, and battle-tested deployment patterns. Apache Flink and Kafka have become de facto standards; all three major cloud providers offer managed streaming services (AWS Managed Service for Apache Flink, Microsoft Fabric Real-Time Intelligence, Google Cloud Dataflow); and the industry recognizes data streaming as a formal software category. Production deployments span banking (fraud detection, real-time risk assessment), payments (sub-10ms decision latency), fintech (Toss processing 7-day frequency capping state at 68GB scale), and operational analytics (billions of events daily). Market evidence signals mainstream adoption: analysts project $146.59B market by 2030 (33% CAGR); enterprises report 764% ROI on implementations (Starbucks), 200+ hour annual savings (Arla); and deployments now address cost-optimization as much as capability. The practice has transitioned past the "whether" question to operational "how"—but that transition remains incomplete. The binding constraint is not technical but organisational: the multi-disciplinary expertise in distributed systems, state management, and streaming semantics that production deployments demand. Large enterprises with dedicated data engineering teams absorb the operational complexity that managed services have reduced but not eliminated. Mid-market adoption lags, constrained not by capability but by the specialized expertise and operational maturity required to productionise these systems reliably. By mid-2026, a market-wide recognition has emerged that continuous streaming carries prohibitive operational overhead for many use cases once justified only by latency requirements—vendors are explicitly guiding customers toward micro-batching and warehouse-native architectures when sub-minute freshness suffices, signaling maturation toward pragmatic adoption boundaries rather than universal real-time.
Vendor consolidation has solidified around Apache Flink as the stateful processing engine and Apache Kafka as the transport layer. AWS has shifted entirely to Managed Service for Apache Flink (replacing Kinesis Data Analytics), Microsoft Fabric earned Forrester Wave leader recognition (Q4 2025), and Flink 2.2.0 introduced ML_PREDICT and VECTOR_SEARCH — embedding LLM inference and vector similarity directly into streaming pipelines. Apache Kafka 4.2.0 GA introduced Share Groups (enabling per-record acknowledgement) and Streams Rebalance Protocol (delivering faster, more stable rebalances), advancing ecosystem maturity. Commercial distributions advanced with Ververica Platform achieving Forrester Leader status with claims of 100B+ events/day, <10ms latency, and 40% TCO reduction versus open-source Flink. Databricks Structured Streaming now offers Real-Time mode, achieving sub-5ms end-to-end latency for operational workloads alongside the traditional micro-batch model. Financial institutions (Rabobank, ING Bank, Capital One, Nationwide Building Society) now run real-time fraud detection and risk management on event-driven Flink pipelines; Riskified processes $60B in annual transaction volume with sub-10ms fraud detection decisions. Beyond finance, PayTech enterprises (Toss) deploy complex state pipelines (7-day frequency capping with 68GB state), PostNL migrates IoT asset tracking to managed Flink, Intuit operates 200+ Kubernetes clusters handling 5B daily messages, and tech giants at ByteDance maintain one of the world's largest documented deployments—70,000+ Flink jobs, 11 million+ resource slots, processing hundreds of trillions records daily. Uber demonstrates Kappa architecture patterns using Kafka and Spark Streaming for unified batch-stream processing enabling multi-team latency/correctness trade-offs for dynamic pricing. The ecosystem around Flink continues advancing: Kubernetes Operator 1.14.0 (March 2026) adds blue-green deployment capabilities for zero-downtime updates; research (ICDE 2026) targets latency reduction through prefetching and cache optimization; and practitioner tooling has matured with comprehensive production guides documenting exactly-once semantics, RocksDB state backends, and sub-millisecond latency architecture patterns.
Market growth reflects sustained enterprise adoption acceleration. May 2026 forecasts project 33% CAGR with $146.59B market by 2030, driven by IoT adoption, real-time AI integration, and edge computing. Customer deployments demonstrate quantified ROI: Starbucks processes 1B+ monthly rows across 17 countries with 764% ROI; Arla saves 1,200+ manual hours annually harmonizing European operations. Practitioner cost analyses document that infrastructure <30% of streaming system cost; the remaining 70% derives from engineering and configuration complexity — confirming that organizational maturity, not technology, is the binding constraint.
Operational friction persists at the seams, however. Integration barriers emerge when combining best-of-breed tools: Flink's exactly-once guarantees require two-phase commit, but ClickHouse lacks full ACID transaction support, making a native connector impossible and forcing latency/correctness trade-offs that organizations must engineer around. IBM documentation from early 2026 details Kubernetes operator failures — JobManager cleanup deleting HA metadata, Java cipher suite restrictions breaking SSL handshakes — that illustrate the configuration complexity lurking beneath managed-service abstractions. Critical scaling challenges emerge at volume: 200k TPS fraud detection systems require 3.2GB state per second with pod-crash recovery and consistency maintenance, revealing why state management expertise remains a gatekeeping competency. Economics become punitive at scale: Kinesis for transitional 100TB/day workloads costs "high five figures per month," pushing organizations to evaluate alternative architectures when data lifetimes are short. Practitioners report checkpoint overhead, schema evolution failures, and write amplification in lakehouse architectures. Regulated sectors face additional headwinds: healthcare and pharmaceutical organisations find that platform speed outpaces validation frameworks like GAMP 5, creating compliance gaps. These barriers keep mid-market adoption tethered to technology-forward organisations with dedicated streaming expertise, even as large enterprises consolidate streaming as standard infrastructure. By June 2026, cost pressures and operational maturity barriers have shifted market sentiment: vendors (Databricks, MotherDuck) are explicitly recommending micro-batching and warehouse-native ingestion for use cases previously defaulting to streaming, acknowledging that continuous processing overhead—often 70% engineering complexity, 30% infrastructure—is only justified when sub-second latency genuinely drives business outcomes. This pragmatic boundary-setting indicates the practice has matured from "whether to stream" to "when streaming is worth its operational cost."
— Uber deployed streaming ingestion (Apache Flink) at petabyte scale, replacing batch with 25% compute reduction and hours-to-minutes freshness improvement across Finance, Delivery, Rider organizations.
— Production supply chain streaming (Kafka+Flink): inventory tracking, SLA protection, disruption response with documented ROI. Shows real-world value and organizational complexity barrier (requires platform engineering expertise).
— Databricks GA feature for streaming checkpoint recovery from failure, addressing production reliability challenge. Documents three recovery approaches (full refresh, preserve with backfill, incremental).
— Databricks production best practices for streaming workloads (Lakeflow Jobs, failure restart, autoscaling guidance, RocksDB state, async checkpointing). Shows ecosystem maturity of lakehouse streaming patterns.
— Databricks guidance: streaming adds complexity (stateful operations, out-of-order handling). Recommends by medallion layer: streaming for Bronze ingestion, batch/incremental for Silver/Gold. Authority guidance on pragmatic streaming adoption.
— Uber case study: 120k events/sec, 5M hexagons, real-time ML features for surge pricing. Demonstrates both capability scale and significant operational burden (backpressure, OOM, optimization expertise required).
— CRITICAL NEGATIVE signal: 2026 market shift away from continuous streaming (Flink, Kafka) toward micro-batching and warehouse-native architectures for cost and complexity reasons. Documents adoption barrier.
— Reference architecture for Netflix-scale recommendation streaming (Kafka→Flink/Spark→Feast): 50ms P99 latency SLOs, 23% engagement uplift, 18% revenue lift documented at 8M customer scale.