Chris Riccomini

Key Insight: MCP is a solution in search of a problem—existing standards like OpenAPI and CLIs already enable LLMs to interact with software systems, making a new protocol unnecessary overhead.

Chris Riccomini argues that the Model Context Protocol (MCP) is an unnecessary new standard when existing solutions like OpenAPI, gRPC, and CLIs already serve the same purpose of teaching LLMs to interact with software systems. He finds MCP's categorization of resources, prompts, and tools arbitrary and confusing, noting that the same flight search example appears in their documentation as both a tool and a resource. While acknowledging arguments about context window limitations and poor documentation, he's skeptical these justify a new protocol, especially as context windows expand rapidly. He concludes that developers should focus on building better tools using existing standards rather than adopting MCP.

Key Insight: The DataFrame ecosystem's fragmentation is fundamentally an abstraction problem—compute lacks the structure that data has—and content-addressed 'compute catalogs' can make expressions portable, cacheable, and reusable across engines and teams.

Chris Riccomini interviews Hussain Sultan, co-founder of Xorq, about the fragmented DataFrame ecosystem and how Xorq addresses it through a multi-engine approach. Sultan explains that Xorq functions as a 'compute catalog' that makes expressions portable, observable, and reusable across different engines like DuckDB, DataFusion, and Pandas. The system uses content-addressed hashing inspired by Nix to enable deterministic caching and artifact reuse across queries, workflows, and users. While currently batch-focused, Xorq's architecture supports out-of-core processing and has plans for streaming engine support. The interview covers how Xorq integrates with ML workloads through Arrow Flight and UDF/UDAF machinery, positioning it as a unified system for data engineering and machine learning pipelines.

Key Insight: The streaming ecosystem has been remarkably stagnant over the past decade, while the most exciting infrastructure innovation is happening at the storage layer with projects like SlateDB enabling new possibilities like zero-copy filesystem forking.

Chris Riccomini provides an update on his various projects after a summer hiatus from newsletter writing. He has been working on the second edition of Designing Data-Intensive Applications, specifically rewriting the batch chapter and updating the streaming chapter. SlateDB development continues with new features like serializable snapshot isolation and transactions, plus Python and Go bindings. The most exciting SlateDB news is ZeroFS, a POSIX-compliant network filesystem built on top of SlateDB. His venture fund Materialized View Capital is 75% deployed and will be fully deployed in 18 months.

Key Insight: Kafka's protocol becoming a de facto standard has created adoption success but also innovation stagnation, and the streaming ecosystem needs unconstrained newcomers willing to rethink fundamentals to spark the next wave of progress.

Chris Riccomini argues that while batch processing has seen dramatic innovation over the past decade, the streaming ecosystem around Kafka has largely stagnated. Despite Kafka and Flink being over 10 years old, they remain the dominant solutions, with most recent progress focused on adoption rather than innovation. The entrenchment of Kafka's protocol as a de facto standard has created network effects that discourage fundamental rethinking. Riccomini sees S2 as a promising challenger because it's the only modern streaming solution not constrained by Kafka's protocol, and suggests technologies like Feldera and differential dataflow libraries could spark the next wave of innovation in streaming, similar to what cloud warehouses and Spark did for batch processing.

Key Insight: Modern IVM engines built on timely dataflow, differential dataflow, and DBSP can automatically convert any batch SQL query into an efficient incremental computation, but the technology's complexity remains a barrier to widespread adoption.

Incremental view maintenance (IVM) is gaining significant momentum with new implementations from Materialize, Feldera, Epsio, and others, as well as PostgreSQL's experimental pg_ivm extension. Traditional approaches to view updates—either recomputing on every read or periodically refreshing entire datasets—are wasteful and slow. IVM solves this by processing only the deltas, potentially making updates millions of times faster. The post traces the evolution from bag algebra approaches to modern frameworks: timely dataflow provides flexible time modeling, differential dataflow adds version tracking for selective computation reuse, and DBSP offers a simpler circuit-based model that can convert any batch SQL to incremental queries. While these technologies underpin many modern IVM systems, adoption challenges remain due to complexity.

Key Insight: AI IDEs can escape commoditization by partnering with software vendors to provide superior LLM experiences for new and proprietary tools, breaking the virtuous cycle that currently favors legacy software.

Chris Riccomini argues that AI IDEs like Windsurf and Cursor face a commoditization threat due to near-zero switching costs from their VS Code foundation. However, he believes differentiation is possible through specialized partnerships and proprietary training data. He identifies a 'virtuous cycle' problem where LLMs perform poorly on new or proprietary software, inadvertently reinforcing adoption of legacy tools. The solution lies in IDE companies partnering with software vendors to provide better LLM support for emerging technologies, and extending this approach to enterprise internal codebases.

Key Insight: The proliferation of specialized data catalogs—registries, service catalogs, information schemas, and discovery tools—is an artificial fragmentation of what should be a unified metadata layer, and market forces are already driving convergence around standards like Iceberg's REST API.

Riccomini argues that the data ecosystem has fractured into too many specialized catalogs—schema registries, service catalogs, information_schema, and traditional data catalogs—that all fundamentally manage metadata about data. Drawing from his experience building Recap, he observes these systems are 80% the same despite serving different use cases. He points to emerging convergence as evidence: Backstage expanding beyond microservices, Datahub adopting Iceberg's REST API, and Confluent's Tableflow speaking Iceberg. While hopeful that Iceberg's REST API could become the lingua franca, he pragmatically predicts we'll end up with catalog gateways instead of true unification.

Key Insight: Incremental view maintenance is evolving from niche streaming use cases to become a fundamental approach for delivering fast, cost-effective analytics by building query-specific data structures rather than optimizing for generic query patterns.

This interview with Epsio CEO Gilad Kleinman explores incremental view maintenance (IVM) and streaming SQL technologies. Kleinman explains how streaming SQL differs from traditional batch processing by incrementally updating query results rather than reprocessing entire datasets. Epsio's approach is distinctive in that it connects to existing PostgreSQL/MySQL databases via CDC, processes changes externally, and writes materialized results back to the source database. The conversation covers how IVM fits into the broader landscape of streaming databases versus stream processors, the technical approaches behind differential dataflow and DBSP, and why IVM is expanding beyond traditional streaming use cases into customer-facing analytics and real-time reporting.

Key Insight: Legacy infrastructure vendors are structurally trapped by their own success—their organizations, tech stacks, and enterprise customers prevent them from building the simpler, cheaper cloud-native solutions that lean AI-enabled startups can ship in months with a fraction of the headcount.

Riccomini argues that legacy infrastructure vendors like Confluent are structurally unable to compete with lean, cloud-native startups like WarpStream. Using Confluent's acquisition of WarpStream as a case study, he explains how organizational baggage, the innovator's dilemma, and Conway's law prevent incumbents from building simpler, cheaper solutions that would cannibalize their existing business. The post contends that cloud-native architectures combined with AI-enabled engineering productivity have created a new class of 'apex predator' startups that can build competitive products with a fraction of the headcount. Riccomini warns that acquiring these nimble competitors may only delay the inevitable commoditization of infrastructure.

Key Insight: S3-based data lakehouses are emerging as the modern, standardized way for SaaS companies to share data with customers, potentially driving broader lakehouse adoption while finally addressing a long-neglected but universal customer need.

Chris Riccomini argues that S3-based data lakes are becoming the modern equivalent of SFTP for sharing data with external customers. Drawing from his experience at WePay, he describes how customer data sharing requests are ubiquitous but historically under-invested in by SaaS companies. He highlights emerging startups like Prequel that focus on exporting customer data to S3 in Parquet format, enabling customers to query their data using modern lakehouse tools. Riccomini suggests this pattern could drive broader lakehouse adoption and give analytics engineers revenue-generating work.

Key Insight: The convergence of mature object storage technology and growing demand from Kubernetes and AI workloads is finally making distributed POSIX-compatible filesystems achievable, even if complete compatibility remains elusive.

The post explores the long-standing challenge of creating a distributed filesystem with POSIX semantics, which would simplify building distributed systems by allowing them to treat remote storage like local files. While object storage like S3 provides the necessary scale and durability, it lacks POSIX compatibility, forcing developers to either integrate S3 directly into their systems or use imperfect FUSE/NFS wrappers. The growth of Kubernetes and AI workloads has intensified demand for such a solution. Riccomini highlights Regatta Storage as a promising contender that could bridge this gap with better POSIX support and built-in caching. He expects increased competition in this space, with enough market opportunity for multiple winners across different use cases.

Key Insight: The economic and technological conditions that justified splitting data warehouse teams into data engineers and analytics engineers have reversed, making reunification both inevitable and necessary.

Riccomini argues that analytics engineers and data engineers should be merged back into a single role, reversing the split that occurred over the past decade. He contends that the analytics engineer role, focused primarily on transformations, doesn't provide enough value to justify dedicated headcount. Several factors drove the original bifurcation: the shift to ELT architectures, Kafka adoption, shift-left philosophy, and vendors creating roles to sell products. Now that ZIRP has ended, data pipelines are commoditized, and LLMs can answer many data questions, the conditions that justified separate roles have changed. Tools like dltHub and tabsdata are emerging to facilitate this reunification.

Key Insight: DuckDB is excellent middleware for columnar analytics, but its deployment model and scalability limitations mean it cannot replace enterprise data warehouses—the most valuable queries require scale that only centralized cloud solutions can provide.

Riccomini argues that despite DuckDB's popularity among data engineers for its portability, speed, and ability to query remote storage, it fundamentally cannot serve as an enterprise data warehouse. The deployment model of installing DuckDB on individual laptops and granting bucket access is untenable for organizations where everyone from PMs to finance needs data access. While most queries are small, the most valuable enterprise queries—financial reconciliation, recommendations, advertising—require scale DuckDB cannot provide. MotherDuck is building the centralized deployment and scaling layer DuckDB needs, but this essentially makes them another cloud data warehouse competing with giants like Snowflake and Databricks. DuckDB excels as middleware, similar to SQLite, but enterprises should view it as a tool rather than a warehouse replacement.

Key Insight: Recent advances in GPU checkpointing, unikernels, and VM migration are finally addressing the specific requirements of modern workloads, setting the stage for more flexible multi-cloud deployments and broader serverless adoption.

Chris Riccomini argues that virtual machines, containers, and serverless technologies are undergoing significant improvements that will reshape cloud computing. He highlights three key areas of advancement: GPU checkpointing (notably NVIDIA's cuda-checkpoint), unikernels enabling faster cold starts, and VM migration tools like Loophole Labs' Architect. These developments address longstanding pain points around slow cold starts, incomplete state snapshots, and poor GPU support. Riccomini predicts these improvements will drive multi-cloud adoption and potentially allow serverless functions to supplant traditional service-oriented architectures.

Key Insight: The democratization of data pipelines through lightweight, Python-native tools like dlt reflects a broader shift where data consumers increasingly want direct control over their data flows, even if it means accepting some organizational messiness.

Riccomini reflects on the evolution of data pipelines, noting that simpler batch ETL approaches are making a comeback for smaller organizations. He references his 2019 QCon talk outlining stages of pipeline maturity, emphasizing that many companies don't need to progress beyond basic batch processing. The post highlights how pipeline ownership has shifted from dedicated data engineering teams to ML, AI, and product engineers who prefer Python-centric tools. He introduces dlt (data load tool) as a promising solution that fits modern workflows while incorporating SDLC best practices. While acknowledging challenges like duplication and unclear ownership, Riccomini sees this democratization of data pipelines as a positive trend driven by data consumers wanting more control.

Key Insight: Modern infrastructure must be built from the ground up to support the full spectrum of deployment options—from embedded libraries to distributed clusters to multi-tenant SaaS—or risk obsolescence.

Riccomini analyzes Confluent's WarpStream acquisition through the lens of a broader industry shift toward flexible deployment options. He argues that modern infrastructure must support multiple deployment models—embedded, single-node, clustered, SaaS, BYOC, and self-hosted—rather than forcing customers into a single approach. Using ClickHouse as the exemplar, he shows how one system can scale from an in-process library to distributed clusters while supporting all deployment modes. The post identifies technical enablers like zero-disk architecture, Rust/C++ adoption, composable data systems, and WebAssembly that make this flexibility achievable. His conclusion is stark: infrastructure vendors without flexible deployment options will fail.

Key Insight: Formal verification will only achieve mainstream adoption when tools prioritize developer ergonomics over mathematical rigor, meeting engineers where they already are.

This interview with JP Kadarkarai, creator of FizzBee, explores the challenges of formal verification in distributed systems and why existing tools like TLA+ have failed to gain mainstream adoption. JP describes how his experience at Google with large-scale systems like GFS, Bigtable, and Spanner revealed the importance of formal verification, while his later work at Lyft and Clumio showed how the industry often adopts distributed architectures without theoretical foundations. FizzBee aims to make formal methods practical by using Python-like syntax instead of TLA+'s mathematical notation, automatically generating visualizations, and enabling model-based testing. The conversation also touches on how AI-assisted coding will increase the need for formal verification tools.

Key Insight: Monoliths don't fail because modularity is impossible—they fail because we've invested in microservice tooling instead of building the tools that would make modular monoliths work.

Riccomini argues that modular monoliths offer a viable alternative to the reflexive jump to microservices that many growing companies make. Drawing on painful personal experiences at LinkedIn and WePay, he contends that microservices often just trade one set of problems for another—tight coupling, complex tooling, and operational overhead. The key insight is that monoliths become spaghetti not because modularity is impossible, but because we lack the tooling to enforce it. He outlines what modular monolith tooling would need: incremental builds, code isolation enforcement, and database boundary management. Rather than an either-or choice, modular monoliths can serve as a stepping stone that extends a monolith's useful life or provides a cleaner path to services.

Key Insight: PostgreSQL's extension architecture enables the database to innovate rapidly in areas like search and analytics while preserving the stability and reliability that made it dominant.

This interview features Philippe Noël, CEO of ParadeDB, discussing how they're building Elasticsearch alternatives as PostgreSQL extensions. ParadeDB offers pg_search for BM25 full-text search and pg_analytics/pg_lakehouse for columnar analytics, allowing users to avoid ETL pipelines to separate search infrastructure. Noël explains that PostgreSQL's extensibility allows it to stay current with data innovations while maintaining core stability. He discusses the recently announced pg_duckdb project, expressing cautious optimism but concern that MotherDuck may be too DuckDB-centric. The company is currently focused on commercial licenses and self-hosted deployments, with plans for BYOC solutions and Postgres platform partnerships.

Key Insight: ClickHouse's buzz comes from exceptional developer experience and operational simplicity, but its long-term viability is threatened by PostgreSQL's OLAP extensions from below and enterprise-scale systems like Pinot from above.

Chris Riccomini investigates the growing enthusiasm around ClickHouse, a realtime OLAP database that has gained significant attention in the data infrastructure community. He identifies three key factors driving the buzz: exceptional speed, remarkably easy installation via a single cURL command, and operational simplicity that allows scaling up on a single machine before needing distributed coordination. However, Riccomini highlights significant challenges including cost concerns (evidenced by Uber's migration away from ClickHouse to Apache Pinot), limited materialized view functionality for complex joins, and the looming threat from PostgreSQL's OLAP extensions like pg_duckdb. He concludes that while ClickHouse is currently a robust solution, its long-term position may be squeezed between PostgreSQL for smaller deployments and Pinot/Druid for enterprise scale.

Key Insight: The rise of object storage has created a gap for purpose-built storage engines that embrace its tradeoffs rather than treating it as a drop-in disk replacement.

Riccomini announces the open-source release of SlateDB, an embedded storage engine built as an LSM tree directly on object storage like S3. The project emerged from his earlier writing about cloud storage tradeoffs and attracted collaborators from companies like Responsive and Jam Socket. SlateDB targets use cases in stateful stream processing, serverless functions, and durable execution where bottomless storage capacity and durability matter more than raw latency. The project is explicitly non-commercial with an Apache 2.0 license and a CNCF sandbox application pending.

Key Insight: Realtime OLAP is not a single problem but a spectrum of use cases with varying requirements for latency, freshness, and query flexibility—and the systems built for each niche are now converging.

Riccomini chronicles his experience building fraud detection systems at WePay, revealing how this work deepened his understanding of realtime OLAP requirements. The fraud detection use case demanded sub-10-second data freshness and single-digit millisecond query responses—far stricter than human-facing dashboards. After evaluating Druid and finding it insufficient for their latency needs, WePay built a custom pipeline using Kafka, BigQuery, Dataflow, and Bigtable. This experience led Riccomini to distinguish between 'human interaction' OLAP (flexible queries, hundreds of milliseconds acceptable) and 'machine interaction' OLAP (pre-defined queries, millisecond response times required). He observes that realtime OLAP appears across domains—fraud, observability, logging, metrics—each with overlapping but distinct requirements, and that these specialized systems are now converging.

Key Insight: The missing piece in the data lakehouse ecosystem is a CDN-like caching layer that brings data closer to applications, and the sidecar pattern perfected by service meshes provides the ideal deployment model for this use case.

In this interview, Luke Kim of Spice AI explains how his startup addresses the gap in serving data lakehouse files in production. Spice AI functions as a CDN for databases, providing a caching and materialization layer that sits between applications and data warehouses/lakes. The architecture prioritizes single-node optimization with horizontal scalability, drawing from Kim's experience building Dapr at Microsoft. Spice can be deployed as sidecars alongside applications, stackable across infrastructure tiers from edge to cloud, minimizing data transit while maintaining query performance. The system supports both proactive materialization and reactive caching, with CDC support for consistency across distributed replicas.

Key Insight: Realtime OLAP systems evolved from rigid pre-materialized views to flexible columnar stores with in-memory buffering, enabling millisecond-latency user-facing analytics without sacrificing query flexibility or data freshness.

Chris Riccomini traces the evolution of realtime OLAP systems through his personal experience at LinkedIn, starting with the Who's Viewed My Profile feature 15 years ago. The initial solution, Avatara, used pre-materialized views stored in Voldemort but suffered from rigid query patterns, high write amplification, and data staleness. This led to the development of Apache Druid by Eric Tschetter at MetaMarkets and Apache Pinot by Kishore Gopalakrishna at LinkedIn, both solving these problems through columnar storage combined with in-memory buffering for recent data. The post explains how these systems enable millisecond-latency queries on fresh data without the limitations of pre-computation, setting the stage for a future discussion of ClickHouse.

Key Insight: The future of B2B software lies in decoupling deployment from management through BYOC, allowing vendors to provide SaaS-like experiences while customers retain control of their data and infrastructure costs.

This interview explores the emerging Bring Your Own Cloud (BYOC) deployment model as an alternative to traditional multi-tenant SaaS. Jon Morehouse argues that B2B software vendors have unnecessarily adopted B2C multi-tenant architectures that many enterprise customers don't actually want, as customers increasingly value data ownership over convenience. BYOC decouples software management from deployment, allowing vendors to manage software that runs in customer environments. The conversation covers practical challenges (installation automation, day-2 tooling, updates) that Nuon aims to solve, and distinguishes between 'vertical clouds' like Snowflake/Confluent and application vendors who should embrace BYOC. Morehouse envisions a future where B2B SaaS vendors only offer BYOC, fundamentally changing SaaS margins to purely R&D and support costs.

Key Insight: The lakehouse ecosystem should let query engines drive catalog API evolution while vendors compete on enterprise features, turning infrastructure standards into a rising tide that lifts all boats.

Chris Riccomini argues that the data lakehouse ecosystem should standardize on Apache Iceberg's REST API for core catalog functionality, with query engines like Trino and DataFusion driving its evolution rather than individual vendors. He distinguishes between 'core' functionality (metadata for query execution) and 'non-core' features (governance, lineage, discovery), suggesting vendors should compete on the latter. The post notes that major players like Databricks Unity, Snowflake Polaris, and Nessie are already adopting Iceberg's REST API. This standardization would allow query engines to deprecate fragmented integrations like Hive Metastore and AWS Glue catalog, creating a cleaner ecosystem.

Key Insight: The gap between vendor marketing and actual open source deliverables in the data lakehouse catalog space suggests these companies are rushing to stake claims without clear strategies or substantial code to back up their announcements.

Riccomini examines the recent open source catalog announcements from Databricks (Unity) and Snowflake (Polaris), finding both fall far short of their marketing promises. Snowflake's Polaris Github repository is essentially empty despite extensive marketing materials. Databricks released an API-compatible rewrite of Unity rather than their actual product, missing critical features like audit capabilities and ML/AI governance. A third-party Rust rewrite (unity-rs) has also emerged, but it's barely started. Riccomini concludes that while these projects will likely mature, the vendors appear to be flailing without a clear strategy.

Key Insight: Catalogs, not table formats, are the strategic control point in the data lakehouse stack because they're where query engines integrate, making them the natural battleground for vendor consolidation.

Data lake catalogs have become a battleground as Snowflake launched Polaris, Databricks open-sourced Unity catalog, and Databricks acquired Tabular. Unlike traditional data catalogs focused on discovery and governance, these new catalogs serve as an information_schema for query engines like Flink, Spark, and Trino. Table formats alone are insufficient—catalogs provide the crucial layer that tracks all databases, tables, and partitions. The strategic importance lies in catalogs being the integration point for query engines, making them a natural stepping stone for vendors to move up or down the stack to protect their positions.

Key Insight: The future of data infrastructure lies in composable systems built from reusable components like DataFusion, while established platforms like Kafka Connect need modernization to remain relevant.

Chris Riccomini explores three interconnected topics in data infrastructure. First, he celebrates ParadeDB's pg_lakehouse extension, which turns PostgreSQL into a data lake query engine using Apache DataFusion and OpenDAL. Second, he examines approaches for serving lakehouse data back to production systems, favoring simple caching services like ROAPI over heavier databases like StarRocks or SingleStore. Finally, he discusses the aging Kafka Connect platform and the need for improvements around connector isolation and scheduling, noting that competitors like Bento and Redpanda Connect are presenting healthy alternatives.

Key Insight: Antithesis should open source their deterministic hypervisor because the real value lies in their surrounding tooling, and community development would accelerate the integration improvements needed for broader adoption.

Chris Riccomini argues that deterministic simulation testing (DST) is gaining momentum, with Antithesis's hypervisor-based approach revolutionizing the field by making any containerized code deterministically testable without modification. However, Antithesis faces adoption challenges due to high costs (~$200k) and integration complexity. Riccomini predicts open source alternatives will inevitably emerge, citing projects like Meta's Hermit and QEMU-based efforts. He advocates that Antithesis should open source their hypervisor since most of their value lies in surrounding tooling, and community development could accelerate integration improvements. The post concludes optimistically with Antithesis confirming they plan to eventually open source their hypervisor.

Key Insight: Google Cloud's dismissive incident response and superficial remediation reveal a company that still treats enterprise customers like consumer software users, failing to understand that trust—not just technical capability—is what enterprises are buying.

Riccomini critiques Google Cloud's handling of a major outage that deleted UniSuper's private cloud for over a week due to a misconfigured parameter. He argues the incident reveals three systemic problems: Google's fundamental misunderstanding of enterprise customers, their superficial incident remediation that only addressed the proximate cause, and growing brand damage as Google Cloud gains a reputation for instability. Despite being a self-professed Google Cloud fan, he finds their incident response tone-deaf—notably lacking any apology while defensively claiming to be 'the most resilient cloud infrastructure in the world.'

Key Insight: S3's missing features like compare-and-swap operations are forcing developers to either abandon AWS's object store for competitors or architect around its limitations with additional metadata layers.

Chris Riccomini argues that S3, despite being an engineering marvel, is falling behind competitors due to missing critical features like compare-and-swap operations, multi-region buckets, and object appends. These gaps are becoming problematic as new infrastructure increasingly uses object storage as their primary persistence layer. S3 Express One Zone has proven disappointing with its directory bucket type, missing standard S3 features, and high storage costs. Developers face a choice: abandon S3 for competitors like GCS that have these features, or build workarounds using additional services like DynamoDB. The post concludes that understanding S3 as an object store rather than a file system, and embracing a separate metadata layer, may be the path forward.

Key Insight: While Parquet's limitations for AI/ML workloads have spawned promising new formats in Nimble and Lance V2, their embrace of extensibility trades flexibility for fragmentation risk—though LV2's implicit reliance on Arrow as the de facto standard offers a pragmatic path forward.

Chris Riccomini examines two new columnar file formats—Nimble from Meta and Lance V2 from LanceDB—that aim to address Apache Parquet's limitations for AI/ML workloads. Parquet struggles with point lookups, large values, wide schemas, and limited encodings that make it poorly suited for machine learning data with long float arrays and multimedia content. Nimble takes an incremental approach with extensible encodings and FlatBuffers for metadata, while LV2 radically eliminates row groups entirely in favor of a minimal ~200 line spec with pluggable type systems. Both formats prioritize extensibility, though this risks fragmentation across implementations. The author concludes that while neither format is production-ready for general OLAP queries, LV2's integration with Arrow and thoughtful design make it particularly compelling for the future.

Key Insight: Swift's evolution toward Rust-like ownership semantics in version 5.9 suggests that even languages with simpler memory models eventually need more explicit control for performance-critical infrastructure work.

The author reflects on Swift's memory management approach compared to Rust's borrow checker after a friend suggested Rust should have adopted Swift's ARC model. Investigation reveals that Apple published an 'Ownership Manifesto' seven years ago proposing Rust-like ownership for Swift, and this work has now materialized in Swift 5.9 with noncopyable types, consume operators, and borrowing/consuming parameter modifiers. The author expresses ambivalence about these changes, appreciating Swift's simpler ARC model while acknowledging that deep infrastructure work requires the performance optimizations that ownership semantics provide.

Key Insight: As object storage gains low latency and atomic operations, databases can persist directly to it by batching writes—but clients must explicitly choose between latency, cost, and durability.

Chris Riccomini argues that object storage (S3, GCS) is becoming the future of database persistence, but current systems require write caches due to latency and atomicity limitations. He introduces the 'LCD theorem'—a tradeoff between Latency, Cost, and Durability when persisting directly to object storage. With improvements like S3 Express and compare-and-swap operations becoming standard, he believes write caches will become unnecessary. He's building SlateDB, a cloud-native LSM tree database that persists directly to object storage, using write batching to manage costs while letting clients choose their tradeoff between waiting for durability or accepting async writes.

Key Insight: PostgreSQL's wire protocol and SQL dialect—not its query engine—will ultimately be its most valuable and enduring contribution to the database ecosystem.

Riccomini examines two competing architectural approaches for PostgreSQL-compatible databases: building extensions on top of PostgreSQL's query engine versus implementing protocol compatibility while building a new engine from scratch. He argues against the view that PostgreSQL is a 'near-perfect kernel,' noting that it constrains what you can build. With maturing components like DataFusion, Velox, and RocksDB, building custom databases is more feasible than ever. He predicts PostgreSQL's protocol, not its query engine, will be its most defining characteristic, while acknowledging both approaches will coexist in the market.

Key Insight: Despite philosophical preferences for alternatives like Zig, ecosystem maturity and tooling quality ultimately matter more than language design when choosing systems programming languages.

Chris Riccomini returns from a break to discuss his renewed attempt at learning Rust, reflecting on the database commoditization conversation sparked by his previous post and Jack Vanlightly's thoughtful follow-up. He describes his 2022 failure with Rust during Advent of Code and how GPT-4 and Gemini have made his current learning experience dramatically better. While he appreciates Rust's mature ecosystem and excellent documentation, he criticizes its thin standard library that requires external packages for basic functionality. He contrasts Rust with Zig, noting that despite preferring Zig's philosophy, its immature ecosystem and questionable package management decisions make it harder to adopt.

Key Insight: When infrastructure becomes successful enough, the protocol transcends any single implementation—and the open source project becomes just one option among many.

Apache Kafka has evolved from an open source project into a de facto protocol, with multiple vendors building their own implementations that diverge significantly from the original Apache codebase. Confluent Cloud's features like tiered storage and Iceberg integration are years ahead of the Apache project, and other implementations like WarpStream, Redpanda, AutoMQ, and S2 are taking radically different approaches while maintaining Kafka protocol compatibility. This pattern mirrors what happened with Redis and PostgreSQL protocols, and even S3. The author argues this is a natural and healthy evolution for successful infrastructure, though it raises questions about companies' responsibilities to their open source roots.

Key Insight: Local-first architecture with conflict-free replicated data types can eliminate the state transfer complexity that dominates modern app development, but requires accepting trade-offs in schema evolution and embracing tighter coupling between components.

This interview explores ElectricSQL, an open-source platform for building local-first software that aims to eliminate the complexity of state transfer between frontend and backend. CEO James Arthur explains how ElectricSQL provides conflict-free and rollback-free programming through Transactional Causal Consistency with CRDTs, allowing developers to write to local databases without coding conflict or sync handlers. The discussion covers schema evolution challenges, where ElectricSQL only supports additive migrations to maintain finality of local writes. Arthur also discusses their choice of Elixir for the sync service, leveraging the BEAM runtime for fault tolerance and concurrency, and introduces PGLite as a future option for unified client-server data models.

Key Insight: Infrastructure tooling is converging—simulation testing is becoming accessible to legacy systems, profiling and tracing are merging, and low-latency object stores may soon absorb the complexity of distributed data systems.

This newsletter edition covers four major developments in data infrastructure. Antithesis has launched a deterministic simulation testing platform that works with any software by using a custom hypervisor, eliminating the need to rewrite code for determinism. Polar Signals has added distributed call trace support to Parca, unifying profiling and tracing in one system. Google Cloud Storage is responding to AWS S3 Express One Zone by planning 100x latency improvements. The post concludes with a highlight of CozoDB, an embedded database combining graph, vector, and relational capabilities with Datalog queries.

Key Insight: Database commoditization isn't the end of innovation—it's the beginning of competition on platforms, verticals, and multi-model architectures that deliver value beyond raw query performance.

Chris Riccomini argues that databases are becoming commoditized as they converge on shared components: PostgreSQL-compatible frontends, object stores like S3 for storage, and formats like Parquet. Rather than viewing this as the death of innovation, he identifies three strategies for differentiation: building comprehensive platforms with features like branching and time travel, targeting specific business verticals like finance or SaaS, and developing multi-model systems that collapse multiple access patterns into fewer systems. He's optimistic that these strategies can combine to create compelling products, and that recent advances in decomposable data systems make previously challenging architectures newly feasible.

Key Insight: Arrow's ecosystem (DataFusion, Ballista, Comet) is positioned to become the dominant composable database infrastructure, making alternatives like Velox largely redundant except for specific migration use cases.

This newsletter covers three main developments in data infrastructure: Apache Arrow's addition of new formats (StringView, ListView, REE) to improve interoperability with Meta's Velox execution engine, the emerging edge database ecosystem with players like Ditto, SKDB, and ElectricSQL, and a UUID performance benchmark comparing storage strategies in PostgreSQL. Riccomini expresses enthusiasm for Arrow and its ecosystem while questioning the necessity of Velox given DataFusion's broader functionality. He highlights ElectricSQL's achievement of running PostgreSQL in the browser as particularly noteworthy for simplifying client-server schema synchronization. The piece concludes with a personal anecdote about UUID storage mistakes and excitement about UUID v7's inclusion in PostgreSQL 17.

Key Insight: Monolithic stream processing platforms that try to serve every use case sacrifice simplicity and user experience, while purpose-built systems for specific use cases offer easier adoption and the flexibility to swap components as better alternatives emerge.

Riccomini reflects on his experience creating Apache Samza at LinkedIn, explaining how the project's architectural decisions—particularly its excessive pluggability and dependency on dynamic deployment systems like YARN—hampered adoption despite technical merits. The challenges with Samza led to internal proposals that ultimately influenced the creation of Kafka Streams and Kafka Connect as separate, purpose-built tools. He argues that stream processing has three distinct use cases (production features, analytical queries, and ETL) with contradictory requirements, and that trying to serve all of them with one monolithic platform creates unnecessary complexity. This historical perspective informs his skepticism of Apache Flink, which he sees as repeating Samza's mistakes by layering on features to become an everything-platform. He advocates for smaller, purpose-built systems that can be swapped out independently when better alternatives emerge.

Key Insight: The translation from programmatic query builders to SQL text and back to query plans is an unnecessary indirection that Substrait could eliminate, making databases truly machine-friendly.

Riccomini proposes that databases should accept Substrait query plans directly instead of text-based SQL, arguing this would eliminate an unnecessary translation layer. Currently, ORMs and query builders generate SQL text that databases must parse back into an intermediate representation—a roundabout process. By allowing applications to send structured query plans directly, databases could support multiple query languages, eliminate parsing overhead, and make programmatic access a first-class citizen. While this approach is gaining traction in data warehousing (DuckDB, Spark Connect), the OLTP space has surprisingly little adoption, presenting an opportunity for someone to build PostgreSQL support.

Key Insight: SQLite's flexible type system and the convergence of durable execution with workflow orchestration both reflect the same engineering truth: different users need different interfaces to the same underlying capabilities.

This newsletter covers three distinct topics: SQLite's nuanced type system with its storage classes, datatypes, and column affinities; the evolution of durable execution and workflow orchestration interfaces from GUIs to code-based SDKs; and HikariCP's impressive JDBC connection pool performance. Riccomini explains how SQLite's type system uses text-matching rules to determine column affinities, leading to sometimes surprising but generally practical behavior. He observes that durable execution frameworks are converging with workflow orchestration tools, with each adopting multiple interface paradigms to serve different users. The piece concludes with appreciation for HikariCP's meticulous performance optimization documentation.

Key Insight: Database disassembly into reusable components will commoditize both performance and features, ultimately threatening Kafka's relevance as OLTP and OLAP systems converge on shared object storage.

The database industry is undergoing a fundamental shift toward disassembly, where monolithic databases are being broken into reusable components like query parsers, optimizers, and storage engines. This trend traces back to Hadoop's separation of compute, data, and control planes eighteen years ago, and continues today with projects like Apache Arrow's DataFusion providing complete database toolkits. PostgreSQL's robust extension API and storage layer have enabled similar disassembly in the RDBMS space, with projects like Neon making even the write-ahead log pluggable. The author predicts this will lead to commoditized data warehouses, unified OLTP/OLAP systems, and poses a serious threat to Kafka's data integration use case as systems converge on object stores.

Key Insight: Data contracts solve CDC's encapsulation problems in theory, but implementing them requires integrated tooling and cultural change around cross-team data ownership—not just stream processors and outbox tables.

Chris Riccomini responds to Gunnar Morling's post about change data capture (CDC) and data contracts, arguing that while data contracts are indeed the right solution to CDC's encapsulation problems, the proposed implementations (outbox pattern and stream processing) don't actually deliver what's needed. He contends that developers need integrated tools that handle schema versioning, compatibility detection, and SLA management—not a patchwork of separate tools. Riccomini emphasizes that the hardest part of data contracts isn't technical but cultural, involving complex cross-team coordination around data ownership and priorities. He concludes by advocating for purpose-built products like Gable that address these holistic challenges rather than just transformation mechanics.

Key Insight: SKDB represents a new approach to reactive applications by combining materialized views, database diffing, and streaming capabilities into a unified database platform.

This is a brief podcast episode announcement introducing an interview with Julien Verlaguet, founder of SkipLabs. The post previews a conversation about SKDB, a reactive database featuring performant materialized views, database diffing, and streaming via ephemeral tables. Chris mentions that SkipLabs has launched and directs readers to their websites for more information. The post is primarily promotional rather than substantive analysis.

Key Insight: Independent convergence on similar data model designs between Zed and Recap suggests these unified schema approaches are solving real problems in the right way.

This newsletter edition covers three main topics: Apple's L4S protocol for reducing network latency in realtime applications, terminal-based benchmarking tools surfaced by the 1brc challenge, and Zed's unified data model compared to Recap. L4S works by having clients signal congestion notification support and servers report queued packets, allowing clients to throttle their send rate. The benchmarking section highlights hyperfine, poop, and flameshow as useful performance analysis tools. Riccomini expresses validation that Zed's data model independently arrived at similar design choices as his Recap project, while noting Zed is more ambitious with its full ecosystem including serialization formats and query languages.

Key Insight: Modern continuous profiling can be achieved with minimal overhead by combining eBPF sampling with columnar storage engines built on Datafusion and Parquet.

Chris Riccomini interviews Frederic Branczyk, founder of Polar Signals, about their always-on, zero-instrumentation profiler. The discussion covers Polar Signals's architecture, particularly Parca and FrostDB. Parca achieves minimal invasiveness by using an eBPF filter that samples the entire OS stack at 19hz. The collected data is stored in FrostDB, an embedded storage engine built on Datafusion and Parquet. The interview also references several influential papers on profiling and optimization from Google.

Key Insight: A viral coding challenge reveals that modern Java has powerful optimization features most developers don't use, and that with NVMe storage, text parsing—not I/O—is now the performance bottleneck.

Gunnar Morling launched the '1 billion row challenge' (1brc), tasking developers with using Java to parse 1 billion rows of temperature data and calculate statistics for each city. The challenge has gone viral, with developers experimenting with modern Java features like SIMD, virtual threads, and memory mapping to optimize performance. The current leader achieved 14 seconds, with hardware variations causing significant performance differences. Riccomini notes that the challenge is essentially reimplementing parts of Apache Arrow, and that NVMe speeds mean the bottleneck is text parsing rather than I/O.

Key Insight: Database isolation levels are ambiguously defined and inconsistently implemented, causing subtle bugs that developers struggle to diagnose because they never suspect the database itself.

The post examines how ANSI SQL '92 database isolation levels are fundamentally broken, synthesizing recent research from Jepsen, Gwen Shapira, and others. The author argues that the 1992 standard left room for unspecified anomalies, and different databases implement isolation levels inconsistently. While some argue developers haven't noticed these issues, the author counters that these bugs are simply hard to detect and reproduce. Drawing from personal experience in payments processing, he emphasizes that database anomalies cause real harm through hours of debugging and unhappy customers. The takeaway is that developers should understand their database's actual behavior rather than trusting isolation level names.

Key Insight: The growing convergence of data systems on S3-based object storage is opening new architectural possibilities for HTAP, streaming, and embedded databases that may reshape the database landscape.

This year-end post serves as a curated reading list, sharing the browser tabs Chris accumulated but couldn't get to throughout 2023. Rather than offering predictions or retrospectives, it catalogs papers and articles across major data infrastructure themes that captured his attention. The post highlights emerging trends including durable execution frameworks, S3-based persistence for streaming systems, HTAP databases, embedded databases at the edge, and PostgreSQL's continued dominance. Chris reflects on how the convergence toward object storage might reshape database architectures, particularly for hybrid workloads. The post also marks the two-month anniversary of Materialized View, with a nod to his most popular piece on durable execution.

Key Insight: Prometheus's single-node architecture is a deliberate design choice that prioritizes simplicity and user choice over built-in distributed storage, enabling an ecosystem of remote backends rather than locking users into one scaling path.

Riccomini explores why there's no true cloud-hosted Prometheus, discovering that its single-node storage design is intentional rather than a limitation. This architecture prioritizes simplicity for getting started while offering flexibility through remote storage integrations when scaling becomes necessary. He then pivots to searching for an embedded time-series database purpose-built for metrics workloads, ultimately discovering FrostDB from PolarSignals. FrostDB uses an LSM tree structure with Apache Arrow and Parquet, designed specifically for high-write, immutable observability data with dynamic schemas.

Key Insight: S3 Express represents AWS's acknowledgment of S3-as-primary-storage use cases, but its pricing means fundamental storage architectures requiring caches and WALs won't change anytime soon.

This newsletter roundup covers three main infrastructure topics: AWS's S3 Express One Zone, zero-downtime VM live migrations, and Google's HighwayHash. Riccomini aggregates community opinions on S3 Express, expressing skepticism about its pricing despite acknowledging it as a useful building block. He highlights Felicitas Pojtinger's standing-ovation Kubecon talk on stateful VM migrations using network block device protocol, which enables microVMs like Firecracker to support live migrations. The post also spotlights HighwayHash, a SIMD-optimized hashing library that achieves 5.2x speedup over SipHash.

Key Insight: The convergence of object storage like S3 with streaming and database systems is creating new architectural patterns that separate storage from compute in increasingly elegant ways.

This newsletter edition covers three major developments in data infrastructure: the growing adoption of S3 as a storage layer for streaming systems, an architectural analysis of Neon DB's innovative PostgreSQL design, and an introduction to SKDB as a novel reactive edge database. The post examines different architectural approaches to S3-based streaming including WarpStream's fully stateless model, tiered storage implementations like AutoMQ and RedPanda, and streaming data lakes like Paimon. Riccomini notes that Kafka finally added tiered storage in version 3.6.0 after Confluent kept it as a proprietary feature. The piece highlights Neon's elegant design that replaces PostgreSQL's storage layer with remote storage interfaces and praises SKDB's ground-up approach to building a concurrent, reactive database with materialized views and subscriptions.

Key Insight: Precision timing, SIMD parallelism, and cross-database transaction protocols represent foundational infrastructure capabilities that enable increasingly sophisticated distributed systems and AI workloads.

This newsletter roundup covers four infrastructure topics: AWS Time Sync Service's new microsecond-level accuracy via hardware clocks on the Nitro platform, an introduction to SIMD (Single Instruction, Multiple Data) and its applications from vector search to LLMs, and the Epoxy paper which presents a protocol for distributed transactions across heterogeneous data stores without requiring XA protocol support. The author connects AWS's precision timing improvements to Google Spanner's TrueTime concept, while noting SIMD's growing relevance for AI workloads. Epoxy offers both isolation and atomicity for cross-database transactions through a shim-based architecture.

Key Insight: The data infrastructure ecosystem is evolving toward S3-based transactional storage and declarative configuration tools, while production streaming workloads are far more diverse than the industry's content focus on SQL and CDC would suggest.

This roundup post covers four data infrastructure topics: the emergence of transactional key-value stores built on S3 for serverless OLTP databases, the surprisingly diverse production use cases for Apache Flink beyond analytics, a balanced rebuttal defending mmap for database storage, and Titan as a new Terraform-like tool for data warehouse configuration. Riccomini argues the industry needs more open source transactional KV stores on S3 to serve as building blocks for serverless databases. He notes his expectation that Flink would be mostly SQL and CDC workloads was wrong—it's used for real-time ads, bot detection, and safety systems. The post also highlights the tension between pragmatic and academic approaches to database design through the mmap debate.

Key Insight: The durable execution bubble can only be justified if second-generation frameworks converge with stream processing to create more user-friendly APIs that expand the market beyond payment processing.

The durable execution space has become overcrowded with startups following Temporal's success, but the market cannot sustain them all without significant expansion. The author argues that frameworks remain too difficult to use, limiting adoption beyond canonical use cases like payment processing. Stream processing represents the biggest opportunity for convergence, as both paradigms face identical challenges around non-determinism, idempotency, and schema evolution. Second-generation frameworks that unify stream processing and durable execution with friendlier APIs could justify the current bubble by expanding the total addressable market.

Key Insight: Deterministic simulation testing—achieved by eliminating non-determinism and using seeded pseudo-random execution—is the gold standard for testing distributed systems, as demonstrated by Dropbox, TigerBeetle, and FoundationDB.

This newsletter edition surveys the resurgence of Viewstamped Replication, a consistency algorithm predating Paxos and Raft, highlighting a new Go implementation alongside existing Zig and Rust versions. Riccomini connects this to the broader theme of deterministic simulation testing, examining approaches from Dropbox, TigerBeetle, and FoundationDB. Dropbox's sync engine rewrite demonstrates how single-threaded control flow enables serialized, reproducible tests. TigerBeetle advocates reducing variability in both IO and time to improve debugging, testing, and runtime performance. FoundationDB's Simulation framework introduces techniques like 'swizzle-clogging' to find deep edge cases. The core takeaway is that stamping out non-determinism and using seeded pseudo-random testing enables reproducible failures.

Key Insight: Modern infrastructure increasingly benefits from separating concerns—whether it's compute from state in stream processing, keys from values in storage engines, or protocol layers in networking—to optimize for new hardware and deployment realities.

This newsletter issue covers four main topics: HTTP/3 and its underlying QUIC protocol, Responsive's new autoscaler for Kafka Streams, the WiscKey paper on SSD-optimized LSM storage, and LittleHorse as a durable execution framework. The author explains how QUIC reimagines TCP with built-in TLS, multiplexing, connection migration, and extensibility. He reflects on his own experience building Apache Samza and the problems with static resource allocation that Responsive's autoscaler addresses. The WiscKey discussion highlights how separating keys from values in LSM trees reduces write amplification for SSDs, an architecture now widely adopted.

Key Insight: SQLite is increasingly being adopted in distributed systems architectures, while purpose-built databases like Nile are pushing multi-tenancy concerns from application code down to the database layer.

This newsletter roundup covers several developments in database infrastructure, with a focus on SQLite's growing role in distributed systems. Bluesky is moving their Personal Data Server from PostgreSQL to a distributed SQLite implementation, signaling SQLite's increasing adoption. Nile launched a serverless Postgres-compatible database purpose-built for SaaS multi-tenancy. The author also explores buffer pool implementations after diving into research on mmap in databases, and highlights Tigris, an open-source serverless NoSQL database built on FoundationDB.

Key Insight: There's clear demand for curated, thoughtful coverage of software infrastructure that goes beyond scattered social media posts.

Chris Riccomini announces the launch of Materialized View, a newsletter focused on software infrastructure. The newsletter originated from a Twitter thread sharing interesting infrastructure projects that gained unexpected traction. Rather than continuing to post scattered content across multiple platforms, Riccomini decided to consolidate his thoughts into a regular newsletter. The publication will cover infrastructure news, developer interviews, stack deep dives, and project highlights on a biweekly schedule.