Ingestion and analytics as separate paths in a lakehouse

When a company has accumulated event feeds, scheduled jobs, object-storage buckets, and one or more reporting stores, its team often looks for a new tool that will make the whole arrangement run faster. A lakehouse may well be part of the answer. However, I start somewhere other than the table format, with a question about what must keep happening when a downstream lookup service, transformation job, catalog, or query cluster is unavailable.

On a recent engagement with a healthcare marketing analytics company, the client's pipeline received data from web, email, and paid-media systems and enriched each event before storing it. For every event, the pipeline called a live enrichment API, which in turn made a series of sequential MySQL stored-procedure calls and a PostgreSQL lookup to resolve campaign and provider data. At the volume involved, we found that the number of database round trips the pipeline made before it could accept an event limited its throughput. The S3 write and the available compute were not the constraint, however.

That design also tied ingestion to a service whose purpose was analytics preparation. Whenever the enrichment service, its connection pool, or either of its databases paused, the pipeline held back otherwise valid raw data until it recovered. In effect, one path accepted each event, looked up its meaning, and prepared it for later reporting, all at once.

In the design we laid out, we separated those responsibilities. On the write path, the ingestion process accepted the raw event and wrote it to object storage under a consistent layout organized by date and source, without waiting for a lookup or a query engine. We also kept the original event, so that we could reapply a later change in enrichment logic to the full history of records.

We handled the lookup data differently. A job produced versioned snapshots of the relatively small campaign and provider lookup tables, so the pipeline no longer made a live request for every event. Each run read a pointer to the current snapshot, so every Spark executor worked from the same input. Spark could then broadcast those tables in memory and join them to incoming records locally. In this way we turned enrichment from a sequence of synchronous network calls, one for every record, into a data transformation with a bounded lookup refresh.

I draw a distinction here between the write path and the analytics path. The write path accepts, retains, and organizes incoming data. It has to remain available while the query cluster is idle or shut down, and while someone maintains the catalog. The analytics path produces cleaned data, enforces the contracts its consumers expect, builds tables, and serves those tables to BI, data-science, and ad-hoc SQL workloads. Both paths can share one object store without sharing the same availability dependencies.

In this case, the raw records formed a Bronze layer, and the broadcast-enriched records formed a Silver layer. A separate job converted the stable Silver data to Apache Iceberg tables, compacting files and registering the resulting tables in a catalog. We ordered the jobs this way so that raw ingestion and enrichment could continue while the catalog was down for maintenance. During such a window, new data became queryable in the analytics layer later than usual; the pipeline lost nothing at the edge of the system.

Iceberg suited the analytics side because it stores table metadata, schema evolution, partitions, and snapshots apart from the object-storage files themselves. A team can therefore use Spark for a transformation job and Trino for interactive SQL without copying the data into a separate proprietary warehouse merely to change query engines. An Iceberg REST catalog such as Apache Polaris serves that shared table metadata to every engine. Polaris also runs outside any one cloud provider, which matters where portability is a requirement.

Not every object-storage collection needs all of these components. For a small, well-defined feed, a direct database load or a modest batch process may serve better. A Bronze/Silver/Gold vocabulary is also a poor substitute for understanding the records and their consumers. So I ask two concrete questions: which data must we write even when the rest of the platform is unavailable, and which work can wait until a read-focused system can do it?

We chose partitions by the same reasoning. If most reports filter by a date range, then date belongs in the layout and in the Iceberg partition strategy. If analysts routinely query a small number of source systems independently, source may belong there as well. We want the query engine to skip files before it begins reading them, without creating so many tiny partitions that maintenance becomes the next throughput problem. So we made that decision from the access pattern of this workload.

A data lake becomes a difficult place to work when raw files, enrichment rules, reporting tables, and ad-hoc access all accumulate without a clear contract between them. When a team keeps the ingestion path available and replayable, and builds a separately operated analytics path over the retained data, it can improve the analytics side, or change its engines entirely, without putting ingestion at risk.

Of course, I can tell you of a time that we opted to separate a feed from the lake entirely, but that is a topic for another discussion.