↓ Skip to main content
Lakehouse: A New Generation of Open Platforms (Armbrust, Ghodsi, Xin, Zaharia, CIDR 2021)
  1. Archive/

Lakehouse: A New Generation of Open Platforms (Armbrust, Ghodsi, Xin, Zaharia, CIDR 2021)

Author
AI Assistant
Curated for Pradithya Aria Pura · AI-written summary of the source material

Weekly Paper Notes — the Seminal Reading of the Week from the 2026-11-22 Data & AI Infra digest. Area: Databases.

Authors: Michael Armbrust, Ali Ghodsi, Reynold Xin, Matei Zaharia (Databricks; Ghodsi also UC Berkeley, Zaharia also Stanford) Published: 11th Annual Conference on Innovative Data Systems Research (CIDR ‘21), January 11–15, 2021, online. PDF

Why the paper still matters
#

This is an eight-page position paper with one chart of results, written by people selling the thing it describes. It’s still worth reading closely, because it states a design clearly enough that the industry could argue about it, copy it, and eventually standardise on it.

The opening claim is blunt: the data warehouse “as we know it today will wane in the coming years,” replaced by a Lakehouse built on open, directly accessible formats such as Parquet and ORC, with first-class support for ML and data science, and state-of-the-art SQL performance.

None of the pieces were new in 2021. Delta Lake had been in development since 2016, Iceberg had started at Netflix, and Hive ACID already tracked table files in an OLTP database. The paper adds the argument that these pieces are enough to drop the warehouse tier, plus a list of techniques for warehouse-class performance without owning the file format.

Last week’s pick was Selinger’s System R optimizer; the per-file min/max statistics this paper keeps in its log exist so an optimizer like that can skip files.

The setup: two tiers and four problems
#

The paper tells the history in three generations (its Figure 1). First-generation platforms ETLed operational data into a schema-on-write warehouse, coupling compute and storage in an on-premises appliance provisioned for peak load that couldn’t hold video, audio or text. The second generation, starting with Hadoop, put raw data into a cheap schema-on-read lake and ETLed a subset into a warehouse such as Teradata. From 2015 cloud object stores (S3, ADLS, GCS) replaced HDFS, with durability the paper puts at “often >10 nines,” but the warehouse stayed downstream, now Redshift or Snowflake. The authors say this two-tier setup is used at “virtually all Fortune 500 enterprises” in their experience.

The paper’s complaint is that data now gets ETLed twice, once into the lake and again into the warehouse, and it lists four resulting problems.

Reliability. Every ETL step between two systems is a chance for a bug, often from small differences in types or SQL dialects between the lake engine and the warehouse.

Data staleness. Warehouse data lags the lake, with new data “frequently taking days to load.” The paper cites a Dimensional Research and Fivetran survey in which 86% of analysts use out-of-date data and 62% wait on engineering resources numerous times per month. The authors call this a step backwards from the first generation, where new operational data was queryable immediately.

Limited support for advanced analytics. TensorFlow, PyTorch and XGBoost read large datasets with non-SQL code, which is slow over ODBC/JDBC. Exporting to files is “a third ETL step!” Running ML on the lake works, but loses ACID transactions, versioning and indexing.

Total cost of ownership. Users pay for continuous ETL, pay twice for storage, and get locked into a proprietary format.

The authors dismiss putting everything in a cloud warehouse with separated storage and compute, because it still handles unstructured data poorly and gives ML no fast direct access. The warehouses’ external tables over Parquet and ORC don’t make lake data easier to manage and often perform poorly. Engines that query the lake directly (Spark SQL, Presto, Hive, Athena) get the access right but lack transactions and indexes.

The mechanism: three ideas stacked on an object store
#

Lakehouse architecture as described in the paper: SQL and DataFrame APIs, a performance layer of caching, auxiliary data and layout, a transactional metadata layer, and open-format files in an object store.
Figure: The Lakehouse stack as the paper sketches it in Section 3 and its Figure 2. The two green layers are what a Lakehouse adds to a plain data lake. Original diagram for this post.

The paper defines a Lakehouse as “a data management system based on low-cost and directly-accessible storage that also provides traditional analytical DBMS management and performance features such as ACID transactions, data versioning, auditing, indexing, caching, and query optimization.” It is upfront that this means giving up some data independence, “a cornerstone of relational DBMS design.” Once clients read Parquet files directly, the storage format is part of the public API and the system can’t silently change it.

A transactional metadata layer
#

The first idea carries most of the weight. Data stays in an object store as ordinary Parquet files, and a metadata layer records which objects are part of each table version. Transactions, versioning and time travel become operations on that metadata, and readers can still open the files directly.

Hive ACID kept the file list in an OLTP DBMS. Delta Lake stores it in the lake itself as a Parquet-format transaction log, which the paper says scales “to billions of objects per table.” Iceberg uses a similar design over Parquet and ORC. Hudi, from Uber, targets streaming ingest but, per the paper, lacked concurrent writers.

Two adoption details stand out. Delta Lake can convert an existing directory of Parquet files into a table with zero copies, by writing a first log entry that references every existing file. And by the paper’s account, Delta Lake grew to cover half of the compute-hours on Databricks in three years.

Data quality and governance live here too. Delta Lake enforces schemas and has a constraints API (a country column limited to a list of values, say), and its client libraries reject or quarantine violating records. For access control, “a metadata layer can check whether a client is allowed to access a table before granting it credentials to read the raw data in the table from a cloud object store, and can reliably log all accesses.”

SQL performance without owning the format
#

A metadata layer gives you management features but not speed. A Lakehouse can’t co-design the format with the engine the way a warehouse does, so the paper proposes three optimizations that leave the data files untouched.

Caching. Because the transaction log says exactly which files are current, it’s safe to cache object-store files on local SSD and RAM; a transaction can tell whether a cached file is still valid. The cache doesn’t have to hold raw Parquet either. Databricks’ cache partially decompresses the data it loads, so hot data sits in a format closer to what a closed warehouse engine would use.

Auxiliary data. The base files must stay standard, but the system fully controls any extra files it keeps next to them. Delta Lake stores column min/max statistics for every data file in the same Parquet file as the transaction log, which allows data skipping when the data is clustered on the filtered column. A Bloom-filter index was in progress at the time of writing.

Data layout. Even with the format fixed, the system decides which records sit together. Delta Lake supports sorting on single columns or on space-filling curves (Z-order and Hilbert) to get locality across several dimensions at once.

The three work together. Hot data, where most queries land, is served from the transcoded cache. For cold data, performance mostly depends on bytes read, and layout plus zone-map-style statistics cut that the way a proprietary warehouse would.

Declarative DataFrames for ML
#

The simplest ML integration is to ask the metadata layer which Parquet files are current and hand over the list. The paper goes further. Spark’s DataFrame API is lazily evaluated: operations such as filtering users to kind == "buyer", projecting three columns and calling fillna(0) build a query plan, and only model.fit(train_set) triggers execution (the paper’s Figure 4). Spark pushes the selections and projections into the Delta Lake data source, which can then use the cache, data skipping and layout to read less. Since DataFrames are how MLlib, GraphFrames, SparkR and Koalas get their input, all of them inherit faster I/O.

The authors admit the limit: APIs like TensorFlow’s tf.data push no query semantics down, and care mostly about overlapping loading with CPU-to-GPU transfers, which warehouses never had to handle.

The evidence: one TPC-DS chart
#

Two bar charts from the paper: TPC-DS power test duration in seconds and cost in dollars at scale factor 30,000, for four anonymised cloud warehouses DW1 to DW4 and Delta Engine on on-demand and spot instances.
Source: Armbrust, Ghodsi, Xin and Zaharia, CIDR 2021, Figure 3. Time to run all 99 TPC-DS queries at SF 30,000 (top) and the customer cost of that run (bottom).

The paper’s only quantitative result compares Delta Engine (Databricks’ new C++ execution engine for Spark, plus the three optimizations above) with four anonymised cloud warehouses on TPC-DS at scale factor 30,000, on comparable 960-vCPU clusters with local SSDs across AWS, Azure and Google Cloud.

Read off the chart, the power-test durations are 2,996 s for DW1, 7,143 s for DW2, 5,793 s for DW3 and 37,283 s for DW4, against 3,302 s for Delta Engine on on-demand instances and 3,252 s on spot. The costs are $153, $286, $206 and $570 for the four warehouses, against $104 on-demand and $56 on spot for Delta Engine.

So DW1 was slightly faster, and Delta Engine was cheapest. The body text’s summary, “comparable or better performance than these systems at a lower price point,” matches the chart. The introduction’s phrasing, that Delta Engine “outperforms leading cloud data warehouses on TPC-DS,” is looser than the data supports.

A footnote matters more than it looks. All systems started with data cached on SSDs where possible, because some warehouses only supported node-attached storage, and Delta Engine was 18% slower from a cold cache. That cold number is the one that tests the paper’s real claim.

This is a vendor benchmark on one workload, with anonymised competitors and no tuning details. It shows the architecture isn’t disqualified on performance, which is all a position paper needs, and not much more.

The open questions the paper lists
#

Each of Sections 3.2 to 3.4 ends with “Future Directions and Alternative Designs,” and Section 4 adds more.

On metadata, Delta Lake keeps its log in the same object store it manages, which keeps things simple and gives the log the store’s availability and bandwidth, but caps transactions per second because of object-store latency. The authors suggest a faster store for metadata may be better in some cases. They also note that Delta Lake, Iceberg and Hudi only support single-table transactions, and that cross-table transactions should be possible. Log format and object sizing are listed as open too.

On performance, they propose new storage formats with more room for layout and indexing, while admitting engines adopt new formats slowly. Which caching, auxiliary-data and layout strategies work best at object-store scale is left open, as is serverless execution.

On ML, the open problems are keeping accelerators fed, “factorized ML” that pushes model logic into SQL joins, and standard data-management interfaces for data scientists. The paper suggests feature stores could sit on Lakehouse transactions and versioning instead of reimplementing them.

Section 4 asks whether a massively parallel serving layer in front of a warehouse could do the same job, and argues it would cost more and run slower than direct object-store access, citing the thin deployment of Hive LLAP. It makes a governance case for open formats (regulators may require searching or deleting old data on short notice) and says the right split between storage format, client libraries and SQL interface “remains to be seen.”

This week: the questions moved up a layer
#

This week’s news shows where those open questions went. On November 16 an Apache Iceberg vote opened to add materialized views to the view spec. On November 18 BigQuery made fine-grained row- and column-level security for Iceberg tables generally available, with its REST catalog vending storage credentials only to principals that have unfiltered access. And Apache Polaris 1.9.0 fixed a bug where commits with an unknown outcome could delete newly written metadata.

None of these asks whether a lake can be a warehouse. They’re about the catalog: what objects it describes, who gets credentials when several engines share the files, and whether commits stay correct when a request times out. The BigQuery rule descends from the paper’s sentence about checking access before granting object-store credentials, plus a wrinkle the paper skipped: a principal with filtered access can’t be handed raw file credentials at all. The Polaris fix is a reminder that a “transactional metadata layer” is a distributed commit protocol with the usual failure modes.

Why this design has outlasted everything around it
#

Delta Engine was later written up as Photon at SIGMOD 2022, and the products in Figure 3 have all changed. What has lasted is the layering in Figure 2.

Putting the table abstraction in a small metadata layer, with bytes left in an open format on cheap storage, separated three things warehouses had bundled: where data lives, which engine reads it, and who decides what a table is. Each can now change on its own schedule, and engines can be swapped without migrating data. The contested piece became the catalog, which the paper treated as part of the metadata layer and which is now a service in its own right, as this week’s BigQuery and Polaris items show.

The paper was also right about which costs were accidental. Double ETL, double storage and lock-in came from the two-tier split, not from analytics, while the hard work of writing correct pipelines remains, as the paper concedes.

Two of its own open questions turned out to matter most. Single-table transactions are a real limit once tables are shared across teams. And a log in the object store bounds commit throughput, which is one reason catalog services now sit in front of table metadata, close to the “faster storage system for the metadata” the paper said might be preferable.

Read alongside
#

Links #

📄 CIDR 2021 paper (PDF)


Summary written from a close read of the CIDR 2021 paper; figures cropped from the PDF are reproduced under fair use for educational commentary.

Related