Matei Zaharia, who started Apache Spark and co-founded Databricks, gave this 30-minute talk (plus Q&A) at Tecton’s apply() 2022 conference for ML engineers. It is the conference-stage version of the argument in the CIDR 2021 Lakehouse paper by Armbrust, Ghodsi, Xin and Zaharia, which is this week’s Seminal Reading and has its own notes at Lakehouse (CIDR 2021). The paper gives the design and the evaluation. The talk is shorter, aimed at people who build ML pipelines, and makes the case in terms of the problems they hit every day: bad data and stale data. Read the paper for the details and watch this for the pitch.
The real problem is the data, not the platform #
Zaharia opens with what Databricks hears from customers when it asks what they want from a data platform. The answers aren’t “faster” or “cheaper”. The top complaints are about the data itself: its quality first, then its timeliness. He cites a Fivetran survey of data analysts in which 60% named data quality as a top challenge, 86% had to use stale data (much of it more than two months old), and essentially all of them had unreliable data sources.
His thesis is that some of this is self-inflicted. Getting good, fresh data is hard anyway, but the shape of today’s data stacks makes it harder, and that shape exists “for historical reasons”, not technical ones.
How we got two tiers #

The history runs in two steps. In the 1980s, companies ETL’d data from operational databases (an airline booking system, say) into a data warehouse built for analytics on historical data. Warehouses were full SQL databases with schemas, indexes, ACID transactions and multi-version control, and that model still supports a large industry.
By the 2010s three problems showed up. Warehouses were SQL- and table-centric and didn’t handle semi-structured or unstructured data. Storing large datasets in them was expensive. And because SQL was the only interface, data science and ML were awkward.
Data lakes answered those: cheap storage like S3, HDFS or Azure Data Lake Storage where you upload anything in any format. Because lakes came out of open source, the data sat in open formats like Parquet, which any engine could read. Zaharia notes that Spark itself got started this way. Data was already in open formats on Hadoop, so he could write a better engine that worked on people’s existing data. The catch is that a lake has almost no management features (“even weaker semantics than a file system”), so companies kept the warehouse and loaded a subset of lake data into it. Today, he says, the lake is where most enterprise bytes end up.
Why two tiers hurt #

The two-tier setup lets you run everything, but it’s far more complex than the 1980s warehouse where an event landed and an analyst could query it right away. He lists three costs:
- Reliability suffers. Two or more storage systems (often with a Kafka bus as well) have different semantics, SQL dialects and data types. Every copy job between them is a chance to corrupt data or fail.
- Timeliness suffers. Extra ETL steps add delay, and a new dataset with no pipeline can’t be analyzed until someone builds one.
- Cost goes up. Data is duplicated, and jobs that only transcode and copy bytes burn cloud credits without doing anything useful.
The lakehouse idea #

The proposal is to build warehouse-style management and performance features directly on lake storage and open formats, and give people a single tier. He sees little technical reason you can’t get good warehouse performance backed by S3, and points out that many modern systems already run on cloud storage underneath.
The resulting picture has the lake at the bottom, a management and performance layer above it that offers transactions and versioning, and two interfaces on top: SQL for tools that speak SQL, and direct file access for high-throughput engines like ML frameworks or Spark. The open question is whether that layer can deliver the performance and governance that justify separate systems today. He takes it in three parts: metadata layers, engine design for SQL, and the ML interface.
Metadata layers and transactions #

Zaharia calls metadata layers the key technology. Several were built around 2016-2017: Delta Lake at Databricks, Apache Iceberg from Netflix, Apache Hudi from Uber, plus proprietary cloud services doing similar things. The mechanism is simple. The layer tracks which files make up each version of a table. A client asks which files belong to the table, gets a list, and reads those files directly.
His example is a delete of all events for one customer. On a plain lake, that means rewriting file1 into file1b, deleting file1, and doing the same for file3, with no way to make the change atomic. A reader in the middle sees a half-updated table, or both file1 and file1b and therefore duplicates. If the job crashes, the table stays corrupt.
Delta Lake fixes this with a log of changes stored in the object store itself, so there’s no extra service to run, and the table is as available as S3. Each log entry says which files make up a version, for example version 2 is files 1, 2 and 3. The update job writes new files that readers ignore because no committed version references them. When it’s done it atomically adds a log entry for version 3. Readers see either all of the change or none of it.
What the log buys you #

Once versions are explicit, several features come almost for free:
- Time travel: query the table as it was a week ago.
- Zero-copy clones: a dev table whose log starts by pointing at the parent’s files and records only its own changes, which he compares to forking in Git.
- Audit history from the log.
- Streaming: treat a table as a stream of changes and use it as a message bus. Latency is worse than Kafka because it sits on S3, so some workloads still need a separate bus, but many don’t.
On adoption, he says Delta Lake is used at thousands of companies to manage exabytes, and that on Databricks the share of write workloads going into Delta went from zero to more than 70% since its 2017 launch. He finds that unusual, since storage formats are normally slow to be adopted.
Getting warehouse performance on open formats #

Better management still isn’t enough if SQL is slow, and warehouses have decades of co-designed storage, compute and query planning behind them. Zaharia’s answer has two parts. Open columnar formats are already good and still evolving, and the metadata layer leaves room for optimizations on the side. He lists four: auxiliary data structures, data layout within files, caching, and execution techniques like vectorization. The first three cut the bytes scanned and the last makes scanning faster.
The worked example is min/max zone maps. Parquet keeps per-column statistics in each file’s footer, but with a million files that means a million I/Os just to decide which files matter. Delta Lake copies those statistics into the transaction log, where a million entries compress to a few megabytes. Reading the snapshot gives all the statistics in one I/O that takes a few hundred milliseconds on S3, and the query planner can then skip every file whose ranges can’t match.
For layout, records likely to be read together can be sorted close to each other, including with space-filling curves that cluster on two dimensions at once. Databricks caches hot data on SSDs in a half-decompressed format that’s faster than plain Parquet on SSD. Its Photon engine is Spark-compatible and uses vectorized CPU instructions. He says Parquet hasn’t been the limiting factor so far.

The slide shows a TPC-DS-derived 100 TB power run, Databricks against Snowflake, which Databricks posted the previous year. Zaharia’s conclusion is that there’s no technical evidence you’d get better warehousing performance by giving up open formats. This is a vendor benchmark presented by the vendor’s co-founder, so treat it as his claim.
Machine learning on the lakehouse #

For an ML audience, this is the section that matters. ML pipelines need data from many systems, and warehouses are a poor source because ML reads whole tables while warehouse SQL interfaces are built for queries that extract small subsets. Pulling a large table over JDBC/ODBC is slow and has to be transcoded into a format the ML engine understands. The workarounds are both bad. Exporting to the lake adds a third ETL step that can break before training starts, and keeping production datasets in both places is more complex still.
With a lakehouse there’s one tier. Many ML frameworks already read Parquet, so a thin shim that tells them which files belong to the current version lets them read those files in parallel straight from cloud storage, with no translation node in between. Spark’s declarative APIs can still optimize queries. Versioning and streaming help with the ML lifecycle too: MLflow records which table version an experiment used, so you can get the same data back later. He names Feast, Tecton and MLflow as tools that work with Delta Lake.
He closes the prepared part on his thesis:
today’s world with many separate data systems i think doesn’t have to be that way for a technical reason it’s that way for historical reasons because of how these evolved
Q&A highlights #
- On assessing data quality: run automated tests (null fractions, out-of-range values) and use versioning to roll back to a good version or hold a bad change out of production. Add periodic end-to-end checks for bugs that produce plausible-looking records. His biggest lever is “a simple architecture with fewer moving pieces.”
- On Snowpark versus Databricks: Snowpark is a proprietary API, while Databricks runs open-source engines and APIs, so Photon runs unmodified Spark jobs and ML jobs use TensorFlow, PyTorch, Horovod or XGBoost.
- On business logic: teams use bronze (raw), silver (streaming-updated derived tables) and gold (vetted) tiers, and an analyst missing a field in gold can query silver or bronze in the same system instead of getting access to another one.
- On on-prem: there’s no Databricks on-prem product, but Spark and Delta Lake run on HDFS.
- On what’s next: concepts that bridge data teams and ML teams, now that feature platforms are spreading beyond advanced tech companies, plus more real-time work.
- On data mesh: shared object storage in one region lets different teams own their data and still compute over all of it efficiently, which makes decentralized ownership easier.
Key takeaways #
- Zaharia’s starting claim is that users’ biggest problems are data quality and timeliness, and that multi-system architectures make both worse by adding copy jobs that can fail and add delay.
- The lake-plus-warehouse split comes from history: warehouses were too expensive and SQL-only for big, messy data, and lakes were cheap and open but had almost no management features.
- A metadata layer that tracks which files belong to each table version is enough to give a data lake ACID transactions, and Delta Lake keeps that log in the object store so no extra service is needed.
- The same versioned log gives you time travel, zero-copy clones, audit history and change streams that can replace a message bus when S3-level latency is acceptable.
- Warehouse-class SQL performance on Parquet comes from statistics kept in the log, data layout such as space-filling curves, SSD caching and a vectorized engine, and Zaharia says the open format itself hasn’t been the bottleneck.
- ML workloads gain the most from a single tier, because they need bulk reads of whole tables, which open files support directly and warehouse SQL interfaces handle badly.
- The TPC-DS comparison in the talk is Databricks’ own benchmark, so the performance claim is best checked against the evaluation in the CIDR paper and independent results.
Source #
- Talk: “Lakehouse: A New Class of Platforms for Data and AI Workloads”
- Speaker: Matei Zaharia (co-founder of Databricks, creator of Apache Spark)
- Event: apply() Conference 2022, hosted by Tecton
- Duration: 37:42
- Video: https://www.youtube.com/watch?v=6j0MazSTLHg