Sahil Walia, Senior Technical Architect at Snowflake, joins host Robert Blumen for a deep dive into Apache Iceberg. The conversation covers OLAP, data formats, data lakes and lake houses, Iceberg as a data format, columnar storage, use cases, reference implementation, read/write, how Iceberg structures metadata, indexing, query planning and performance issues.
Brought to you by IEEE Computer Society and IEEE Software magazine.
Show Notes
Related Episodes
- SE Radio 328: Bruce Momjian on the Postgres Query Planner
- SE Radio 364: Peter Zaitsev on Choosing the Right Open Source Database
- SE Radio 393: Jay Kreps on Enterprise Integration Architecture with a Kafka Event Log
Related Links
- Kafka in Action by Dylan Scott, Viktor Gamov, Dave Klein Kafka in Action
- Viktor Gamov blog, talks gamov.io , Viktor Gamov
- Vitkrot Gamov github gAmUssA – Overview
- Iceberg 101 course by Dremio Apache Iceberg 101 – Your Guide to Learning Apache Iceberg Concepts and Practices | Dremio
- Sahil Walia’s LinkedIn profile
- Sahil Walia – Medium
- Sahil Walia’s Google Scholar page
- Apache Iceberg – Apache Iceberg™
- apache iceberg: the definitive guide Apache Iceberg: The Definitive Guide
Transcript
Transcript brought to you by IEEE Software magazine.
This transcript was automatically generated. To suggest improvements in the text, please contact [email protected] and include the episode number.
Robert Blumen 00:00:35 For Software Engineering Radio, this is Robert Blumen. I have with me today Sahil Walia. Sahil is a Senior Technical Architect at Snowflake, and he has published many papers on Data Engineering and Artificial Intelligence. Today, Sahil and I will be talking about Apache Iceberg. Sahil, welcome to Software Engineering Radio.
Sahil Walia 00:00:59 Thanks for having me, Robert. Excited to be here.
Robert Blumen 00:01:02 So anything else you’d like to say to listeners about your background before we get into the topic?
Sahil Walia 00:01:08 Absolutely. Thank you first of all for the opportunity, Robert. I’m Sahil Walia. I’m a Senior Technical Architect working with Snowflake. I specialize in data engineering workloads and specifically open lakehouse architectures. I try to migrate customers outside of their old architectures into the contemporary new architectures like Lakehouse, Iceberg, and make sure it’s interoperable for other environments.
Robert Blumen 00:01:34 So Sahil, you just threw out a bunch of buzzwords that sound very relevant to the topic today. Apache Iceberg, we’ll be getting into defining what all of those mean. Let’s start out with a background question. You can run these OAP workloads less efficiently on a general purpose database. What is the cutover point where you want to switch to a specialized OAP for your workload?
Sahil Walia 00:02:01 Absolutely. For the audience who’s not familiar, OLTP is essentially transactional. OLAP is essentially analytical and there’s no certain benchmark I would put, but there are certain criteria I would look for. So, one of the examples I would look for is when you’re looking for latency budget of milliseconds, you want to stay in OLTP world. But when your latency budget is in about seconds and minutes and your audience is mostly analytical training the ML workloads own data, and it’s not dependent on transactional data, I would move towards the OLAP. Another thing that I look for is where you have read heavy aggregations and you have wide tables. Those are essentially a few of the good candidates where an OLAP would make more sense. But if you need primary key enforcements and for example, you want to do frequent single row write in that scenario, I would maintain the OLTP.
Robert Blumen 00:03:02 We’re talking about workloads with latency budget of many seconds to minutes. What are the size of data sets that require that amount of time?
Sahil Walia 00:03:12 Sure. If I understand your question right, Robert, you’re asking at what latency we should be thinking about moving towards the OLAP. Is that right?
Robert Blumen 00:03:19 No, I’m saying if you have a workload that you’re willing to wait for several minutes, you must be dealing with a very large amount of data. How big is that data set?
Sahil Walia 00:03:30 Oh, that can start from anywhere about gigabytes to terabytes and petabytes. Iceberg, which the topic we are going to talk about essentially can handle petabytes of data and Netflix, who are the, where the creators created Iceberg, they actually benchmarked on petabytes of data. Think of petabytes as about, I would say 220 million songs. So that’s a big data set.
Robert Blumen 00:03:54 What are some other domains where you have data sets of that size?
Sahil Walia 00:03:59 Banking is one of them, healthcare is one of them. And then clickstream data essentially meaning how you’re clicking on the app or interface. And then healthcare is another one of them.
Robert Blumen 00:04:09 Let’s now talk about some of the terms that will enable us to have a conversation about Iceberg Data Lake. And in recent years, I’m hearing about data lake houses. Explain what those are.
Sahil Walia 00:04:24 Sure. The way I would think from Data Lake and Data Lake houses, to take a step back first to a data warehouse. So essentially data warehouse is anything which manages your analytical workloads. They were essentially meant to be very structured schema on right kind of thing and optimized for BI. And then later came aware of Data Lake with a cloud word wherein you had S3 ADLS. What you could do is you could manage your row files into a cloud bucket, dumped into a cheap storage. But the challenge that the teams were dealing with was it was essentially creating a data dump, right? And data swamp. And they were not able to manage it. There was no structure that the teams were used to and the schema on right behaviors the team were used to in the data warehouse world. So then essentially the challenge came and we like how can we get benefits of the both world?
Sahil Walia 00:05:18 And that’s where data lakehouse come in. So, the analogy I usually use is that a data warehouse is like a filing cabinet while a data lake is kind of a U-Haul storage. And you can put that U-Haul storage. And within that U-Haul storage, I think you can put a data warehouse, which is a filing cabinet and you can manage it properly. So, you have the lake house storage, which is cheaper, but you have warehouse like a structure and warehouse like transactions. So, data should be easily accessible. You have guarantees associated to it. Let’s say business is running on it and it provides you with a lot more clarity. Happy to double click here, Robert.
Robert Blumen 00:05:59 Yeah, I want to go into that a little bit. So, what I understood is the data warehouse, you’re talking about a schema that has in variance where it’s much easier to extract data from it because you know how the data aligned with the data types are. And you go to the other extreme which you’re calling a swamp, which doesn’t sound like a great living space where data might not have been regularized, clean, normalized, therefore much harder to query. Is that roughly what you said?
Sahil Walia 00:06:29 That’s correct. Absolutely right. And data warehouses were essentially curated, right? So there had to be a use case and you’ll create a data warehouse, but data lakes weren’t curated for a reason. And what was essentially happening is it was upend only. You were putting files. The great thing was you could put any sort of files in those kind of things and you will manage them at the rate time. But when you’re managing at the rate time, it created a lot more of burden. So, let’s go back with the same analogy. Say you have put your winter coat in that giant storage locker, but now you need to find it and in order for it to serve you, it just takes so much time.
Robert Blumen 00:07:11 So the point I want to explore here about these data formats versus the processing, if I think about Postgres, it can be, and for most people it is a black box, there’s an engine and it sits on top of some kind of file format. I don’t have to know what that file format is in order to use Postgres. And there are tools like change data capture and some replication products that understand Postgres log formats and can read them. But for the most part, these are very tightly coupled things. Now, I wouldn’t then say, well, I’m going to get my Postgres file formats and run Oracle on top of it. It just, they’re very, one is engineered for the other. I have a quote here from Confluence website, which I’ll read. It says, in recent years, we’ve witnessed the rise of headless data infrastructure. Companies are building a more open lake house in cloud object storage is accessible by any number of tools. Is this tight coupling from the database world between the format and the engine? Is that breaking down to where you mix and match?
Sahil Walia 00:08:15 Yeah, so I think you’re bringing the right point with the right background in the wall of data. A lot of the things were good, but they were tightly coupled. So, you have your storage engine and then you have your compute engine and any other engine vectorized enhancements are built on top of it. But as a data team, you wouldn’t have access to it. So, you couldn’t make decisions or you couldn’t change the lever of compute or change the lever of storage what you wanted. Right? So, the conf blog you’re talking about is yes, this is about the headless data infrastructure, which essentially means in my mind is your data lives separate to your compute and now you can scale your storage and compute separately. It doesn’t stop only there. The advantages now team get is say you have data in S3, right? It can be accessed by multiple engines now. You don’t have to maintain multiple copies, right? And that’s the way it’s kind of headless. And when you can bring your own compute, say Spark, or let’s say you want to do Databricks or you want to do let’s say Snowflake, you can connect any of them to that data and start running your say analytics workload on top of it.
Robert Blumen 00:09:31 Okay, I think we’re ready now to talk about what is Apache Iceberg? Where does it fit into this landscape we’ve been sketching out?.
Sahil Walia 00:09:41 Yeah, we did talk about data lake houses and Apache Iceberg enabled that. And we did talk about the headless infrastructure, right? Apache Iceberg enables that. So, let’s dissect it a little bit, right? But to provide a little bit of background, it started with Netflix. In Netflix they were dealing with petabyte skills of data. They were having a lot of issues with their hive systems at the time, right? So, they wanted something which can help them manage their metadata better. And in short term, I would put Iceberg as an open table format. It doesn’t stop there. It defines how you organize your data files, but also the metadata files. So multiple engines can safely read and write the same table as I was alluding to, right? And it has few key components. So, one is called Iceberg spec or specification. And the way I would think of it is like by the committers or by the committee, they put a rule document, right? And that rule document is something engines follow, it tells you the standard or it tells you how you need to follow the specification for it to be interoperable, right? And then there are libraries, essentially libraries are essentially Java, Python, Go to boil it down. It essentially means that it’s the Iceberg implementation for Java, Iceberg implementation for Python or let’s say Iceberg implementation for REST. And the last thing is the REST catalog. It allows the standardized HTTP API catalog operations. So, it lets other engines connect via HTTP commands. To put it simply.
Robert Blumen 00:11:19 Is this something like XML where it was an open data format and any provider could implement bindings in any language they want?
Sahil Walia 00:11:30 Yes, they can implement. But right now the bindings that you talked about are limited to as a remember Java, Python, Rust, and Go. So those are the four ones right now I think there’s a C++ one as well. But these implementations, what I have seen in my experience are not something a lot of team invest time on, but what they invest time on is how those are integrated with the catalogs. Because catalogs are something which enables that accessibility of the data or writing to that data.
Robert Blumen 00:12:02 I think I should ask you then explain what is the catalog, what’s in it?
Sahil Walia 00:12:07 Sure. A catalog does a lot of things, but in the world of Iceberg, it essentially says when an engine goes to a catalog, it tells you, okay, catalog, I need this table. It contains the rich metadata of that particular table. So then engines can go through that JSON files in technical world and inspect and understand how it should prune the queries if it is reading it and how it should write the files if it is writing it. And then they do other work as well. For example, governance and those kind of things. I don’t want to go into that, but catalog is basically a point access to that particular table.
Robert Blumen 00:12:45 Is this similar to what might be called schema introspection in some data products?
Sahil Walia 00:12:52 I may not be able to answer that honestly. Do you want to double click on that? Do you want to give an example?
Robert Blumen 00:12:58 Yeah. And most SQL systems you can say describe table or show schema for table and it will show you yes. Columns, constraints.
Sahil Walia 00:13:07 Yeah. So that’s how he does it. Thanks for the clarification. So when you are basically going to that particular describe statement, what essentially engine is doing is inspecting that JSON file and that JSON file has a lot more details. Happy to jump into details like what those can have and how it’s the whole architecture.
Robert Blumen 00:13:26 We might come back to that. I want to do some more high level. You’ve talked about the format that the files can live pretty much anywhere on S3 or any type of blob storage or on a regular file system for situating Apache Iceberg in the landscape. Is there anything you would compare it to or is it the first of its kind?
Sahil Walia 00:13:49 Nothing comes to my mind in relation to the data as such data field, but I would put it in the world of USBC. So USBC is something you can plug into your MacBook, you can plug into your laptop, you can charge your phone with it. So, it’s kind of a common connector. And that’s the way I would think about it.
Robert Blumen 00:14:08 What is the history? Was this a consortium of companies that all had the same problem or was it invented by an open-source engineer?
Sahil Walia 00:14:17 Sure. So, it started with Netflix, but there has been active contribution in its early years by Apple and by other data companies. And these contributions, essentially when it was becoming successful, led it to be donating to Apache Software Foundation. It got incubated and then it got accepted. And it has been part of the Apache Software Foundation.
Robert Blumen 00:14:42 We’re talking about these agnostic layers. You can store it wherever you want on any cloud, any file system, anything can read it. One of the reasons that the storage format and the engine were so tightly coupled is that the storage was designed in a particular way to reflect the physical nature of how data was getting moved off of some kind of persistent format and into memory. Taking into account the Io rates, the number of spindles, the adjacency of things on discs such that the performance of the engine would be reasonable, especially for large data sets. Do we give up something in performance if we stick it on S3, which no one outside of Amazon even knows how it works.
Sahil Walia 00:15:29 That’s a very valid question. And in early years when the Iceberg, so currently Iceberg is on V3 spec, just like any versions of iPhone. Similarly on the V3 spec, in the early years of Iceberg, there were a lot of discrepancies in terms of benchmarks, right? And a lot of engines that were closed, they knew their files and they could optimize for it better. But in the last I would say three to four years, there has been convergence towards Iceberg. And even the vendors now are pushing hard going towards the Iceberg and making their engines more compatible or I would say faster towards that. And the recent bar benchmarks to answer your question have been showing these days that it’s effectively same or sometimes better, the performance. In terms of S3, you are absolutely right, you are leaving the world of vendors in term of lock-in, but you’re still locked in turn kind of with AWS and AWS has your S3 and that’s where you manage your storage.
Sahil Walia 00:16:37 But the good part with S3 is you can do, first of all, it’s very cheap, about $23 per terabyte. So, it gives them cheaper rates compared to vendors, right? And then companies manage their own data. So tomorrow, let’s say you are working with vendor X, but with the innovation vendor X has been very behind, right? Tomorrow a brand-new company comes with a shiny object, you want to move to that vendor Y, you still hold your data, you still manage your data, you just need to change the engine. So now you don’t have to do those messy migrations wherein you basically have a lot of table stakes, high product which goes down, and you do a proper planning. Rather it should be idly a plug and play depending on how data moves, the world moves.
Robert Blumen 00:17:25 The point I’m trying to get at is to the extent that you have all this interoperability and abstraction layers, you make fewer guarantees about how things work versus Postgres or Oracle that owns the entire stack all the way down to the file system, they can make very strong assumptions. So, when you talk about vendors and benchmarks, do you mean the vendors of the engines have all tuned their engines, or do you mean the storage vendors like Amazon have said, hey, Iceberg runs really well on S3, we’ve done some magic stuff inside S3 that’s going to make it very friendly to your Iceberg formats.
Sahil Walia 00:18:04 Okay, thanks for clarifying question. Yeah, I mean that data vendors, like for example, Oracle, Snowflake, Databricks as an example, not particularly S3 and these, I’m just not limiting the storage to S3. You can do ADLS if you want Azure, you can do Google Cloud as well, right? And can you repeat your question? Did I answer your question or
Robert Blumen 00:18:22 My question is, large workloads move petabytes of data. Ultimately, you’re constrained by the Io rate of the physical device or devices where your data is, you can only move data off of the device at whatever rate it supports. And is there a uniformity of Io capacity across all different substrates? Sure. Or people say, well, yeah, I mean three’s slow or blob storage is slow, but I can live with that because it’s cheap and it’s a trade-off we make.
Sahil Walia 00:18:58 Sure, yeah, let’s talk about it. So over here, S3 has its own limitations. So, let’s say if you’re calling the S3 API, you can only do so much of list operations. So, say you have records, which are billions in numbers, right on the original table, but you only need a thousand of them. So, the beauty of Iceberg is it has very rich metadata. So, in the world of let’s say high or anything, what you would do is you’ll go to that particular director and list all the files in that directory. But the metadata is so rich in Iceberg that it’s at the file level. So, what really happens is instead of making those thousands API calls or a hundred API calls, Iceberg is really making only 10 or 15 API calls. And that essentially helps you fetch data faster and then that helps you with your S3 throttling and everything.
Robert Blumen 00:19:51 Okay? What I think you might be saying is maybe on Postgres Oracle, if you had a poorly designed query and it’s doing a table scan, it’s a 50-gigabyte table and it ends up needing to read 50 gigabytes that the design of Iceberg, it’s so smart that it can have, you’re reading one gigabyte that has the gigabyte you need. And so that’s inherently faster than reading 50 gigabytes regardless of how fast your Io is on that substrate. Something like that.
Sahil Walia 00:20:22 Exactly. Yeah. So, there are two components to write. One is the metadata being rich and another is your engine as well as the warehouse and the computer using on that, right? So, let’s say if you have a larger compute, you’ll have more RAM and those kinds of things, but you’re also limited with the metadata as well. So, it’s combination of both. I will marry them together. I wouldn’t separate them together in terms of optimizing it, right? And finding the data faster. But your point on the Iceberg, I plan to cover that if we have time, the Iceberg architecture, how the metadata is laid out, but the metadata is so rich and it at file level that it lets you prune or basically skip the files which are not relevant at the backend.
Robert Blumen 00:21:27 Well let’s do this. Let’s talk about some of the queries and the metadata. When you’re using Iceberg, what does the interface look like to a programmer? Are you doing select or something that looks familiar?
Sahil Walia 00:21:42 Yeah, the data engineers or the engineers who are working with Iceberg, they don’t have to understand what Iceberg implementation look like. They just need to worry about the SQL or the Python they know. So as an example, if you know select queries, insert queries, update queries, the other things are managed by the engines, right? Because they’re integrating with the REST catalog. Now let’s talk about it. So, let’s say if you’re running a select query in SQL, it’s pretty much the same select query you will run in Iceberg. If you’re running the insert query, you’ll run the same insert query and it’s kind of antsy standard SQL that you run. So, you don’t have to learn that. What you need to learn is about the architecture, veterans, right? How the Iceberg handles data, how Iceberg handles metadata, how read operations happens, how right operation happens, what metadata files are. Because if you don’t know those things, you may not be able to optimize for your workloads
Robert Blumen 00:22:37 And reading about Iceberg, there’s discussion of time, travel queries. Can you explain what that is?
Sahil Walia 00:22:44 Absolutely. So, time travel is basically, I want to go in the past at a certain time, and you can do it so many ways, right? You can say, I want to go back and see how the data looked like five days ago and assume you have a lot more events and you’re doing something. Let’s say you’re doing a machine learning model or you’re doing some DML and your code has changed now, and your data looks a little different. Or you’ll be like, oh my God, I messed it up, I want to back up. I want to go back to the old state, to see how it looks. So, what you can basically do is select from that event stable as an example, timestamp as off, and you provide that particular timestamp. Not only that, but you can also provide other things, right? For example, what you can do is there’s something called snapshot.
Sahil Walia 00:23:30 So every commit you do is a snapshot and it goes with an ID and you will say select, start from that again, the even stable. And you’ll say, I need this particular version, right? And you’ll say version as of and you provide that version. Okay? So, it lets you go into the past to get that particular state. And this really is very powerful when you want to inspect data. Let’s say you have GDPR compliance, right? And it really lets you see how your data changed and sometime mistakenly mess up the data, right? You can go back and recover it from the past. You don’t have to do rollbacks. It automatically is a feature of Iceberg.
Robert Blumen 00:24:09 The idea of GDPR compliance as an example from this, sounds very interesting. I didn’t quite get how that example illustrates GDPR. Could you explain that in a bit more detail?
Sahil Walia 00:24:22 Sure. In GDPR, essentially there’s a concept of right to be forgotten and you want your records to be deleted, you don’t want to be remembered, right? And what you can do is if you can go back and show that this particular time, that particular query was deleted, it may not be the right example here, but another thing you could do is you need to also remove the metadata related to that. If you can remove that, it deletes all the link between that parquet file that you have stored and the metadata and you can show what you have done in the past.
Robert Blumen 00:24:53 That is an interesting example. What I think you meant is somebody’s auditing you or a customer says, can you prove that I’ve been forgotten? You could show that five days ago the record was there or as of five days ago the record was there and as of today it’s not. But the data might still be somewhere on disc. It’s simply inaccessible because nothing links to it?
Sahil Walia 00:25:16 That’s why I was saying like it may not be the right example here because you really have to delete the data, the Parquet files and the metadata link. So, there shouldn’t be any memory there. But the good idea will, here is like if you do timestamp as off it gets into the weeds of it, right? If you do timestamp as of it should not be able to show you anything because you want all of the data to be deleted for compliance reasons.
Robert Blumen 00:25:38 Okay. That’s an interesting example. Maybe not a correct example. Do you have any other examples of a use case for the time travel queries?
Sahil Walia 00:25:45 Sure. Think of an example. Team ran a delete job that was too broad, right? It was supposed to run delete 500 records, but it deleted 2 million records, right? So, with Iceberg you can run a command like rollback as we do in the old worlds, right? And you roll back the previous snapshot, you tell it, okay, this is the previous snapshot I want to go to, and your data is back in seconds.
Robert Blumen 00:26:07 That is a very timely example Sahil, because we now have stories reaching the tech news about people enabling AI agents to talk to their database, the AI agent has inadvertently deleted too broadly. That would be a very useful feature.
Sahil Walia 00:26:25 Yes, it is very, very much beneficial and I have seen a lot of AI workloads really value it. No one wants to be in a situation where wherein you have you deleted and make that mess and then have to explain I executed it without even reading it. Like it was too broad. But it’s tool in your toolbox that really helps you feel secure.
Robert Blumen 00:26:44 You mentioned that it supports inserts and updates. Clearly your database product is of no use if there’s no way of getting data into it. But is the overall architecture highly read, optimized at the expense of rights performance being sacrificed?
Sahil Walia 00:27:03 It’s essential Iceberg. I would put it for analytical workloads first of all. So, and secondly, I would assume most of the workload would be for the read purposes, but it doesn’t mean you can’t do, write? So, the writes I would imagine in my world would be more of a bulk write instead of transactional writes. Okay, say it won’t be a teller like writes but it’ll be, oh you’re doing bulk million rows write now. Those writes can be happening too with; there are two things that happen. There’s a concept of copy on, right? And merge on read. So, if you want, I can go deep into it and basically it’s defines how the reads what, should be done or how the writes should be done. And just a little bit of primer copy on writes essentially means let’s say if you’re doing any some DML operation, right?
Sahil Walia 00:27:51 It’ll rewrite the new file. So, assume you have thousands of packet files in one Parquet file, which is a hundred MB or a hundred GB. You have 1 million records, but you only want to change one record. What it’ll do is it’ll rewrite the whole new file, right? It’s great in terms of read because it’ll be faster and referencing to the new file. And there’s another concept of merge on read in which what it does is it maintains the same files but adds a kind of a sticky note to that file or this particular record was deleted. But when you’re reading your compute engine is computing that, oh I need to do this 10 records minus one and present you the nine records and that’s how you define optimize for your workloads.
Robert Blumen 00:28:38 I would like now to get more of a deep dive into the read and write paths, including the structure of the catalog and the metadata. I think this would be a good time to ask you to define Parquet because you’ve mentioned that a few times and I think it will be important. So, let’s do Parquet first and then do the architecture and metadata.
Sahil Walia 00:29:02 I would think of Parquet as a file format. So just like you have .xlsx for Excel, you’ll have .parquet the advantage or before going to Advantage, I’ll describe it, it’s columnar storage meaning say you have a table or excel sheet that we are aware of and it has one column with names, it’s stored by the names, it doesn’t store by the row, so it’s columnar, but it’s also not only that it can be compressed. So, it’s at least three to four times smaller in size than your Excel file per se, right? So, it’s, you can store a lot more data in a lot smaller file, right? And each column is stored contiguously on disk. So, when it is being read, it’s faster because it’s stored together, right? The column. And then another thing that it has advantages of is that it’s self-prescribing, meaning the schema is in there in the file footer of the parquet and the compression is highly effective. Another thing, I don’t want to go super technical, but something called predicate push down wherein you’re telling like what filter inside the file you want to do without doing the full scan, which lets you be more performant when you’re reading it, right? And then another thing that I really like ballpark is, or other engineers do as well, you can have complex types like trucks, and maps if you want to do that, right? So when you’re doing any reads, it read only the age column as an example, but also note that be performant when reading it.
Robert Blumen 00:30:34 I could guess that column or formats are highly compressible because particularly domain specific data tends to have a very few values or a small range. Now the predicate push down is that something where the file may be organized in such a way that if you’re looking for give one example greater than 10, it can figure out it doesn’t need to read the entire file to get values in that range. So, it reduces the amount of Io?
Sahil Walia 00:31:04 Exactly. That’s precisely right. You reduce the Io and with that predicate pushed on, it goes only to that block where the data is and within package called row groups and those kind of things in the whole architecture and goes there and find that information.
Robert Blumen 00:31:18 What I’d like to do, I’ll give you some discretion in how you want to organize this but walk through either read or write paths or both and to incorporate the parquet files, the catalog, the metadata, how they are all working together to give the user the query. How did you want to approach that?
Sahil Walia 00:31:40 I can give an example of a read, but before that I think it’s, it’ll be great service for the audience that we explain what the Iceberg architecture looks like so that when I talk about it they’ll follow, right?
Robert Blumen 00:31:52 Okay, let’s do architecture and then read.
Sahil Walia 00:31:55 Sure. So, I’ll think of it in three layers, right? And it’s a tree structure I would think of, on the very top would set your catalogs, right? And it can be any catalog that you can plug in a lot of Iceberg compatible rest catalogs. To put it simply, again, you can use any catalog to talk to Iceberg tables, right? And in simple words, as I explained before, catalogs answer the question, where is table X, right? It points to the current metadata file and the catalog will have the information about the metadata JSON file. It has the latest version of that table JSON file. So, say if the table has changed, it will know which the latest metadata file is, and then that metadata JSON file, in fact in turn has information about the manifest list. And manifest list is an AVRIL file. It has information about your, let’s say partitions in a table, right?
Sahil Walia 00:32:55 And what it does is it helps you prune the partitions or help you prune the files you don’t need. And it has first level of pruning here. And then that manifest list in turn has information about the manifest files. It’s again an AVRIL file and that manifest file, what it does is maintain the statistics about the data files, the Parquet files we were talking about. And those manifest files have per file stats. So, so let’s say I have a hundred Parquet files for each Parquet file, it’ll have details about what the main and the max values by column, how many rows were there in in particular Parquet file, right? What kind of operation was it? So, it maintains all of that rich metadata. So that’s the kind of a whole architecture and it’s very rich in that. So now let’s talk about the read pattern as an example, right?
Sahil Walia 00:33:48 So now I want to read where my table events is. I’m going to go and I’m going to write a command select star form events. As soon as I do that, the catalog will answer the question, where is the latest file for events? It finds that the latest version is V5 metadata JSON, that’s the latest file. And in that JSON file it’ll have bunch of information, but within that bunch of information will be one key thing, which is called snapshot. And it’ll tell you what latest snapshot did. We were talking about it as part of the time travel a minute ago. And that snapshot will have a pointer to the manifest list and that manifest list will have all the information for the partitions and anything you need. And then it’ll take me, okay, this is the AVRIL file. Now let’s go to that file and it’ll have the manifest files there. When it goes to the manifest file, it’ll have all the details about the files it needs to read. So, say I did select extra appointments, but now I want to do select extra appointments where date is July of 2025, right? So, it knows which file has the min and the max value of July and it’ll only go and read that file so it doesnít have to pass through the directories or anything. It’s just so rich in metadata that didn’t pass through and only read those data files at the bottom.
Robert Blumen 00:35:15 The example you gave there covers a column in which the column is a time value. So, to avoid having time appear twice, say we were measuring some value, that’s not time like the state in the United States, the time dimension that’s expressed through the time travel, if I understand that, is in this tree at the snapshot layer. So, if you have a time travel, you will go down to that point in the tree where the snapshot corresponds to your time travel cutoff? That’s a guess, is that correct?
Sahil Walia 00:35:54 Yes. So, the metadata JSON will have that snapshot log and in that snapshot log it’s essentially a log but also a timeline, right? So, when we do as of that particular timeline, that’s what it uses as it’s cutoff time.
Robert Blumen 00:36:09 Okay, so you took us through read and our talking a little bit about time travel read the other major operation would be write. So, can you walk us through the write going through all those layers of the tree?
Sahil Walia 00:36:27 Absolutely. So, say you are an engine, you are trying to do NDML operation or trying to do an insert, right? You generally, you can define what size of a file you would like to populate in the storage. Let’s say you want to maintain 256 GBO files, right? And you want to write 2, 5, 6, 0. So it’ll create 10 files for you. And the way it does it, it starts from bottom up. So first it will write the files in your storage object storage, but other engine querying wouldn’t know because it hasn’t written that snapshot or that metadata yet. As soon as it writes it, it’ll write the manifest file in manifest file, then it’ll populate, okay, these are the files I populated, these are the min. and the max. values per column, right? And other commit time and those kinds of things. And then once it creates that manifest file, it’ll create a manifest list and that manifest list will be like, okay, this is the manifest file I need to refer and this is part of X snapshot.
Sahil Walia 00:37:31 And then it will populate the partition information, say, write if you have any partition information, it’ll populate that there. Once it’s done with all of this, it’ll try to write a very final thing, the metadata JSON and that’s where it does compare and swap. It starts with the latest version and then compares. And so, what it does is it tries to figure out, okay, I started writing those 10 files, but in the meantime did the metadata JSON change? If it changed, it has to redo the newer version of the JSON, which otherwise would conflict or otherwise, it’ll just do version plus one, do JSON and write that metadata JSON file and that will become the current file. Going on forward, they’ll read isolation, anything that is coming into read it, we’ll start reading from that particular commit.
Robert Blumen 00:38:22 We want our writes to be atomic. Now this process you described there are multiple files that need to be changed. Something could fail before it’s completed. The process, does it depend on the final step being created and that the system can tolerate, let’s say, dangling references or files with nothing pointing to them from prior failures?
Sahil Walia 00:38:46 It solves all the problems that you talked about there. These were essentially the problem with the high as well, right? So, there are two kinds of things. One is read isolation, one is write. In terms of read, what it does is for the audience, if you start at 2:00 PM reading something and at two 1:00 PM something changed, but it, if you result at 2:05, you still see the result at 2:00 PM. It locked you there at the very first time you started reading. But for the write, the way it does it, it’s optimistic, write Isolation. To put it simply, say robot is trying to write and Sahil is trying to write, whoever first writes get that JSON, the other person will get the rejection, and it’ll go to that engine, okay, Sahil Robert committed, go back and start writing the engines. But what it doesn’t have to do is to rewrite a file, it only has to write the metadata, the files stay the same, but assume something happens, the world crashes and everything and say it’s all of the operation leave in between, you have those orphan files but they are not rendered anywhere because there’s no reference in the metadata.
Sahil Walia 00:39:52 So they won’t give you corrupt reads. The only thing you will have to do is clean up your storage and that operation has to be done to remove the orphan files if you identify. But those are very rare scenarios.
Robert Blumen 00:40:05 Does Iceberg support anything for managing the size of data? Either I no longer need historical time travel beyond a certain date in the past, or I can de-REST the data by going from every single change to once a week or something like Postgres vacuum that is just cleaning up spaces created by write operations?
Sahil Walia 00:40:32 Yeah, there’s something called snapshot expiry. So, say you are doing a lot of DML operation every 15 minutes, and you don’t necessarily need snapshot from a week ago or a week ago or later, right? What you do is you can configure through engines that I want to expire or remove the snapshots after a certain date. And that’s where you will see, okay, you can’t go back beyond a certain date. And that’s a decision team makes depending on their workloads, what they want to do, those kind of things, right? And similarly for the, I think you talked about the vacuum, there’s something or vacuum full in Postgres if I’m not wrong. So essentially what it means is say you have committed small files over a period of time and over the weekend you get smaller chunk of files for example. But over the weekend you get a proper trend. But what you want to do is you want to make sure your whole bucket is clean and you don’t want to maintain small files because it still has to read, it still has to do everything. What you can do is you can merge all small files into a standard large files, which is best for your read patterns and it’s called data compaction. And a lot of engines provide that kind of service when you do managed service with them. But you can manage by yourself as well.
Robert Blumen 00:41:46 Most databases have something like the query planner that decides what order to access different tables in a Join. We haven’t talked about Joins, but does Iceberg support Joins and does it have a query planner or is that process you described where it’s walking through the metadata to find what it needs? Is that the query planner?
Sahil Walia 00:42:08 Yes. So, Iceberg handles all of those things to in engine. What it does is it tells you, okay, this is the metadata I have, and this is the metadata you should pass through to do the query fetch. But it doesn’t tell you this is how you should do it. It tells you the spec, it tells you that this is what I have, this is how you can traverse through the tree of the metadata. But it doesn’t tell you, okay, this is up to the engines to implement.
Robert Blumen 00:42:38 So far, we’ve been talking about a single table or we haven’t been specific. Does Iceberg support Joins?
Sahil Walia 00:42:46 Yes, Iceberg does support Joins. You can connect multiple tables together to have a rendered result as you need. You can join Iceberg to Iceberg or in case of several engines, let’s say have their custom table format, you can connect Iceberg to a custom table format. But that’s something engine does. But to answer your question, yes.
Robert Blumen 00:43:04 So my next question is going to be is there a query planner? But since Iceberg is not an engine, it’s just a format, what’s the right question I should be asking?
Sahil Walia 00:43:17 So at the first we answered the question like it does support join. Yes. And to a follow up question, it’s very much engine dependent. Iceberg does provide the stats for better joint planning, right? But it doesn’t execute the Joins. Your engine has to plan those Joins architectural data query plan per se based on that. But Iceberg doesn’t prescribe that.
Robert Blumen 00:43:39 Are there multiple engines in the marketplace from different vendors? Are they commercial? Are they open source?
Sahil Walia 00:43:47 Yeah, there are multiple engine options you have, you have proprietary ones, the commercial ones, and you have the open ones. You can use Spark, the Flink, the Trino, which is open source. But you can use Snowflake, you can use Databricks, you can use anything which is compatible to the Iceberg Spec can use it.
Robert Blumen 00:44:07 You also mentioned there are multiple options for catalog and I was reading about this, I found this list consisting of Hadoop, Hive, AWS Glue and some others. What are some of the more popular catalogs?
Sahil Walia 00:44:22 Sure. AWS Glue is one of them. And AWS Glue recently became compatible with IRC, which is IS Progress Compatible, right? And then you have Snowflake, Horizon, Databricks, Unity catalog and Hive Meta Store. But Hive Meta Store is usually, I haven’t seen being used a lot just because with Cloud at Hive Meta Store was built better for the HDFS file and everything, but it’s one of the options as well.
Robert Blumen 00:44:51 What are some differentiators between different catalogs?
Sahil Walia 00:44:56 Very good question. So, it really varies what you are trying to do. So let me put it this way. As we talked about Iceberg Spec, there’s several versions of it. Version one, version two, version three. So as an example, the current version that is GA is version three, which supports nanoseconds time support. So, note all catalogs would support it. So, you have to think through how the implementations are, what you’re looking into. Say Spark is writing it, Snowflake is reading it, can that catalog support it, right? And then another thing you want to see is how the RBACs are handled. So, say since this is interoperable, you want to make sure that your governance gets applied from one side to another, right? So, say you have access via Spark, but you don’t have access via Snowflake to a certain area. You want to make sure you have the right roles and the governance is handled via the catalog. How strong are they with the governance, how they went, the credentials to the storage, those aspects. So, it’s a variety and I would evaluate in a way when, what’s my use case?
Robert Blumen 00:46:00 I want to move now for the last part of the interview talking about the bigger picture of Iceberg in the application architecture for an entire application. The idea of a CRUD application is extremely widespread. Almost everything we use on the internet for business is some kind of CRUD application that writes into OLTP. And by application, I mean something has a database, a middle tier, and a user interface. What would be, some examples of a database application where Iceberg is the data layer. What is that application doing? What is its architecture?
Sahil Walia 00:46:39 I can give one of my examples that I recently implemented. We had clickstream data coming in, and the clickstream data landed in Postgres. Postgres is OLTP. And then from there it landed via the Pulsar — Apache Pulsar is again for streaming as well as for managing the queues. What it did is it holds data for about 15 minutes and then pushes to Snowflake. And then when it lands to Snowflake, that’s where you will do a lot of your cleaning up, transformations you want to do, and you prepare it or curate it for other teams. And then ultimately the teams wanted to implement the ML walk or the model on the Spark side and that’s where the interoperability came in. They were able to use the Snowflake engine to do a lot from a transformation and loading for the right tables, but the ML teams could leverage the Spark engine for the ML workloads by just accessing the table that were written by Snowflake on S3.
Sahil Walia 00:47:39 And to summarize this, I would put it in a way wherein your multiple teams need access to the same data, but you don’t want to have a redundant storage in terms of just okay for the same use case. You don’t want to have that; it’s better you don’t do it. Secondly, I have seen a lot of teams doing similar transformation or almost similar transformation just to give different kinds of data to different teams, right? It reduces that. Another thing is if you have use cases wherein you want to do anything analytical, that’s where I will use it. Again, AI is one of the use cases that’s getting very much popular in which what happens is you can have Git-like branching and you can do A/B testing on the data with Nessie catalog and you do branch the data into two layers and do analysis on both layers.
Robert Blumen 00:48:32 By AI being an application, are we talking more about training or fine tuning or about using the data to drive inference of models you already have in order to evaluate the models? Or what do we mean by AI?
Sahil Walia 00:48:49 Okay, just the training the data. If you’re training the data with dataset A versus B and you have a branch like structured — data branch, not a Git branch, data branch-like structure, you can have A and B comparison while training and do analysis.
Robert Blumen 00:49:02 Well Sahil, we’ve covered Iceberg from beginning to end. Before we wrap up, anything else about Iceberg that you’d like listeners to take away?
Sahil Walia 00:49:13 What I would say is a very exciting field and it’s a very exciting world of the Iceberg. Now the Iceberg team is working and collaborating on the version V4. It’s not final yet, but it’s going towards an even better product. And another thing that I have been noticing in industry is that vendors are converging to Iceberg as a standard. They are now not limiting themselves to their own proprietary one. So, I think it’s a good time to adopt. Another thing that I would recommend teams to take away is be more involved in Apache’s dev list for the Iceberg. You’ll get to know how it’s developing, what the discussion themes are, what the common themes are, and you’ll get to know how things are developing in the world of Iceberg. It’s a truly, truly exciting time to be learning Iceberg.
Robert Blumen 00:50:01 Before we sign off, would you like to direct listeners anywhere they can find you, your publications or any projects you are involved with?
Sahil Walia 00:50:11 Absolutely. You can reach out to me on my LinkedIn. My LinkedIn is Sahil Walia as you see my name. And then Syracuse, Syracuse, New York. And I can be reached, I have a medium page where I publish articles, and you can look at my Google Scholar as well.
Robert Blumen 00:50:27 Sahil, thank you very much and thank you for speaking to Software Engineering Radio.
Sahil Walia 00:50:32 Thank you Robert. It was great chatting with you. I appreciate it.
Robert Blumen 00:50:35 For Software Engineering Radio, this has been Robert Blumen. Thank you for listening.
[End of Audio]
