Apache Iceberg‘s architecture, features multiple snapshots (aka versions) that gives us cool features like time-travel querying and rollback functionality. I show the flexibility of this in a prior post. There’s a belief out there that having a long tail of prior versions available affects performance, but it actually doesn’t.
Most queries are simply working against the current version and the query engine gets the name of the metadata file from the Iceberg catalog and doesn’t have to worry about all the other metadata files hanging around to support that long line of prior snapshots. The consequence of having all those snapshots is ultimately going to be the footprint of the underlying data files that have to stay on your data lake to support each & every older version.
This post is focused on showing how you can very quickly be keeping around 2-10, OR MORE, times the data size of your current version and how expiring snapshots on some regular schedule is going to allow you to fit into some acceptable norm. The cloud providers will let you keep as much data as you want, but they seem to keep asking us to pay for it, too.
Creating new versions
v1: Initial inserts
In this logical scenario, version 1 of an Iceberg table was created with the insert of 1.5TB of data that was spread across 3 files of 500MB each.
v2: Delete a few rows
When a delete statement was run that identified 3 rows from file A and another row from file C, the new version has to reference the same files from version 1, plus a small “delete file” that references the row locations pointing to the 4 records being deleted. The good news is that you still only need to store 1.5TB of data.
This is because the data lake files themselves are immutable (can’t be updated once created, but can be deleted) AND that Iceberg DML operations utilizes a Merge-on-Read, instead of Copy-on-Write, strategy to account for deletes and updates. Need a primer? Here’s more on MoR vs CoW.
v3: Compaction
Iceberg has a cool feature for tackling compaction (i.e. replacing many smaller files with fewer larger ones). This process can also resolve the delete files by including their modifications in the newly rewritten files that replace the older ones. In the scenario below, the process created better versions of files A & C that ended up without the records logically deleted in version 2.
At this point, we have 2.5TB of data on the lake even though version 3 only references 1.5TB of data files.
v4: Update one row
When a single record that is physically stored in file B is updated, a positional delete file is created as well as a new data file that has the full record as it would be after the update was applied to it. The MoR strategy doesn’t cause the overall footprint to go beyond the current 2.5TB.
v5: Compaction
This time, the compaction rewrites file B along that actually stores the updated record from version 4. The newly rewritten file I is yet another 500MB of data being stored bringing our total up to 3TB.
Expiring snapshots
Ultimately, coupling the older version snapshot expirations along with the orphan file cleanup action allows us to start reclaiming data lake storage space as older files end up no longer being referenced by any versions.
Eliminate v1
Getting rid of version 1 does not eliminate the need to keep all existing files and does not save any data lake storage space.
Eliminate v2
We get to eliminate two of the older 500MB files (A & C), along with the small delete file (D), lowering the overall footprint to 2TB.
Eliminate v3
Yep, we are still referencing all the files that we were before expiring this snapshot.
Eliminate v4
Finally, we are back down to 1.5TB of overall data lake storage being needed to able to use the single remaining version.
Conclusion
Every scenario will be a bit different, but the volume & velocity characteristics of your ingestion pipeline will heavily impact what happens. The frequency & amount of your DML statements will also contribute to a particular table’s situation. Finally, the need to keep track of a particular number of prior versions will also impact your particular scenario.
Likely, you’ll settle on some general standard for the majority of your Iceberg tables, but again, keep in mind the consequence of data lake storage requirements for all of your largest tables. Periodically validate that your table maintenance routines are achieving the results you need to support prior snapshot features WITHOUT breaking the bank on your storage costs.