Format Wars: Why I'm Starting to Care About Apache Iceberg's v4 Choices

Last month, I was debugging a data pipeline at 2 AM in Islamabad when my colleague asked: "Should we migrate to Iceberg?" I didn't have a good answer. We were running on Delta Lake, things worked, and switching felt like moving the entire house while still living in it. But the more I read about what's happening in the data lakehouse community, the more I realize this question isn't just technical, it's about whether these projects can evolve without breaking the trust of people building production systems on them.

The Apache Iceberg community just wrapped up a critical week of discussion about their v4 table specification, and honestly? It's the most pragmatic engineering conversation I've seen in the data infrastructure space in months. Not because they have all the answers, but because they're asking the right questions before shipping breaking changes.

The Real Problem: Format Evolution at Scale

Here's what I learned from this week's discussions: when you're designing a data format that thousands of organizations depend on, you can't just evolve it the way we evolve our side projects. Every choice cascades.

The Iceberg community is debating whether v4 manifests should be Parquet-only instead of supporting both Parquet and Avro. On the surface, this sounds like an implementation detail. But dig deeper and you see the tension: supporting two formats means every tool integrating with Iceberg has to make decisions about which one to handle. That sounds like flexibility, but it's actually decision tax masquerading as options.

Steven Wu's proposal for Parquet-only got support from folks like Russell Spitzer and Manu Zhang, who framed it as a step toward converging on a single file format across the entire stack. The reasoning is solid, Avro can't efficiently support projection reads on manifest files, including column statistics. Why maintain support for something that's fundamentally constrained?

The Migration Question That Keeps Me Up

Where things got interesting was Shawn Chang's concern about the upgrade path from v3 to v4. The current design makes upgrading O(1)-your root manifest just points to existing v3 metadata while new writes produce v4 files. No expensive rewrites, no downtime.

But Chang worried about what happens at scale. If you've got petabyte-scale tables in five years with v3, v4, and v5 data coexisting, your query performance becomes unpredictable. You're hitting a blend of metadata formats, each with different performance characteristics. Anoop Johnson defended the O(1) approach because expensive rewrites would add friction, but I think both sides are right about different things.

This is the pattern I've seen in production: everyone wants cheap upgrades until they don't. Eventually you pay the cost anyway, except you pay it in unpredictable query latency instead of controlled maintenance windows.

Performance Wins and Smart API Design

There was one concrete win this week that genuinely impressed me: Varun Lakhyani's benchmarks for integrating EagerInputFile into manifest reading showed 25-55% reduction in S3 manifest read times. That's not theoretical, that's real speed on real cloud storage.

What I appreciated even more was how the community decided where to put this optimization. Rather than spraying it everywhere, they chose the FileReader API specifically for Parquet, because the performance win is tied to Parquet's behavior. That's disciplined API design. Don't solve a general problem with a general solution when the specific problem has a specific cause.

My Take: Format Decisions Are Trust Decisions

Here's what really struck me about this week's conversations: the Iceberg community is essentially asking, "How do we make changes without breaking the people who depend on us?"

That's maturity. That's not what I see everywhere in the JavaScript ecosystem, where we upgrade frameworks every other week and deprecate APIs with six months' notice. The data infrastructure world can't move that fast because the cost of migration is exponential with data volume.

I'd lean toward the Parquet-only decision for v4. Yes, it narrows options today, but it simplifies the integration matrix for every tool downstream. Every simplification is a place where bugs don't hide.

The upgrade path concerns me more. I'd rather see a clearer guidance on when and how to rewrite metadata, even if it costs more upfront, than deal with performance cliffs years later.

What This Means for Us

If you're building data pipelines, you should care about how your formats evolve. Not because you need to migrate immediately, but because you need to understand your vendor's philosophy. Does Iceberg's approach resonate with how you want to manage your infrastructure? Do you trust their prioritization?

These weekly discussions are how projects earn that trust.

Source: This post was inspired by "Apache Data Lakehouse Weekly: August 5 - August 12, 2026" by Dev.to. Read the original article