[Improvement] Improve iceberg rest catalog service

[yulongfufu]

What would you like to be improved?

The Iceberg REST Catalog Service is pivotal for our company's transition to a unified metadata system, and I believe it should be our primary focus (Only when the REST Catalog offers significantly more benefits compared to the Hive Catalog will we have the motivation to proceed with the metadata migration).
Here are some of my thoughts on potential improvements to the Iceberg REST Catalog Service:

1. Fine-grained lock orchestration on the service side: By orchestrating commits, we can reduce the number of lock calls and decrease the commit failure rate. Furthermore, for append-only semantics, it is actually possible to operate without locks.
2. Unified caching: From the service's perspective, we should add global caching for Iceberg tables.
3. Taking over more Iceberg metadata structures into the REST Catalog Service: this would not only provide higher access performance but also allow for unified handling of Iceberg version compatibility.

As we are currently evaluating options for a unified metadata solution, we are eager to actively participate in its development :)

How should we improve?

Give me a moment to think

[jerryshao]

Yes, there are a bunch of things we can do to differentiate from the traditional HMS-based Iceberg catalog, do you have any plan about what you would do in the next step 😄 ?

[yulongfufu]

Yes, there are a bunch of things we can do to differentiate from the traditional HMS-based Iceberg catalog, do you have any plan about what you would do in the next step 😄 ?

Sorry, I just started exploring this direction as well. I’m getting familiar with Iceberg. First, we want to start using the REST catalog, but I’m currently stuck on selecting a solution. If I come up with a specific plan in the next few days, I will update you promptly. Let me think about it for now. 😊