Tracking: rust version of iceberg sink.

[liurenjie1024]

This issue tracks the implementation of rust version of iceberg sink.

Phase 1

In phase 1, we will focus on implementing an append-only iceberg sink.

Iceberg rust client

• Implement parquet file appender. icelake-io/icelake#37
• Implement data file writer. icelake-io/icelake#38
• Implement append only task writer. icelake-io/icelake#74
• Tracking: Implement table transaction api. icelake-io/icelake#39
• Allow user to config parquet writer. icelake-io/icelake#142

Risingwave sink

• Implement CoordinatedSinkWriter for iceberg sink. #10642
• Implement TwoPhaseCommitSinkCommitter for iceberg sink. #10643
• Add test for different types for iceberg sink. #11650
• Check iceberg sink config to allow user to do more customization. #11652
• Add integration tests for new iceberg sink. #11775
• Rename iceberg_v2 to iceberg. #11776

Phase 2

In phase 2, we we add update/delete for iceberg sink.

Iceberg rust client

• Implement sorted position delete file writer. icelake-io/icelake#40
• Implement equality delete file writer. icelake-io/icelake#41
• Tracking: Implement catalog api icelake-io/icelake#172
• Switch to catalog api so that we can support rest catalog. #12221
• Implement delta writer. icelake-io/icelake#206

Risingwave sink

• Tracking: Implement table transaction api. icelake-io/icelake#39
• Add observability for iceberg sink. #12352
• Improve data skew problem. #12016
• Implement delta task writer. #12326
• Setup e2e demo for iceberg sink. #12721
• Improve iceberg commit logic to so that we can avoid restarting streaming job when commit conflicts happens. #13155
• Improve the error message of iceberg sink. #13897
• Add shuffle in iceberg sink to reduce small files as much as possible. #13898
• Decouple iceberg commit from risingwave commit. #13899
• Support all kinds of iceberg catalog. #13900

[liurenjie1024]

cc @ZENOTME @wenym1

[Xuanwo]

Let's rock!

[github-actions]

This issue has been open for 60 days with no activity.

If you think it is still relevant today, and needs to be done in the near future, you can comment to update the status, or just manually remove the no-issue-activity label.

You can also confidently close this issue as not planned to keep our backlog clean.
Don't worry if you think the issue is still valuable to continue in the future.
It's searchable and can be reopened when it's time. 😄

[xxchan]

track on individual issues instead.