[#5624] feat(bundles): support ADLS credential provider

[orenccl]

What changes were proposed in this pull request?

Add ADLS credential provider

Why are the changes needed?

Fix: #5624

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Added a unit test and verified successful access to the ADLS container.

Supplementary Information

Chose to use the following versions of Azure libraries:

• azure-identity = "1.13.1"
• azure-storage-file-datalake = "12.20.0"
• azure-core-http-okhttp = "1.12.0"

Instead of the latest versions because, although the official documentation states support for Java 8 and later, the latest versions appear to have been compiled with Java 21. This caused compilation issues in the Gravitino environment. Therefore, downgraded and tested to find the latest usable versions.

[orenccl]

Hi @FANNG1 , could you please take a look when you have a moment

[FANNG1]

seems there are class conflictes with netty in Gravitino and netty in azure, My suggestion is to shadow the related packages in azure-bundle like what is done in gcp-bundle

tasks.withType(ShadowJar::class.java) {
  isZip64 = true
  configurations = listOf(project.configurations.runtimeClasspath.get())
  archiveClassifier.set("")

  // Relocate dependencies to avoid conflicts
  relocate("org.apache.httpcomponents", "org.apache.gravitino.gcp.shaded.org.apache.httpcomponents")
  relocate("org.apache.commons", "org.apache.gravitino.gcp.shaded.org.apache.commons")
  relocate("com.google", "org.apache.gravitino.gcp.shaded.com.google")
  relocate("com.fasterxml", "org.apache.gravitino.gcp.shaded.com.fasterxml")
}

[FANNG1]

What is the purpose of IcebergConstants?

IcebergConstants defines the properties for Iceberg catalog and IcebergRESTServer which are used in Gravitino server, Gravitino IcebergRESTServer, Spark connector, etc.

What variables should be included in IcebergCatalogWrapper?

Besides the credential information defined in xxTokenCredential, the other credential related information like endpoint region we'd better pass to the client too, so the client side no need to do extra configuration. For ADSL, I don't known whether it needs some extra configuration(I guess not need), maybe you could try to test whether it works without extra config.

[tengqm]

overall lgtm.
a few nits for consideration.

[orenccl]

Hi @FANNG1 ,
Thank you for your help earlier, which allowed me to make progress.
This PR is ready for review now, please take a look. 🙇‍♂️
I will update the documentation within a few days in this PR.

[FANNG1]

@orenccl I don't have enough time to test and review this week, may delay until next week, is that ok for you?

[orenccl]

Hi, @FANNG1
Sure, totally fine. Thanks for letting me know!

[orenccl]

Originally, I was able to pass the unit tests. However, when I built the Iceberg REST server Docker image and tested it with Spark, I ran into an error.

INSERT INTO testDatabase.test_azure VALUES (1), (2)
24/12/06 22:49:49 ERROR TorrentBroadcast: Store broadcast broadcast_0 fail, remove all pieces of the broadcast
24/12/06 22:49:49 ERROR SparkSQLDriver: Failed in [INSERT INTO testDatabase.test_azure VALUES (1), (2)]
java.io.NotSerializableException: com.azure.storage.common.StorageSharedKeyCredential
Serialization stack:
        - object not serializable (class: com.azure.storage.common.StorageSharedKeyCredential, value: com.azure.storage.common.StorageSharedKeyCredential@63f38fb1)

I found that this is a known issue. Please refer to:

• Issue #10245
• Pull Request #10045

This issue has been addressed in Iceberg version 1.6.0.

How should we handle this problem?

[orenccl]

Using the account name and account key to obtain StorageSharedKeyCredential doesn't work in the Iceberg 1.5.2 + Spark environment; only SAS tokens or DefaultAzureCredentialBuilder can be used.

However, the Gravitino server currently cannot use tokens to write metadata information, and DefaultAzureCredentialBuilder requires users to set environment variables, which cannot be configured through Gravitino conf.

---

Additional Testing Observations

After encountering this issue, I retested it on another PC that does not have the AWS CLI installed. The unit test failed during the Insert operation.

Previously, the tests might have passed due to some interference from the AWS CLI. However, under normal conditions, the tests should have failed.

For reference, please see: AWS Java SDK Developer Guide - Credentials.

In simple terms: It seems the AWS SDK automatically reads the SAS token temporarily stored in the AWS CLI.

[FANNG1]

The main problem of the current implementation is passing Azure secret key not the token(token name is not correct) to the client side, The StorageSharedKeyCredential key problems exists in the client side not the Iceberg REST server side, which means we could update the Iceberg version when testing. or update the Iceberg version in the server side is also ok but I prefer to do it in another PR

[FANNG1]

After correct the token name for Iceberg, I still failed to run pass the IT for auth failure, seems the token is not generated correctly(I also failed to do the Azure operation with the generated token in the python client). could you dig out the reason?

[orenccl]

Updated a version that corrects the token key and passes the IT test.
However, this version's SAS token does not specify a path.
Currently, specifying a path in the SAS token causes it to fail. I am still working on resolving this issue.

[FANNG1]

LGTM, just one comment, @yuqi1129 do you have time to review? Some changes related to fileset azure properties.

[FANNG1]

@orenccl , merged to main, it's an import feature for Gravitino, thanks for your work! There are some subsequent issues related like support AzureSecertKeyCredential, support ADLS credential in python client, do you want to continue work on it?

[orenccl]

@FANNG1
Thank you for providing help and reviewing!
Yes, I would like to continue working on it. I can open an issue later today, or if you create it first, please tag me and assign it to me!

[FANNG1]

@FANNG1 Thank you for providing help and reviewing! Yes, I would like to continue working on it. I can open an issue later today, or if you create it first, please tag me and assign it to me!

thanks, you could create the issue when you are free

[orenccl]

I think when we implement AzureSecretKeyCredential, we need to upgrade the Iceberg version.
We should upgrade to at least version 1.6.0, as it supports AzureSecretKeyCredential.

For supporting ADLS credentials in the Python client, should we refer to PR #5209 ?

[FANNG1]

I think when we implement AzureSecretKeyCredential, we need to upgrade the Iceberg version. We should upgrade to at least version 1.6.0, as it supports AzureSecretKeyCredential.

For supporting ADLS credentials in the Python client, should we refer to PR #5209 ?

Update Iceberg version to 1.6.0 seems not required because the decrease problems exists in Iceberg REST client side (Spark or Flink) not Iceberg REST server side, but whether updating Iceberg or not both ok to me.

For ADLS credentials in Python client, please refer to #5890