[#5973] feat(hadoop-catalog): Support credential when using fileset catalog with cloud storage #5974

yuqi1129 · 2024-12-24T13:13:35Z

What changes were proposed in this pull request?

Support dynamic credential in obtaining cloud storage fileset.

Why are the changes needed?

Static key are not very safe, we need to optimize it.

Fix: #5973

Does this PR introduce any user-facing change?

N/A

How was this patch tested?

N/A

yuqi1129 · 2024-12-24T13:15:29Z

This PR depends on #5620, #5806 and #5971

...ts/filesystem-hadoop3/src/test/java/org/apache/gravitino/filesystem/hadoop/TestGvfsBase.java

yuqi1129 · 2024-12-27T12:01:55Z

@FANNG1 ,
Please help to solve the problem about token permission for OSS and S3, besides, GCS token for Java client seems to have some problems, please see: #6028

FANNG1 · 2024-12-29T01:09:00Z

Do you plan to support the static credential like S3SecretKey and some storage properties not included in the credential like s3-region in the new PR?

bundles/aliyun/src/main/java/org/apache/gravitino/oss/fs/OSSSessionCredentialProvider.java

...org/apache/gravitino/filesystem/hadoop/integration/test/GravitinoVirtualFileSystemGCSIT.java

yuqi1129 · 2024-12-30T12:01:22Z

@FANNG1 , Please help to solve the problem about token permission for OSS and S3, besides, GCS token for Java client seems to have some problems, please see: #6028

solved

yuqi1129 · 2024-12-30T12:17:13Z

Do you plan to support the static credential like S3SecretKey and some storage properties not included in the credential like s3-region in the new PR?

The current PR also support static credential as s3 endpoint(can replace s3-region) is a required parameter

jerryshao · 2025-01-06T03:26:02Z

bundles/aliyun/build.gradle.kts

@@ -39,6 +39,10 @@ dependencies {
  implementation(project(":catalogs:hadoop-common")) {
    exclude("*")
  }
+  implementation(project(":clients:client-java-runtime", configuration = "shadow"))


Again, why this should be implementation, AFAIK, the bundles are typically used with gvfs, we already included this client runtime in gvfs, what's the reason to do this here?

Let me think a bit.

Changed as suggested.

jerryshao · 2025-01-06T03:34:32Z

bundles/aliyun/src/main/java/org/apache/gravitino/oss/fs/OSSCredentialProvider.java

+
+  public OSSCredentialProvider(URI uri, Configuration conf) {
+    this.filesetIdentifier =
+        conf.get(GravitinoVirtualFileSystemConfiguration.GVFS_FILESET_IDENTIFIER);


How do you make sure the behavior is correct if you have multiple filesets with different FSs, have you verified?

If you have multiple filesets with the same FS, will they share the same CredentialProvider or use the different ones?

In this PR, each fileset will have a FileSystem instance.

It's better to verify the behavior there're multiple fileset for same CredentialProvider, because the CredentialProvider seems are shared by all filesets.

jerryshao · 2025-01-06T03:41:46Z

bundles/aliyun/src/main/java/org/apache/gravitino/oss/fs/OSSCredentialProvider.java

+    FilesetCatalog filesetCatalog = client.loadCatalog(catalog).asFilesetCatalog();
+
+    Fileset fileset = filesetCatalog.loadFileset(NameIdentifier.of(idents[2], idents[3]));
+    Credential[] credentials = fileset.supportsCredentials().getCredentials();


It actually returns credentials, why do you insist on XXXCredentialProvider, not XXXCredentialsProvider?

We will eventually only use one of them. I have no preference on XXXCredentialProvider or XXXCredentialsProvider, If it matters, I can change it.

We will soon support one fileset mapping to multiple storage locations in the next version, which will return multiple credentials, please think of this scenarios carefully, and don't do the assumptions based on your own, and refactor them again in the next version.

OKay, I took it.

jerryshao · 2025-01-06T03:43:06Z

bundles/aliyun/src/main/java/org/apache/gravitino/oss/fs/OSSCredentialProvider.java

+          new DefaultCredentials(
+              configuration.get(Constants.ACCESS_KEY_ID),
+              configuration.get(Constants.ACCESS_KEY_SECRET));
+      return;


What if user didn't configure the AKSK? Should they still configure the AKSK when credential vending is enabled?

If we can't get any crendential from Gravitino server(dynamic or static), we will try using AKSK. If ASKS has not been set, errors will occur.

Should they still configure the AKSK when credential vending is enabled?

I prefer to keep the way that only support AKSK in version 0.7.0

Do you need a flag to enable credential vending on the client side like Iceberg client?

Have you added an option in the Icebert client to use credentials or not? I want to make credentials transparent to client, so I'm hesitant to add this option in the client side. @jerryshao do you have any comments on this point?

The iceberg community added an option in the Icebert client to control whether enable credential vending.

The iceberg community added an option in the Icebert client to control whether enable credential vending.

GVFS supports multiple storage at the same time and aims to shield the difference. I'm afraid the case is quite different from that in Iceberg rest server.

jerryshao · 2025-01-06T03:51:13Z

bundles/aliyun/src/main/java/org/apache/gravitino/oss/fs/OSSCredentialProvider.java

+            .findFirst();
+    if (dynamicCredential.isPresent()) {
+      return dynamicCredential;
+    }


If you use Optional, please use monad style (functional programming), not using if..., which is not elegant.

OK, let me make some adjustments.

FANNG1 · 2025-01-06T07:18:38Z

bundles/aliyun/src/main/java/org/apache/gravitino/oss/fs/OSSFileSystemProvider.java

@@ -60,7 +61,15 @@ public FileSystem getFileSystem(Path path, Map<String, String> config) throws IO
      hadoopConfMap.put(OSS_FILESYSTEM_IMPL, AliyunOSSFileSystem.class.getCanonicalName());
    }

+    if (!hadoopConfMap.containsKey(Constants.CREDENTIALS_PROVIDER_KEY)


please use a function, it's hard to understand

FANNG1 · 2025-01-06T07:37:27Z

bundles/gcp/src/main/java/org/apache/gravitino/gcs/fs/GCSCredentialProvider.java

+
+  @Override
+  public Configuration getConf() {
+    return this.configuration;


please remove this..

jerryshao · 2025-01-06T09:09:59Z

bundles/aliyun/build.gradle.kts

@@ -29,6 +29,7 @@ dependencies {
  compileOnly(project(":catalogs:catalog-common"))
  compileOnly(project(":catalogs:catalog-hadoop"))
  compileOnly(project(":core"))
+  compileOnly(project(":clients:client-java-runtime", configuration = "shadow"))


Why do you use the runtime jar for compileOnly dependency?

jerryshao · 2025-01-07T06:53:52Z

bundles/gcp/src/main/java/org/apache/gravitino/gcs/fs/GCSFileSystemProvider.java

+      if (accessTokenProvider.getAccessToken() != null) {
+        configuration.set(GCS_TOKEN_PROVIDER_IMPL, GCSCredentialsProvider.class.getName());
+      }
+    }


I think we should decide whether we should let users to set AKSK or credential file if credential vending is not enabled. IMO, I think we should not enable XXXCredentialsProvider is credential vending is not enabled in the server side; if it is enabled, then we should fully rely on credential vending, no need to configure the user-side credentials again.

WDYT? @FANNG1

+1, this is more clear, if credential vending is enabled in the server side, use server credentials; if not, use credentials in client side.

jerryshao · 2025-01-07T10:49:05Z

catalogs/hadoop-common/build.gradle.kts

@@ -23,6 +23,7 @@ plugins {

 // try to avoid adding extra dependencies because it is used by catalogs and connectors.
 dependencies {
+  implementation(project(":api"))


Why hadoop-common depends on api?

Gvfs credential provider inerface located in this module and it needs interface credential which in module api

Support using dynamic credential

c424d8e

yuqi1129 marked this pull request as draft December 24, 2024 13:13

yuqi1129 added 6 commits December 26, 2024 14:07

Merge branch 'main' of github.com:apache/gravitino into issue_5973

d7a031c

Fix again.

5b648e8

fix

0c61a48

Merge branch 'main' of github.com:datastrato/graviton into issue_5973

44425e9

fix

18f6ff6

Fix

8fc56ae

yuqi1129 marked this pull request as ready for review December 27, 2024 11:51

yuqi1129 commented Dec 27, 2024

View reviewed changes

...ts/filesystem-hadoop3/src/test/java/org/apache/gravitino/filesystem/hadoop/TestGvfsBase.java Outdated Show resolved Hide resolved

Fix

00fa098

yuqi1129 self-assigned this Dec 27, 2024

Fix

682705d

yuqi1129 requested a review from FANNG1 December 28, 2024 01:19

Fix

50a4d15

FANNG1 reviewed Dec 29, 2024

View reviewed changes

bundles/aliyun/src/main/java/org/apache/gravitino/oss/fs/OSSSessionCredentialProvider.java Outdated Show resolved Hide resolved

yuqi1129 added 4 commits December 29, 2024 18:34

Fix

7f0a99b

Merge branch 'main' of github.com:datastrato/graviton into issue_5973

fede4a8

fix conflict

22bea5c

fix

f4f2287

FANNG1 reviewed Dec 30, 2024

View reviewed changes

...org/apache/gravitino/filesystem/hadoop/integration/test/GravitinoVirtualFileSystemGCSIT.java Outdated Show resolved Hide resolved

fix

20f7ec6

Polish code.

e8814b0

yuqi1129 added 2 commits December 30, 2024 20:49

Merge branch 'main' of github.com:datastrato/graviton into issue_5973

9b7bf01

fix

06b192b

Merge remote-tracking branch 'me/issue_5973' into issue_5973

2d8f4dc

jerryshao reviewed Jan 6, 2025

View reviewed changes

resolve comments.

193f467

FANNG1 reviewed Jan 6, 2025

View reviewed changes

yuqi1129 added 2 commits January 6, 2025 16:21

fix

a1f0989

Merge remote-tracking branch 'me/issue_5973' into issue_5973

d75b67e

jerryshao reviewed Jan 6, 2025

View reviewed changes

yuqi1129 added 10 commits January 6, 2025 19:57

fix

4fb6e79

fix comments.

755a474

Merge remote-tracking branch 'me/issue_5973' into issue_5973

5251a85

fix

3bb9252

Optimize

614464f

fix

e481c8d

fix

0a97fc7

Merge branch '5472' of github.com:yuqi1129/gravitino into issue_5973

9bb28a8

optimize again.

7e74fc9

Merge remote-tracking branch 'me/issue_5973' into issue_5973

c853de2

jerryshao reviewed Jan 7, 2025

View reviewed changes

The new framework of credential vending in fileset.

571c1e9

jerryshao reviewed Jan 7, 2025

View reviewed changes

yuqi1129 added 9 commits January 7, 2025 23:27

Refactor the framework of using credential in fileset

e27f9cd

Merge branch 'main' of github.com:apache/gravitino into issue_5973

37a5735

fix

bad0c87

fix

ca51efb

fix

bf1290c

fix typo

a1b249f

fix the checkstyle problem.

a722080

fix

03395c8

fix

274ec3e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[#5973] feat(hadoop-catalog): Support credential when using fileset catalog with cloud storage #5974

[#5973] feat(hadoop-catalog): Support credential when using fileset catalog with cloud storage #5974

yuqi1129 commented Dec 24, 2024

yuqi1129 commented Dec 24, 2024

yuqi1129 commented Dec 27, 2024

FANNG1 commented Dec 29, 2024

yuqi1129 commented Dec 30, 2024

yuqi1129 commented Dec 30, 2024 •

edited

Loading

jerryshao Jan 6, 2025

yuqi1129 Jan 6, 2025

yuqi1129 Jan 6, 2025

jerryshao Jan 6, 2025

yuqi1129 Jan 6, 2025

FANNG1 Jan 6, 2025

jerryshao Jan 6, 2025

yuqi1129 Jan 6, 2025

jerryshao Jan 6, 2025

yuqi1129 Jan 6, 2025

jerryshao Jan 6, 2025

yuqi1129 Jan 6, 2025

yuqi1129 Jan 6, 2025

FANNG1 Jan 6, 2025

yuqi1129 Jan 6, 2025

FANNG1 Jan 6, 2025

yuqi1129 Jan 6, 2025

jerryshao Jan 6, 2025 •

edited

Loading

yuqi1129 Jan 6, 2025

FANNG1 Jan 6, 2025

FANNG1 Jan 6, 2025

yuqi1129 Jan 6, 2025

jerryshao Jan 6, 2025

jerryshao Jan 7, 2025

FANNG1 Jan 7, 2025

jerryshao Jan 7, 2025

yuqi1129 Jan 7, 2025

[#5973] feat(hadoop-catalog): Support credential when using fileset catalog with cloud storage #5974

Are you sure you want to change the base?

[#5973] feat(hadoop-catalog): Support credential when using fileset catalog with cloud storage #5974

Conversation

yuqi1129 commented Dec 24, 2024

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

yuqi1129 commented Dec 24, 2024

yuqi1129 commented Dec 27, 2024

FANNG1 commented Dec 29, 2024

yuqi1129 commented Dec 30, 2024

yuqi1129 commented Dec 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jerryshao Jan 6, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuqi1129 commented Dec 30, 2024 •

edited

Loading

jerryshao Jan 6, 2025 •

edited

Loading