Skip to content

Commit

Permalink
[#6070][#5649] docs(core): add credential vending document (#6071)
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?
move credential vending related document from iceberg-rest-server part
to a separate file, then fileset could refer to it.

### Why are the changes needed?
Fix: #6070 
Fix: #5649 

### Does this PR introduce _any_ user-facing change?
no

### How was this patch tested?
just document
  • Loading branch information
FANNG1 authored Jan 7, 2025
1 parent b3a848b commit 32df91f
Show file tree
Hide file tree
Showing 3 changed files with 215 additions and 45 deletions.
27 changes: 23 additions & 4 deletions docs/hadoop-catalog.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,12 @@ Hadoop 3. If there's any compatibility issue, please create an [issue](https://g

Besides the [common catalog properties](./gravitino-server-config.md#apache-gravitino-catalog-properties-configuration), the Hadoop catalog has the following properties:

| Property Name | Description | Default Value | Required | Since Version |
|---------------|-------------------------------------------------|---------------|----------|---------------|
| `location` | The storage location managed by Hadoop catalog. | (none) | No | 0.5.0 |
| Property Name | Description | Default Value | Required | Since Version |
|------------------------|----------------------------------------------------|---------------|----------|------------------|
| `location` | The storage location managed by Hadoop catalog. | (none) | No | 0.5.0 |
| `credential-providers` | The credential provider types, separated by comma. | (none) | No | 0.8.0-incubating |

Please refer to [Credential vending](./security/credential-vending.md) for more details about credential vending.

Apart from the above properties, to access fileset like HDFS, S3, GCS, OSS or custom fileset, you need to configure the following extra properties.

Expand All @@ -50,6 +53,8 @@ Apart from the above properties, to access fileset like HDFS, S3, GCS, OSS or cu
| `s3-access-key-id` | The access key of the AWS S3. | (none) | Yes if it's a S3 fileset. | 0.7.0-incubating |
| `s3-secret-access-key` | The secret key of the AWS S3. | (none) | Yes if it's a S3 fileset. | 0.7.0-incubating |

Please refer to [S3 credentials](./security/credential-vending.md#s3-credentials) for credential related configurations.

At the same time, you need to place the corresponding bundle jar [`gravitino-aws-bundle-${version}.jar`](https://repo1.maven.org/maven2/org/apache/gravitino/gravitino-aws-bundle/) in the directory `${GRAVITINO_HOME}/catalogs/hadoop/libs`.

#### GCS fileset
Expand All @@ -60,6 +65,8 @@ At the same time, you need to place the corresponding bundle jar [`gravitino-aws
| `default-filesystem-provider` | The name default filesystem providers of this Hadoop catalog if users do not specify the scheme in the URI. Default value is `builtin-local`, for GCS, if we set this value, we can omit the prefix 'gs://' in the location. | `builtin-local` | No | 0.7.0-incubating |
| `gcs-service-account-file` | The path of GCS service account JSON file. | (none) | Yes if it's a GCS fileset. | 0.7.0-incubating |

Please refer to [GCS credentials](./security/credential-vending.md#gcs-credentials) for credential related configurations.

In the meantime, you need to place the corresponding bundle jar [`gravitino-gcp-bundle-${version}.jar`](https://repo1.maven.org/maven2/org/apache/gravitino/gravitino-gcp-bundle/) in the directory `${GRAVITINO_HOME}/catalogs/hadoop/libs`.

#### OSS fileset
Expand All @@ -72,6 +79,8 @@ In the meantime, you need to place the corresponding bundle jar [`gravitino-gcp-
| `oss-access-key-id` | The access key of the Aliyun OSS. | (none) | Yes if it's a OSS fileset. | 0.7.0-incubating |
| `oss-secret-access-key` | The secret key of the Aliyun OSS. | (none) | Yes if it's a OSS fileset. | 0.7.0-incubating |

Please refer to [OSS credentials](./security/credential-vending.md#oss-credentials) for credential related configurations.

In the meantime, you need to place the corresponding bundle jar [`gravitino-aliyun-bundle-${version}.jar`](https://repo1.maven.org/maven2/org/apache/gravitino/gravitino-aliyun-bundle/) in the directory `${GRAVITINO_HOME}/catalogs/hadoop/libs`.


Expand All @@ -84,6 +93,8 @@ In the meantime, you need to place the corresponding bundle jar [`gravitino-aliy
| `azure-storage-account-name ` | The account name of Azure Blob Storage. | (none) | Yes if it's a Azure Blob Storage fileset. | 0.8.0-incubating |
| `azure-storage-account-key` | The account key of Azure Blob Storage. | (none) | Yes if it's a Azure Blob Storage fileset. | 0.8.0-incubating |

Please refer to [ADLS credentials](./security/credential-vending.md#adls-credentials) for credential related configurations.

Similar to the above, you need to place the corresponding bundle jar [`gravitino-azure-bundle-${version}.jar`](https://repo1.maven.org/maven2/org/apache/gravitino/gravitino-azure-bundle/) in the directory `${GRAVITINO_HOME}/catalogs/hadoop/libs`.

:::note
Expand Down Expand Up @@ -146,7 +157,8 @@ The Hadoop catalog supports creating, updating, deleting, and listing schema.
| `authentication.impersonation-enable` | Whether to enable impersonation for this schema of the Hadoop catalog. | The parent(catalog) value | No | 0.6.0-incubating |
| `authentication.type` | The type of authentication for this schema of Hadoop catalog , currently we only support `kerberos`, `simple`. | The parent(catalog) value | No | 0.6.0-incubating |
| `authentication.kerberos.principal` | The principal of the Kerberos authentication for this schema. | The parent(catalog) value | No | 0.6.0-incubating |
| `authentication.kerberos.keytab-uri` | The URI of The keytab for the Kerberos authentication for this scheam. | The parent(catalog) value | No | 0.6.0-incubating |
| `authentication.kerberos.keytab-uri` | The URI of The keytab for the Kerberos authentication for this schema. | The parent(catalog) value | No | 0.6.0-incubating |
| `credential-providers` | The credential provider types, separated by comma. | (none) | No | 0.8.0-incubating |

### Schema operations

Expand All @@ -166,6 +178,13 @@ Refer to [Schema operation](./manage-fileset-metadata-using-gravitino.md#schema-
| `authentication.type` | The type of authentication for Hadoop catalog fileset, currently we only support `kerberos`, `simple`. | The parent(schema) value | No | 0.6.0-incubating |
| `authentication.kerberos.principal` | The principal of the Kerberos authentication for the fileset. | The parent(schema) value | No | 0.6.0-incubating |
| `authentication.kerberos.keytab-uri` | The URI of The keytab for the Kerberos authentication for the fileset. | The parent(schema) value | No | 0.6.0-incubating |
| `credential-providers` | The credential provider types, separated by comma. | (none) | No | 0.8.0-incubating |

Credential providers can be specified in several places, as listed below. Gravitino checks the `credential-provider` setting in the following order of precedence:

1. Fileset properties
2. Schema properties
3. Catalog properties

### Fileset operations

Expand Down
Loading

0 comments on commit 32df91f

Please sign in to comment.