[#5831] fix(CLI): Fix CLi gives unexpected output when input tag set command #5897

Abyss-lord · 2024-12-18T03:52:41Z

What changes were proposed in this pull request?

Running the command gcli tag set --metalake metalake_demo or gcli tag remove --metalake metalake_demo gives unexpected output.it should give some help information.

Why are the changes needed?

Fix: #5831

Does this PR introduce any user-facing change?

NO

How was this patch tested?

local test + UT.

gcli tag set -m demo_metalake
# Missing --tag option.

gcli tag set -m demo_metalake --tag tagA
# Missing --name option.

gcli tag remove -m demo_metalake
# Missing --tag option.

gcli tag remove -m demo_metalake --tag tagA
# Missing --name option.

gcli tag set --tag tagA --property test
# Command cannot be executed. The set command only supports configuring tag properties or attaching tags to entity.

gcli tag set --tag tagA --value val1
# Command cannot be executed. The set command only supports configuring tag properties or attaching tags to entity.

gcli tag set --metalake demo_metalake --name Hive_catalog --tag tagB tagC
# Hive_catalog now tagged with tagA,tagB,tagC

gcli tag remove --metalake demo_metalake --name Hive_catalog --tag tagA tagC
# Hive_catalog removed tag(s): [tagA, tagC], now tagged with [tagB]

gcli tag remove --metalake demo_metalake --name Hive_catalog --tag tagA tagB tagC
# Hive_catalog removed tag(s): [tagA, tagC], now tagged with []

Abyss-lord · 2024-12-18T03:54:52Z

hi @justinmclean @xunliu , Could you please take a look when you have time?

clients/cli/src/main/java/org/apache/gravitino/cli/ErrorMessages.java

clients/cli/src/main/java/org/apache/gravitino/cli/GravitinoCommandLine.java

…a tag Running the command gcli tag set --metalake metalake_demo or gcli tag remove --metalake metalake_demo gives unexpected output.it should give some help information.

…a tag fix failed test case.

…a tag fix some problems according to reviewer.

…a tag Change the code due to new pr merged.

justinmclean · 2024-12-19T23:36:10Z

This is not making 100% sense to me. There are a couple of things I think can be improved here:

The output of bin/gcli.sh tag set --metalake demo_metalake should be Missing --name option. as the name argument is missing. The existing code should already do this before this function is called?
The hasEntity logic seems wrong, and I am not sure what you are trying to check here. Could teh function have a better name?
In hasEntity I'm not sure why you are checking 3 parts of the name when all sorts of entities can be tagged. Also columns will have a four part name.
Also there are hasCatalogName, hasTableName, etc methods that would be better to use, if where needed to be used?

…pache#5783) ### What changes were proposed in this pull request? Add grant and revoke privileges to the Gravitino CLI. ### Why are the changes needed? To complete the role commands. Fix: apache#5162 ### Does this PR introduce _any_ user-facing change? No but it adds two more commands. ### How was this patch tested? Tested locally.

…pache#5903) ### What changes were proposed in this pull request? Changed to exit with -1 if an error occurs. Not that testing code that calls System.exit can be difficult, so during testing it thows an exception and doesn't exit the VM. Testing also uncovered one bug in updating column default values and this was fixed. ### Why are the changes needed? So scripts can test if the gcli command works. Fix: apache#5896 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Tested locally. --------- Co-authored-by: Shaofeng Shi <[email protected]>

…che#5919) ### What changes were proposed in this pull request? Fix to grant privilege for the metalake ### Why are the changes needed? Fix: apache#5892 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Add a UT.

…ce implement (apache#5918) ### What changes were proposed in this pull request? 1. Remove AuthorizationPlugin single instance implement in the `BaseAuthorizaiton.java` 2. Updatge ITs codes. ### Why are the changes needed? Fix: apache#5916 ### Does this PR introduce _any_ user-facing change? N/A ### How was this patch tested? CI Passed.

### What changes were proposed in this pull request? fix comments to have correct entity ### Why are the changes needed? fix copy and paste error Fix: # N/A ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? N/A

…n user is not specified in Gravitino CLI (apache#5921) ### What changes were proposed in this pull request? Fix missing leading error message when user is not specified in Gravitino CLI, include all commands. ### Why are the changes needed? Fix: apache#5828 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? ```bash bin/gcli.sh user create --metalake demo_metalake # Missing --user option. bin/gcli.sh user details --metalake demo_metalake # Missing --user option. bin/gcli.sh user delete --metalake demo_metalake # Missing --user option. bin/gcli.sh user list --metalake demo_metalake # anonymous ```

…n group is not specified (apache#5920) ### What changes were proposed in this pull request? Slightly missing leading error message when group is not specified, it should give some hints to user. ### Why are the changes needed? Fix: apache#5829 ### Does this PR introduce _any_ user-facing change? NO ### How was this patch tested? ```bash bin/gcli.sh group create --metalake demo_metalake # Missing --group option. bin/gcli.sh group details --metalake demo_metalake # Missing --group option. bin/gcli.sh group delete --metalake demo_metalake # Missing --group option. ```

…down (apache#5935) ### What changes were proposed in this pull request? Avoid other catalogs' privileges are pushed down. For example, if a role has two catalogs. One catalog has select table, the other catalog has create table. The plugin will make the role can create and select table at the same time. ### Why are the changes needed? Fix: apache#5934 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Add a UT

…ache#5938) ### What changes were proposed in this pull request? Support Azure account key credential ### Why are the changes needed? Fix: apache#5894 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Unit Test IcebergRESTADLSAccountKeyIT at iceberg 1.6.0

…pache#5923) ### What changes were proposed in this pull request? Removed the systematic validations in Iceberg Config and modified the instantiation of the warehouse attribute and uri attribute in the IcebergCatalogWrapper to conform with the new possibility (REST). ### Why are the changes needed? The bug was blocking the creation of a rest catalog. Fix: [apache#5756](apache#5756) ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? ./gradlew build && compileDistribution && assembleDistribution + creation of the docker image there : https://hub.docker.com/r/fsalhi2/gravitino Started gravitino with such configs : ``` ### Gravitino General Settings gravitino.auxService.names = iceberg-rest gravitino.iceberg-rest.classpath = iceberg-rest-server/libs, iceberg-rest-server/conf ### HTTP Server gravitino.iceberg-rest.host = 0.0.0.0 gravitino.iceberg-rest.httpPort = 9001 ### Storage gravitino.iceberg-rest.io-impl = org.apache.iceberg.aws.s3.S3FileIO gravitino.iceberg-rest.s3-access-key-id = XXXXX gravitino.iceberg-rest.s3-secret-access-key = XXXXXX gravitino.iceberg-rest.s3-path-style-access = true gravitino.iceberg-rest.s3-endpoint = http://minio:9000/ gravitino.iceberg-rest.s3-region = us-east-1 ### JDBC gravitino.iceberg-rest.catalog-backend = jdbc gravitino.iceberg-rest.uri = jdbc:mysql://mysql:3306/ gravitino.iceberg-rest.warehouse = s3://lake/catalog gravitino.iceberg-rest.jdbc.user = root gravitino.iceberg-rest.jdbc.password = XXXXXX gravitino.iceberg-rest.jdbc-driver = com.mysql.cj.jdbc.Driver ``` Was able to create a catalog through Web UI and start working on the scheme.

…5908) ### What changes were proposed in this pull request? This PR adds the ModelOperationDispatcher logic in core. ### Why are the changes needed? This is a part of work to support model management. Fix: apache#5794 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Added UTs to test.

### What changes were proposed in this pull request? Fix the wrong possible values. Remove "COLUMN" from metadataObjectType possible values in get/set owner. ### Why are the changes needed? Fix: apache#5911 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Just documents.

…Delete a table with a missing table name (apache#5906) ### What changes were proposed in this pull request? Fix CLI throws an obscure error when Delete a table with a missing table name.it should give clearer hints. ### Why are the changes needed? Fix: apache#5827 ### Does this PR introduce _any_ user-facing change? NO ### How was this patch tested? #### local bash test ```bash bin/gcli.sh table delete --metalake demo_metalake Missing required argument(s): catalog, schema, table bin/gcli.sh table delete --metalake demo_metalake --name Hive_catalog # output: Missing required argument(s): schema, table bin/gcli.sh table delete --metalake demo_metalake --name Hive_catalog.default # output: Missing required argument(s): table bin/gcli.sh table details --metalake demo_metalake --name Hive_catalog.default # output # Malformed entity name. # Missing required argument(s): table ``` #### Unit test add some test case to test whether the command fuse is executed correctly.

…I when missing schema (apache#5939) ### What changes were proposed in this pull request? Fix schema arguments validation just like table commands. running command like `gcli schema details --metalake metalake_demo --name catalog_postgres -I` give Missleading error message as below. ```bash Malformed entity name. Invalid string to encode: null ``` It should display a friendly error message. ### Why are the changes needed? Fix: apache#5926 ### Does this PR introduce _any_ user-facing change? NO ### How was this patch tested? ```bash bin/gcli.sh schema details --m demo_metalake -i # output # Missing --name option. # Missing --name option. # Missing required argument(s): catalog, schema bin/gcli.sh schema details --m demo_metalake --name Hive_catalog -i # output # Malformed entity name. # Missing required argument(s): schema bin/gcli.sh schema details --m demo_metalake --name Hive_catalog.default -i # ouput: default,Default Hive database bin/gcli.sh schema list --m demo_metalake -i # output: # Missing --name option. # Missing required argument(s): catalog bin/gcli.sh schema list --m demo_metalake -i --name Hive_catalog # correct output ```

…on and rename catalog (apache#5949) ### What changes were proposed in this pull request? Fix the issue of renaming catalogs or metalakes. ### Why are the changes needed? Fix: apache#5947 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Add UT.

apache#5945) ### What changes were proposed in this pull request? Fix columns details command produces no output, when command without --audit option, cli should tell user that command is not support. ### Why are the changes needed? Fix: apache#5928 ### Does this PR introduce _any_ user-facing change? NO ### How was this patch tested? local test.

…tput (apache#5941) ### What changes were proposed in this pull request? If a user has no roles then no output is produced from `user\group details` command, it should give some help information. ### Why are the changes needed? Fix: apache#5929 ### Does this PR introduce _any_ user-facing change? NO ### How was this patch tested? ```bash bin/gcli.sh user details -m demo_metalake --user test_user # output: User has no roles. bin/gcli.sh group details -m demo_metalake --group group_no_role # output: Groups has no roles. ``` --------- Co-authored-by: Qiming Teng <[email protected]>

…ark connector (apache#5952) ### What changes were proposed in this pull request? 1. Most code work is implemented in apache#5938 apache#5737 including catalog properties convert and add Iceberg azure bundle jar, this PR mainly about test and document. 2. Remove hidden properties of the cloud secret key from the Iceberg catalog, as Gravitino doesn't have an unified security management yet and Iceberg REST server need to fetch catalog cloud properties to initiate `IcebergWrapper` dymaticly. Another benefit is spark connector does not need to specify the secret key explictly. Supports ADLS for Iceberg catalog and spark connector ### Why are the changes needed? Fix: apache#5954 ### Does this PR introduce _any_ user-facing change? Yes, the user no need to specify the cloud secret key in spark connector. ### How was this patch tested? test in local enviroment

### What changes were proposed in this pull request? Revert helm-chart related descriptions in `docs/how-to-use-the-playground.md`. ### Why are the changes needed? Make documentation match `README.md` of `apache/gravitino-playground` . ### Does this PR introduce _any_ user-facing change? Yes, documentation is affacted. ### How was this patch tested? Manually.

…e#5786) ### What changes were proposed in this pull request? 1. Add Chain auth plugin module 1. Add auth common module 3. Add Chain authorization Ranger Hive and Ranger HDFS ITs ### Why are the changes needed? Fix: apache#5775 ### Does this PR introduce _any_ user-facing change? N/A ### How was this patch tested? Add ITs

…nagement of paimon table in paimon spark connector (apache#5860) ### What changes were proposed in this pull request? Add tests for partitionManagement of paimon table in paimon spark connector ### Why are the changes needed? Fix: apache#4714 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? New ITs. --------- Co-authored-by: caican <[email protected]>

…handle methods (apache#5972) ### What changes were proposed in this pull request? refactor the validation logic of all entities and add test case, just like validation of table command apache#5906 . A hint is provided when the user's output is missing the required arguments. for example: ```bash gcli column list -m demo_metalake, --name Hive_catalog # Malformed entity name. # Missing required argument(s): schema, table gcli column details -m demo_metalake, --name Hive_catalog --audit # Malformed entity name. # Missing required argument(s): schema, table, column gcli user delete -m demo_metalake Missing --user option. ``` Currently, the Role command needs to be refactored and opened as a separate issue ### Why are the changes needed? Fix: apache#5861 ### Does this PR introduce _any_ user-facing change? NO ### How was this patch tested? local test

) ### What changes were proposed in this pull request? Improve output when CLI add an unknown tags. ### Why are the changes needed? Fix: apache#5930 ### Does this PR introduce _any_ user-facing change? NO ### How was this patch tested? local test

…atalog (apache#5682) ### What changes were proposed in this pull request? Support credential vending for fileset catalog 1. add `credential-providers` properties for the fileset catalog, schema, and fileset. 2. try to get `credential-providers` from the order of fileset, schema, and catalog. 3. The user could set multi-credential providers ### Why are the changes needed? Fix: apache#5620 ### Does this PR introduce _any_ user-facing change? will add document after this PR is merged ### How was this patch tested? Add IT and test with local setup Gravitino server

…ent (apache#5948) ### What changes were proposed in this pull request? This PR adds the server-side REST endpoint for model management. ### Why are the changes needed? This is a part of model management for Gravitino. Fix: apache#5817 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Add UTs for this PR.

…tion-common module (apache#5994) ### What changes were proposed in this pull request? Move the JdbcAuthorizationPlugin to authorization-common module ### Why are the changes needed? Fix: apache#5993 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Just refactor.

### What changes were proposed in this pull request? Make some methods public, let some classes to reuse some methods. ### Why are the changes needed? This is a follow-up PR. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? No need. Just refactor.

…apache#5999) ### What changes were proposed in this pull request? Add more configurations for the config API ### Why are the changes needed? Fix apache#5987 Front end can use this config option to optimize the user experience. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? ![image](https://github.com/user-attachments/assets/a5f26c61-9e10-4dfa-bbcd-54cb887b1b71)

…er (apache#5995) ### What changes were proposed in this pull request? add credential cache for Gravitino server, not support for Iceberg rest server yet. ### Why are the changes needed? Fix: apache#4398 ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? testing in local env, get credential from Gravitino server and see whether it's fetched from remote or local cache

…core jars that does not contains hadoop-{aws,gcp,aliyun,azure} (apache#5806) ### What changes were proposed in this pull request? Provide another kind of bundle jars that does not contains hadoop-{aws,gcp,aliyun,azure} like aws-mini, gcp-mini. ### Why are the changes needed? To make it works in a wide range of Hadoop version Fix: apache#5585 ### Does this PR introduce _any_ user-facing change? N/A ### How was this patch tested? Existing UTs and ITs

…ot equal to the credential type of ADLSTokenCredential (apache#5990) ### What changes were proposed in this pull request? ADLSTokenCredentialProvider use the credential type from ADLSTokenCredential ### Why are the changes needed? Fix: apache#5989 ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? add test

…ng multiple values (apache#5988) ### What changes were proposed in this pull request? In `GravitinoOptions`, the role option supports multiple values, but the handleRoleCommand implementation currently only processes a single role value. CLI should support create and delete multiple roles simultaneously. ### Why are the changes needed? Fix: apache#5985 ### Does this PR introduce _any_ user-facing change? NO ### How was this patch tested? local test + UT ```bash gcli role create -m demo_metalake --role role1 role2 role3 # role1, role2, role3 created gcli role delete -m demo_metalake --role role1 role2 role3 # role1, role2, role3 deleted. gcli role delete -m demo_metalake --role role1 role2 role3 unknown # role1, role2, role3 deleted, but unknown is not deleted. gcli role details -m demo_metalake # Missing --role option. gcli role details -m demo_metalake --role roleA roleB # Exception in thread "main" java.lang.IllegalArgumentException: details requires only one role, but multiple are currently passed. ```

…6015) ### What changes were proposed in this pull request? In the `ListMetalakes` and `ListCatalogs` methods, retain the use of `output(metalakes)` and `output(catalogs)`. If metalakes or catalogs are empty arrays, they will be handled by the `output` method in `PlainFormat` and `TableFormat`. ### Why are the changes needed? Issue: apache#6014 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? No.

…a tag 1. Change validation code; 2. add method to FullName; 3. fix some error of ErrorMessages;

…a tag Add some test case.

Abyss-lord · 2024-12-29T05:17:19Z

Hi @justinmclean @shaofengshi , I’ve finished updating the code. Please take a look at the PR again when you have time.

clients/cli/src/main/java/org/apache/gravitino/cli/FullName.java

clients/cli/src/main/java/org/apache/gravitino/cli/commands/UntagEntity.java

clients/cli/src/main/java/org/apache/gravitino/cli/GravitinoCommandLine.java

…a tag fix some error

Abyss-lord · 2024-12-30T02:26:17Z

Hi @justinmclean , I’ve finished updating the code. Please take a look at the PR again when you have time.

justinmclean

Looks good, thanks for this.

justinmclean reviewed Dec 18, 2024

View reviewed changes

clients/cli/src/main/java/org/apache/gravitino/cli/ErrorMessages.java Outdated Show resolved Hide resolved

justinmclean reviewed Dec 18, 2024

View reviewed changes

clients/cli/src/main/java/org/apache/gravitino/cli/GravitinoCommandLine.java Outdated Show resolved Hide resolved

justinmclean reviewed Dec 18, 2024

View reviewed changes

clients/cli/src/main/java/org/apache/gravitino/cli/GravitinoCommandLine.java Outdated Show resolved Hide resolved

justinmclean assigned Abyss-lord Dec 18, 2024

Abyss-lord force-pushed the fix-tag-command branch 2 times, most recently from 2857789 to 328bc1f Compare December 19, 2024 05:53

Abyss-lord added 4 commits December 19, 2024 19:02

[apache#5831] fix(CLI): Fix CLi gives unexpected output when setting …

ee7f792

…a tag Running the command gcli tag set --metalake metalake_demo or gcli tag remove --metalake metalake_demo gives unexpected output.it should give some help information.

[apache#5831] fix(CLI): Fix CLi gives unexpected output when setting …

e9d6f0c

…a tag fix failed test case.

[apache#5831] fix(CLI): Fix CLi gives unexpected output when setting …

e7f16d3

…a tag fix some problems according to reviewer.

[apache#5831] fix(CLI): Fix CLi gives unexpected output when setting …

d435355

…a tag Change the code due to new pr merged.

Abyss-lord force-pushed the fix-tag-command branch from 328bc1f to d435355 Compare December 19, 2024 11:02

justinmclean and others added 17 commits December 29, 2024 13:14

FANNG1 and others added 19 commits December 29, 2024 13:14

[apache#5831] fix(CLI): Fix CLi gives unexpected output when setting …

cfe6691

…a tag 1. Change validation code; 2. add method to FullName; 3. fix some error of ErrorMessages;

[apache#5831] fix(CLI): Fix CLi gives unexpected output when setting …

79ac312

…a tag Add some test case.

Merge branch 'main' into fix-tag-command

d0f79d8

Abyss-lord requested a review from justinmclean December 29, 2024 05:16

justinmclean reviewed Dec 30, 2024

View reviewed changes

clients/cli/src/main/java/org/apache/gravitino/cli/FullName.java Outdated Show resolved Hide resolved

justinmclean reviewed Dec 30, 2024

View reviewed changes

clients/cli/src/main/java/org/apache/gravitino/cli/commands/UntagEntity.java Outdated Show resolved Hide resolved

justinmclean reviewed Dec 30, 2024

View reviewed changes

clients/cli/src/main/java/org/apache/gravitino/cli/GravitinoCommandLine.java Show resolved Hide resolved

[apache#5831] fix(CLI): Fix CLi gives unexpected output when setting …

d4ba7cc

…a tag fix some error

Abyss-lord requested a review from justinmclean December 30, 2024 02:25

justinmclean approved these changes Dec 31, 2024

View reviewed changes

justinmclean merged commit 9a5fef9 into apache:main Dec 31, 2024
25 checks passed

Abyss-lord deleted the fix-tag-command branch December 31, 2024 01:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[#5831] fix(CLI): Fix CLi gives unexpected output when input tag set command #5897

[#5831] fix(CLI): Fix CLi gives unexpected output when input tag set command #5897

Abyss-lord commented Dec 18, 2024 •

edited

Loading

Abyss-lord commented Dec 18, 2024

justinmclean commented Dec 19, 2024 •

edited

Loading

Abyss-lord commented Dec 29, 2024

Abyss-lord commented Dec 30, 2024

justinmclean left a comment

[#5831] fix(CLI): Fix CLi gives unexpected output when input tag set command #5897

[#5831] fix(CLI): Fix CLi gives unexpected output when input tag set command #5897

Conversation

Abyss-lord commented Dec 18, 2024 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Abyss-lord commented Dec 18, 2024

justinmclean commented Dec 19, 2024 • edited Loading

Abyss-lord commented Dec 29, 2024

Abyss-lord commented Dec 30, 2024

justinmclean left a comment

Choose a reason for hiding this comment

Abyss-lord commented Dec 18, 2024 •

edited

Loading

justinmclean commented Dec 19, 2024 •

edited

Loading