Add feature to ignore Iceberg tables #185

javsanbel2 · 2024-11-27T11:32:50Z

Fixes #

…m-opensource:ExpediaGroup/beekeeper into feature/prevent-actions-on-iceberg-tables

Integrated LocalStack with containers. Rule is from Junit 4

…ionTest

…aryIntegrationTest

...rc/main/java/com/expediagroup/beekeeper/metadata/cleanup/handler/ExpiredMetadataHandler.java

...java/com/expediagroup/beekeeper/scheduler/apiary/filter/IcebergTableListenerEventFilter.java

…eeper/scheduler/apiary/filter/IcebergTableListenerEventFilter.java Co-authored-by: Jay Green-Stevens <[email protected]>

...piary/src/main/java/com/expediagroup/beekeeper/scheduler/apiary/service/SchedulerApiary.java

.../com/expediagroup/beekeeper/scheduler/apiary/filter/IcebergTableListenerEventFilterTest.java

…ithub.com/ExpediaGroup/beekeeper into feature/prevent-actions-on-iceberg-tables

...piary/src/main/java/com/expediagroup/beekeeper/scheduler/apiary/service/SchedulerApiary.java

jmnunezizu · 2024-11-27T16:55:09Z

...er-cleanup/src/main/java/com/expediagroup/beekeeper/cleanup/validation/IcebergValidator.java

+      Map<String, String> tableParameters = client.getTableProperties(databaseName, tableName);
+
+      if (isIcebergTablePredicate.test(tableParameters)) {
+        throw new BeekeeperIcebergException(


nit: Maybe using IllegalStateException would have been clearer.

We are purposely using a custom exception to handle the exception in the main class later on.

...er-core/src/main/java/com/expediagroup/beekeeper/core/predicate/IsIcebergTablePredicate.java

...rc/main/java/com/expediagroup/beekeeper/metadata/cleanup/handler/ExpiredMetadataHandler.java

...java/com/expediagroup/beekeeper/scheduler/apiary/filter/IcebergTableListenerEventFilter.java

CHANGELOG.md

Co-authored-by: Jay Green-Stevens <[email protected]>

CHANGELOG.md

Co-authored-by: Jay Green-Stevens <[email protected]>

Hamza Jugon and others added 30 commits November 20, 2024 12:10

Updated to BK-core

8a00255

Updated to path-cleanup

6baeee4

Update PagingCleanupServiceTest.java

7090dc4

cleanup

0f99aa6

Merge branch 'feature/prevent-actions-on-iceberg-tables' of github.co…

9bbd52d

…m-opensource:ExpediaGroup/beekeeper into feature/prevent-actions-on-iceberg-tables

cleanup 2

1e339c9

main business logic

62c68d5

adding exception

b6c718f

Add DB & table name to exception message

c5ad343

Update IcebergValidator.java

fd454a8

Create IcebergValidatorTest.java

e41ac6c

Update HiveMetadataCleanerTest.java

a2939d5

Updating and adding S3PathCleaner tests

223e086

Integrated LocalStack with containers. Rule is from Junit 4

Adding IcebergValidator to constructors

1f6e360

Updating Junit imports

1905b0a

Update SchedulerApiaryTest.java

efca2c9

Update CommonBeans

d16bc0a

clean-up add comment

9bf7248

Remove extra deletion

4c45de2

adding beans

61c2f88

fix tests

4e0b82b

fixing it tests for metadata cleanup

e548873

fix path cleanup

8b1ca85

fix main problem with tests

631502b

Fix BeekeeperDryRunPathCleanupIntegrationTest

c1a7c96

revert changes to fix BeekeeperExpiredMetadataSchedulerApiaryIntegrat…

90c2871

…ionTest

Added missing properties to fix BeekeeperUnreferencedPathSchedulerApi…

45fcc26

…aryIntegrationTest

Add integration test for metadatacleanup

06914d1

Update metadataHandler to catch beekeeperException

a09e9f1

cleanup

8c1ce38

Hamza Jugon added 2 commits November 27, 2024 15:03

Revert versions used for testing & changelog

f300e60

Revert testing version

bacd477

patduin reviewed Nov 27, 2024

View reviewed changes

...rc/main/java/com/expediagroup/beekeeper/metadata/cleanup/handler/ExpiredMetadataHandler.java Outdated Show resolved Hide resolved

patduin reviewed Nov 27, 2024

View reviewed changes

...java/com/expediagroup/beekeeper/scheduler/apiary/filter/IcebergTableListenerEventFilter.java Outdated Show resolved Hide resolved