Implement function to get updated partitions #24

huynguyent · 2024-02-05T12:46:20Z

Address issue #23

The logic for exclude_optimize_operations mentioned in the issue is more complicated than I expected. I'll implement it in another PR

I also had to regenerate the lock file because of this error when running poetry install

Installing dependencies from lock file
Warning: poetry.lock is not consistent with pyproject.toml. You may be getting improper dependencies. Run `poetry lock [--no-update]` to fix it.

Because levi depends on deltalake (^0.9) which doesn't match any versions, version solving failed.

MrPowers

LGTM!

zeotuan · 2024-09-05T07:56:05Z

levi/__init__.py

@@ -96,3 +99,13 @@ def boundary_parser(boundary):
    else:
        raise ValueError(
            f"{boundary} cannot be parsed.  Valid prefixes are <, <=, >, and >=.  Two values can be delimited with a dash.  Valid examples include <10kb and 10kb-5gb.")
+
+def updated_partitions(delta_table: DeltaTable, start_time: Optional[datetime.datetime] = None, end_time: Optional[datetime.datetime] = None) -> dict[str, str]:


The return type might be wrong here. I think this should have been List[dict[str, str]]

huynguyent added 4 commits February 5, 2024 22:40

Regenerate poetry lock

11842dc

Implement updated_partitions

a1c460f

Add test_updated_partitions_without_time_filter

417b23a

Add test_updated_partitions_with_time_filter

5f83d35

MrPowers self-requested a review February 5, 2024 12:55

MrPowers approved these changes Feb 5, 2024

View reviewed changes

MrPowers merged commit 180f88c into mrpowers-io:main Feb 5, 2024

huynguyent deleted the get-updated-partitions branch February 5, 2024 13:15

zeotuan reviewed Sep 5, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement function to get updated partitions #24

Implement function to get updated partitions #24

huynguyent commented Feb 5, 2024

MrPowers left a comment

zeotuan Sep 5, 2024

Implement function to get updated partitions #24

Implement function to get updated partitions #24

Conversation

huynguyent commented Feb 5, 2024

MrPowers left a comment

Choose a reason for hiding this comment

zeotuan Sep 5, 2024

Choose a reason for hiding this comment