[dagster-fivetran] Implement base poll method in FivetranClient #26060
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Nov 20, 2024
![graphite-app[bot]](https://avatars.githubusercontent.com/in/158384?s=60&v=4){kind=small}
python_modules/libraries/dagster-fivetran/dagster_fivetran/resources.py
Outdated
Show resolved
Hide resolved
maximearmstrong
force-pushed
the
maxime/implement-resync-base-method-fivetran-client
branch
from
cadbe37
to
90123af
Compare
maximearmstrong
force-pushed
the
maxime/implement-base-poll-method-fivetran-client
branch
from
469b919
to
6fff27d
Compare
dpeng817
approved these changes
Nov 20, 2024
| if connector.last_sync_completed_at > previous_sync_completed_at: | ||
| break | ||
|
|
||
| if poll_timeout and datetime.now() > poll_start + timedelta(seconds=poll_timeout): |
| # Sleep for the configured time interval before polling again. | ||
| time.sleep(poll_interval) | ||
|
|
||
| if not connector.is_last_sync_successful: |
| @@ -59,3 +61,10 @@ def test_basic_resource_request( | |||
| client.start_resync(connector_id=connector_id, resync_parameters={"property1": ["string"]}) | |||
| assert len(all_api_mocks.calls) == 3 | |||
| assert f"{connector_id}/schemas/tables/resync" in all_api_mocks.calls[2].request.url | |||
|
|
|||
| # poll calls | |||
| all_api_mocks.calls.reset() | |||
There was a problem hiding this comment.
I know that this is just re-originating a method that exists prior to this PR, but I'm just looking at all the unexercised code paths [1] and [2]; would be nice to test failure conditions here if it's not hard. Won't block review on that, especially since the problem exists beyond just this PR
There was a problem hiding this comment.
That's totally fair - done in a55e620
maximearmstrong
force-pushed
the
maxime/implement-resync-base-method-fivetran-client
branch
from
90123af
to
b81d516
Compare
maximearmstrong
force-pushed
the
maxime/implement-base-poll-method-fivetran-client
branch
from
6fff27d
to
77163b2
Compare
maximearmstrong
force-pushed
the
maxime/implement-resync-base-method-fivetran-client
branch
from
b81d516
to
004f258
Compare
maximearmstrong
force-pushed
the
maxime/implement-base-poll-method-fivetran-client
branch
from
77163b2
to
a55e620
Compare
maximearmstrong
force-pushed
the
maxime/implement-resync-base-method-fivetran-client
branch
from
004f258
to
03b4f71
Compare
maximearmstrong
force-pushed
the
maxime/implement-base-poll-method-fivetran-client
branch
from
a55e620
to
7ff0145
Compare
This was referenced Nov 21, 2024
Open
Merged
[5/n][dagster-fivetran] Implement
FivetranWorkspaceData to FivetranConnectorTableProps method
#25797
Merged
This was referenced Nov 21, 2024
maximearmstrong
force-pushed
the
maxime/implement-resync-base-method-fivetran-client
branch
from
03b4f71
to
6855c14
Compare
maximearmstrong
force-pushed
the
maxime/implement-base-poll-method-fivetran-client
branch
from
3bdb351
to
0d4c5c3
Compare
This was referenced Nov 23, 2024
maximearmstrong
force-pushed
the
maxime/implement-resync-base-method-fivetran-client
branch
from
6855c14
to
2021ba0
Compare
maximearmstrong
force-pushed
the
maxime/implement-base-poll-method-fivetran-client
branch
from
0d4c5c3
to
919e38e
Compare
maximearmstrong
force-pushed
the
maxime/implement-resync-base-method-fivetran-client
branch
from
2021ba0
to
3df2067
Compare
maximearmstrong
force-pushed
the
maxime/implement-base-poll-method-fivetran-client
branch
from
919e38e
to
b85bbe8
Compare
maximearmstrong
force-pushed
the
maxime/implement-resync-base-method-fivetran-client
branch
from
3df2067
to
22bfe6c
Compare
maximearmstrong
force-pushed
the
maxime/implement-base-poll-method-fivetran-client
branch
from
b85bbe8
to
c7250c1
Compare
maximearmstrong
force-pushed
the
maxime/implement-resync-base-method-fivetran-client
branch
from
22bfe6c
to
ffab308
Compare
maximearmstrong
force-pushed
the
maxime/implement-base-poll-method-fivetran-client
branch
from
c7250c1
to
9161976
Compare
maximearmstrong
force-pushed
the
maxime/implement-resync-base-method-fivetran-client
branch
from
ffab308
to
16121e5
Compare
maximearmstrong
force-pushed
the
maxime/implement-base-poll-method-fivetran-client
branch
from
9161976
to
13b8e6d
Compare
maximearmstrong
force-pushed
the
maxime/implement-resync-base-method-fivetran-client
branch
from
16121e5
to
8a48241
Compare
maximearmstrong
force-pushed
the
maxime/implement-base-poll-method-fivetran-client
branch
from
13b8e6d
to
f039336
Compare
Base automatically changed from
maxime/implement-resync-base-method-fivetran-client
to
master
November 27, 2024 13:44
maximearmstrong
force-pushed
the
maxime/implement-base-poll-method-fivetran-client
branch
from
f039336
to
4b0d41a
Compare
maximearmstrong
deleted the
maxime/implement-base-poll-method-fivetran-client
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary & Motivation
This PR reworks legacy oll method and implements it in the
FivetranClient:poll_syncis added based on legacypoll_syncFivetranConnectorpropertiesTests mock the request API calls and make sure that all calls are made.
How I Tested These Changes
Additional unit tests with BK