Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Source Intercom: Fix incremental sync state issues #49936

Open
wants to merge 29 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
620f5eb
Source Intercom: migrate to manifest only
btkcodedev Oct 22, 2024
6800bbf
chore: auto-fix lint and format issues
octavia-squidington-iii Oct 22, 2024
efcdd3b
update docs
btkcodedev Oct 22, 2024
8bebea4
Merge branch 'master' into btkcodedev/intercomMigrateManifestOnly
btkcodedev Dec 9, 2024
b0d4537
Merge branch 'master' into btkcodedev/intercomMigrateManifestOnly
btkcodedev Dec 12, 2024
75c6c7c
update manifest
btkcodedev Dec 12, 2024
0921b3d
Merge branch 'master' into btkcodedev/intercomMigrateManifestOnly
btkcodedev Dec 12, 2024
bc85586
add stream hash after testing without custom components
btkcodedev Dec 12, 2024
f657d4d
test failing streams
btkcodedev Dec 12, 2024
0b17a7a
Merge branch 'master' into btkcodedev/intercomMigrateManifestOnly
btkcodedev Dec 12, 2024
e1b4716
update incremental parent
btkcodedev Dec 12, 2024
9b9daf9
update test scenarios
btkcodedev Dec 12, 2024
7714da3
change incremental dependency
btkcodedev Dec 15, 2024
d83859d
revert back custom components
btkcodedev Dec 16, 2024
cbc9a62
update incremental sync
btkcodedev Dec 16, 2024
d85b794
feat: re-implement unit tests
ChristoGrab Dec 16, 2024
d896f60
chore: version bump
ChristoGrab Dec 17, 2024
205ba72
update abnormal state values
btkcodedev Dec 17, 2024
2fc0c70
Merge branch 'btkcodedev/intercomMigrateManifestOnly' of https://gith…
btkcodedev Dec 17, 2024
fa00fd6
revert base image version bump due to failure
btkcodedev Dec 17, 2024
9a14f96
retest: incremental sync
btkcodedev Dec 19, 2024
c51e8c1
Merge branch 'master' into btkcodedev/intercomMigrateManifestOnly
btkcodedev Dec 19, 2024
25473f6
update version and catalog
btkcodedev Dec 19, 2024
228b21f
update version and catalog
btkcodedev Dec 19, 2024
c3203a7
fix incremental sync params
btkcodedev Dec 19, 2024
ef69c5a
Merge branch 'btkcodedev/intercomMigrateManifestOnly' of https://gith…
btkcodedev Dec 19, 2024
1bc4d5e
revert companies ids
btkcodedev Dec 19, 2024
ca1b8c7
fix version
btkcodedev Dec 19, 2024
3a00582
Merge branch 'master' into btkcodedev/intercomMigrateManifestOnly
btkcodedev Dec 19, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,7 @@ acceptance_tests:
- config_path: "secrets/config.json"
expect_records:
path: "integration_tests/expected_records.jsonl"
empty_streams:
- name: conversation_parts
bypass_reason: Deeply nested response which could not be seeded with sandbox
empty_streams: []
incremental:
tests:
- config_path: "secrets/config.json"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,10 @@
"name": "company_segments"
},
"stream_state": {
"updated_at": 7626086649
"updated_at": 7626086649,
"companies": {
"updated_at": 1676517777
}
}
}
},
Expand All @@ -50,7 +53,10 @@
"name": "conversation_parts"
},
"stream_state": {
"updated_at": 7626086649
"updated_at": 7626086649,
"conversations": {
"updated_at": 1676462031
}
}
}
},
Expand Down
65 changes: 23 additions & 42 deletions airbyte-integrations/connectors/source-intercom/manifest.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -651,29 +651,18 @@ definitions:
{{ record['updated_at'] >= stream_state.get('prior_state',
{}).get('updated_at', 0) - (config.get('lookback_window', 0) *
86400)}}
partition_router:
type: SubstreamPartitionRouter
parent_stream_configs:
- type: ParentStreamConfig
parent_key: id
partition_field: id
stream:
$ref: "#/definitions/streams/conversations"
incremental_sync:
type: DatetimeBasedCursor
type: CustomIncrementalSync
class_name: >-
source_declarative_manifest.components.IncrementalSubstreamSlicerCursor
cursor_field: updated_at
cursor_datetime_formats:
- "%s"
- "%ms"
datetime_format: "%s"
start_datetime:
type: MinMaxDatetime
datetime: "{{ config[\"start_date\"] }}"
datetime_format: "%Y-%m-%dT%H:%M:%SZ"
end_datetime:
type: MinMaxDatetime
datetime: "{{ now_utc().strftime('%Y-%m-%dT%H:%M:%SZ') }}"
datetime_format: "%Y-%m-%dT%H:%M:%SZ"
parent_complete_fetch: true
parent_stream_configs:
- type: ParentStreamConfig
stream:
$ref: "#/definitions/streams/conversations"
parent_key: id
partition_field: id
transformations:
- type: AddFields
fields:
Expand Down Expand Up @@ -722,29 +711,18 @@ definitions:
{{ record['updated_at'] >= stream_state.get('prior_state',
{}).get('updated_at', 0) - (config.get('lookback_window', 0) *
86400)}}
partition_router:
type: SubstreamPartitionRouter
parent_stream_configs:
- type: ParentStreamConfig
parent_key: id
partition_field: id
stream:
$ref: "#/definitions/streams/companies"
incremental_sync:
type: DatetimeBasedCursor
type: CustomIncrementalSync
class_name: >-
source_declarative_manifest.components.IncrementalSubstreamSlicerCursor
cursor_field: updated_at
cursor_datetime_formats:
- "%s"
- "%ms"
datetime_format: "%s"
start_datetime:
type: MinMaxDatetime
datetime: "{{ config[\"start_date\"] }}"
datetime_format: "%Y-%m-%dT%H:%M:%SZ"
end_datetime:
type: MinMaxDatetime
datetime: "{{ now_utc().strftime('%Y-%m-%dT%H:%M:%SZ') }}"
datetime_format: "%Y-%m-%dT%H:%M:%SZ"
parent_complete_fetch: true
parent_stream_configs:
- type: ParentStreamConfig
stream:
$ref: "#/definitions/streams/companies"
parent_key: id
partition_field: id
schema_loader:
type: InlineSchemaLoader
schema:
Expand Down Expand Up @@ -875,8 +853,11 @@ metadata:
conversations:
- errorHandler
- incrementalSync
conversation_parts:
- incrementalSync
company_segments:
- errorHandler
- incrementalSync
testedStreams:
activity_logs:
hasRecords: true
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ data:
connectorSubtype: api
connectorType: source
definitionId: d8313939-3782-41b0-be29-b3ca20d8dd3a
dockerImageTag: 0.9.0-rc.1
dockerImageTag: 0.10.0
dockerRepository: airbyte/source-intercom
documentationUrl: https://docs.airbyte.com/integrations/sources/intercom
githubIssueLabel: source-intercom
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/sources/intercom.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,7 @@ The Intercom connector should not run into Intercom API limitations under normal

| Version | Date | Pull Request | Subject |
|:--------|:-----------|:---------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------|
| 0.10.0 | 2024-12-19 | [49936](https://github.com/airbytehq/airbyte/pull/49936) | Migrate to manifest-only format |
| 0.9.0-rc.1 | 2024-12-17 | [47240](https://github.com/airbytehq/airbyte/pull/47240) | Migrate to manifest-only format |
| 0.8.3 | 2024-12-12 | [48979](https://github.com/airbytehq/airbyte/pull/48979) | Update dependencies |
| 0.8.2 | 2024-10-29 | [47919](https://github.com/airbytehq/airbyte/pull/47919) | Update dependencies |
Expand Down
Loading