Source Intercom: Fix incremental sync state issues

[btkcodedev]

What

Source Intercom: Migrate to manifest only format with components

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Dec 19, 2024 7:22pm

[btkcodedev]

One update: I've tried to use older custom substream slicer, but the connection check is failing, so its a bummer

[btkcodedev]

I'll try different combinations and base versions

[btkcodedev]

Update: I tweaked with partition_id of parent and it was kind of success when tested with local

[btkcodedev]

@ChristoGrab Can you come again for those state messages?
Docker read, doesn't give any state messages right? Or am I missing something?
docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-intercom:dev read --config /secrets/config.json --catalog /integration_tests/incremental_catalog.json

[ChristoGrab]

@btkcodedev State messages do get logged when running with docker, they'll show up at the end of each stream's record output in your terminal. Easy way to find it is by searching for "type":"STATE" in the connector's output. So for conversation_parts, we get:

{
  "type":"STATE",
  "state":{
    "type":"STREAM",
    "stream":{
      "stream_descriptor":{
        "name":"conversation_parts"
      },
      "stream_state":{
        "states":[
          {"partition":{"id":"1", "parent_slice":{}}, "cursor":{"updated_at":7626086649}},
          {"partition":{"id":"60","parent_slice":{}},"cursor":{"updated_at":7626086649}},
          {"partition":{"id":"61","parent_slice":{}},"cursor":{"updated_at":7626086649}},
          {"partition":{"id":"63","parent_slice":{}},"cursor":{"updated_at":7626086649}},
          {"partition":{"id":"64","parent_slice":{}},"cursor":{"updated_at":7626086649}},
          {"partition":{"id":"65","parent_slice":{}},"cursor":{"updated_at":7626086649}},
          {"partition":{"id":"66","parent_slice":{}},"cursor":{"updated_at":7626086649}},
          {"partition":{"id":"67","parent_slice":{}},"cursor":{"updated_at":7626086649}},
          {"partition":{"id":"68","parent_slice":{}},"cursor":{"updated_at":7626086649}},
          {"partition":{"id":"69","parent_slice":{}},"cursor":{"updated_at":7626086649}},
          {"partition":{"id":"59","parent_slice":{}},"cursor":{"updated_at":7626086649}}
        ]
      }
    }
  }
}

What's interesting here is that conversation_parts does use per-partition states (as in, a separate cursor value is tracked for every partition/parent slice). Haven't dug into the code, but I presume this was added here: #46658.

However, the existing state object in on our sandbox connector looks closer to the format used in our abnormal_state test, where we track the parent state and a single value for the whole of conversation_parts:

  {
    "type": "STREAM",
    "stream": {
      "stream_descriptor": {
        "name": "conversation_parts",
        "namespace": null
      },
      "stream_state": {
        "updated_at": 1689067774,
        "prior_state": {
          "updated_at": 1689067774,
          "prior_state": {
            "updated_at": 1689067774,
            "conversations": {
              "updated_at": 1689068230
            }
          },
          "conversations": {
            "updated_at": 1689068230
          }
        },
        "conversations": {
          "updated_at": 1689068230
        }
      }
    }
  }

This is likely due to the custom component altering the way state is tracked, but where things get confusing for me is the interaction of per-partition state with the custom component, and why there was a regression in the live connection that I tested with progressive rollout that didn't match up with the success of the CAT tests... I'm running another round of tests on your latest changes, will let you know soon as I have more insights.