Source Intercom: Migrate to manifest only format with components

[btkcodedev]

What

Closes https://github.com/airbytehq/airbyte-internal-issues/issues/11089
Source Intercom: Migrate to manifest-only formats with components

How

Migrated to inline schemas
Migrated spec
Migrated to manifest only with ci

Migration report

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Dec 17, 2024 10:28pm

[btkcodedev]

/format-fix

Format-fix job started... Check job output.

✅ Changes applied successfully. (6800bbf)

[ChristoGrab]

@btkcodedev Danylo and I were going over the changes and it does seem like streams which were previously reading successfully are now throwing 422 errors (company_attributes stream is an example). I recommend running the connector locally to test it out, it's possible the issue is with the custom component code not functioning after the CDK upgrade, we may need to make some updates. Let us know if you have any trouble with testing!

[btkcodedev]

@ChristoGrab I've made some changes and tested the streams in local, removed some unwanted parameters and fully revised code to new version
Errors cleared!

[ChristoGrab]

@btkcodedev Looks like we're getting very close! CAT tests show both the incremental substreams (conversation_parts and company_segments) failing on the future-state test. Looking at the abnormal_state file, I think the failure is at least in part due to the way the state object is constructed in the sample. Since they're substreams, I would expect the state object to use per-partition states, and look something like:

  {
    "type": "STREAM",
    "stream": {
      "stream_descriptor": {
        "name": "company_segments"
      },
      "stream_state": {
        "states": [
          {
            "partition": {
              "id": "<some_parent_id>",
              "parent_slice": {}
            },
            "cursor": {
              "updated_at": 9626086649
            }
          },
          {
            "partition": {
              "id": "<some_other_parent_id>",
              "parent_slice": {}
            },
            "cursor": {
              "updated_at": 9626086649
            }
          }
        ]
      }
    }
  }    

This is probably a good starting point to investigate to make sure the acceptance test is giving us accurate results.

As a heads up, I tried playing with the state for company_segments, but was still seeing the same records returned, so I don't think this is just a problem with the sample state in the tests.

[btkcodedev]

@ChristoGrab Necessary tests passed with Intercom,
Please continue review
I think its ready.

[btkcodedev]

Aha, We've got an interesting condition here @ChristoGrab.
If we change the base image to 6.10.0, Then unit tests are passing, but the incremental sync is failing while checking in abnormal values
Conversely, if the base image is 5.14.0, then acceptance tests and abnormal large value checks are passing, but the unit tests fail. 😸

So my opinion is to remove the unit tests as it is purely manifest with components right now.

[ChristoGrab]

/format-fix

Format-fix job started... Check job output.

✅ Changes applied successfully. (c7dc09f)

[ChristoGrab]

@btkcodedev Taking a look! You're right, and I think the issue may be that we switched to using concurrent reads in CDK 6 by default, which may have an impact on incremental substream behavior. I commented out the unit tests for now, running a fresh round of tests to verify.

[ChristoGrab]

Marking as a release candidate for progressive rollout. Thanks for your hard work @btkcodedev!

[btkcodedev]

Continued in #49936