CI: update --use-local-cdk arg in airbyte-ci, adds --use-cdk-ref arg

[aaronsteers]

What

This PR updates --use-local-cdk behavior for Python connectors and adds --use-cdk-ref flag.

Testing here:

• ci: use new --use-local-cdk method for testing connectors airbyte-python-cdk#40

How

Review guide

User Impact

Can this PR be safely reverted and rolled back?

• YES 💚
• NO ❌

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Nov 13, 2024 0:09am

[aaronsteers]

@alafanechere - This is only a start.
Do you have time to assist?

If poetry add doesn't work, we could revert to the pip inject method, but just with an alternate path.

[alafanechere]

Requesting changes as your directory mounting logic will not work as it is. Feel free to force merge once you fixed it.
And please bump the pipelines package version + add an entry to the changelog in README.md

[aaronsteers]

@alafanechere - Tests are looking good so I'm yolo-merging this to try to resolve a couple other issues. Will gather data as we run in a variety of environments, and will try to resolve what comes up.

[aaronsteers]

Intentionally un-deleting branch (for now) because we have a CDK pipeline pinned to it.

[sentry-io]

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

• ‼️ FileExistsError: Local CDK not found at '/home/runner/work/airbyte/airbyte-python-cdk' pipelines.dagger.actions.python.common in apply... View Issue

_{Did you find this useful? React with a 👍 or 👎}