Support for Prefect 3

[Udit107710]

Topic

Make Airbyte compatible with Prefect 3

Relevant information

Since we are blocked by parallelism support we are running Prefect + Pyairbyte and want to upgrade to Prefect 3 but are blocked on the dependency of airbyte-cdk (which is a dependency of pyairbyte) on jsonschema = "~3.2.0" and pendulum = "<3.0.0".
Prefect 3 brings a lot of performance improvements to utilise pyairbyte to the fullest.
Moreover,

1. jsonschema 4.0.0 was released on Sep 30, 2021 and not upgrading does not make sense.
2. pendulum 3.0.0 brings lots of performance improvement with extensions rewritten in Rust

Upgrading these core dependencies ensures compatibility across various other integrations with airbyte.

[marcosmarxm]

@aaronsteers moving this issue t PyAirbyte project

[Udit107710]

I don't think this issue can be solved by Pyairbyte since pyairbyte does not declare dependency on jsonschema = "~3.2.0" and pendulum = "<3.0.0"
These dependencies arise from airbyte-cdk which pyairbyte utilises internally.

[aaronsteers]

@Udit107710 - Thanks for raising this issue. I agree with the intention of bumping jsonschema and pendulum refs. As noted, we'd have to first bump them in the CDK and then we could bump their references here.

The jsonschema library is pretty good with backwards compatibility and support for specific draft versions in each release. I vaguely recall some pendulum breaking changes when I last looked at this, but those are probably minimal...

[Udit107710]

There are breaking changes on Pendulum and jsonschema
I have local changes related to airbyte-cdk. I was able to install the latest versions but that caused multiple tests to fail. The issue with jsonschema caused was not something I was able to solve and have no way forward. The issue with pendulum requires changing the format of datetime across the entire module. I will raise a PR soon on aribyte-cdk and we can review and plan new changes.

Furthermore, I think bumping up the dependency in airbyte-cdk will also require making other airbyte packages like airbyte-ci compatible with the new versions since these must be getting used together.

[Udit107710]

Draft PR for the changes: airbytehq/airbyte#45677

[aaronsteers]

Logged these for tracking:

• Add support for Pendulum 3.0 #397
• Add support for latest version of jsonschema library #398

[aman-immersa]

@Udit107710 would like to learn more about your usage Pyairbyte with Prefect 3.0 since we're attempting something similar.