Error installing PyAirbyte with dbt-core

[GiselaH]

Here are the contents of my Pipfile

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
airbyte = "*"
dbt-common = "*"
dbt-core = "*"
jsonschema = "*"

[requires]
python_version = "3.10"

My error message after trying to install

CRITICAL:pipenv.patched.pip._internal.resolution.resolvelib.factory:Cannot install -r /var/folders/ny/p19nr54d7tq1dy4pkpx9h2880000gn/T/pipenv-n0ks9g1u-requirements/pipenv-twk0qus_-constraints.txt (line 3), -r 
/var/folders/ny/p19nr54d7tq1dy4pkpx9h2880000gn/T/pipenv-n0ks9g1u-requirements/pipenv-twk0qus_-constraints.txt (line 4) and jsonschema because these package versions have conflicting dependencies.
INFO:pipenv.patched.pip._internal.resolution.resolvelib.factory:
The conflict is caused by:
    The user requested jsonschema
    airbyte 0.2.0 depends on jsonschema==3.2.0
    dbt-common 1.12.0 depends on jsonschema<5.0 and >=4.0
    The user requested jsonschema
    airbyte 0.2.0 depends on jsonschema==3.2.0
    dbt-common 1.11.0 depends on jsonschema<5.0 and >=4.0
    The user requested jsonschema
    airbyte 0.2.0 depends on jsonschema==3.2.0
    dbt-common 1.10.0 depends on jsonschema<5.0 and >=4.0
    The user requested jsonschema
    airbyte 0.2.0 depends on jsonschema==3.2.0
    dbt-common 1.9.0 depends on jsonschema<5.0 and >=4.0
    The user requested jsonschema
    airbyte 0.2.0 depends on jsonschema==3.2.0
    dbt-common 1.8.0 depends on jsonschema<5.0 and >=4.0
    The user requested jsonschema
    airbyte 0.2.0 depends on jsonschema==3.2.0
    dbt-common 1.7.0 depends on jsonschema<5.0 and >=4.0
    The user requested jsonschema
    airbyte 0.2.0 depends on jsonschema==3.2.0
    dbt-common 1.6.0 depends on jsonschema<5.0 and >=4.0
    The user requested jsonschema
    airbyte 0.2.0 depends on jsonschema==3.2.0
    dbt-common 1.5.0 depends on jsonschema<5.0 and >=4.0
    The user requested jsonschema
    airbyte 0.2.0 depends on jsonschema==3.2.0
    dbt-common 1.4.0 depends on jsonschema<5.0 and >=4.0
    The user requested jsonschema
    airbyte 0.2.0 depends on jsonschema==3.2.0
    dbt-common 1.3.0 depends on jsonschema<5.0 and >=4.0
    The user requested jsonschema
    airbyte 0.2.0 depends on jsonschema==3.2.0
    dbt-common 1.2.0 depends on jsonschema<5.0 and >=4.0
    The user requested jsonschema
    airbyte 0.2.0 depends on jsonschema==3.2.0
    dbt-common 1.1.0 depends on jsonschema<5.0 and >=4.0
    The user requested jsonschema
    airbyte 0.2.0 depends on jsonschema==3.2.0
    dbt-common 1.0.4 depends on jsonschema<5.0 and >=4.0
    The user requested jsonschema
    airbyte 0.2.0 depends on jsonschema==3.2.0
    dbt-common 1.0.3 depends on jsonschema<5.0 and >=4.0
    The user requested jsonschema
    airbyte 0.2.0 depends on jsonschema==3.2.0
    dbt-common 1.0.2 depends on jsonschema<5.0 and >=4.0
    The user requested jsonschema
    airbyte 0.2.0 depends on jsonschema==3.2.0
    dbt-common 1.0.1 depends on jsonschema<5.0 and >=4.0
    The user requested jsonschema
    airbyte 0.2.0 depends on jsonschema==3.2.0
    dbt-common 1.0.0 depends on jsonschema<5.0 and >=4.0
    The user requested jsonschema
    airbyte 0.2.0 depends on jsonschema==3.2.0
    dbt-common 0.1.6 depends on jsonschema~=4.0
    The user requested jsonschema
    airbyte 0.2.0 depends on jsonschema==3.2.0
    dbt-common 0.1.5 depends on jsonschema~=4.0
    The user requested jsonschema
    airbyte 0.2.0 depends on jsonschema==3.2.0
    dbt-common 0.1.4 depends on jsonschema~=4.0
    The user requested jsonschema
    airbyte 0.2.0 depends on jsonschema==3.2.0
    dbt-common 0.1.3 depends on jsonschema~=4.0
    The user requested jsonschema
    airbyte 0.2.0 depends on jsonschema==3.2.0
    dbt-common 0.1.1 depends on jsonschema~=4.0
    The user requested jsonschema
    airbyte 0.2.0 depends on jsonschema==3.2.0
    dbt-common 0.1.0 depends on jsonschema~=4.0

[aaronsteers]

Hi, @GiselaH . We are tracking this related item here:

• Add support for latest version of jsonschema library #398

It will need to be delivered first in the CDK and then we could apply here in PyAirbyte.

[GiselaH]

Thanks @aaronsteers! Do you have any recommendations for how we can workaround this ourselves for the time being?

[aaronsteers]

@GiselaH - since dbt is intended to be used as a CLI and not directly as a Python library, I'd generally recommend installing dbt into a separate virtual environment using pipx or uvx - or manually if you prefer self-managed venvs.

[aaronsteers]

Something like this might work...

pipx install --preinstall=dbt-postgres dbt-core
dbt --help

[cbini]

Just put in a PR to update jsonschema to a compatible version (4.17.3). Passing all tests locally so 🤞 ...

[aaronsteers]

CDK PR Merged here:

• chore(deps): bump jsonschema to "~4.17.3" airbyte-python-cdk#68

And in PyAirbyte here:

• Chore: Bump cdk to 6.6.4 #540

@cbini - Let us know if this unblocks you. If so, we can close this issue as resolve. Thanks very much for your help on this!

As a sidebar, I'll caution again that this may only be a short-term fix, as package versions are often drifting apart from each other. I'll still recommend (long-term) looking for a pipx, uvx, or virtual environment solution to install CLI "apps" like dbt. This makes it easier to avoid conflicts between the CLI apps and libraries like PyAirbyte that need to be called directly from Python. If you are interested in following up on this in more detail, we can discuss more here:

• Add guidance for avoiding version conflicts with connectors and other Python CLI apps #543

[cbini]

@aaronsteers Yup! I'm able to resolve all dependencies now.

Appreciate the advice. It gets a little muddy for my use case, where I'm trying to use Dagster to orchestrate extraction via PyAirbyte and transformation via Dagster's dbt integration, which relies on the having the dbt Python libraries installed. I can think of some ways to work around this if it does become a problem down the line.

[aaronsteers]

Awesome - thanks for circling back! I'll close this as resolved.