Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix psycopg2 install #139

Merged
merged 28 commits into from
Apr 15, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
a5f4b89
install build tools first
mikealfare Apr 4, 2024
f6c3507
add psycopg2 env variable
mikealfare Apr 4, 2024
2242da9
move psycopg2 env variable to be in-line with pip download command
mikealfare Apr 4, 2024
4158457
move psycopg2 env variable to be in-line with pip install command
mikealfare Apr 4, 2024
c3c8610
move psycopg2 env variable to be in-line with pip install command
mikealfare Apr 4, 2024
8f7c3ec
remove env var
mikealfare Apr 4, 2024
cd5fddb
shorten test cases to speed up ci, revert this
mikealfare Apr 4, 2024
4511fc4
download the source distribution for `dbt-postgres`, as is done in pr…
mikealfare Apr 4, 2024
4693357
add a test for the download process to ensure we always get the corre…
mikealfare Apr 4, 2024
2e557cf
add all test version back in to test_create.py
mikealfare Apr 4, 2024
9c521c1
revert moving the environment variable
mikealfare Apr 15, 2024
a1ebb25
align current and 1.8 req files
mikealfare Apr 15, 2024
94f8834
revert moving environment variable
mikealfare Apr 15, 2024
f800814
revert moving environment variable
mikealfare Apr 15, 2024
0e76b55
revert moving environment variable
mikealfare Apr 15, 2024
1cafc5f
revert moving environment variable
mikealfare Apr 15, 2024
ff620ed
revert moving environment variable
mikealfare Apr 15, 2024
20a635c
limit psycopg2 check to linux
mikealfare Apr 15, 2024
53f07b0
update download test to be platform based
mikealfare Apr 15, 2024
7cd2ed1
revert formatting change
mikealfare Apr 15, 2024
020f8e9
fix syntax
mikealfare Apr 15, 2024
e796512
fix syntax
mikealfare Apr 15, 2024
e35bbe0
fix syntax
mikealfare Apr 15, 2024
86b7a64
fix syntax
mikealfare Apr 15, 2024
ea7bcd2
remove forgotten test step
mikealfare Apr 15, 2024
d23fa94
remove 1.8.0b1 from tests since we know this does not work with the n…
mikealfare Apr 15, 2024
fe2cc93
update main reqs file to reflect 1.8
mikealfare Apr 15, 2024
7a8f47a
formatting
mikealfare Apr 15, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
hatchling<2.0
dbt-common @ git+https://github.com/dbt-labs/dbt-common.git@main
dbt-adapters @ git+https://github.com/dbt-labs/dbt-adapters.git@main
dbt-core @ git+https://github.com/dbt-labs/dbt-core.git@main#subdirectory=core
Expand All @@ -12,8 +11,9 @@ dbt-databricks @ git+https://github.com/databricks/[email protected]
dbt-trino @ git+https://github.com/starburstdata/dbt-trino.git@master
dbt-fabric @ git+https://github.com/microsoft/dbt-fabric.git@main
dbt-synapse @ git+https://github.com/microsoft/dbt-synapse.git@master
grpcio-status~=1.47.0
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was from older versions and no longer applies. It must have been left in here accidentally and didn't cause an issue until now. It causes a version conflict currently. The version that BQ installs is 1.62.0. It was removed from the specific minor reqs files in 1.5.


pyasn1-modules~=0.2.1
pyarrow!=12.0.1
pyarrow~=14.0.1
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was also an old holdover. It looks like this change was made in the minor reqs files in 1.4.

pydantic~=1.10
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This got added in 1.4.7.

pyodbc==4.0.39 --no-binary pyodbc
snowflake-connector-python~=3.0,!=3.0.4
4 changes: 3 additions & 1 deletion release_creation/bundle/test_archive_install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,8 @@ python -m pip install -r "${requirements_file}" \
--find-links ./bundle_pkg_test \
--pre
dbt --version
# make sure psycopg2 is installed, but not psycopg2-binary
# make sure psycopg2 is installed, but not psycopg2-binary (linux only)
if "$OSTYPE" == linux*; then
echo -n "Checking psycopg2 install..."
if ! pip freeze | grep psycopg2; then
echo "psycopg2 is not installed!"
Expand All @@ -27,5 +28,6 @@ if pip freeze | grep psycopg2-binary; then
echo "psycopg2-binary is installed and should not be!"
exit 1
fi
fi
echo ok
deactivate
2 changes: 1 addition & 1 deletion test/integration/bundle/test_create.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@

@pytest.mark.parametrize(
argnames="test_version",
argvalues=["0.0.0", "1.3.0", "1.4.0", "1.5.0", "1.6.0", "1.7.0b1", "1.7.0", "1.8.0b1"],
argvalues=["0.0.0", "1.3.0", "1.4.0", "1.5.0", "1.6.0", "1.7.0b1", "1.7.0"],
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can't currently test 1.8.0b1 since we know this will fail due to the metadata issue in the versions that's published to PyPI. We need to add it back once we publish the next dbt-postgres beta.

)
def test_generate_bundle_creates_a_bundle_with_valid_version(
test_version
Expand Down
37 changes: 37 additions & 0 deletions test/integration/bundle/test_download.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
from pathlib import Path
import platform
from semantic_version import Version
import sys

import pytest

from release_creation.bundle.bundle_config import get_bundle_config
from release_creation.bundle.create import _download_packages


@pytest.mark.parametrize("test_version", ["0.0.0"])
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can't currently test 1.8.0b1 since we know this will fail due to the metadata issue in the versions that's published to PyPI. We need to add it once we publish the next dbt-postgres beta.

def test_correct_version_of_psycopg2(test_version):
if sys.version_info == (3, 10) and Version.coerce(test_version) >= Version.coerce(
"1.5.0"
):
pytest.skip("We run test for Python 3.11 with version 1.5.0+ of the bundle")

bundle_configuration = get_bundle_config(Version.coerce(test_version))
_download_packages(bundle_configuration)

tmp_dir = Path(bundle_configuration.py_version_tmp_path)
assert tmp_dir.is_dir()
psycopg2_is_found = False
psycopg2_binary_is_found = False
for file in tmp_dir.iterdir():
if "psycopg2" in file.name:
if "psycopg2_binary" not in file.name:
psycopg2_is_found = True
else:
psycopg2_binary_is_found = True
if platform.system() == "Linux":
assert psycopg2_is_found and not psycopg2_binary_is_found
elif platform.system() == "Darwin":
assert psycopg2_binary_is_found and not psycopg2_is_found
else:
assert False, f"Unexpected platform: {platform.system()}"