Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add and configure pre-commit #62

Merged
merged 16 commits into from
Apr 19, 2024
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 10 additions & 20 deletions .github/workflows/code-quality.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,16 +9,14 @@ on:
workflow_dispatch:
inputs:
dbt_adapters_branch:
description: "The branch of dbt-adapters to evaluate"
type: string
required: true
default: "main"
description: "The branch of dbt-adapters to evaluate"
type: string
default: "main"
workflow_call:
inputs:
dbt_adapters_branch:
description: "The branch of dbt-adapters to evaluate"
type: string
required: true
mikealfare marked this conversation as resolved.
Show resolved Hide resolved
default: "main"

permissions: read-all
Expand All @@ -27,34 +25,26 @@ permissions: read-all
concurrency:
group: ${{ github.workflow }}-${{ github.event_name }}-${{ contains(github.event_name, 'pull_request') && github.event.pull_request.head.ref || github.sha }}
cancel-in-progress: true
env:
# set DBT_ADAPTERS_BRANCH to the input value if the event is a workflow_dispatch (workflow_call uses the same event_name),
# otherwise use 'main'
DBT_ADAPTERS_BRANCH: ${{ github.event_name == 'workflow_dispatch' && inputs.dbt_adapters_branch || 'main' }}

jobs:
code-quality:
name: Code Quality
runs-on: ubuntu-latest


steps:
- name: Check out repository
uses: actions/checkout@v4
with:
persist-credentials: false

- name: Update Adapters and Core branches
if: ${{ contains(github.event_name, 'workflow_') }}
shell: bash
run: |
./.github/scripts/update_dev_packages.sh \
$DBT_ADAPTERS_BRANCH \
"main"
run: ./.github/scripts/update_dev_packages.sh ${{ inputs.dbt_adapters_branch }} "main"

- name: Setup `hatch`
uses: dbt-labs/dbt-adapters/.github/actions/setup-hatch@main

- name: Run linters
run: hatch run lint:all
uses: dbt-labs/dbt-adapters/.github/actions/setup-hatch@config/add-precommit
mikealfare marked this conversation as resolved.
Show resolved Hide resolved

- name: Run typechecks
run: hatch run typecheck:all
- name: Run code quality
shell: bash
run: hatch run code-quality
mikealfare marked this conversation as resolved.
Show resolved Hide resolved
10 changes: 1 addition & 9 deletions .github/workflows/integration-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ jobs:
PGDATABASE: postgres

- name: Setup `hatch`
uses: dbt-labs/dbt-adapters/.github/actions/setup-hatch@main
uses: dbt-labs/dbt-adapters/.github/actions/setup-hatch@config/add-precommit
mikealfare marked this conversation as resolved.
Show resolved Hide resolved
with:
python-version: ${{ matrix.python-version }}

Expand All @@ -101,14 +101,6 @@ jobs:
POSTGRES_TEST_DATABASE: dbt
POSTGRES_TEST_THREADS: 4

- name: Publish results
mikealfare marked this conversation as resolved.
Show resolved Hide resolved
uses: dbt-labs/dbt-adapters/.github/actions/publish-results@main
if: always()
with:
source-file: "results.csv"
file-name: "integration_results"
python-version: ${{ matrix.python-version }}

psycopg2-check:
name: "Test psycopg2 build version"
runs-on: ${{ matrix.scenario.platform }}
Expand Down
10 changes: 1 addition & 9 deletions .github/workflows/unit-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -32,18 +32,10 @@ jobs:
persist-credentials: false

- name: Setup `hatch`
uses: dbt-labs/dbt-adapters/.github/actions/setup-hatch@main
uses: dbt-labs/dbt-adapters/.github/actions/setup-hatch@config/add-precommit
mikealfare marked this conversation as resolved.
Show resolved Hide resolved
with:
python-version: ${{ matrix.python-version }}

- name: Run unit tests
run: hatch run unit-tests:all
shell: bash

- name: Publish results
mikealfare marked this conversation as resolved.
Show resolved Hide resolved
uses: dbt-labs/dbt-adapters/.github/actions/publish-results@main
if: always()
with:
source-file: "results.csv"
file-name: "unit_results"
python-version: ${{ matrix.python-version }}
30 changes: 30 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
default_language_version:
mikealfare marked this conversation as resolved.
Show resolved Hide resolved
python: python3

repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.4.0
hooks:
- id: check-yaml
args: [--unsafe]
- id: check-json
- id: end-of-file-fixer
- id: trailing-whitespace
- id: check-case-conflict
- repo: local
hooks:
- id: black
name: black
entry: black
language: system
types: [python]
- id: flake8
name: flake8
entry: flake8
language: system
types: [python]
- id: mypy
name: mypy
entry: mypy
language: system
types: [python]
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,7 @@ Remember to commit and push the file that's created.

### Signing the CLA

> **_NOTE:_** All contributors to `dbt-postgres` must sign the
> **_NOTE:_** All contributors to `dbt-postgres` must sign the
> [Contributor License Agreement](https://docs.getdbt.com/docs/contributor-license-agreements)(CLA).

Maintainers will be unable to merge contributions until the contributor signs the CLA.
Expand Down
30 changes: 17 additions & 13 deletions dbt/adapters/postgres/relation.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,19 +20,23 @@

@dataclass(frozen=True, eq=False, repr=False)
class PostgresRelation(BaseRelation):
renameable_relations: FrozenSet[RelationType] = field(default_factory=lambda: frozenset(
{
RelationType.View,
RelationType.Table,
RelationType.MaterializedView,
}
))
replaceable_relations: FrozenSet[RelationType] = field(default_factory=lambda: frozenset(
{
RelationType.View,
RelationType.Table,
}
))
renameable_relations: FrozenSet[RelationType] = field(
default_factory=lambda: frozenset(
{
RelationType.View,
RelationType.Table,
RelationType.MaterializedView,
}
)
)
replaceable_relations: FrozenSet[RelationType] = field(
default_factory=lambda: frozenset(
{
RelationType.View,
RelationType.Table,
}
)
)

def __post_init__(self):
# Check for length of Postgres table/view names.
Expand Down
31 changes: 9 additions & 22 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -59,31 +59,21 @@ path = "dbt/adapters/postgres/__version__.py"
dependencies = [
"dbt-adapters @ git+https://github.com/dbt-labs/dbt-adapters.git",
"dbt-common @ git+https://github.com/dbt-labs/dbt-common.git",
]

[tool.hatch.envs.lint]
mikealfare marked this conversation as resolved.
Show resolved Hide resolved
detached = true
dependencies = [
"black>=24.3",
"flake8",
"Flake8-pyproject",
]
[tool.hatch.envs.lint.scripts]
all = [
"black",
"flake8",
]
black = "python -m black ."
flake8 = "python -m flake8 ."

[tool.hatch.envs.typecheck]
dependencies = [
"pre-commit==3.7.0",
"mypy",
"types-protobuf",
"types-pytz",
"types-PyYAML",
]
[tool.hatch.envs.typecheck.scripts]
all = "python -m mypy ."
post-install-commands = ["pre-commit install"]
[tool.hatch.envs.default.scripts]
code-quality = "pre-commit run --all-files" # runs others via hooks
black = "python -m black ."
flake8 = "python -m flake8 ."
mypy = "python -m mypy ."

[tool.hatch.envs.unit-tests]
dependencies = [
Expand All @@ -94,7 +84,6 @@ dependencies = [
"pytest-xdist",
]
extra-dependencies = [
# TODO: remove `dbt-core` dependencies from unit tests
"dbt-adapters @ git+https://github.com/dbt-labs/dbt-adapters.git",
"dbt-common @ git+https://github.com/dbt-labs/dbt-common.git",
"dbt-core @ git+https://github.com/dbt-labs/dbt-core.git#subdirectory=core",
Expand Down Expand Up @@ -125,8 +114,6 @@ dependencies = [
"twine",
"check-wheel-contents",
]


[tool.hatch.envs.build.scripts]
check-all = [
"- check-wheel",
Expand Down Expand Up @@ -174,4 +161,4 @@ env_files = ["test.env"]
testpaths = [
"tests/functional",
"tests/unit",
]
]
11 changes: 0 additions & 11 deletions tests/functional/artifacts/test_artifacts.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
from datetime import datetime
import jsonschema
import os

Expand All @@ -12,16 +11,6 @@
)
import pytest

from tests.functional.artifacts.expected_manifest import (
expected_references_manifest,
expected_seeded_manifest,
expected_versions_manifest,
)
from tests.functional.artifacts.expected_run_results import (
expected_references_run_results,
expected_run_results,
expected_versions_run_results,
)
from tests.functional.utils import run_dbt, run_dbt_and_capture


Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
import json
import os

from dbt.tests.util import write_file
from dbt_common.exceptions import CompilationError
Expand Down
5 changes: 1 addition & 4 deletions tests/functional/dbt_debug/test_dbt_debug.py
Original file line number Diff line number Diff line change
@@ -1,9 +1,6 @@
import pytest
import os
import re
import yaml

from tests.functional.utils import run_dbt, run_dbt_and_capture
from tests.functional.utils import run_dbt_and_capture

MODELS__MODEL_SQL = """
seled 1 as id
Expand Down
4 changes: 3 additions & 1 deletion tests/functional/dbt_runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,13 +25,15 @@ def assert_run_results_have_compiled_node_attributes(


class dbtTestRunner(dbtRunner):
exit_assertions: List[Callable[[List[str], dbtRunnerResult], None]]

def __init__(
self,
manifest: Optional[Manifest] = None,
callbacks: Optional[List[Callable[[EventMsg], None]]] = None,
exit_assertions: Optional[List[Callable[[List[str], dbtRunnerResult], None]]] = None,
):
self.exit_assertions = exit_assertions if exit_assertions else _STANDARD_ASSERTIONS
self.exit_assertions = exit_assertions if exit_assertions else _STANDARD_ASSERTIONS # type: ignore
super().__init__(manifest, callbacks)

def invoke(self, args: List[str], **kwargs) -> dbtRunnerResult:
Expand Down
Loading
Loading