feat: conditionally run pipelines in debug mode #862
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…-way-to-run-pipelines-in-debug-mode
This was referenced Dec 3, 2024
qgerome
reviewed
Dec 3, 2024
Sure, added some and provided a cleaner parsing |
qgerome
approved these changes
Dec 4, 2024
qgerome
deleted the
HEXA-1076-implement-a-way-to-run-pipelines-in-debug-mode
branch
nazarfil
pushed a commit
that referenced
this pull request
Dec 9, 2024
* Add log level to model + wire it in env * Remove todo * Merge * Fix * Fix * Wire backend * Wire backend * Wire backend * Fix wiring * Add tests * Working test * Simplify test * Simplify test * Add to view * Add safe parsing and tests * Cleaner tests
nazarfil
added a commit
that referenced
this pull request
Dec 9, 2024
* chore(deps): refresh pip-compile outputs * chore: set manifest version to 0.71.7 * chore: release 0.71.8 Release-As: 0.71.8 * chore: adds packages * chore: release with correct previous version * chore: adds release pattern * Revert "chore: adds release pattern" This reverts commit 171720f. * chore(main): release 0.71.8 (#786) * fix(Minio): Do not raise cors related errors that are not supported by minio * chore(main): release 0.71.9 (#788) * chore(CI): Build & push image on main * fix: Cors pipelines webhook (#762) * fix(Dev): Upgrade dockerpy to fix the pipelines runner in dev * fix(Pipelines): Webhook can be called from frontend * chore: release 0.72.0 Release-As: 0.72.0 * chore(main): release 0.72.0 (#789) * chore: Make configurable the DB hostname in the container (#791) Co-authored-by: Christophe Philemotte <[email protected]> * fix(analytics): PipelineRunUser should be excluded (#794) * fix(analytics): PipelineRunUser should be excluded * fix tests * chore(main): release 0.72.1 (#792) * Fix: pipeline run user perms (#795) * fix: Use a real error in the except clause * fix(Analytics): The url has to end with a slash * fix(Auth): PipelineRunUser was not able to get the datasets from the SDK. Fixes OPENHEXA-16J * chore(main): release 0.72.2 (#796) * fix: disable buffered consumer for mixpanel in order to find where it's broken * chore(main): release 0.72.3 (#797) * chore(mixpanel): just use the default consumer for first release * fix(Analytics): use pipeline code for id (#801) * fix(Analytics): use pipeline code for id * fix(Analytics): scheduled pipeline_run events was not tracked * fix: version does not always exist (when it's a jupyter notebook * chore(main): release 0.72.4 (#798) * fix(Analytics): Notebook pipelines do not have a last_version, never. OPENHEXA-173 * chore(main): release 0.72.5 (#805) * chore: reset release-please sections * chore(main): release 0.72.6 (#806) * fix(analytics): remove the version.id that crashes for notebook pipelines * fix(analytics): uuid is not serializable * chore(main): release 0.72.7 (#807) * chore: change the changelog to the default one * fix(Datasets): increase slug size and only add suffix if needed (#802) * fix(Datasets): increase slug size and only add suffix if needed * fix(Datasets): use slug for dataset checks * fix(Workspaces): Try to create buckets/datasets/workspaces using their slug is taken --------- Co-authored-by: Quentin Gérôme <[email protected]> * fix(Statics): Remove cors decorator (#809) * chore(docker): set app image platform to linux/amd64 * fix(CORS): Add a decorator to disable CORS on specific endpoints * fix: replaces dot with dash for bucket creation * chore(main): release 0.72.8 (#811) * fix(Dataset): save sample as JSON instead of JSON string * Revert "fix(Dataset): save sample as JSON instead of JSON string" This reverts commit 54e5214. * chore: update postgresql to 13-3.3 * fix(User): removed reference to deprecated method to generate password * fix(User): removed reference to deprecated method to generate password * fix(User): reimplements the password * chire: debug in ci/cd * test(User): removed the test for admin page * chore: release 0.71.8 Release-As: 0.71.8 * chore: release 0.72.0 Release-As: 0.72.0 * chore(deps): bump cryptography from 43.0.0 to 43.0.1 (#804) Bumps [cryptography](https://github.com/pyca/cryptography) from 43.0.0 to 43.0.1. - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](pyca/cryptography@43.0.0...43.0.1) --- updated-dependencies: - dependency-name: cryptography dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): update dependency boto3 to v1.35.21 (#787) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * feat(Storage): Filesystem-based storage backend for files * chore(docker): Add .ruff_cache & .venv to ignore folder for docker * chore(CI): Copy the .env.dist to .env for the tests * feat(Storage): Implement a filesystem storage to store the datasets files in a mounted directory * Fix datasets worker * Manage proxy settings in base.py settings and docker image build args (#810) This allows to reduce the number of env vars to pass and manage secured connections. Co-authored-by: Christophe Philemotte <[email protected]> * fix: enforce file size to be an integer * chore: Adapt code based on Nazar's comments * chore: Do not silently handle the ValueError in case of wrong value --------- Co-authored-by: Christophe Philemotte <[email protected]> Co-authored-by: Christophe Philemotte <[email protected]> Co-authored-by: nazarfil <[email protected]> * chore(docker): Entrypoint 'fixtures' can be called without being in DEBUG mode * chore(storage): Disable by default the max size of the request when in local hosting mode * feat(Core): add query to return password requirements (#817) * chore(main): release 0.73.0 (#815) * Fix: Fix wrong mgt of env var TRUST_FORWARDED_PROTO (#816) Co-authored-by: Christophe Philemotte <[email protected]> * chore(main): release 0.73.1 (#818) * fix(GCP): 'updated' was not passed to StorageObject for the prefixes * chore(main): release 0.73.2 (#819) * fix(GCP): Bucket cannot contain '.' or be a recognized top-level domain (#820) * chore(main): release 0.73.3 (#821) * fix(DB): Starting from PostgreSQL 15+ it's needed to grant all on the public schema to allow creation and deletion of tables by the user (#822) * chore(deps): update dependency django to v5.0.9 [security] (#826) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * feat(Metadata): add generic metadata and link with dataset, version, files (#800) * feat(Metadata): implements metadata module, extends Dataset, DatasetVersionFile with metadata Closes PATHWAYS-143: dataset metadata annotation in OpenHEXA back-end * fix: migration * fix: tests * fix: metadata generation * fix: removed linked object * chore: rename attributes * fix: rmeoved has permission from model * fix: renamed to opaqueId * try generic relation * fix: adds indexing migration * fix: adds indexing migration * chore: adds todos * fix: moves permissions to model * fix: implements opaqueId graphql resolver * fix: removes target * fix: fixes tests * fix: tests dataset * chore: merged mirations * chore: improve the process * fix tests dataset * fix tests metadata * fix: fixes opaqueId to targetId * fix: encodes key in base64 * test: tests add system attribute * adds properties to the file * modifies profiling test with proeprties * fix: value comparison for profiling * fix: added test to valdiate attributes have been copied --------- Co-authored-by: Quentin Gérôme <[email protected]> * chore(main): release 0.74.0 (#823) * fix(cors): CORS_URL_PREFIXES was invalid since we added the analytics * chore(main): release 0.74.1 (#828) * fix: CORS regex was not correct and the dev configuration was redefining unnecessary variables * fix: Move removed variable from dev to the test configuration * fix(Pipelines): Run outputs can be a StorageObject instance (#830) * chore(main): release 0.74.2 (#829) * fix(datasets): fix the backward compatibility field 'uploadUrl' on CreateDatasetVersionFileResult (#831) Typo in the name of the property to get the content type from * chore(main): release 0.74.3 (#832) * fix(datasets): move the save() in its own try..except * fix(datasets): return the dataset_file_sample to not break tests * fix(datasets): When saving the sample in DB, we need a custom encoder that supports dates & uuids (#836) * chore(main): release 0.74.4 (#834) * fix(CORS): Include the files/up & files/dl to the cors exempted urls (#835) * chore(Datasets): add a downloadUrl on DatasetVersionFile to easily get the files * chore(datasets): Add a property on DatasetVersionFile to get the size of an file * chore(security): Configure the X_FRAME_OPTIONS to deny all frames * chore(Datasets): add a downloadUrl on DatasetVersionFile to easily get the files * chore(datasets): Add a property on DatasetVersionFile to get the size of an file (#838) * chore(main): release 0.74.5 (#837) * fix: fills in empty fields with NaN (#840) * fix: fills in empty field with NaN --------- Co-authored-by: Quentin Gérôme <[email protected]> * chore(main): release 0.74.6 (#842) * chore(datasets): Add fixtures * chore: Release 0.74.7 Release-As: 0.74.7 * chore(main): release 0.74.7 (#843) * fix(settings): In case of local hosting with a proxy, set the proxy url as the base_url as well * chore(debug): Add instructions on how to setup a debugger for VSCode & PyCharm (#847) * chore(debug): Add instructions on how to setup a debugger for VSCode & PyCharm * chore:Allow all hosts (this helps with debugging with 0.0.0.0) --------- Co-authored-by: YolanFery <[email protected]> * feat(Pipelines): Pipeline notifications v2 (#846) * feat(PipelineNotifications): Add notification events type * feat(Pipelines): update pipeline recipient * feat(Pipelines): delete pipeline notifications recipient * refacto(Pipelines): only check if pipeline has recipients * chore(Pipelines): rename notification_event to notification_level and simplify process * feat(Pipelines): allow user to mute notifications for a run * Tests(Pipelines): tests recipients notification level --------- Co-authored-by: Cheikh Gueye Wane <[email protected]> * chore: Make the Docker image "rootless" (#799) The previous image set root as the default user. This means that a process has the root rights and can create files with this id in a mounted volume from the host. This is a common attack vector. With the present change, the container will run under a normal user `openhexa` without any root permission. It's also possible to set the UID and GID of the containerized process and session. The pipeline runner requires the GID that corresponds to the host Docker group. Noticed that doesn't make Docker rootless. This requires to prepare the Docker engine to run in a normal userspace. This goes beyond the present change. Co-authored-by: Christophe Philemotte <[email protected]> * fix: Remove and prevent saving pipeline versions with the same name (#849) * fix: Remove and prevent saving versions with the same name * Working test * Fix atomicity * TODOS * Clean test * Clean error * Clean error * Test * Test * Fix test * Split * chore(main): release 0.75.0 (#845) * fix: adds byte field to json serialzer (#851) * fix: adds byte field to json serialzer * move encoding to separate method * move encoding back to if statement * chore(main): release 0.75.1 (#852) * chore(deps): bump werkzeug from 3.0.3 to 3.0.6 (#844) Bumps [werkzeug](https://github.com/pallets/werkzeug) from 3.0.3 to 3.0.6. - [Release notes](https://github.com/pallets/werkzeug/releases) - [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst) - [Commits](pallets/werkzeug@3.0.3...3.0.6) --- updated-dependencies: - dependency-name: werkzeug dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): update dependency boto3 to v1.35.66 (#814) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * chore(IP): Set REMOTE_ADDR from the X-Forwarded-For header if present (#854) * chore: Release 0.75.2 Release-As: 0.75.2 * chore(main): release 0.75.2 (#855) * fix: Set CMD command as json arguments * fix: skips geometry and processes wkb (#858) * fix: skips geometry and processes wkb * removed repeated test * fix: skipped bytes * fix: skipped bytes * fix: silent email address trim (#848) * fix: silent email address trim * fix: revert previous implemenation and use existing string util * refactoring: remove comments && dead code * chore: clean . cache and .local files * chore: clean . cache and .local files * chore(CI): Support hotfix/* branches for release-please * feat!: Set a default version number for each pipeline version, make version name optional (#857) * Model change * TODOs * Migrations * Tets migration * Test autoincrement * Test autoincrement fix * Missing migration * Admin add version number * Remove TODO * GrpahQL returns the verison number * Fix migration * Fix migration * GraphQl, Improve Admin * Remove TODOs * Display name * Version name * Version name fix * Rename version name * Rename version name * Clean migrations * Clean model * Fix graphql * Fix version name bug * Fix test and returned version name * Fix test * Fix test * Add a revert script in case of empty values * Change version_name format * Add a case for unit test * Modify slightly the name * Add a check of names * fix migration * fix migration test * Fix remove byte profiling (#863) * feat: skip any byte column for profiling, adds number of rows * feat: skip any byte column for profiling, adds number of column * fix(datasets): Set the correct engine for xls files & do not crash when DatasetVersionFile do not exist anymore (#864) * feat: conditionally run pipelines in debug mode (#862) * Add log level to model + wire it in env * Remove todo * Merge * Fix * Fix * Wire backend * Wire backend * Wire backend * Fix wiring * Add tests * Working test * Simplify test * Simplify test * Add to view * Add safe parsing and tests * Cleaner tests * chore(deps): update dependency django to v4.2.17 [security] (#867) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * fix : Pipeline Version parameters were converted to snake_case with invalid format (#865) * Convert names by default * Fix bug * Upgrade recommended * Upgrade recommended * Upgrade recommended * Remove usage of deprecated convert kwargs * Convert camel case to snake case * Add unit test * More renames * Revert a useless rename * chore(deps): refresh pip-compile outputs * chore: release 0.71.8 Release-As: 0.71.8 * chore: release 0.72.0 Release-As: 0.72.0 * fix(User): removed reference to deprecated method to generate password * fix(User): removed reference to deprecated method to generate password --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: nazarfil <[email protected]> Co-authored-by: Quentin Gérôme <[email protected]> Co-authored-by: Quentin Gérôme <[email protected]> Co-authored-by: Christophe Philemotte <[email protected]> Co-authored-by: Christophe Philemotte <[email protected]> Co-authored-by: Cheikh Gueye Wane <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: YolanFery <[email protected]> Co-authored-by: Cheikh Gueye Wane <[email protected]> Co-authored-by: Steve Lebleu <[email protected]>
Suspect IssuesThis pull request was deployed and Sentry observed the following issues:
Did you find this useful? React with a 👍 or 👎 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We want to provide a way for users to decide if they want to generate debug messages for the pipeline run. By default they will not be stored and shown.
Changes
HEXA_LOG_LEVELto the docker imagesHow/what to test
The flag is correctly set when creating a run and correctly passed to the pipeline run image
Screenshots / screencast
screen-capture.3.webm
Related PRs