Fix PyPI Code Deployment workflow #2750

barshaul · 2024-11-25T13:23:49Z

Fixed self-hosted error:

💥 maturin failed
  Caused by: The given list of python interpreters is invalid
  Caused by: Python interpreter `python3.13` doesn't exist

By upgrading maturin version in cargo.toml
https://github.com/valkey-io/valkey-glide/actions/runs/11997257172/job/33442732358

Fixed macos imports error by setting maturin version for macos wheels build to 0.14.14, last version where it worked with macos (need to open a bug for maturin)
Fixed macos error:

Failed
  Caused by: The given list of python interpreters is invalid
  Caused by: Python interpreter `python3.8` doesn't exist

By setting up all relevant python versions

Avi steps:

Above maturin 0.14.17 the imports breaking in tests pointing on circular dependencies.
The reason is probably that the project folder name (python) is the same name as the source dir, and that the lib (rust code) is the same name as the python module (glide), those leading to this state. Need to refactor the structure, out of scope. ⇒
Managed to find a solution, using a different name in the cargo.toml from glide to glide_rs, and this way can change the rust code to create a module called glide_rs.
Import it around using glide_rs as a separate module instead of .glide, which is path based and create circular or broken style.
Based on reading docs, the issue with mac was solved in 0.15, will run a try pointing to the latest. In case it won't work, will pin on 0.14 until refactor.
My try was wrong, i used lower version for the rc, so the rc it tried to rlease was the older one with higher v.
Will take seconed look tommorow.
We removed support from 3.8 so removed it back.
Simplified the way we use the --pre tag so its pass as we want.

Issue link

This Pull Request is linked to issue (URL): #2762

Checklist

Before submitting the PR make sure the following are checked:

This Pull Request is related to one issue.
Commit message has a detailed description of what changed and why.
N/A Tests are added or updated.
N/A CHANGELOG.md and documentation files are updated.
Destination branch is correct - main or release
Commits will be squashed upon merging.

Yury-Fridlyand

__init__ is empty in the wheel - so nothing could be imported from glide.
Same with other 1.2 RCs.
Credits to @tjzhang-BQ for this finding

Yury-Fridlyand

😌 wheels are not empty now. Can you try running CD on this branch to validate?

python/src/lib.rs

Yury-Fridlyand · 2024-11-27T00:54:21Z

node/npm/glide/package.json

+        "copy-declaration-files": "cp ../../build-ts/*.d.ts build-ts/ && cp ../../build-ts/src/*.d.ts build-ts/src/ && cp ../../build-ts/src/server-modules/*.d.ts build-ts/src/server-modules/",
+        "build": "tsc && mkdir -p build-ts/src && mkdir -p build-ts/src/server-modules && npm run copy-declaration-files"


This is already added in #2754. Please update your branch.

Yury-Fridlyand · 2024-11-27T00:57:25Z

.github/workflows/pypi-cd.yml

@@ -250,7 +277,11 @@ jobs:
                  python -m venv venv
                  source venv/bin/activate
                  pip install -U pip
-                  pip install ${PIP_PRE} valkey-glide
+                  if [[ "${{ env.PIP_PRE }}" == "true" ]]; then


THis should work as it was as I understand. Do you change it because it didn't work or for another reason?

it didnt work

acarbonetto

works with release from https://github.com/valkey-io/valkey-glide/actions/runs/12041257002

There seems to be unrelated changes in this PR that we should split out.

acarbonetto · 2024-11-27T02:06:58Z

.github/workflows/pypi-cd.yml

@@ -174,10 +177,10 @@ jobs:
              if: startsWith(matrix.build.NAMED_OS, 'darwin')
              uses: PyO3/maturin-action@v1
              with:
-                  maturin-version: latest
+                  maturin-version: 0.14.17


python/src/lib.rs

acarbonetto · 2024-11-27T02:31:59Z

python/python/tests/conftest.py

@@ -12,7 +12,7 @@
    ReadFrom,
    ServerCredentials,
 )
-from glide.exceptions import ClosingError, RequestError


acarbonetto · 2024-11-27T02:33:55Z

.github/workflows/pypi-cd.yml

+                  sudo apt-get update
+                  sudo apt-get install -y redis-server
+
+            - name: Install engine MacOS


these commands are under install-engine, no?

acarbonetto · 2024-11-27T02:34:31Z

.github/workflows/pypi-cd.yml

@@ -217,6 +220,10 @@ jobs:
            matrix:
                build: ${{ fromJson(needs.load-platform-matrix.outputs.PLATFORM_MATRIX) }}
        steps:
+            - name: Setup self-hosted runner access
+              if: ${{ matrix.build.TARGET == 'aarch64-unknown-linux-gnu' }}
+              run: sudo chown -R $USER:$USER /home/ubuntu/actions-runner/_work/valkey-glide


can this cause a permissions issue?

Signed-off-by: Andrew Carbonetto <[email protected]> Signed-off-by: avifenesh <[email protected]>

Signed-off-by: Andrew Carbonetto <[email protected]>

barshaul had a problem deploying to AWS_ACTIONS November 25, 2024 13:23 — with GitHub Actions Error

barshaul force-pushed the fix_pypi branch from efa4f66 to 61e45a3 Compare November 25, 2024 13:25

barshaul temporarily deployed to AWS_ACTIONS November 25, 2024 13:25 — with GitHub Actions Inactive

barshaul had a problem deploying to AWS_ACTIONS November 25, 2024 13:25 — with GitHub Actions Failure

avifenesh force-pushed the fix_pypi branch from 61e45a3 to 1c51178 Compare November 25, 2024 19:46

avifenesh had a problem deploying to AWS_ACTIONS November 25, 2024 19:46 — with GitHub Actions Error

avifenesh force-pushed the fix_pypi branch from 1c51178 to 45bb1de Compare November 25, 2024 19:46

avifenesh had a problem deploying to AWS_ACTIONS November 25, 2024 19:46 — with GitHub Actions Error

avifenesh had a problem deploying to AWS_ACTIONS November 25, 2024 19:47 — with GitHub Actions Error

avifenesh force-pushed the fix_pypi branch from 45bb1de to 34f0165 Compare November 25, 2024 20:42

avifenesh had a problem deploying to AWS_ACTIONS November 25, 2024 20:43 — with GitHub Actions Error

avifenesh force-pushed the fix_pypi branch from 34f0165 to 9f02eca Compare November 25, 2024 21:05

avifenesh had a problem deploying to AWS_ACTIONS November 25, 2024 21:05 — with GitHub Actions Failure

avifenesh had a problem deploying to AWS_ACTIONS November 25, 2024 21:05 — with GitHub Actions Error

avifenesh force-pushed the fix_pypi branch from 9f02eca to 7651926 Compare November 26, 2024 02:15

avifenesh had a problem deploying to AWS_ACTIONS November 26, 2024 02:15 — with GitHub Actions Error

avifenesh had a problem deploying to AWS_ACTIONS November 26, 2024 02:16 — with GitHub Actions Error

avifenesh force-pushed the fix_pypi branch from 08bce30 to d9f137d Compare November 26, 2024 02:22

avifenesh had a problem deploying to AWS_ACTIONS November 26, 2024 02:22 — with GitHub Actions Error

Yury-Fridlyand added python Python wrapper CI/CD CI/CD related labels Nov 26, 2024

avifenesh had a problem deploying to AWS_ACTIONS November 26, 2024 19:11 — with GitHub Actions Error

Yury-Fridlyand requested changes Nov 26, 2024

View reviewed changes

Yury-Fridlyand added the Release blocker Can't release without. label Nov 26, 2024

avifenesh force-pushed the fix_pypi branch from 09da101 to 4bde22c Compare November 27, 2024 00:16

avifenesh had a problem deploying to AWS_ACTIONS November 27, 2024 00:16 — with GitHub Actions Error

avifenesh temporarily deployed to AWS_ACTIONS November 27, 2024 00:16 — with GitHub Actions Inactive

Yury-Fridlyand approved these changes Nov 27, 2024

View reviewed changes

acarbonetto approved these changes Nov 27, 2024

View reviewed changes

Node: Add server-modules to npm release (#2754)

1acc108

Signed-off-by: Andrew Carbonetto <[email protected]> Signed-off-by: avifenesh <[email protected]>

acarbonetto force-pushed the fix_pypi branch from 4bde22c to 1acc108 Compare November 27, 2024 02:41

acarbonetto had a problem deploying to AWS_ACTIONS November 27, 2024 02:41 — with GitHub Actions Error

Revert noop change in imports

e0b5f2d

Signed-off-by: Andrew Carbonetto <[email protected]>

acarbonetto had a problem deploying to AWS_ACTIONS November 27, 2024 02:44 — with GitHub Actions Error

acarbonetto had a problem deploying to AWS_ACTIONS November 27, 2024 02:45 — with GitHub Actions Error

acarbonetto merged commit 031dfe5 into release-1.2 Nov 27, 2024
53 of 58 checks passed

acarbonetto deleted the fix_pypi branch November 27, 2024 04:07

yipin-chen mentioned this pull request Nov 27, 2024

Python rc tests fail on macos #2728

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix PyPI Code Deployment workflow #2750

Fix PyPI Code Deployment workflow #2750

barshaul commented Nov 25, 2024 •

edited by acarbonetto

Loading

Yury-Fridlyand left a comment

Yury-Fridlyand left a comment

Yury-Fridlyand Nov 27, 2024

Yury-Fridlyand Nov 27, 2024

avifenesh Nov 27, 2024

acarbonetto left a comment

acarbonetto Nov 27, 2024

acarbonetto Nov 27, 2024

acarbonetto Nov 27, 2024

acarbonetto Nov 27, 2024

		"copy-declaration-files": "cp ../../build-ts/.d.ts build-ts/ && cp ../../build-ts/src/.d.ts build-ts/src/ && cp ../../build-ts/src/server-modules/*.d.ts build-ts/src/server-modules/",
		"build": "tsc && mkdir -p build-ts/src && mkdir -p build-ts/src/server-modules && npm run copy-declaration-files"

Fix PyPI Code Deployment workflow #2750

Fix PyPI Code Deployment workflow #2750

Conversation

barshaul commented Nov 25, 2024 • edited by acarbonetto Loading

Issue link

Checklist

Yury-Fridlyand left a comment

Choose a reason for hiding this comment

Yury-Fridlyand left a comment

Choose a reason for hiding this comment

Yury-Fridlyand Nov 27, 2024

Choose a reason for hiding this comment

Yury-Fridlyand Nov 27, 2024

Choose a reason for hiding this comment

avifenesh Nov 27, 2024

Choose a reason for hiding this comment

acarbonetto left a comment

Choose a reason for hiding this comment

acarbonetto Nov 27, 2024

Choose a reason for hiding this comment

acarbonetto Nov 27, 2024

Choose a reason for hiding this comment

acarbonetto Nov 27, 2024

Choose a reason for hiding this comment

acarbonetto Nov 27, 2024

Choose a reason for hiding this comment

barshaul commented Nov 25, 2024 •

edited by acarbonetto

Loading