Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fix] nvm_get_arch: proper value for alpine linux #3212

Merged
merged 1 commit into from
Nov 2, 2023

Conversation

asolopovas
Copy link
Contributor

@ljharb I have updated, nvm_get_arch to return proper value for alpine, please let me know if that would work.

Copy link
Member

@ljharb ljharb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We’ll need some unit tests for nvm_get_arch, and ideally some for nvm_download that ensures the proper URL is produced for this arch.

@ljharb ljharb marked this pull request as draft September 30, 2023 23:52
ksh5a

This comment was marked as spam.

@asolopovas
Copy link
Contributor Author

How do you want to do it, modify existing ones, or create a separate test in tests folder?

@ljharb
Copy link
Member

ljharb commented Oct 4, 2023

@asolopovas please make a new file for each test

@asolopovas
Copy link
Contributor Author

@ljharb how can I create test for nvm_get_arch function if it passes the existing unit test? I can create nvm_get_arch_unofficial unit test but there is very little point to it, because I would have to mock alpine os detection, which is already done by detection of /etc/alpine-release file. It is there by default in each alpine and existing NVM_NODEJS_ORG_MIRROR functionality allows you to set it to https://unofficial-builds.nodejs.org/download/release. So it kind of works without additional tests... But if you have different idea please critique

@ljharb
Copy link
Member

ljharb commented Oct 10, 2023

There won't be an /etc/alpine-release file in all the places tests are run; so, we'd need to chroot to a fake filesystem where the file doesn't exist, and another where it does, and confirm that the detection is, and is not, musl-x64 in the appropriate scenarios.

@asolopovas
Copy link
Contributor Author

asolopovas commented Oct 10, 2023

There won't be an /etc/alpine-release file in all the places tests are run; so, we'd need to chroot to a fake filesystem where the file doesn't exist, and another where it does, and confirm that the detection is, and is not, musl-x64 in the appropriate scenarios.

As far as I know /etc/alpine-release file exist in each docker alpine container, I am already using nvm in my alpine images, currently I need to sort of hack in order to install it into alpine container. Here is example of my script which already works, but if that little if statement was present everything that would be necessary is to assign env variable for NVM_NODEJS_ORG_MIRROR and default installer would work with alpine containers without any hacky modifaction:

#!/bin/bash

OS=$(awk '/^ID=/' /etc/os-release | sed -e 's/ID=//' -e 's/"//g' | tr '[:upper:]' '[:lower:]')

NODE_VERSION=${NODE_VERSION:-18.18.0}
NVM_VERSION=${NVM_VERSION:-v0.39.5}
NVM_DIR="$HOME/.nvm"
FORCE=${FORCE:-false}

if [ "$FORCE" = "true" ]; then
    echo "Force install nodejs"
    rm -rf $NVM_DIR
fi

PROFILE=/dev/null bash -c "curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/${NVM_VERSION}/install.sh | bash"

if [ "$OS" = "alpine" ]; then
    chmod +x $NVM_DIR/nvm.sh
    sed -i '/nvm_get_arch() {/,/^}$/c\nvm_get_arch() { nvm_echo "x64-musl"; }' $NVM_DIR/nvm.sh
fi

source "$NVM_DIR/nvm.sh"
nvm install $NODE_VERSION

so assuming if NVM_NODEJS_ORG_MIRROR=https://unofficial-builds.nodejs.org/download/release, the download file will correctly resolve at https://unofficial-builds.nodejs.org/download/release/v18.18.0/node-v18.18.0-linux-x64-musl.tar.xz

@ljharb
Copy link
Member

ljharb commented Oct 10, 2023

Sorry, let me clarify. Here's the 4 tests I'd expect:

  1. mock out nvm_get_arch to return musl-x64, and:
    1. ensure that nothing loads for, say, nvm ls-remote 18 with the default mocks
    2. make a new mock file that simulates NVM_NODEJS_ORG_MIRROR=https://unofficial-builds.nodejs.org/download/release for ls-remote, and ensure that the expected thing loads
  2. use chroot - which makes a fake /:
    1. lacking /etc/alpine-release such that nvm_get_arch returns something !== "musl-x64"
    2. having /etc/alpine-release such that nvm_get_arch returns "musl-x64"

That way, the tests should pass on both a non-alpine and an alpine machine.

@asolopovas
Copy link
Contributor Author

Thanks. I think I understood you now

@asolopovas asolopovas marked this pull request as ready for review October 11, 2023 23:13
@asolopovas
Copy link
Contributor Author

I managed to write test both to test in fake file system with chroot as well as I did an example where it tries to retrive nvm ls-remote 18.18.0, seems to work please review.

Copy link
Member

@ljharb ljharb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made some tweaks to the tests for POSIX-compliance but otherwise LGTM!

@ljharb
Copy link
Member

ljharb commented Nov 1, 2023

The failing tests suggest that this change actually breaks installation on alpine :-/

@ljharb ljharb marked this pull request as draft November 1, 2023 18:57
@asolopovas
Copy link
Contributor Author

asolopovas commented Nov 1, 2023

The failing tests suggest that this change actually breaks installation on alpine :-/

It shouldn't I didn't touch the test suite but it should contain env variable NVM_NODEJS_ORG_MIRROR=https://unofficial-builds.nodejs.org/download/release for tests to pass. Do you want me to look into it?

@ljharb
Copy link
Member

ljharb commented Nov 1, 2023

The issue is that nvm_get_arch is now returning a different value on alpine, so our existing alpine tests aren't able to install node. Please do look into it.

@asolopovas
Copy link
Contributor Author

asolopovas commented Nov 1, 2023

The issue is that nvm_get_arch is now returning a different value on alpine, so our existing alpine tests aren't able to install node. Please do look into it.

Ok, I think I found the culprit, I mixed musl-x64 with x64-musl even though had it correct in my script don't know how I missed it.

tests

[ "${ARCH_WITH_ALPINE}" = "x64-musl" ] || die "Expected x64-musl for alpine environment but got ${ARCH_WITH_ALPINE}"

ARCH_WITHOUT_ALPINE=$(sudo chroot "${CHROOT_WITHOUT_ALPINE}" /bin/sh -c ". ./nvm.sh && nvm_get_arch")
[ "${ARCH_WITHOUT_ALPINE}" != "x64-musl" ] || die "Did not expect x64-musl for non-alpine environment"

and nvm.sh:

  if [ -f "/etc/alpine-release" ]; then
    NVM_ARCH=x64-musl
  fi

as to the test in order to modify test I need to edit your wsl_matrix and preferably strip Alpine from wsl_matrix and relocate it to wsl_matrix_unofficial, otherwise if I add NVM_NODEJS_ORG_MIRROR: https://unofficial-builds.nodejs.org/download/release to evn section it will start breaking Debian and Ubuntu test. Is it that Ok for me to proceed this way?

@ljharb
Copy link
Member

ljharb commented Nov 1, 2023

Modifying existing tests makes it a breaking change; if official node versions can be installed in alpine then that's not something we can break.

@asolopovas
Copy link
Contributor Author

asolopovas commented Nov 1, 2023

its not exactly breaking anything this should clarify what I mean:

  wsl_matrix:
    name: 'WSL nvm install'
    defaults:
      run:
          shell: wsl-bash {0}
    runs-on: windows-latest
    env:
      WSLENV: NVM_INSTALL_GITHUB_REPO:NVM_INSTALL_VERSION:/p
    strategy:
      fail-fast: false
      matrix:
        wsl-distrib:
          - Debian
          - Ubuntu-18.04
        npm-node-version:
          - '--lts'
          - '14'
          - '12'
          - '11'
          - '10'
        method:
          - ''
          - 'script'
    steps:
      - uses: Vampire/setup-wsl@v1
        with:
          distribution: ${{ matrix.wsl-distrib }}
          additional-packages: bash git curl ca-certificates wget
      - name: Retrieve nvm on WSL
        run: |
          if [ -z "${{ matrix.method }}" ]; then
            curl -fsSLo- "https://raw.githubusercontent.com/${NVM_INSTALL_GITHUB_REPO}/${NVM_INSTALL_VERSION}/install.sh" | bash
          else
            curl -fsSLo- "https://raw.githubusercontent.com/${NVM_INSTALL_GITHUB_REPO}/${NVM_INSTALL_VERSION}/install.sh" | METHOD="${{matrix.method}}" bash
          fi
          . "$HOME/.nvm/nvm.sh"
          nvm install ${{ matrix.npm-node-version }}

  wsl_matrix_unofficial:
    name: 'WSL nvm install'
    defaults:
      run:
          shell: wsl-bash {0}
    runs-on: windows-latest
    env:
      WSLENV: NVM_INSTALL_GITHUB_REPO:NVM_INSTALL_VERSION:/p
      NVM_NODEJS_ORG_MIRROR: https://unofficial-builds.nodejs.org/download/release
    strategy:
      fail-fast: false
      matrix:
        wsl-distrib:
          - Alpine
        npm-node-version:
          - '--lts'
          - '14'
          - '12'
          - '11'
          - '10'
        method:
          - ''
          - 'script'
    steps:
      - uses: Vampire/setup-wsl@v1
        with:
          distribution: ${{ matrix.wsl-distrib }}
          additional-packages: bash git curl ca-certificates wget
      - name: Retrieve nvm on WSL
        run: |
          if [ -z "${{ matrix.method }}" ]; then
            curl -fsSLo- "https://raw.githubusercontent.com/${NVM_INSTALL_GITHUB_REPO}/${NVM_INSTALL_VERSION}/install.sh" | bash
          else
            curl -fsSLo- "https://raw.githubusercontent.com/${NVM_INSTALL_GITHUB_REPO}/${NVM_INSTALL_VERSION}/install.sh" | METHOD="${{matrix.method}}" bash
          fi
          . "$HOME/.nvm/nvm.sh"
          nvm install ${{ matrix.npm-node-version }}

because Alpine supported only with unofficial NVM_NODEJS_ORG_MIRROR in order not to change anything in Ubuntu and Debian we have to test it separately.

@ljharb
Copy link
Member

ljharb commented Nov 1, 2023

If I'm understanding right, you want to stop testing the official mirror on alpine, and test the unofficial one instead.

i'm saying that installs currently work with the official mirror, which is why that's what's tested, and that needs to continue to work.

@asolopovas
Copy link
Contributor Author

asolopovas commented Nov 1, 2023

official mirror does not support alpine. it hasn't got the musl build.

@asolopovas
Copy link
Contributor Author

so what is the point of testing it against official mirror if build for alpine does not exist in it?

@ljharb
Copy link
Member

ljharb commented Nov 1, 2023

the tests currently successfully download and install a version of node on alpine. Does that node version not work?

@asolopovas
Copy link
Contributor Author

in reality it would be good to support the hole list of them, even they are unofficial they are built and hosted by nodejs why should it matter If I want to use alpine for my docker thats the only way I get it if not build it from source :
https://unofficial-builds.nodejs.org/download/release/v20.9.0/

node-v20.9.0-headers.tar.gz
node-v20.9.0-headers.tar.xz
node-v20.9.0-linux-armv6l.tar.gz
node-v20.9.0-linux-armv6l.tar.xz
node-v20.9.0-linux-loong64.tar.gz
node-v20.9.0-linux-loong64.tar.xz
node-v20.9.0-linux-riscv64.tar.gz
node-v20.9.0-linux-riscv64.tar.xz
node-v20.9.0-linux-x64-glibc-217.tar.gz
node-v20.9.0-linux-x64-glibc-217.tar.xz
node-v20.9.0-linux-x64-musl.tar.gz
node-v20.9.0-linux-x64-musl.tar.xz
node-v20.9.0-linux-x64-pointer-compression.tar.gz
node-v20.9.0-linux-x64-pointer-compression.tar.xz

@asolopovas
Copy link
Contributor Author

asolopovas commented Nov 1, 2023

I don't know how test can be successful if Alpine test can't pass without musl built.
https://nodejs.org/dist/v20.9.0/:

node-v20.9.0-aix-ppc64.tar.gz
node-v20.9.0-arm64.msi
node-v20.9.0-darwin-arm64.tar.gz
node-v20.9.0-darwin-arm64.tar.xz
node-v20.9.0-darwin-x64.tar.gz
node-v20.9.0-darwin-x64.tar.xz
node-v20.9.0-headers.tar.gz
node-v20.9.0-headers.tar.xz
node-v20.9.0-linux-arm64.tar.gz
node-v20.9.0-linux-arm64.tar.xz
node-v20.9.0-linux-armv7l.tar.gz
node-v20.9.0-linux-armv7l.tar.xz
node-v20.9.0-linux-ppc64le.tar.gz
node-v20.9.0-linux-ppc64le.tar.xz
node-v20.9.0-linux-s390x.tar.gz
node-v20.9.0-linux-s390x.tar.xz
node-v20.9.0-linux-x64.tar.gz
node-v20.9.0-linux-x64.tar.xz
node-v20.9.0-win-arm64.7z
node-v20.9.0-win-arm64.zip
node-v20.9.0-win-x64.7z
node-v20.9.0-win-x64.zip
node-v20.9.0-win-x86.7z
node-v20.9.0-win-x86.zip
node-v20.9.0-x64.msi
node-v20.9.0-x86.msi
node-v20.9.0.pkg
node-v20.9.0.tar.gz
node-v20.9.0.tar.xz

@ljharb
Copy link
Member

ljharb commented Nov 1, 2023

I understand that you're saying "without compiling", but if it currently works with compiling, then that's functionality we can't break.

@ljharb
Copy link
Member

ljharb commented Nov 1, 2023

hmm, i'm trying out verifying in these tests that node -v works - is this the error that means official node is broken on alpine? https://github.com/ljharb/nvm/actions/runs/6726177984/job/18281961710

@asolopovas
Copy link
Contributor Author

your test failing not because it doesn't build it but because it trys to pull it from wrong location, with separate test it will not be failing but passing:

Binary download from https://nodejs.org/dist/v20.9.0/node-v20.9.0-linux-musl-x64.tar.gz failed, trying source.
grep: /root/.nvm/.cache/bin/node-v20.9.0-linux-musl-x64/node-v20.9.0-linux-musl-x64.tar.gz: No such file or directory
Provided file to checksum does not exist.
/root/.nvm/nvm.sh: line 2133: TMPDIR: unbound variable
Error: Process completed with exit code 1.

@ljharb
Copy link
Member

ljharb commented Nov 1, 2023

right, i understand that with your change, and the unofficial mirror, the tests will pass.

i'm trying to understand why the fact that your change makes the existing tests fail, is acceptable. is that perhaps because currently, with the official mirror, nvm installs node fine except that the node it installs ends up broken and unusable?

@asolopovas
Copy link
Contributor Author

this is what it does currently on alpine docker images:

nvm install 18.18.0
Downloading and installing node v18.18.0...
Downloading https://nodejs.org/dist/v18.18.0/node-v18.18.0-linux-x64.tar.xz...
####################################################################################################### 100.0%
Computing checksum with sha256sum
Checksums matched!
Now using node v18.18.0
Creating default alias: default -> 18.18.0 (-> v18.18.0 *)
bash-5.1# node --version
bash: /root/.nvm/versions/node/v18.18.0/bin/node: No such file or directory
bash-5.1#

@ljharb
Copy link
Member

ljharb commented Nov 1, 2023

in docker, scoping can get weird; what if you run nvm install 18.18.0 && node --version?

@asolopovas
Copy link
Contributor Author

asolopovas commented Nov 1, 2023

bash-5.1# pwd
/root/.nvm/versions/node/v18.18.0/bin
bash-5.1# ls
corepack node npm npx
bash-5.1# ./node
bash: ./node: No such file or directory

@asolopovas
Copy link
Contributor Author

look I been using node in alpine for very long time it does not work with official version and official download, it only works with musl builds

@asolopovas
Copy link
Contributor Author

and by the way node file has executable permissions it just it throws this strange error No such file or directory

@ljharb
Copy link
Member

ljharb commented Nov 1, 2023

Thanks for bearing with me, I'm just trying to understand, since I don't use alpine :-)

It seems like I'll be removing those Alpine tests on master, and then rebasing this PR, which should then pass. I'll also add alpine tests with the unofficial mirror as part of rebasing.

@asolopovas
Copy link
Contributor Author

No problem at all, I have already updated everything if you want I can push it?

@ljharb
Copy link
Member

ljharb commented Nov 1, 2023

Thanks, that'd be great!

@ljharb ljharb marked this pull request as ready for review November 1, 2023 23:54
@asolopovas
Copy link
Contributor Author

tests still are not picking up mirror var

@ljharb ljharb changed the title Proper nvm_get_arch value for alpine linux [Fix] nvm_get_arch: proper value for alpine linux Nov 2, 2023
@ljharb ljharb merged commit ef7fc2f into nvm-sh:master Nov 2, 2023
127 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants