Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

.zuul, playbooks: Run the CI on Arch Linux #1588

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

debarshiray
Copy link
Member

@debarshiray
Copy link
Member Author

/cc @Foxboron

@Foxboron
Copy link
Collaborator

Foxboron commented Nov 5, 2024

Cool, this looks good to me

@debarshiray debarshiray force-pushed the wip/rishi/issues-1438 branch from ae7f071 to 9d882b5 Compare November 6, 2024 00:05
@debarshiray
Copy link
Member Author

recheck

@danpawlik
Copy link
Contributor

recheck

@danpawlik
Copy link
Contributor

Updated image once again.

@danpawlik
Copy link
Contributor

recheck

@debarshiray
Copy link
Member Author

recheck

playbooks/arch/dependencies.yaml Outdated Show resolved Hide resolved
nodes:
- name: arch
label: arch-linux
pre-run: playbooks/arch/setup.yaml
Copy link
Contributor

@danpawlik danpawlik Nov 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so if something will fail in pre-run, you might have RETRY_LIMIT.
Maybe try to do:

- job:
    name: system-test-arch-commands-options
    description: Run Toolbx's commands-options system tests on Arch Linux
    timeout: 6300
    nodeset:
      nodes:
        - name: arch
          label: arch-linux
    run:
      - playbooks/arch/setup.yaml
      - playbooks/system-test-commands-options.yaml

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was intentional to use pre-run for downloads that can fail due to passing network problems, outdated caches, etc..

We use zuul.attempts > 1 to decide if the DNF cache needs to be forcibly updated on Fedora Rawhide nodes, or to resolve problems arising from the DNF5 transition.

Downloading the Go modules is another thing that has proven to be flaky and benefits from this. It's still not 100% robust, but a lot more than what it used to be.

@debarshiray
Copy link
Member Author

@Foxboron It seems that installing Podman or Toolbx on an Arch Linux host doesn't install fuse-overlayfs. Am I reading it correctly?

We had recently removed fuse-overlayfs as a weak dependency of Podman, through the containers-common package, in Fedora as part of an effort to deprecate it, only to later give up and restored it.

Users with Podman containers and images that predate the use of the Linux kernel's overlay file system still need fuse-overlayfs. Otherwise it leads to:
https://discussion.fedoraproject.org/t/rpm-ostree-update-breaks-toolbox-fedora-40/120095
#1512

Another problem is if Podman is supposed to be usable on a live media that itself uses the Linux kernel's overlayfs. It's not possible to use overlayfs on top of another overlayfs.

This led to a discussion that ended up restoring fuse-overlayfs:
coreos/fedora-coreos-tracker#1749
containers/common#2203

It's fine to drop fuse-overlayfs if these scenarios don't matter for Arch Linux. Otherwise, it might be important.

@debarshiray debarshiray force-pushed the wip/rishi/issues-1438 branch 4 times, most recently from be501ee to 565b46e Compare November 12, 2024 17:00
@debarshiray debarshiray force-pushed the wip/rishi/issues-1438 branch from 565b46e to 7b7b8a9 Compare November 12, 2024 21:04
@debarshiray debarshiray force-pushed the wip/rishi/issues-1438 branch from 7b7b8a9 to 1ac6b53 Compare November 14, 2024 01:38
@debarshiray debarshiray force-pushed the wip/rishi/issues-1438 branch from 1ac6b53 to e8315a1 Compare November 15, 2024 20:50
This is a step towards running the CI on Arch Linux.

containers#1438
@debarshiray debarshiray force-pushed the wip/rishi/issues-1438 branch from e8315a1 to 6491082 Compare November 17, 2024 10:27
@debarshiray debarshiray force-pushed the wip/rishi/issues-1438 branch from 6491082 to eed02a8 Compare November 17, 2024 15:45
@debarshiray
Copy link
Member Author

The runtime-environment tests are failing with:

arch | not ok 3 network: /etc/resolv.conf inside the default container
arch | # tags: arch-fedora runtime-environment
arch | # (from function `assert_line' in file test/system/libs/bats-assert/src/assert.bash, line 488,
arch | #  in test file test/system/203-network.bats, line 72)
arch | #   `assert_line --index 0 "/run/host/etc/resolv.conf"' failed
arch | # ~ ~/src/github.com/containers/toolbox
arch | #
arch | # -- line differs --
arch | # index    : 0
arch | # expected : /run/host/etc/resolv.conf
arch | # actual   : /run/host/run/systemd/resolve/stub-resolv.conf
arch | # --
arch | #
...
...
arch | not ok 36 ulimit: real-time non-blocking time (hard)
arch | # tags: arch-fedora runtime-environment
arch | # (from function `assert_line' in file test/system/libs/bats-assert/src/assert.bash, line 488,
arch | #  in test file test/system/210-ulimit.bats, line 45)
arch | #   `assert_line --index 0 "$limit"' failed
arch | #
arch | # -- line differs --
arch | # index    : 0
arch | # expected : unlimited
arch | # actual   :
arch | # --
arch | #

I might have to install an Arch Linux host to see how things are on the host.

@debarshiray debarshiray force-pushed the wip/rishi/issues-1438 branch from eed02a8 to f900585 Compare November 17, 2024 18:34
@debarshiray debarshiray force-pushed the wip/rishi/issues-1438 branch from f789d30 to bb1d09c Compare November 18, 2024 19:10
Some Arch Linux hosts have /etc/resolv.conf as an absolute symbolic link
to /run/systemd/resolve/stub-resolv.conf, instead of being a relative
symbolic link to ../run/systemd/resolve/stub-resolv.conf or a regular
file.  eg., the images built by arch-boxes [1].

This changes the target that the Toolbx container's /etc/resolv.conf
points at and confuses the tests [2].

Ideally, these host operating systems should be fixed to use relative
symbolic links.  This is highlighted by skipping the tests, because
there's no point in failing them until that happens.

This is a step towards running the CI on Arch Linux.

[1] https://gitlab.archlinux.org/archlinux/arch-boxes
    https://geo.mirror.pkgbuild.com/images/latest/

[2] Commit 88a95b0
    containers@88a95b07af335be2
    containers#187

containers#1438
@debarshiray debarshiray force-pushed the wip/rishi/issues-1438 branch from bb1d09c to 4d72443 Compare November 18, 2024 20:09
@debarshiray
Copy link
Member Author

The runtime-environment tests are failing with:

arch | not ok 3 network: /etc/resolv.conf inside the default container
arch | # tags: arch-fedora runtime-environment
arch | # (from function `assert_line' in file test/system/libs/bats-assert/src/assert.bash, line 488,
arch | #  in test file test/system/203-network.bats, line 72)
arch | #   `assert_line --index 0 "/run/host/etc/resolv.conf"' failed
arch | # ~ ~/src/github.com/containers/toolbox
arch | #
arch | # -- line differs --
arch | # index    : 0
arch | # expected : /run/host/etc/resolv.conf
arch | # actual   : /run/host/run/systemd/resolve/stub-resolv.conf
arch | # --
arch | #

The problem with /etc/resolv.conf is that the images generated by arch-boxes have /etc/resolv.conf as an absolute symbolic link, instead of relative.

I have now added a note to the Toolbx tests to detect such cases without failing. I will send a merge request to arch-boxes once I get access to gitlab.archlinux.org.

@debarshiray
Copy link
Member Author

There's also this failure that I didn't notice before:

arch | not ok 7 network: DNS inside the default container
arch | # tags: arch-fedora runtime-environment
arch | # (from function `assert_success' in file test/system/libs/bats-assert/src/assert.bash, line 114,
arch | #  in test file test/system/203-network.bats, line 228)
arch | #   `assert_success' failed
arch | # ~ ~/src/github.com/containers/toolbox
arch | #
arch | # -- command failed --
arch | # status : 127
arch | # output :
arch | # --
arch | #

@debarshiray
Copy link
Member Author

I need to context switch from this for a few days. So, I have merged the finished patches from this pull request through #1597

@debarshiray
Copy link
Member Author

The runtime-environment tests are failing with:

arch | not ok 3 network: /etc/resolv.conf inside the default container
arch | # tags: arch-fedora runtime-environment
arch | # (from function `assert_line' in file test/system/libs/bats-assert/src/assert.bash, line 488,
arch | #  in test file test/system/203-network.bats, line 72)
arch | #   `assert_line --index 0 "/run/host/etc/resolv.conf"' failed
arch | # ~ ~/src/github.com/containers/toolbox
arch | #
arch | # -- line differs --
arch | # index    : 0
arch | # expected : /run/host/etc/resolv.conf
arch | # actual   : /run/host/run/systemd/resolve/stub-resolv.conf
arch | # --
arch | #

The problem with /etc/resolv.conf is that the images generated by arch-boxes have /etc/resolv.conf as an absolute symbolic link, instead of relative.

I have now added a note to the Toolbx tests to detect such cases without failing. I will send a merge request to arch-boxes once I get access to gitlab.archlinux.org.

There was already an attempt to change /etc/resolv.conf into a relative symbolic link:
https://gitlab.archlinux.org/archlinux/arch-boxes/-/merge_requests/194
https://gitlab.archlinux.org/archlinux/arch-boxes/-/commit/ff22bfc49ac60001849fcce73fe91e5fbabcf03a

... but it had to be reverted:
https://gitlab.archlinux.org/archlinux/arch-install-scripts/-/issues/55
https://gitlab.archlinux.org/archlinux/arch-boxes/-/commit/5768351a8f93e102fb8ab5d248fd35fbe53863ff

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants