zfs-2.2.7 patchset

.github/CONTRIBUTING.md

@@ -145,22 +145,18 @@ Once everything is in good shape and the details have been worked out you can re
 Any required reviews can then be finalized and the pull request merged.
 
 #### Tests and Benchmarks
-* Every pull request will by tested by the buildbot on multiple platforms by running the [zfs-tests.sh and zloop.sh](
+* Every pull request is tested using a GitHub Actions workflow on multiple platforms by running the [zfs-tests.sh and zloop.sh](
 https://openzfs.github.io/openzfs-docs/Developer%20Resources/Building%20ZFS.html#running-zloop-sh-and-zfs-tests-sh) test suites.
+`.github/workflows/scripts/generate-ci-type.py` is used to determine whether the pull request is nonbehavior, i.e., not introducing behavior changes of any code, configuration or tests. If so, the CI will run on fewer platforms and only essential sanity tests will run. You can always override this by adding `ZFS-CI-Type` line to your commit message:
+  * If your last commit (or `HEAD` in git terms) contains a line `ZFS-CI-Type: quick`, quick mode is forced regardless of what files are changed.
+  * Otherwise, if any commit in a PR contains a line `ZFS-CI-Type: full`, full mode is forced.
 * To verify your changes conform to the [style guidelines](
 https://github.com/openzfs/zfs/blob/master/.github/CONTRIBUTING.md#style-guides
 ), please run `make checkstyle` and resolve any warnings.
-* Static code analysis of each pull request is performed by the buildbot; run `make lint` to check your changes.
-* Test cases should be provided when appropriate.
-This includes making sure new features have adequate code coverage.
+* Code analysis is performed by [CodeQL](https://codeql.github.com/) for each pull request.
+* Test cases should be provided when appropriate.  This includes making sure new features have adequate code coverage.
 * If your pull request improves performance, please include some benchmarks.
-* The pull request must pass all required [ZFS
-Buildbot](http://build.zfsonlinux.org/) builders before
-being accepted. If you are experiencing intermittent TEST
-builder failures, you may be experiencing a [test suite
-issue](https://github.com/openzfs/zfs/issues?q=is%3Aissue+is%3Aopen+label%3A%22Type%3A+Test+Suite%22).
-There are also various [buildbot options](https://openzfs.github.io/openzfs-docs/Developer%20Resources/Buildbot%20Options.html)
-to control how changes are tested.
+* The pull request must pass all CI checks before being accepted.
 
 ### Testing
 All help is appreciated! If you're in a position to run the latest code

.github/workflows/checkstyle.yaml

@@ -4,6 +4,10 @@ on:
   push:
   pull_request:
 
+concurrency:
+  group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
+  cancel-in-progress: true
+
 jobs:
   checkstyle:
     runs-on: ubuntu-22.04
@@ -13,15 +17,11 @@ jobs:
         ref: ${{ github.event.pull_request.head.sha }}
     - name: Install dependencies
       run: |
-        # https://github.com/orgs/community/discussions/47863
-        sudo apt-mark hold grub-efi-amd64-signed
-        sudo apt-get update --fix-missing
-        sudo apt-get upgrade
-        sudo xargs --arg-file=${{ github.workspace }}/.github/workflows/build-dependencies.txt apt-get install -qq
-        sudo xargs --arg-file=${{ github.workspace }}/.github/workflows/checkstyle-dependencies.txt apt-get install -qq
-        sudo python3 -m pip install --quiet flake8
-        sudo apt-get clean
-
+        # for x in lxd core20 snapd; do sudo snap remove $x; done
+        sudo apt-get purge -y snapd google-chrome-stable firefox
+        ONLY_DEPS=1 .github/workflows/scripts/qemu-3-deps.sh ubuntu22
+        sudo apt-get install -y cppcheck devscripts mandoc pax-utils shellcheck
+        sudo python -m pipx install --quiet flake8
         # confirm that the tools are installed
         # the build system doesn't fail when they are not
         checkbashisms --version
@@ -31,8 +31,13 @@ jobs:
         shellcheck --version
     - name: Prepare
       run: |
+        sed -i '/DEBUG_CFLAGS="-Werror"/s/^/#/' config/zfs-build.m4
         ./autogen.sh
+    - name: Configure
+      run: |
         ./configure
+    - name: Make
+      run: |
         make -j$(nproc) --no-print-directory --silent
     - name: Checkstyle
       run: |

.github/workflows/codeql.yml

@@ -4,10 +4,14 @@ on:
   push:
   pull_request:
 
+concurrency:
+  group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
+  cancel-in-progress: true
+
 jobs:
   analyze:
     name: Analyze
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-22.04
     permissions:
       actions: read
       contents: read
@@ -27,15 +31,15 @@ jobs:
       uses: actions/checkout@v4
 
     - name: Initialize CodeQL
-      uses: github/codeql-action/init@v2
+      uses: github/codeql-action/init@v3
       with:
         config-file: .github/codeql-${{ matrix.language }}.yml
         languages: ${{ matrix.language }}
 
     - name: Autobuild
-      uses: github/codeql-action/autobuild@v2
+      uses: github/codeql-action/autobuild@v3
 
     - name: Perform CodeQL Analysis
-      uses: github/codeql-action/analyze@v2
+      uses: github/codeql-action/analyze@v3
       with:
         category: "/language:${{matrix.language}}"

.github/workflows/scripts/README.md

@@ -0,0 +1,14 @@
+
+Workflow for each operating system:
+- install qemu on the github runner
+- download current cloud image of operating system
+- start and init that image via cloud-init
+- install dependencies and poweroff system
+- start system and build openzfs and then poweroff again
+- clone build system and start 2 instances of it
+- run functional testings and complete in around 3h
+- when tests are done, do some logfile preparing
+- show detailed results for each system
+- in the end, generate the job summary
+
+/TR 14.09.2024

.github/workflows/scripts/generate-ci-type.py

@@ -0,0 +1,107 @@
+#!/usr/bin/env python3
+
+"""
+Determine the CI type based on the change list and commit message.
+
+Prints "quick" if (explicity required by user):
+- the *last* commit message contains 'ZFS-CI-Type: quick'
+or if (heuristics):
+- the files changed are not in the list of specified directories, and
+- all commit messages do not contain 'ZFS-CI-Type: full'
+
+Otherwise prints "full".
+"""
+
+import sys
+import subprocess
+import re
+
+"""
+Patterns of files that are not considered to trigger full CI.
+Note: not using pathlib.Path.match() because it does not support '**'
+"""
+FULL_RUN_IGNORE_REGEX = list(map(re.compile, [
+    r'.*\.md',
+    r'.*\.gitignore'
+]))
+
+"""
+Patterns of files that are considered to trigger full CI.
+"""
+FULL_RUN_REGEX = list(map(re.compile, [
+    r'cmd.*',
+    r'configs/.*',
+    r'META',
+    r'.*\.am',
+    r'.*\.m4',
+    r'autogen\.sh',
+    r'configure\.ac',
+    r'copy-builtin',
+    r'contrib',
+    r'etc',
+    r'include',
+    r'lib/.*',
+    r'module/.*',
+    r'scripts/.*',
+    r'tests/.*',
+    r'udev/.*'
+]))
+
+if __name__ == '__main__':
+
+    prog = sys.argv[0]
+
+    if len(sys.argv) != 3:
+        print(f'Usage: {prog} <head_ref> <base_ref>')
+        sys.exit(1)
+
+    head, base = sys.argv[1:3]
+
+    def output_type(type, reason):
+        print(f'{prog}: will run {type} CI: {reason}', file=sys.stderr)
+        print(type)
+        sys.exit(0)
+
+    # check last (HEAD) commit message
+    last_commit_message_raw = subprocess.run([
+        'git', 'show', '-s', '--format=%B', 'HEAD'
+    ], check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+
+    for line in last_commit_message_raw.stdout.decode().splitlines():
+        if line.strip().lower() == 'zfs-ci-type: quick':
+            output_type('quick', f'explicitly requested by HEAD commit {head}')
+
+    # check all commit messages
+    all_commit_message_raw = subprocess.run([
+        'git', 'show', '-s',
+        '--format=ZFS-CI-Commit: %H%n%B', f'{head}...{base}'
+    ], check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    all_commit_message = all_commit_message_raw.stdout.decode().splitlines()
+
+    commit_ref = head
+    for line in all_commit_message:
+        if line.startswith('ZFS-CI-Commit:'):
+            commit_ref = line.lstrip('ZFS-CI-Commit:').rstrip()
+        if line.strip().lower() == 'zfs-ci-type: full':
+            output_type('full', f'explicitly requested by commit {commit_ref}')
+
+    # check changed files
+    changed_files_raw = subprocess.run([
+        'git', 'diff', '--name-only', head, base
+    ], check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    changed_files = changed_files_raw.stdout.decode().splitlines()
+
+    for f in changed_files:
+        for r in FULL_RUN_IGNORE_REGEX:
+            if r.match(f):
+                break
+        else:
+            for r in FULL_RUN_REGEX:
+                if r.match(f):
+                    output_type(
+                        'full',
+                        f'changed file "{f}" matches pattern "{r.pattern}"'
+                        )
+
+    # catch-all
+    output_type('quick', 'no changed file matches full CI patterns')