Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

set minimum minikube legacy version to v1.26.0 for containerd cri #17220

Merged
merged 1 commit into from
Sep 10, 2023
Merged

set minimum minikube legacy version to v1.26.0 for containerd cri #17220

merged 1 commit into from
Sep 10, 2023

Conversation

prezha
Copy link
Contributor

@prezha prezha commented Sep 9, 2023

fixes #17219

solution is to use at least minikube v1.26.0 for this test to be able to pass

details:

v1.25.2 (and earlier versions)

$ make integration -e TEST_ARGS="-minikube-start-args='--driver=docker --container-runtime=containerd' -test.run TestMissingContainerUpgrade --cleanup=false"
go test -ldflags="-X k8s.io/minikube/pkg/version.version=v1.31.2 -X k8s.io/minikube/pkg/version.isoVersion=v1.31.0-1692872107-17120 -X k8s.io/minikube/pkg/version.gitCommitID="14c9e44bc46f135f12e8131cd386499871695824-dirty" -X k8s.io/minikube/pkg/version.storageProvisionerVersion=v5" -v -test.timeout=90m ./test/integration --tags="integration " -minikube-start-args='--driver=docker --container-runtime=containerd' -test.run TestMissingContainerUpgrade --cleanup=false 2>&1 | tee "./out/testout_14c9e44bc.txt"
Found 16 cores, limiting parallelism with --test.parallel=9
=== RUN   TestMissingContainerUpgrade
=== PAUSE TestMissingContainerUpgrade
=== CONT  TestMissingContainerUpgrade
    version_upgrade_test.go:324: (dbg) Run:  /tmp/minikube-v1.25.2.2226404152.exe start -p missing-upgrade-755862 --memory=2200 --driver=docker --container-runtime=containerd
    version_upgrade_test.go:324: (dbg) Done: /tmp/minikube-v1.25.2.2226404152.exe start -p missing-upgrade-755862 --memory=2200 --driver=docker --container-runtime=containerd: (1m1.945083447s)
    version_upgrade_test.go:333: (dbg) Run:  docker stop missing-upgrade-755862
    version_upgrade_test.go:333: (dbg) Done: docker stop missing-upgrade-755862: (10.54743632s)
    version_upgrade_test.go:338: (dbg) Run:  docker rm missing-upgrade-755862
    version_upgrade_test.go:344: (dbg) Run:  out/minikube start -p missing-upgrade-755862 --memory=2200 --alsologtostderr -v=1 --driver=docker --container-runtime=containerd
    version_upgrade_test.go:344: (dbg) Non-zero exit: out/minikube start -p missing-upgrade-755862 --memory=2200 --alsologtostderr -v=1 --driver=docker --container-runtime=containerd: exit status 90 (1m39.726507602s)
...
                stderr:
                time="2023-09-09T00:17:56Z" level=fatal msg="getting the runtime version: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
                I0909 01:18:12.622117 2431489 ssh_runner.go:195] Run: sudo /usr/bin/crictl version
                I0909 01:18:12.649987 2431489 out.go:177] 
                W0909 01:18:12.653521 2431489 out.go:239] X Exiting due to RUNTIME_ENABLE: Failed to start container runtime: Temporary Error: sudo /usr/bin/crictl version: Process exited with status 1
                stdout:
...
--- FAIL: TestMissingContainerUpgrade (176.65s)
FAIL
Tests completed in 2m56.645994233s (result code 1)
FAIL    k8s.io/minikube/test/integration        176.685s
FAIL

note: unknown service runtime.v1alpha2.RuntimeService error indicates that the cri is disabled, and, indeed, there's disabled_plugins = ["cri"] from default containerd's config:

$ minikube ssh -p missing-upgrade-755862 sudo cat /etc/containerd/config.toml
#   Copyright 2018-2020 Docker Inc.

#   Licensed under the Apache License, Version 2.0 (the "License");
#   you may not use this file except in compliance with the License.
#   You may obtain a copy of the License at

#       http://www.apache.org/licenses/LICENSE-2.0

#   Unless required by applicable law or agreed to in writing, software
#   distributed under the License is distributed on an "AS IS" BASIS,
#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#   See the License for the specific language governing permissions and
#   limitations under the License.

disabled_plugins = ["cri"]

#root = "/var/lib/containerd"
#state = "/run/containerd"
#subreaper = true
#oom_score = 0

#[grpc]
#  address = "/run/containerd/containerd.sock"
#  uid = 0
#  gid = 0

#[debug]
#  address = "/run/containerd/debug.sock"
#  uid = 0
#  gid = 0
#  level = "info"

and so, we cannot generateContainerdConfig(), which then leaves default containerd intact, following with expected failure with the above error

v1.26.0+

$ make integration -e TEST_ARGS="-minikube-start-args='--driver=docker --container-runtime=containerd' -test.run TestMissingContainerUpgrade --cleanup=false"
go test -ldflags="-X k8s.io/minikube/pkg/version.version=v1.31.2 -X k8s.io/minikube/pkg/version.isoVersion=v1.31.0-1692872107-17120 -X k8s.io/minikube/pkg/version.gitCommitID="14c9e44bc46f135f12e8131cd386499871695824-dirty" -X k8s.io/minikube/pkg/version.storageProvisionerVersion=v5" -v -test.timeout=90m ./test/integration --tags="integration " -minikube-start-args='--driver=docker --container-runtime=containerd' -test.run TestMissingContainerUpgrade --cleanup=false 2>&1 | tee "./out/testout_14c9e44bc.txt"
Found 16 cores, limiting parallelism with --test.parallel=9
=== RUN   TestMissingContainerUpgrade
=== PAUSE TestMissingContainerUpgrade
=== CONT  TestMissingContainerUpgrade
    version_upgrade_test.go:324: (dbg) Run:  /tmp/minikube-v1.26.0.1879240289.exe start -p missing-upgrade-233051 --memory=2200 --driver=docker --container-runtime=containerd
    version_upgrade_test.go:324: (dbg) Done: /tmp/minikube-v1.26.0.1879240289.exe start -p missing-upgrade-233051 --memory=2200 --driver=docker --container-runtime=containerd: (1m8.102403871s)
    version_upgrade_test.go:333: (dbg) Run:  docker stop missing-upgrade-233051
    version_upgrade_test.go:333: (dbg) Done: docker stop missing-upgrade-233051: (10.599567423s)
    version_upgrade_test.go:338: (dbg) Run:  docker rm missing-upgrade-233051
    version_upgrade_test.go:344: (dbg) Run:  out/minikube start -p missing-upgrade-233051 --memory=2200 --alsologtostderr -v=1 --driver=docker --container-runtime=containerd
    version_upgrade_test.go:344: (dbg) Done: out/minikube start -p missing-upgrade-233051 --memory=2200 --alsologtostderr -v=1 --driver=docker --container-runtime=containerd: (51.522710075s)
    helpers_test.go:183: skipping cleanup of missing-upgrade-233051 (--cleanup=false)
--- PASS: TestMissingContainerUpgrade (133.79s)
PASS
Tests completed in 2m13.788954604s (result code 0)
ok      k8s.io/minikube/test/integration        133.822s

here, thanks to how the kicbase was built (via Dockerfile), we have expected containerd config (patched to our needs):

$ minikube ssh -p missing-upgrade-233051 sudo cat /etc/containerd/config.toml
version = 2
root = "/var/lib/containerd"                                     
state = "/run/containerd"
oom_score = 0      
# imports

[grpc]
  address = "/run/containerd/containerd.sock"
  uid = 0
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216

[debug]
  address = ""
  uid = 0
  gid = 0
  level = ""

[metrics]
  address = ""
  grpc_histogram = false

[cgroup]
  path = ""

[plugins]
  [plugins."io.containerd.monitor.v1.cgroups"]
    no_prometheus = false
  [plugins."io.containerd.grpc.v1.cri"]
    stream_server_address = ""
    stream_server_port = "10010"
    enable_selinux = false
    sandbox_image = "registry.k8s.io/pause:3.7"
    stats_collect_period = 10
    enable_tls_streaming = false
    max_container_log_line_size = 16384
    restrict_oom_score_adj = false

    [plugins."io.containerd.grpc.v1.cri".containerd]
      discard_unpacked_layers = true
      snapshotter = "overlayfs"
      [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]
        runtime_type = "io.containerd.runc.v2"
      [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime]
        runtime_type = ""
        runtime_engine = ""
        runtime_root = ""
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
          runtime_type = "io.containerd.runc.v2"
          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
            SystemdCgroup = true

    [plugins."io.containerd.grpc.v1.cri".cni]
      bin_dir = "/opt/cni/bin"
      conf_dir = "/etc/cni/net.d"
      conf_template = ""
    [plugins."io.containerd.grpc.v1.cri".registry]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
          endpoint = ["https://registry-1.docker.io"]
  [plugins."io.containerd.service.v1.diff-service"]
    default = ["walking"]
  [plugins."io.containerd.gc.v1.scheduler"]
    pause_threshold = 0.02
    deletion_threshold = 0
    mutation_threshold = 100
    schedule_delay = "0s"
    startup_delay = "100ms"

note: i was thinking about making minikube more resilient to this and support older versions by changing our code to supply our initial containerd config if we see that the default one is present instead, but since we change which containerd version and then also change the initial config (format & values) we supply over time, that's not a viable option

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Sep 9, 2023
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: prezha

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 9, 2023
@prezha
Copy link
Contributor Author

prezha commented Sep 9, 2023

/ok-to-test

@k8s-ci-robot k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Sep 9, 2023
@minikube-pr-bot
Copy link

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 17220) |
+----------------+----------+---------------------+
| minikube start | 51.2s    | 50.5s               |
| enable ingress | 27.5s    | 27.4s               |
+----------------+----------+---------------------+

Times for minikube ingress: 27.1s 28.0s 27.5s 27.2s 27.6s
Times for minikube (PR 17220) ingress: 27.1s 29.6s 27.2s 24.7s 28.6s

Times for minikube (PR 17220) start: 49.8s 50.0s 50.6s 50.3s 51.8s
Times for minikube start: 49.8s 50.7s 53.6s 51.3s 50.5s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 17220) |
+----------------+----------+---------------------+
| minikube start | 22.8s    | 22.9s               |
| enable ingress | 21.4s    | 21.0s               |
+----------------+----------+---------------------+

Times for minikube (PR 17220) start: 21.5s 22.4s 24.6s 21.7s 24.5s
Times for minikube start: 22.2s 21.0s 21.4s 25.0s 24.2s

Times for minikube (PR 17220) ingress: 20.9s 22.9s 20.8s 20.8s 19.8s
Times for minikube ingress: 21.8s 20.8s 22.9s 20.9s 20.8s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 17220) |
+----------------+----------+---------------------+
| minikube start | 23.8s    | 21.9s               |
| enable ingress | 31.3s    | 29.1s               |
+----------------+----------+---------------------+

Times for minikube ingress: 32.4s 31.3s 31.4s 30.3s 31.3s
Times for minikube (PR 17220) ingress: 31.3s 31.3s 31.3s 32.3s 19.3s

Times for minikube start: 24.1s 22.6s 23.6s 24.5s 24.2s
Times for minikube (PR 17220) start: 22.9s 23.5s 20.7s 20.9s 21.6s

@minikube-pr-bot
Copy link

These are the flake rates of all failed tests.

Environment Failed Tests Flake Rate (%)
Docker_Linux_containerd TestAddons/parallel/Headlamp (gopogh) 0.00 (chart)
KVM_Linux TestStartStop/group/default-k8s-diff-port/serial/VerifyKubernetesImages (gopogh) 2.45 (chart)
Hyperkit_macOS TestNetworkPlugins/group/bridge/Start (gopogh) 2.56 (chart)
QEMU_macOS TestFunctional/parallel/CpCmd (gopogh) 2.65 (chart)
QEMU_macOS TestFunctional/parallel/FileSync (gopogh) 2.65 (chart)
QEMU_macOS TestFunctional/parallel/NonActiveRuntimeDisabled (gopogh) 2.65 (chart)
QEMU_macOS TestFunctional/parallel/UpdateContextCmd/no_minikube_cluster (gopogh) 2.65 (chart)
QEMU_macOS TestFunctional/parallel/Version/components (gopogh) 2.65 (chart)
QEMU_macOS TestFunctional/serial/LogsCmd (gopogh) 2.65 (chart)
KVM_Linux TestStartStop/group/default-k8s-diff-port/serial/AddonExistsAfterStop (gopogh) 3.07 (chart)
KVM_Linux TestStartStop/group/default-k8s-diff-port/serial/Pause (gopogh) 3.07 (chart)
KVM_Linux TestStartStop/group/default-k8s-diff-port/serial/SecondStart (gopogh) 3.07 (chart)
KVM_Linux TestStartStop/group/default-k8s-diff-port/serial/UserAppExistsAfterStop (gopogh) 3.07 (chart)
QEMU_macOS TestFunctional/parallel/UpdateContextCmd/no_clusters (gopogh) 3.31 (chart)
QEMU_macOS TestFunctional/parallel/CertSync (gopogh) 3.97 (chart)
QEMU_macOS TestFunctional/parallel/DashboardCmd (gopogh) 3.97 (chart)
QEMU_macOS TestFunctional/parallel/DockerEnv/bash (gopogh) 3.97 (chart)
QEMU_macOS TestFunctional/parallel/ImageCommands/ImageBuild (gopogh) 3.97 (chart)
QEMU_macOS TestFunctional/parallel/ImageCommands/ImageListJson (gopogh) 3.97 (chart)
QEMU_macOS TestFunctional/parallel/ImageCommands/ImageListTable (gopogh) 3.97 (chart)
QEMU_macOS TestFunctional/parallel/ImageCommands/ImageLoadDaemon (gopogh) 3.97 (chart)
QEMU_macOS TestFunctional/parallel/ImageCommands/ImageLoadFromFile (gopogh) 3.97 (chart)
QEMU_macOS TestFunctional/parallel/ImageCommands/ImageReloadDaemon (gopogh) 3.97 (chart)
QEMU_macOS TestFunctional/parallel/ImageCommands/ImageSaveToFile (gopogh) 3.97 (chart)
QEMU_macOS TestFunctional/parallel/ImageCommands/ImageTagAndLoadDaemon (gopogh) 3.97 (chart)
QEMU_macOS TestFunctional/parallel/NodeLabels (gopogh) 3.97 (chart)
QEMU_macOS TestFunctional/parallel/ServiceCmd/DeployApp (gopogh) 3.97 (chart)
QEMU_macOS TestFunctional/parallel/ServiceCmd/Format (gopogh) 3.97 (chart)
QEMU_macOS TestFunctional/parallel/ServiceCmd/HTTPS (gopogh) 3.97 (chart)
QEMU_macOS TestFunctional/parallel/ServiceCmd/JSONOutput (gopogh) 3.97 (chart)
More tests... Continued...

Too many tests failed - See test logs for more details.

To see the flake rates of all tests by environment, click here.

@afbjorklund
Copy link
Collaborator

That configuration is from docker, not from containerd.

We need something that removes it (or more like : doesn't disable it), when using containerd

@medyagh medyagh merged commit a46ec98 into kubernetes:master Sep 10, 2023
27 of 29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@medyagh medyagh Awaiting requested review from medyagh

@spowelljr spowelljr Awaiting requested review from spowelljr

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

TestMissingContainerUpgrade fails for containerd cri with 'unknown service runtime.v1alpha2.RuntimeService'
5 participants
@prezha @k8s-ci-robot @minikube-pr-bot @afbjorklund @medyagh