test: allow running functional tests for different k8s versions #18564

ComradeProgrammer · 2024-04-02T20:34:20Z

FIX #18508
test: allow running functional tests for different k8s versions
It also fix a small bug in minikube tunnel. Now we also check the machine name stored in the tunnel.json when preventing establishing duplicated tunnel.

Before: functional test can only be run for default specified k8s version

After:

When running the whold functional test, now functional tests can be run for multiple k8s version(default, newest).
Functional tests for non-default k8s versions are named as " TestFunctionalNewestKuberentes/Versionxxx/parallel/xxx", while tests for default versions remains unchanged.

Here are the example outputs(most tests are disabled for simplification)

make functional
go build  -tags "" -ldflags="-X k8s.io/minikube/pkg/version.version=v1.33.0-beta.0 -X k8s.io/minikube/pkg/version.isoVersion=v1.33.0-1711559712-18485 -X k8s.io/minikube/pkg/version.gitCommitID="2d03356f4cfd5e00101854f26afd0711a65bd178-dirty" -X k8s.io/minikube/pkg/version.storageProvisionerVersion=v5" -o out/minikube k8s.io/minikube/cmd/minikube
go test -ldflags="-X k8s.io/minikube/pkg/version.version=v1.33.0-beta.0 -X k8s.io/minikube/pkg/version.isoVersion=v1.33.0-1711559712-18485 -X k8s.io/minikube/pkg/version.gitCommitID="2d03356f4cfd5e00101854f26afd0711a65bd178-dirty" -X k8s.io/minikube/pkg/version.storageProvisionerVersion=v5" -v -test.timeout=20m ./test/integration --tags="integration "  -test.run TestFunctional 2>&1 | tee "./out/testout_2d03356f4.txt"
Found 8 cores, limiting parallelism with --test.parallel=2
=== RUN   TestFunctional
=== RUN   TestFunctional/serial
=== RUN   TestFunctional/serial/CopySyncFile
    functional_test.go:1869: local sync path: /Users/tjm/.minikube/files/etc/test/nested/copy/14959/hosts
=== RUN   TestFunctional/serial/StartWithProxy
    functional_test.go:2248: (dbg) Run:  out/minikube start -p functional-270000 --memory=4000 --apiserver-port=8441 --wait=all 
    functional_test.go:2248: (dbg) Done: out/minikube start -p functional-270000 --memory=4000 --apiserver-port=8441 --wait=all : (34.611209708s)
=== RUN   TestFunctional/parallel
=== RUN   TestFunctional/delete_addon-resizer_images
    functional_test.go:207: (dbg) Run:  docker rmi -f gcr.io/google-containers/addon-resizer:1.8.8
    functional_test.go:207: (dbg) Run:  docker rmi -f gcr.io/google-containers/addon-resizer:functional-270000
=== RUN   TestFunctional/delete_my-image_image
    functional_test.go:215: (dbg) Run:  docker rmi -f localhost/my-image:functional-270000
=== RUN   TestFunctional/delete_minikube_cached_images
    functional_test.go:223: (dbg) Run:  docker rmi -f minikube-local-cache-test:functional-270000
=== NAME  TestFunctional
    helpers_test.go:175: Cleaning up "functional-270000" profile ...
    helpers_test.go:178: (dbg) Run:  out/minikube delete -p functional-270000
    helpers_test.go:178: (dbg) Done: out/minikube delete -p functional-270000: (1.969108417s)
--- PASS: TestFunctional (36.75s)
    --- PASS: TestFunctional/serial (34.61s)
        --- PASS: TestFunctional/serial/CopySyncFile (0.00s)
        --- PASS: TestFunctional/serial/StartWithProxy (34.61s)
    --- PASS: TestFunctional/parallel (0.00s)
    --- PASS: TestFunctional/delete_addon-resizer_images (0.08s)
    --- PASS: TestFunctional/delete_my-image_image (0.04s)
    --- PASS: TestFunctional/delete_minikube_cached_images (0.04s)
=== RUN   TestFunctionalNewestKuberentes
=== RUN   TestFunctionalNewestKuberentes/Versionv1.30.0-beta.0
=== RUN   TestFunctionalNewestKuberentes/Versionv1.30.0-beta.0/serial
=== RUN   TestFunctionalNewestKuberentes/Versionv1.30.0-beta.0/serial/CopySyncFile
    functional_test.go:1869: local sync path: /Users/tjm/.minikube/files/etc/test/nested/copy/14959/hosts
=== RUN   TestFunctionalNewestKuberentes/Versionv1.30.0-beta.0/serial/StartWithProxy
    functional_test.go:2248: (dbg) Run:  out/minikube start -p functional-400000 --memory=4000 --apiserver-port=8441 --wait=all  --kubernetes-version=v1.30.0-beta.0
    functional_test.go:2248: (dbg) Done: out/minikube start -p functional-400000 --memory=4000 --apiserver-port=8441 --wait=all  --kubernetes-version=v1.30.0-beta.0: (33.801479542s)
=== RUN   TestFunctionalNewestKuberentes/Versionv1.30.0-beta.0/parallel
=== RUN   TestFunctionalNewestKuberentes/Versionv1.30.0-beta.0/delete_addon-resizer_images
    functional_test.go:207: (dbg) Run:  docker rmi -f gcr.io/google-containers/addon-resizer:1.8.8
    functional_test.go:207: (dbg) Run:  docker rmi -f gcr.io/google-containers/addon-resizer:functional-400000
=== RUN   TestFunctionalNewestKuberentes/Versionv1.30.0-beta.0/delete_my-image_image
    functional_test.go:215: (dbg) Run:  docker rmi -f localhost/my-image:functional-400000
=== RUN   TestFunctionalNewestKuberentes/Versionv1.30.0-beta.0/delete_minikube_cached_images
    functional_test.go:223: (dbg) Run:  docker rmi -f minikube-local-cache-test:functional-400000
=== NAME  TestFunctionalNewestKuberentes/Versionv1.30.0-beta.0
    helpers_test.go:175: Cleaning up "functional-400000" profile ...
    helpers_test.go:178: (dbg) Run:  out/minikube delete -p functional-400000
    helpers_test.go:178: (dbg) Done: out/minikube delete -p functional-400000: (1.919934375s)
--- PASS: TestFunctionalNewestKuberentes (35.88s)
    --- PASS: TestFunctionalNewestKuberentes/Versionv1.30.0-beta.0 (35.88s)
        --- PASS: TestFunctionalNewestKuberentes/Versionv1.30.0-beta.0/serial (33.80s)
            --- PASS: TestFunctionalNewestKuberentes/Versionv1.30.0-beta.0/serial/CopySyncFile (0.00s)
            --- PASS: TestFunctionalNewestKuberentes/Versionv1.30.0-beta.0/serial/StartWithProxy (33.80s)
        --- PASS: TestFunctionalNewestKuberentes/Versionv1.30.0-beta.0/parallel (0.00s)
        --- PASS: TestFunctionalNewestKuberentes/Versionv1.30.0-beta.0/delete_addon-resizer_images (0.08s)
        --- PASS: TestFunctionalNewestKuberentes/Versionv1.30.0-beta.0/delete_my-image_image (0.04s)
        --- PASS: TestFunctionalNewestKuberentes/Versionv1.30.0-beta.0/delete_minikube_cached_images (0.04s)
PASS
Tests completed in 1m12.628801166s (result code 0)
ok      k8s.io/minikube/test/integration        73.292s

k8s-ci-robot · 2024-04-02T20:34:30Z

Hi @ComradeProgrammer. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

minikube-bot · 2024-04-02T20:39:01Z

Can one of the admins verify this patch?

medyagh

@ComradeProgrammer can we make it in a way that the TestName will show different when it fails ?

I rather we have a new test like this

func TestFunctional(t *testing.T) {
k8sversion= DEFAULT_ONE
///. the current test
commonFunctiona() ...
}

func TestFunctionalNewestKuberentes(t *testing.T) {
k8sversion= NEWEST ONES
// here it should call the commonTest
commonFunctiona() ...

}

// generic common test that wilill be called by others
func commonFunctionalt *testing.T) {

}

ComradeProgrammer · 2024-04-03T15:41:46Z

@ComradeProgrammer can we make it in a way that the TestName will show different when it fails ?

I rather we have a new test like this

func TestFunctional(t *testing.T) { k8sversion= DEFAULT_ONE ///. the current test commonFunctiona() ... }

func TestFunctionalNewestKuberentes(t *testing.T) { k8sversion= NEWEST ONES // here it should call the commonTest commonFunctiona() ...

}

// generic common test that wilill be called by others func commonFunctionalt *testing.T) {

}

Revised. Now they are named TestFunctional/serial/<testname> and TestFunctionalNewestKuberentes/Versionxxx/serial/<testname>

Versionxxx is added so that we don't need to change anything if we want to run a specified functional test on an active minikube cluster( tests for newest version won't be triggered when we use something like TEST_ARGS="-test.run TestFunctional/parallel/MountCmd" )

test/integration/functional_test.go

medyagh · 2024-04-09T16:52:47Z

/ok-to-test

pkg/minikube/tunnel/registry.go

…tional test

k8s-ci-robot · 2024-07-03T22:31:15Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ComradeProgrammer
Once this PR has been reviewed and has the lgtm label, please assign prezha for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

minikube-pr-bot · 2024-07-04T03:43:42Z

These are the flake rates of all failed tests.

Environment	Failed Tests	Flake Rate (%)
QEMU_macOS	TestAddons/Setup (gopogh)	8.33 (chart)
QEMU_macOS	TestFunctional/parallel/CertSync (gopogh)	8.33 (chart)
QEMU_macOS	TestFunctional/parallel/CpCmd (gopogh)	8.33 (chart)
QEMU_macOS	TestFunctional/parallel/FileSync (gopogh)	8.33 (chart)
QEMU_macOS	TestFunctional/parallel/NonActiveRuntimeDisabled (gopogh)	8.33 (chart)
QEMU_macOS	TestFunctional/parallel/SSHCmd (gopogh)	8.33 (chart)
QEMU_macOS	TestFunctional/parallel/UpdateContextCmd/no_changes (gopogh)	8.33 (chart)
QEMU_macOS	TestFunctional/parallel/UpdateContextCmd/no_clusters (gopogh)	8.33 (chart)
QEMU_macOS	TestFunctional/parallel/UpdateContextCmd/no_minikube_cluster (gopogh)	8.33 (chart)
QEMU_macOS	TestFunctional/parallel/Version/components (gopogh)	8.33 (chart)
QEMU_macOS	TestFunctional/serial/CacheCmd/cache/cache_reload (gopogh)	8.33 (chart)
QEMU_macOS	TestFunctional/serial/CacheCmd/cache/verify_cache_inside_node (gopogh)	8.33 (chart)
QEMU_macOS	TestFunctional/serial/KubeContext (gopogh)	8.33 (chart)
QEMU_macOS	TestFunctional/serial/KubectlGetPods (gopogh)	8.33 (chart)
QEMU_macOS	TestFunctional/serial/LogsCmd (gopogh)	8.33 (chart)
QEMU_macOS	TestFunctional/serial/LogsFileCmd (gopogh)	8.33 (chart)
QEMU_macOS	TestFunctional/serial/MinikubeKubectlCmd (gopogh)	8.33 (chart)
QEMU_macOS	TestFunctional/serial/MinikubeKubectlCmdDirectly (gopogh)	8.33 (chart)
QEMU_macOS	TestFunctional/serial/SoftStart (gopogh)	8.33 (chart)
QEMU_macOS	TestFunctional/serial/StartWithProxy (gopogh)	8.33 (chart)
QEMU_macOS	TestErrorSpam/setup (gopogh)	10.00 (chart)
QEMU_macOS	TestFunctional/parallel/DashboardCmd (gopogh)	10.00 (chart)
QEMU_macOS	TestFunctional/parallel/DockerEnv/bash (gopogh)	10.00 (chart)
QEMU_macOS	TestFunctional/parallel/ImageCommands/ImageBuild (gopogh)	10.00 (chart)
QEMU_macOS	TestFunctional/parallel/ImageCommands/ImageListJson (gopogh)	10.00 (chart)
QEMU_macOS	TestFunctional/parallel/ImageCommands/ImageListShort (gopogh)	10.00 (chart)
QEMU_macOS	TestFunctional/parallel/ImageCommands/ImageListTable (gopogh)	10.00 (chart)
QEMU_macOS	TestFunctional/parallel/ImageCommands/ImageListYaml (gopogh)	10.00 (chart)
QEMU_macOS	TestFunctional/parallel/ImageCommands/ImageLoadDaemon (gopogh)	10.00 (chart)
QEMU_macOS	TestFunctional/parallel/ImageCommands/ImageLoadFromFile (gopogh)	10.00 (chart)
More tests...	Continued...

Too many tests failed - See test logs for more details.

To see the flake rates of all tests by environment, click here.

pkg/minikube/tunnel/registry.go

test/integration/functional_test.go

Co-authored-by: Steven Powell <[email protected]>

spowelljr

You have lint issues, could you run gofmt -w -s .

medyagh · 2024-12-16T19:34:53Z

/ok-to-test

medyagh · 2024-12-16T19:35:29Z

lets fix the lint and rebase

minikube-pr-bot · 2024-12-16T20:56:08Z

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 18564) |
+----------------+----------+---------------------+
| minikube start | 51.2s    | 50.6s               |
| enable ingress | 17.2s    | 18.2s               |
+----------------+----------+---------------------+

Times for minikube start: 50.0s 52.0s 50.7s 50.7s 52.4s
Times for minikube (PR 18564) start: 49.2s 50.2s 52.4s 48.9s 52.1s

Times for minikube ingress: 19.0s 15.5s 19.5s 16.0s 16.0s
Times for minikube (PR 18564) ingress: 18.5s 18.6s 18.5s 20.0s 15.5s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 18564) |
+----------------+----------+---------------------+
| minikube start | 23.7s    | 23.4s               |
| enable ingress | 12.7s    | 12.6s               |
+----------------+----------+---------------------+

Times for minikube start: 24.5s 24.0s 25.1s 24.0s 20.8s
Times for minikube (PR 18564) start: 23.6s 23.9s 21.3s 23.7s 24.6s

Times for minikube (PR 18564) ingress: 12.8s 12.3s 12.8s 12.8s 12.3s
Times for minikube ingress: 12.3s 12.8s 12.3s 12.8s 13.3s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 18564) |
+----------------+----------+---------------------+
| minikube start | 21.7s    | 21.0s               |
| enable ingress | 35.6s    | 38.8s               |
+----------------+----------+---------------------+

Times for minikube start: 22.8s 20.1s 22.8s 23.1s 19.7s
Times for minikube (PR 18564) start: 22.9s 22.3s 20.0s 20.2s 19.6s

Times for minikube ingress: 38.8s 38.8s 38.8s 22.8s 38.8s
Times for minikube (PR 18564) ingress: 38.8s 38.8s 38.8s 38.8s 38.8s

minikube-pr-bot · 2024-12-16T23:17:23Z

Here are the number of top 10 failed tests in each environments with lowest flake rate.

Environment	Test Name	Flake Rate
Docker_Linux_docker_arm64 (2 failed)	TestScheduledStopUnix(gopogh)	2.27% (chart)
Docker_Linux_docker_arm64 (2 failed)	TestStartStop/group/old-k8s-version/serial/SecondStart(gopogh)	12.12% (chart)
Docker_Linux_crio_arm64 (3 failed)	TestMultiControlPlane/serial/RestartCluster(gopogh)	10.77% (chart)
Docker_Linux_crio (3 failed)	TestFunctional/parallel/ImageCommands/ImageLoadFromFile(gopogh)	5.34% (chart)
Docker_macOS (8 failed)	TestAddons/parallel/Ingress(gopogh)	Unknown
Docker_macOS (8 failed)	TestAddons/serial/Volcano(gopogh)	0.00% (chart)
Docker_macOS (8 failed)	TestAddons/parallel/CSI(gopogh)	0.00% (chart)
Docker_macOS (8 failed)	TestAddons/parallel/LocalPath(gopogh)	0.00% (chart)
Docker_macOS (8 failed)	TestFunctional/parallel/PersistentVolumeClaim(gopogh)	0.00% (chart)
Docker_macOS (8 failed)	TestFunctional/parallel/MySQL(gopogh)	0.00% (chart)
Docker_macOS (8 failed)	TestFunctional/parallel/TunnelCmd/serial/WaitService/Setup(gopogh)	0.00% (chart)
Docker_macOS (8 failed)	TestFunctional/parallel/TunnelCmd/serial/AccessDirect(gopogh)	0.00% (chart)

Besides the following environments also have failed tests:

Docker_Cloud_Shell: 3 failed (gopogh)
QEMU_macOS: 156 failed (gopogh)
KVM_Linux_crio: 11 failed (gopogh)
Hyperkit_macOS: 16 failed (gopogh)

To see the flake rates of all tests by environment, click here.

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Apr 2, 2024

k8s-ci-robot requested review from sharifelgamal and spowelljr April 2, 2024 20:34

k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Apr 2, 2024

medyagh requested changes Apr 2, 2024

View reviewed changes

ComradeProgrammer force-pushed the version_functional_test branch from 2d03356 to 9279e98 Compare April 3, 2024 15:40

k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Apr 3, 2024

spowelljr requested changes Apr 3, 2024

View reviewed changes

test/integration/functional_test.go Outdated Show resolved Hide resolved

test/integration/functional_test.go Outdated Show resolved Hide resolved

test/integration/functional_test.go Outdated Show resolved Hide resolved

ComradeProgrammer force-pushed the version_functional_test branch from 9279e98 to 224bcb6 Compare April 3, 2024 19:44

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Apr 3, 2024

ComradeProgrammer force-pushed the version_functional_test branch from 224bcb6 to 8167a9a Compare April 3, 2024 19:45

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Apr 9, 2024

This comment has been minimized.

Sign in to view

test: allow running functional tests for different k8s versions

981083d

ComradeProgrammer force-pushed the version_functional_test branch from 8167a9a to 981083d Compare April 16, 2024 21:41

This comment has been minimized.

Sign in to view

ComradeProgrammer force-pushed the version_functional_test branch from 24a9e9e to 4722894 Compare April 17, 2024 08:06

This comment has been minimized.

Sign in to view

ComradeProgrammer force-pushed the version_functional_test branch 2 times, most recently from 1e215c5 to f4e30a8 Compare May 20, 2024 18:43

This comment has been minimized.

Sign in to view

medyagh approved these changes May 22, 2024

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 22, 2024

medyagh requested changes May 22, 2024

View reviewed changes

pkg/minikube/tunnel/registry.go Outdated Show resolved Hide resolved

k8s-ci-robot removed the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 22, 2024

medyagh reviewed Jun 17, 2024

View reviewed changes

pkg/minikube/tunnel/registry.go Outdated Show resolved Hide resolved

ComradeProgrammer force-pushed the version_functional_test branch 2 times, most recently from 39a928e to 7070dbd Compare June 30, 2024 15:10

This comment has been minimized.

Sign in to view

fix: fix incomplete cleanup in tunnel.json and zombie process in func…

b139639

…tional test

ComradeProgrammer force-pushed the version_functional_test branch from 7070dbd to b139639 Compare July 3, 2024 22:31

This comment has been minimized.

Sign in to view

spowelljr requested changes Oct 1, 2024

View reviewed changes

pkg/minikube/tunnel/registry.go Outdated Show resolved Hide resolved

test/integration/functional_test.go Outdated Show resolved Hide resolved

This comment has been minimized.

Sign in to view

Apply suggestions from code review

7b31a43

Co-authored-by: Steven Powell <[email protected]>

ComradeProgrammer force-pushed the version_functional_test branch from 266b4c3 to 7b31a43 Compare November 14, 2024 20:39

spowelljr requested changes Nov 15, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: allow running functional tests for different k8s versions #18564

test: allow running functional tests for different k8s versions #18564

ComradeProgrammer commented Apr 2, 2024 •

edited

Loading

k8s-ci-robot commented Apr 2, 2024

minikube-bot commented Apr 2, 2024

medyagh left a comment

ComradeProgrammer commented Apr 3, 2024 •

edited

Loading

medyagh commented Apr 9, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

k8s-ci-robot commented Jul 3, 2024

This comment has been minimized.

minikube-pr-bot commented Jul 4, 2024

This comment has been minimized.

This comment has been minimized.

spowelljr left a comment

medyagh commented Dec 16, 2024

medyagh commented Dec 16, 2024

minikube-pr-bot commented Dec 16, 2024

minikube-pr-bot commented Dec 16, 2024

test: allow running functional tests for different k8s versions #18564

Are you sure you want to change the base?

test: allow running functional tests for different k8s versions #18564

Conversation

ComradeProgrammer commented Apr 2, 2024 • edited Loading

k8s-ci-robot commented Apr 2, 2024

minikube-bot commented Apr 2, 2024

medyagh left a comment

Choose a reason for hiding this comment

ComradeProgrammer commented Apr 3, 2024 • edited Loading

medyagh commented Apr 9, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

k8s-ci-robot commented Jul 3, 2024

This comment has been minimized.

minikube-pr-bot commented Jul 4, 2024

This comment has been minimized.

This comment has been minimized.

spowelljr left a comment

Choose a reason for hiding this comment

medyagh commented Dec 16, 2024

medyagh commented Dec 16, 2024

minikube-pr-bot commented Dec 16, 2024

minikube-pr-bot commented Dec 16, 2024

ComradeProgrammer commented Apr 2, 2024 •

edited

Loading

ComradeProgrammer commented Apr 3, 2024 •

edited

Loading