Adds a command to generate the zz_generated.deepcopy.go files for the…

… apis (googleforgames#3900)

* Adds a command to generate the zz_generated.deepcopy.go files for the apis
* Combines CRD client and deepcopy code generation into one script
* Updates boilerplate headers to 2024
* Generated code from gen-all-sdk-grpc
* Small update to template comment

agones-{extensions,allocator}: Make servers context aware (googleforgames#3845)

* agones-{extensions,allocator}: Make servers context aware, add gRPC health check

* adds an `httpserver` utility package to handle the `Run` function
that controller/extensions use. Make that context aware using the same
method as https.Run:
https://github.com/googleforgames/agones/blob/dfa414e5e4da37798833bbf8c33919acb5f3c2ea/pkg/util/https/server.go#L127-L130

* also plumbs context-awareness through the allocator
run{Mux,REST,GRPC} functions.

* adds a gRPC health server to the allocator, calls .Shutdown() on it
during graceful termination - this seems to push the client off correctly.

Tested with e2e in a loop.

Towards googleforgames#3853

* Move from context.Background()

* Use Shutdown/GracefulStop

* Relax deadline slightly (original had none), also delete pod from GetAllocatorClient

Update Slack invite link (googleforgames#3896)

Looks like the Slack invite link expired somewhere along the way, so
this is a new, shiny updated one!

.github/ISSUE_TEMPLATE/kubernetes_update.md

@@ -26,7 +26,7 @@ List of items to do for upgrading to {version_1} {version_2} {version_3}
 - [ ] Update CRD API reference to {version_2}
     - [ ] Update links to k8s documentation in `site/assets/templates/crd-doc-config.json`
     - [ ] Regenerate crd api reference docs - `make gen-api-docs`
-    - [ ] Regenerate crd client libraries - `make gen-crd-client`
+    - [ ] Regenerate crd client libraries and generated code - `make gen-crd-code`
 - [ ] Regenerate Kubernetes resource includes (e.g. ObjectMeta, PodTemplateSpec)
     - [ ] Start a cluster with `make gcloud-test-cluster` (this cluster will use Kubernetes {version_2}), uninstall agones using `helm uninstall agones -n agones-system`, and then run  `make gen-embedded-openapi` and `make gen-install`
 - [ ] Update documentation for creating clusters and k8s API references to align with the above clusters versions and the k8s API version
@@ -41,11 +41,11 @@ List of items to do for upgrading to {version_1} {version_2} {version_3}
     - [ ] Update the `grpc_release_tag` in the SDK [base image grpc version](https://github.com/googleforgames/agones/blob/main/build/includes/sdk.mk).
     - [ ] Update the gRPC version number in C++ gRPC Dependency documentation [here](https://github.com/googleforgames/agones/blob/main/site/content/en/docs/Guides/Client%20SDKs/cpp.md).
     - [ ] Update the gRPC version
-      ([Dockerfile](https://github.com/googleforgames/agones/blob/main/examples/cpp-simple/Dockerfile)) and 
+      ([Dockerfile](https://github.com/googleforgames/agones/blob/main/examples/cpp-simple/Dockerfile)) and
       increment the image tag
       ([Makefile](https://github.com/googleforgames/agones/blob/main/examples/cpp-simple/Makefile)) in the C++
       `cpp-simple` example.
-    - [ ] Regenerate all client sdks: [make gen-all-sdk-grpc](https://github.com/googleforgames/agones/blob/main/build/README.md#make-gen-all-sdk-grpc) 
+    - [ ] Regenerate all client sdks: [make gen-all-sdk-grpc](https://github.com/googleforgames/agones/blob/main/build/README.md#make-gen-all-sdk-grpc)
           This can take 20 minutes or so, as the above changes force a rebuild. Plan your day accordingly 😃.
     - [ ] Regenerate allocated API endpoints: [make gen-allocation-grpc](https://github.com/googleforgames/agones/blob/main/build/README.md#make-gen-allocation-grpc)
 - [ ] Confirm the update works as expected by running e2e tests

build/Makefile

@@ -738,12 +738,12 @@ gen-install: $(ensure-build-image)
 		--set agones.crds.cleanupOnDelete=false \
 		> $(mount_path)/install/yaml/install.yaml'
 
-# Generate the client for our CustomResourceDefinition
-gen-crd-client: $(ensure-build-image)
+# Generate the client, conversions, deepcopy, and defaults code for our CustomResourceDefinition
+gen-crd-code: $(ensure-build-image)
 	docker run --rm \
 		$(common_mounts) -w $(workdir_path) $(build_tag) bash -c "\
 		$(git_safe) && \
-		/root/gen-crd-client.sh && \
+		/root/gen-crd-code.sh && \
 		cd $(workdir_path)/pkg && goimports -w ."
 
 # Run a bash shell with the developer tools in it. (Creates the image if it doesn't exist)
@@ -848,14 +848,23 @@ update-go-deps:
 	$(DOCKER_RUN) go mod tidy
 	$(DOCKER_RUN) go mod vendor
 
-test-gen-crd-client:
-	mkdir -p build/tmp
-	mv ../pkg/client build/tmp
-	make gen-crd-client
-	diff_output=$$(diff -bBr build/tmp/client ../pkg/client); \
-	if [ -z "$$diff_output" ]; then \
-		echo "No differences found. Deleting build/tmp"; \
+test-gen-crd-code:
+	mkdir -p build/tmp/apis | mkdir -p build/tmp/client; \
+	cp -r ../pkg/apis/* build/tmp/apis | cp -r ../pkg/client/* build/tmp/client; \
+	make gen-crd-code; \
+	$(MAKE) diff-directory DIRECTORY=apis
+	$(MAKE) diff-directory DIRECTORY=client
+	# Delete build/tmp if the directory is empty
+	if [ ! "$(ls -A build/tmp)" ]; then \
+		echo "No differences found. Deleting empty directory build/tmp."; \
 		rm -r build/tmp; \
+	fi
+
+diff-directory:
+	diff_output=$$(diff -bBr build/tmp/$(DIRECTORY) ../pkg/$(DIRECTORY)); \
+	if [ -z "$$diff_output" ]; then \
+		echo "No differences found. Deleting build/tmp/$(DIRECTORY)"; \
+		rm -r build/tmp/$(DIRECTORY); \
 	else \
 		echo "Differences found."; \
 		echo "$$diff_output"; \

build/README.md

@@ -64,7 +64,7 @@ Table of Contents
         * [make build-agones-sdk-image](#make-build-agones-sdk-image)
         * [make gen-install](#make-gen-install)
         * [make gen-embedded-openapi](#make-gen-embedded-openapi)
-        * [make gen-crd-client](#make-gen-crd-client)
+        * [make gen-crd-code](#make-gen-crd-code)
         * [make gen-allocation-grpc](#make-gen-allocation-grpc)
         * [make gen-all-sdk-grpc](#make-gen-all-sdk-grpc)
         * [make gen-sdk-grpc](#make-gen-sdk-grpc)
@@ -585,7 +585,7 @@ make test-e2e-integration ARGS='-run TestGameServerReserve'
 Run controller failure portion of the end-to-end tests.
 
 #### `make test-e2e-allocator-crash`
-Run allocator failure portion of the end-to-end test. 
+Run allocator failure portion of the end-to-end test.
 
 #### `make setup-prometheus`
 
@@ -675,8 +675,8 @@ Generate the embedded OpenAPI specs for existing Kubernetes Objects, such as `Po
 
 This should be run against a clean or brand new cluster, as external CRD's or schemas could cause errors to occur.
 
-#### `make gen-crd-client`
-Generate the Custom Resource Definition client(s)
+#### `make gen-crd-code`
+Generate the Custom Resource Definition client(s), conversions, deepcopy, and defaults code.
 
 #### `make gen-allocation-grpc`
 Generate the allocator gRPC code

build/boilerplate.go.txt

@@ -1,4 +1,4 @@
-// Copyright 2023 Google LLC All Rights Reserved.
+// Copyright 2024 Google LLC All Rights Reserved.
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.

build/boilerplate.yaml.txt

@@ -1,4 +1,4 @@
-# Copyright 2023 Google LLC All Rights Reserved.
+# Copyright 2024 Google LLC All Rights Reserved.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

build/build-image/gen-crd-client.sh → build/build-image/gen-crd-code.sh

@@ -38,3 +38,11 @@ kube::codegen::gen_client \
   /go/src/agones.dev/agones/pkg/apis
 
 echo "CRD client code generation complete."
+
+echo "Generating CRD conversions, deepcopy, and defaults code..."
+
+kube::codegen::gen_helpers \
+  --boilerplate /go/src/agones.dev/agones/build/boilerplate.go.txt \
+  /go/src/agones.dev/agones/pkg/apis
+
+echo "CRD conversions, deepcopy, and defaults code generation complete."

cloudbuild.yaml

@@ -116,10 +116,10 @@ steps:
   # Preventing Broken PR Merges in CI
   #
   - name: make-docker
-    id: test-gen-crd-client
+    id: test-gen-crd-code
     waitFor: [pull-build-image]
     dir: build
-    args: [test-gen-crd-client]
+    args: [test-gen-crd-code]
 
   #
   # Runs the linter -- but also builds the build image, if not able to download
@@ -128,7 +128,7 @@ steps:
     id: lint
     waitFor:
       - pull-build-image
-      - test-gen-crd-client
+      - test-gen-crd-code
       - test-gen-all-sdk-grpc
     dir: build
     args: [lint]  # pull the build image if it exists

cmd/allocator/main.go

@@ -35,6 +35,8 @@ import (
 	"google.golang.org/grpc"
 	"google.golang.org/grpc/codes"
 	"google.golang.org/grpc/credentials"
+	grpchealth "google.golang.org/grpc/health"
+	"google.golang.org/grpc/health/grpc_health_v1"
 	"google.golang.org/grpc/keepalive"
 	"google.golang.org/grpc/status"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
@@ -52,6 +54,7 @@ import (
 	"agones.dev/agones/pkg/gameserverallocations"
 	"agones.dev/agones/pkg/gameservers"
 	"agones.dev/agones/pkg/util/fswatch"
+	"agones.dev/agones/pkg/util/httpserver"
 	"agones.dev/agones/pkg/util/runtime"
 	"agones.dev/agones/pkg/util/signals"
 )
@@ -218,19 +221,18 @@ func main() {
 	health, closer := setupMetricsRecorder(conf)
 	defer closer()
 
-	// http.DefaultServerMux is used for http connection, not for https
-	http.Handle("/", health)
-
 	kubeClient, agonesClient, err := getClients(conf)
 	if err != nil {
 		logger.WithError(err).Fatal("could not create clients")
 	}
 
+	listenCtx, cancelListenCtx := context.WithCancel(context.Background())
+
 	// This will test the connection to agones on each readiness probe
 	// so if one of the allocator pod can't reach Kubernetes it will be removed
 	// from the Kubernetes service.
-	ctx, cancelCtx := context.WithCancel(context.Background())
 	podReady = true
+	grpcHealth := grpchealth.NewServer() // only used for gRPC, ignored o/w
 	health.AddReadinessCheck("allocator-agones-client", func() error {
 		if !podReady {
 			return errors.New("asked to shut down, failed readiness check")
@@ -245,16 +247,15 @@ func main() {
 	signals.NewSigTermHandler(func() {
 		logger.Info("Pod shutdown has been requested, failing readiness check")
 		podReady = false
+		grpcHealth.Shutdown()
 		time.Sleep(conf.ReadinessShutdownDuration)
-		cancelCtx()
-		logger.Infof("Readiness shutdown duration has passed, context cancelled")
-		time.Sleep(1 * time.Second) // allow a brief time for cleanup, but force exit if main doesn't
-		os.Exit(0)
+		cancelListenCtx()
 	})
 
 	grpcUnallocatedStatusCode := grpcCodeFromHTTPStatus(conf.httpUnallocatedStatusCode)
 
-	h := newServiceHandler(ctx, kubeClient, agonesClient, health, conf.MTLSDisabled, conf.TLSDisabled, conf.remoteAllocationTimeout, conf.totalRemoteAllocationTimeout, conf.allocationBatchWaitTime, grpcUnallocatedStatusCode)
+	workerCtx, cancelWorkerCtx := context.WithCancel(context.Background())
+	h := newServiceHandler(workerCtx, kubeClient, agonesClient, health, conf.MTLSDisabled, conf.TLSDisabled, conf.remoteAllocationTimeout, conf.totalRemoteAllocationTimeout, conf.allocationBatchWaitTime, grpcUnallocatedStatusCode)
 
 	if !h.tlsDisabled {
 		cancelTLS, err := fswatch.Watch(logger, tlsDir, time.Second, func() {
@@ -294,51 +295,62 @@ func main() {
 
 	// If grpc and http use the same port then use a mux.
 	if conf.GRPCPort == conf.HTTPPort {
-		runMux(h, conf.HTTPPort)
+		runMux(listenCtx, workerCtx, h, grpcHealth, conf.HTTPPort)
 	} else {
 		// Otherwise, run each on a dedicated port.
 		if validPort(conf.HTTPPort) {
-			runREST(h, conf.HTTPPort)
+			runREST(listenCtx, workerCtx, h, conf.HTTPPort)
 		}
 		if validPort(conf.GRPCPort) {
-			runGRPC(h, conf.GRPCPort)
+			runGRPC(listenCtx, h, grpcHealth, conf.GRPCPort)
 		}
 	}
 
-	// Finally listen on 8080 (http) and block the main goroutine
-	// this is used to serve /live and /ready handlers for Kubernetes probes.
-	err = http.ListenAndServe(":8080", http.DefaultServeMux)
-	logger.WithError(err).Fatal("allocation service crashed")
+	// Finally listen on 8080 (http), used to serve /live and /ready handlers for Kubernetes probes.
+	healthserver := httpserver.Server{Logger: logger}
+	healthserver.Handle("/", health)
+	go func() { _ = healthserver.Run(listenCtx, 0) }()
+
+	// TODO: This is messy. Contexts are the wrong way to handle this - we should be using shutdown,
+	// and a cascading graceful shutdown instead of multiple contexts and sleeps.
+	<-listenCtx.Done()
+	logger.Infof("Listen context cancelled")
+	time.Sleep(5 * time.Second)
+	cancelWorkerCtx()
+	logger.Infof("Worker context cancelled")
+	time.Sleep(1 * time.Second)
+	logger.Info("Shut down allocator")
 }
 
 func validPort(port int) bool {
 	const maxPort = 65535
 	return port >= 0 && port < maxPort
 }
 
-func runMux(h *serviceHandler, httpPort int) {
+func runMux(listenCtx context.Context, workerCtx context.Context, h *serviceHandler, grpcHealth *grpchealth.Server, httpPort int) {
 	logger.Infof("Running the mux handler on port %d", httpPort)
 	grpcServer := grpc.NewServer(h.getMuxServerOptions()...)
 	pb.RegisterAllocationServiceServer(grpcServer, h)
+	grpc_health_v1.RegisterHealthServer(grpcServer, grpcHealth)
 
 	mux := runtime.NewServerMux()
 	if err := pb.RegisterAllocationServiceHandlerServer(context.Background(), mux, h); err != nil {
 		panic(err)
 	}
 
-	runHTTP(h, httpPort, grpcHandlerFunc(grpcServer, mux))
+	runHTTP(listenCtx, workerCtx, h, httpPort, grpcHandlerFunc(grpcServer, mux))
 }
 
-func runREST(h *serviceHandler, httpPort int) {
+func runREST(listenCtx context.Context, workerCtx context.Context, h *serviceHandler, httpPort int) {
 	logger.WithField("port", httpPort).Info("Running the rest handler")
 	mux := runtime.NewServerMux()
 	if err := pb.RegisterAllocationServiceHandlerServer(context.Background(), mux, h); err != nil {
 		panic(err)
 	}
-	runHTTP(h, httpPort, mux)
+	runHTTP(listenCtx, workerCtx, h, httpPort, mux)
 }
 
-func runHTTP(h *serviceHandler, httpPort int, handler http.Handler) {
+func runHTTP(listenCtx context.Context, workerCtx context.Context, h *serviceHandler, httpPort int, handler http.Handler) {
 	cfg := &tls.Config{}
 	if !h.tlsDisabled {
 		cfg.GetCertificate = h.getTLSCert
@@ -356,21 +368,29 @@ func runHTTP(h *serviceHandler, httpPort int, handler http.Handler) {
 	}
 
 	go func() {
+		go func() {
+			<-listenCtx.Done()
+			_ = server.Shutdown(workerCtx)
+		}()
+
 		var err error
 		if !h.tlsDisabled {
 			err = server.ListenAndServeTLS("", "")
 		} else {
 			err = server.ListenAndServe()
 		}
 
-		if err != nil {
+		if err == http.ErrServerClosed {
+			logger.WithError(err).Info("HTTP/HTTPS server closed")
+			os.Exit(0)
+		} else {
 			logger.WithError(err).Fatal("Unable to start HTTP/HTTPS listener")
 			os.Exit(1)
 		}
 	}()
 }
 
-func runGRPC(h *serviceHandler, grpcPort int) {
+func runGRPC(ctx context.Context, h *serviceHandler, grpcHealth *grpchealth.Server, grpcPort int) {
 	logger.WithField("port", grpcPort).Info("Running the grpc handler on port")
 	listener, err := net.Listen("tcp", fmt.Sprintf(":%d", grpcPort))
 	if err != nil {
@@ -380,11 +400,22 @@ func runGRPC(h *serviceHandler, grpcPort int) {
 
 	grpcServer := grpc.NewServer(h.getGRPCServerOptions()...)
 	pb.RegisterAllocationServiceServer(grpcServer, h)
+	grpc_health_v1.RegisterHealthServer(grpcServer, grpcHealth)
 
 	go func() {
+		go func() {
+			<-ctx.Done()
+			grpcServer.GracefulStop()
+		}()
+
 		err := grpcServer.Serve(listener)
-		logger.WithError(err).Fatal("allocation service crashed")
-		os.Exit(1)
+		if err != nil {
+			logger.WithError(err).Fatal("allocation service crashed")
+			os.Exit(1)
+		} else {
+			logger.Info("allocation server closed")
+			os.Exit(0)
+		}
 	}()
 }
 

cmd/controller/main.go

@@ -20,7 +20,6 @@ import (
 	"encoding/json"
 	"fmt"
 	"io"
-	"net/http"
 	"os"
 	"path/filepath"
 	"strings"
@@ -54,6 +53,7 @@ import (
 	"agones.dev/agones/pkg/gameservers"
 	"agones.dev/agones/pkg/gameserversets"
 	"agones.dev/agones/pkg/metrics"
+	"agones.dev/agones/pkg/util/httpserver"
 	"agones.dev/agones/pkg/util/runtime"
 	"agones.dev/agones/pkg/util/signals"
 )
@@ -171,7 +171,7 @@ func main() {
 	agonesInformerFactory := externalversions.NewSharedInformerFactory(agonesClient, defaultResync)
 	kubeInformerFactory := informers.NewSharedInformerFactory(kubeClient, defaultResync)
 
-	server := &httpServer{}
+	server := &httpserver.Server{Logger: logger}
 	var rs []runner
 	var health healthcheck.Handler
 
@@ -547,10 +547,6 @@ type runner interface {
 	Run(ctx context.Context, workers int) error
 }
 
-type httpServer struct {
-	http.ServeMux
-}
-
 func whenLeader(ctx context.Context, cancel context.CancelFunc, logger *logrus.Entry, doLeaderElection bool, kubeClient *kubernetes.Clientset, namespace string, start func(_ context.Context)) {
 	if !doLeaderElection {
 		start(ctx)
@@ -600,22 +596,3 @@ func whenLeader(ctx context.Context, cancel context.CancelFunc, logger *logrus.E
 		},
 	})
 }
-
-func (h *httpServer) Run(_ context.Context, _ int) error {
-	logger.Info("Starting http server...")
-	srv := &http.Server{
-		Addr:    ":8080",
-		Handler: h,
-	}
-	defer srv.Close() // nolint: errcheck
-
-	if err := srv.ListenAndServe(); err != nil {
-		if err == http.ErrServerClosed {
-			logger.WithError(err).Info("http server closed")
-		} else {
-			wrappedErr := errors.Wrap(err, "Could not listen on :8080")
-			runtime.HandleError(logger.WithError(wrappedErr), wrappedErr)
-		}
-	}
-	return nil
-}