Add graceful shutdown to sansshell proxy (#468)

SIGTERM is sent by orchestration platforms like Kubernetes to initiate graceful termination. We can listen for it and use grpc's GracefulStop, which will refuse new RPCs and continue serving existing RPCs. This graceful behavior can give a better user experience for ongoing streaming RPCs by allowing them to run during termination.

I'm only doing this for proxy and not server because server tends to have a lifetime coupled to the node it serves and proxy tends to have a lifetime separate from servers.

I've intentionally put this as behavior that unconditionally happens instead of as an option. I'm having trouble thinking of a case where someone wouldn't want this. We can turn it into an option if we find a reason to do so.

cmd/proxy-server/server/server.go

@@ -28,6 +28,8 @@ import (
 	"net/http"
 	"net/http/pprof"
 	"os"
+	"os/signal"
+	"syscall"
 
 	"github.com/go-logr/logr"
 	grpc_prometheus "github.com/grpc-ecosystem/go-grpc-middleware/providers/prometheus"
@@ -461,6 +463,18 @@ func Run(ctx context.Context, opts ...Option) {
 		s(g)
 	}
 
+	// React to interrupt signals by shutting down gracefully.
+	// This tends to improve the proxy behavior when running on platforms like Kubernetes.
+	// The proxy will continue to serve streaming RPCs during graceful shutdown.
+	sigCh := make(chan os.Signal, 2)
+	signal.Notify(sigCh, syscall.SIGTERM, syscall.SIGINT)
+	go func() {
+		s := <-sigCh
+		signal.Stop(sigCh)
+		rs.logger.Info("beginning graceful shutdown", "signal", s)
+		g.GracefulStop()
+	}()
+
 	rs.logger.Info("initialized proxy service", "credsource", rs.credSource)
 	rs.logger.Info("serving..")