Merge pull request #11 from kz33/caicloud

Merge remote-tracking branch 'upstream/master' into caicloud

README.md

@@ -10,25 +10,26 @@
 
 ## Project Status
 
-**Project status:** *beta* 
+**Project status:** *beta*
 
 **Current API version:** *`v1beta2`*
 
 **If you are currently using the `v1beta1` version of the APIs in your manifests, please update them to use the `v1beta2` version by changing `apiVersion: "sparkoperator.k8s.io/<version>"` to `apiVersion: "sparkoperator.k8s.io/v1beta2"`. You will also need to delete the `previous` version of the CustomResourceDefinitions named `sparkapplications.sparkoperator.k8s.io` and `scheduledsparkapplications.sparkoperator.k8s.io`, and replace them with the `v1beta2` version either by installing the latest version of the operator or by running `kubectl create -f manifest/crds`.**
 
-Customization of Spark pods, e.g., mounting arbitrary volumes and setting pod affinity, is implemented using a Kubernetes [Mutating Admission Webhook](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/), which became beta in Kubernetes 1.9. The mutating admission webhook is disabled by default if you install the operator using the Helm [chart](https://github.com/helm/charts/tree/master/incubator/sparkoperator). Check out the [Quick Start Guide](docs/quick-start-guide.md#using-the-mutating-admission-webhook) on how to enable the webhook.
+Customization of Spark pods, e.g., mounting arbitrary volumes and setting pod affinity, is implemented using a Kubernetes [Mutating Admission Webhook](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/), which became beta in Kubernetes 1.9. The mutating admission webhook is disabled by default if you install the operator using the Helm [chart](charts/spark-operator-chart). Check out the [Quick Start Guide](docs/quick-start-guide.md#using-the-mutating-admission-webhook) on how to enable the webhook.
 
 ## Prerequisites
 
 * Version >= 1.13 of Kubernetes to use the [`subresource` support for CustomResourceDefinitions](https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/#subresources), which became beta in 1.13 and is enabled by default in 1.13 and higher.
 
 ## Installation
 
-The easiest way to install the Kubernetes Operator for Apache Spark is to use the Helm [chart](https://github.com/helm/charts/tree/master/incubator/sparkoperator).
+The easiest way to install the Kubernetes Operator for Apache Spark is to use the Helm [chart](charts/spark-operator-chart/).
 
 ```bash
-$ helm repo add incubator https://charts.helm.sh/incubator --force-update
-$ helm install incubator/sparkoperator --namespace spark-operator
+$ helm repo add spark-operator https://googlecloudplatform.github.io/spark-on-k8s-operator
+
+$ helm install my-release spark-operator/spark-operator --namespace spark-operator --create-namespace
 ```
 
 This will install the Kubernetes Operator for Apache Spark into the namespace `spark-operator`. The operator by default watches and handles `SparkApplication`s in every namespaces. If you would like to limit the operator to watch and handle `SparkApplication`s in a single namespace, e.g., `default` instead, add the following option to the `helm install` command:
@@ -37,7 +38,7 @@ This will install the Kubernetes Operator for Apache Spark into the namespace `s
 --set sparkJobNamespace=default
 ```
 
-For configuration options available in the Helm chart, please refer to [Configuration](https://github.com/helm/charts/tree/master/incubator/sparkoperator#configuration).
+For configuration options available in the Helm chart, please refer to the chart's [README](charts/spark-operator-chart/README.md).
 
 ## Version Matrix
 
@@ -48,34 +49,34 @@ The following table lists the most recent few versions of the operator.
 | `latest` (master HEAD) | `v1beta2` | 1.13+ | `3.0.0` | `latest` |
 | `v1beta2-1.2.0-3.0.0` | `v1beta2` | 1.13+ | `3.0.0` | `v1beta2-1.2.0-3.0.0` |
 | `v1beta2-1.1.2-2.4.5` | `v1beta2` | 1.13+ | `2.4.5` | `v1beta2-1.1.2-2.4.5` |
-| `v1beta2-1.0.1-2.4.4` | `v1beta2` | 1.13+ | `2.4.4` | `v1beta2-1.0.1-2.4.4` | 
+| `v1beta2-1.0.1-2.4.4` | `v1beta2` | 1.13+ | `2.4.4` | `v1beta2-1.0.1-2.4.4` |
 | `v1beta2-1.0.0-2.4.4` | `v1beta2` | 1.13+ | `2.4.4` | `v1beta2-1.0.0-2.4.4` |
 | `v1beta1-0.9.0` | `v1beta1` | 1.13+ | `2.4.0` | `v2.4.0-v1beta1-0.9.0` |
 
 When installing using the Helm chart, you can choose to use a specific image tag instead of the default one, using the following option:
 
 ```
---set operatorVersion=<operator image tag>
+--set image.tag=<operator image tag>
 ```
 
 ## Get Started
 
-Get started quickly with the Kubernetes Operator for Apache Spark using the [Quick Start Guide](docs/quick-start-guide.md). 
+Get started quickly with the Kubernetes Operator for Apache Spark using the [Quick Start Guide](docs/quick-start-guide.md).
 
 If you are running the Kubernetes Operator for Apache Spark on Google Kubernetes Engine and want to use Google Cloud Storage (GCS) and/or BigQuery for reading/writing data, also refer to the [GCP guide](docs/gcp.md).
 
 For more information, check the [Design](docs/design.md), [API Specification](docs/api-docs.md) and detailed [User Guide](docs/user-guide.md).
 
 ## Overview
 
-The Kubernetes Operator for Apache Spark aims to make specifying and running [Spark](https://github.com/apache/spark) applications as easy and idiomatic as running other workloads on Kubernetes. It uses 
-[Kubernetes custom resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) 
+The Kubernetes Operator for Apache Spark aims to make specifying and running [Spark](https://github.com/apache/spark) applications as easy and idiomatic as running other workloads on Kubernetes. It uses
+[Kubernetes custom resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/)
 for specifying, running, and surfacing status of Spark applications. For a complete reference of the custom resource definitions, please refer to the [API Definition](docs/api-docs.md). For details on its design, please refer to the [design doc](docs/design.md). It requires Spark 2.3 and above that supports Kubernetes as a native scheduler backend.
 
 The Kubernetes Operator for Apache Spark currently supports the following list of features:
 
 * Supports Spark 2.3 and up.
-* Enables declarative application specification and management of applications through custom resources. 
+* Enables declarative application specification and management of applications through custom resources.
 * Automatically runs `spark-submit` on behalf of users for each `SparkApplication` eligible for submission.
 * Provides native [cron](https://en.wikipedia.org/wiki/Cron) support for running scheduled applications.
 * Supports customization of Spark pods beyond what Spark natively is able to do through the mutating admission webhook, e.g., mounting ConfigMaps and volumes, and setting pod affinity/anti-affinity.
@@ -84,8 +85,8 @@ The Kubernetes Operator for Apache Spark currently supports the following list o
 * Supports automatic retries of failed submissions with optional linear back-off.
 * Supports mounting local Hadoop configuration as a Kubernetes ConfigMap automatically via `sparkctl`.
 * Supports automatically staging local application dependencies to Google Cloud Storage (GCS) via `sparkctl`.
-* Supports collecting and exporting application-level metrics and driver/executor metrics to Prometheus. 
+* Supports collecting and exporting application-level metrics and driver/executor metrics to Prometheus.
 
 ## Contributing
 
-Please check [CONTRIBUTING.md](CONTRIBUTING.md) and the [Developer Guide](docs/developer-guide.md) out. 
+Please check [CONTRIBUTING.md](CONTRIBUTING.md) and the [Developer Guide](docs/developer-guide.md) out.

charts/spark-operator-chart/Chart.yaml

@@ -1,7 +1,7 @@
 apiVersion: v2
 name: spark-operator
 description: A Helm chart for Spark on Kubernetes operator
-version: 1.0.2
+version: 1.0.5
 appVersion: v1beta2-1.2.0-3.0.0
 keywords:
   - spark

charts/spark-operator-chart/README.md

@@ -17,10 +17,34 @@ The previous `spark-operator` Helm chart hosted at [helm/charts](https://github.
 
 - This repository **only** supports Helm chart installations using Helm 3+ since the `apiVersion` on the chart has been marked as `v2`.
 - Previous versions of the Helm chart have not been migrated, and the version has been set to `1.0.0` at the onset. If you are looking for old versions of the chart, it's best to run `helm pull incubator/sparkoperator --version <your-version>` until you are ready to move to this repository's version.
+- Several configuration properties have been changed, carefully review the [values](#values) section below to make sure you're aligned with the new values.
 
 ## Installing the chart
 
-TBD
+```shell
+
+$ helm repo add spark-operator https://googlecloudplatform.github.io/spark-on-k8s-operator
+
+$ helm install my-release spark-operator/spark-operator
+```
+
+This will create a release of `spark-operator` in the default namespace. To install in a different one:
+
+```shell
+$ helm install -n spark my-release spark-operator/spark-operator
+```
+
+Note that `helm` will fail to install if the namespace doesn't exist. Either create the namespace beforehand or pass the `--create-namespace` flag to the `helm install` command.
+
+## Uninstalling the chart
+
+To uninstall `my-release`:
+
+```shell
+$ helm uninstall my-release
+```
+
+The command removes all the Kubernetes components associated with the chart and deletes the release, except for the `crds`, those will have to be removed manually.
 
 ## Values
 
@@ -42,10 +66,16 @@ TBD
 | metrics.enable | bool | `true` | Enable prometheus mertic scraping |
 | metrics.endpoint | string | `"/metrics"` | Metrics serving endpoint |
 | metrics.port | int | `10254` | Metrics port |
+| metrics.portName | string | `metrics` | Metrics port name |
 | metrics.prefix | string | `""` | Metric prefix, will be added to all exported metrics |
 | nameOverride | string | `""` | String to partially override `spark-operator.fullname` template (will maintain the release name) |
 | nodeSelector | object | `{}` | Node labels for pod assignment |
 | podAnnotations | object | `{}` | Additional annotations to add to the pod |
+| podMonitor.enable | bool| `false` | Submit a prometheus pod monitor for operator's pod. Note that prometheus metrics should be enabled as well.|
+| podMonitor.labels | object | `{}` | Pod monitor labels |
+| podMonitor.jobLabel | string | `spark-operator-podmonitor` | The label to use to retrieve the job name from |
+| podMonitor.podMetricsEndpoint.scheme | string | `http` | Prometheus metrics endpoint scheme |
+| podMonitor.podMetricsEndpoint.interval | string | `5s` | Interval at which metrics should be scraped |
 | podSecurityContext | object | `{}` | Pod security context |
 | rbac.create | bool | `true` | Create and use `rbac` resources |
 | replicaCount | int | `1` | Desired number of pods, leaderElection will be enabled if this is greater than 1 |

charts/spark-operator-chart/README.md.gotmpl

@@ -17,10 +17,34 @@ The previous `spark-operator` Helm chart hosted at [helm/charts](https://github.
 
 - This repository **only** supports Helm chart installations using Helm 3+ since the `apiVersion` on the chart has been marked as `v2`.
 - Previous versions of the Helm chart have not been migrated, and the version has been set to `1.0.0` at the onset. If you are looking for old versions of the chart, it's best to run `helm pull incubator/sparkoperator --version <your-version>` until you are ready to move to this repository's version.
+- Several configuration properties have been changed, carefully review the [values](#values) section below to make sure you're aligned with the new values.
 
 ## Installing the chart
 
-TBD
+```shell
+
+$ helm repo add spark-operator https://googlecloudplatform.github.io/spark-on-k8s-operator
+
+$ helm install my-release spark-operator/spark-operator
+```
+
+This will create a release of `spark-operator` in the default namespace. To install in a different one:
+
+```shell
+$ helm install -n spark my-release spark-operator/spark-operator
+```
+
+Note that `helm` will fail to install if the namespace doesn't exist. Either create the namespace beforehand or pass the `--create-namespace` flag to the `helm install` command.
+
+## Uninstalling the chart
+
+To uninstall `my-release`:
+
+```shell
+$ helm uninstall my-release
+```
+
+The command removes all the Kubernetes components associated with the chart and deletes the release, except for the `crds`, those will have to be removed manually.
 
 {{ template "chart.valuesSection" . }}
 

charts/spark-operator-chart/crds/sparkoperator.k8s.io_scheduledsparkapplications.yaml

@@ -3614,6 +3614,8 @@ spec:
                           maximum: 49151
                           minimum: 1024
                           type: integer
+                        portName:
+                          type: string
                       required:
                       - jmxExporterJar
                       type: object

charts/spark-operator-chart/crds/sparkoperator.k8s.io_sparkapplications.yaml

@@ -3600,6 +3600,8 @@ spec:
                       maximum: 49151
                       minimum: 1024
                       type: integer
+                    portName:
+                      type: string
                   required:
                   - jmxExporterJar
                   type: object

charts/spark-operator-chart/templates/deployment.yaml

@@ -48,7 +48,7 @@ spec:
           {{- toYaml .Values.securityContext | nindent 10 }}
         {{- if .Values.metrics.enable }}
         ports:
-          - name: metrics
+          - name: {{ .Values.metrics.portName | quote }}
             containerPort: {{ .Values.metrics.port }}
         {{ end }}
         args:

charts/spark-operator-chart/templates/prometheus-podmonitor.yaml

@@ -0,0 +1,19 @@
+{{ if and .Values.metrics.enable .Values.podMonitor.enable }}
+apiVersion: monitoring.coreos.com/v1
+kind: PodMonitor
+metadata:
+  name: {{ include "spark-operator.name" . -}}-podmonitor
+  labels: {{ toYaml .Values.podMonitor.labels | nindent 4 }}
+spec:
+  podMetricsEndpoints:
+    - interval: {{ .Values.podMonitor.podMetricsEndpoint.interval }}
+      port: {{ .Values.metrics.portName | quote }}
+      scheme: {{ .Values.podMonitor.podMetricsEndpoint.scheme }}
+  jobLabel: {{ .Values.podMonitor.jobLabel }}
+  namespaceSelector:
+    matchNames:
+      - {{ .Release.Namespace }}
+  selector:
+    matchLabels:
+      {{- include "spark-operator.selectorLabels" . | nindent 6 }}
+{{ end }}

charts/spark-operator-chart/values.yaml

@@ -75,11 +75,26 @@ metrics:
   enable: true
   # -- Metrics port
   port: 10254
+  # -- Metrics port name
+  portName: metrics
   # -- Metrics serving endpoint
   endpoint: /metrics
   # -- Metric prefix, will be added to all exported metrics
   prefix: ""
 
+# -- Prometheus pod monitor for operator's pod.
+podMonitor:
+  # -- If enabled, a pod monitor for operator's pod will be submitted. Note that prometheus metrics should be enabled as well.
+  enable: false
+  # -- Pod monitor labels
+  labels: {}
+  # -- The label to use to retrieve the job name from
+  jobLabel: spark-operator-podmonitor
+  # -- Prometheus metrics endpoint properties. `metrics.portName` will be used as a port
+  podMetricsEndpoint:
+    scheme: http
+    interval: 5s
+
 # nodeSelector -- Node labels for pod assignment
 nodeSelector: {}
 

docs/developer-guide.md

@@ -38,11 +38,11 @@ To update the auto-generated code, run the following command. (This step is only
 $ hack/update-codegen.sh
 ```
 
-To update the auto-generated CRD definitions, run the following command:
+To update the auto-generated CRD definitions, run the following command. After doing so, you must update the list of required fields under each `ports` field to add the `protocol` field to the list. Skipping this step will make the CRDs incompatible with Kubernetes v1.18+.
 
 ```bash
 $ GO111MODULE=off go get -u sigs.k8s.io/controller-tools/cmd/controller-gen
-$ controller-gen crd:trivialVersions=true,maxDescLen=0 paths="./pkg/apis/sparkoperator.k8s.io/v1beta2" output:crd:artifacts:config=./manifest/crds/
+$ controller-gen crd:trivialVersions=true,maxDescLen=0,crdVersions=v1beta1 paths="./pkg/apis/sparkoperator.k8s.io/v1beta2" output:crd:artifacts:config=./manifest/crds/
 ```
 
 You can verify the current auto-generated code is up to date with: