Skip to content

Commit

Permalink
Merge branch 'develop' into helm-auth-proxy-service-label-fix
Browse files Browse the repository at this point in the history
  • Loading branch information
Astroknut authored May 17, 2023
2 parents 027450d + 64c45a1 commit 48c7ec1
Show file tree
Hide file tree
Showing 24 changed files with 434 additions and 24 deletions.
1 change: 1 addition & 0 deletions docs/AppFramework.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ Utilizing the App Framework requires one of the following remote storage provide
### Prerequisites for S3 based remote object storage
* Create role and role-binding for splunk-operator service account, to provide read-only access for S3 credentials.
* The remote object storage credentials provided as a kubernetes secret, or in an IAM role.
* If you are using [interface VPC endpoints](https://docs.aws.amazon.com/vpc/latest/privatelink/create-interface-endpoint.html) with DNS enabled to access AWS S3, please update the corresponding volume endpoint URL with one of the `DNS names` from the endpoint. Please ensure that the endpoint has access to the S3 buckets using the credentials configured. Similarly other endpoint URLs with access to the S3 buckets can also be used.

### Prerequisites for Azure Blob remote object storage
* The remote object storage credentials provided as a kubernetes secret.
Expand Down
1 change: 1 addition & 0 deletions docs/SmartStore.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ The Splunk Operator includes a method for configuring a SmartStore remote storag

* SmartStore configuration is supported on these Custom Resources: Standalone and ClusterManager.
* SmartStore support in the Splunk Operator is limited to Amazon S3 & S3-API-compliant object stores only if you are using the CRD configuration for S3 as described below."
* For Amazon S3, if you are using [interface VPC endpoints](https://docs.aws.amazon.com/vpc/latest/privatelink/create-interface-endpoint.html) with DNS enabled to access AWS S3, please update the corresponding volume endpoint URL with one of the `DNS names` from the endpoint. Please ensure that the endpoint has access to the S3 buckets using the credentials configured. Similarly other endpoint URLs with access to the S3 buckets can also be used.
* Use of GCS with SmartStore is supported by using configuration via Splunk App.
* Specification allows definition of SmartStore-enabled indexes only.
* Already existing indexes data should be migrated from local storage to the remote store as a pre-requisite before configuring those indexes in the Custom Resource of the Splunk Operator. For more details, please see [Migrate existing data on an indexer cluster to SmartStore](https://docs.splunk.com/Documentation/Splunk/latest/Indexer/MigratetoSmartStore#Migrate_existing_data_on_an_indexer_cluster_to_SmartStore).
Expand Down
11 changes: 5 additions & 6 deletions docs/SplunkOperatorUpgrade.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,10 +28,10 @@ A Splunk Operator for Kubernetes upgrade might include support for a later versi
wget -O splunk-operator-namespace.yaml https://github.com/splunk/splunk-operator/releases/download/2.2.1/splunk-operator-namespace.yaml
```
2. (Optional) Review the file and update it with your specific customizations used during your install.
3. Upgrade the Splunk Operator.
2. (Optional) Review the file and update it with your specific customizations used during your install.

3. Upgrade the Splunk Operator.​
```
kubectl apply -f splunk-operator-namespace.yaml --server-side --force-conflicts
```
Expand All @@ -42,7 +42,6 @@ After applying the yaml, a new operator pod will be created and the existing ope
kubectl get pods
NAME READY STATUS RESTARTS AGE
splunk-operator-controller-manager-75f5d4d85b-8pshn 1/1 Running 0 5s
```
If a Splunk Operator release changes the custom resource (CRD) API version, the administrator is responsible for updating their Custom Resource specification to reference the latest CRD API version.
Expand Down Expand Up @@ -73,11 +72,11 @@ wget -O operator-upgarde.sh https://github.com/splunk/splunk-operator/releases/d
wget -O splunk-operator-install.yaml https://github.com/splunk/splunk-operator/releases/download/1.1.0/splunk-operator-install.yaml
```

3. (Optional) Review the file and update it with your specific customizations used during your install.
3. (Optional) Review the file and update it with your specific customizations used during your install.

4. Upgrade the Splunk Operator.

Set KUBECONFIG and run [operator-upgrade.sh](https://github.com/splunk/splunk-operator/releases/download/2.1.0/operator-upgrade.sh) script with the following mandatory arguments
Set KUBECONFIG and run the already downloaded `operator-upgrade.sh` script with the following mandatory arguments

* `current_namespace` current namespace where operator is installed
* `manifest_file`: path to 1.1.0 Splunk Operator manifest file
Expand Down
12 changes: 6 additions & 6 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,10 @@ go 1.19

require (
github.com/aws/aws-sdk-go v1.42.16
github.com/go-logr/logr v1.2.3
github.com/go-logr/logr v1.2.4
github.com/google/go-cmp v0.5.9
github.com/minio/minio-go/v7 v7.0.16
github.com/onsi/ginkgo/v2 v2.9.2
github.com/onsi/ginkgo/v2 v2.9.5
github.com/onsi/gomega v1.27.6
github.com/pkg/errors v0.9.1
github.com/prometheus/client_golang v1.14.0
Expand Down Expand Up @@ -65,13 +65,13 @@ require (
go.uber.org/atomic v1.7.0 // indirect
go.uber.org/multierr v1.6.0 // indirect
golang.org/x/crypto v0.1.0 // indirect
golang.org/x/net v0.8.0 // indirect
golang.org/x/net v0.10.0 // indirect
golang.org/x/oauth2 v0.0.0-20220223155221-ee480838109b // indirect
golang.org/x/sys v0.6.0 // indirect
golang.org/x/sys v0.8.0 // indirect
golang.org/x/term v0.5.0 // indirect
golang.org/x/text v0.8.0 // indirect
golang.org/x/text v0.9.0 // indirect
golang.org/x/time v0.3.0 // indirect
golang.org/x/tools v0.7.0 // indirect
golang.org/x/tools v0.9.1 // indirect
gomodules.xyz/jsonpatch/v2 v2.2.0 // indirect
google.golang.org/appengine v1.6.7 // indirect
google.golang.org/protobuf v1.28.1 // indirect
Expand Down
10 changes: 10 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,8 @@ github.com/go-logr/logr v1.2.0/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbV
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
github.com/go-logr/logr v1.2.3 h1:2DntVwHkVopvECVRSlL5PSo9eG+cAkDCuckLubN+rq0=
github.com/go-logr/logr v1.2.3/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
github.com/go-logr/logr v1.2.4 h1:g01GSCwiDw2xSZfjJ2/T9M+S6pFdcNtFYsp+Y43HYDQ=
github.com/go-logr/logr v1.2.4/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
github.com/go-logr/zapr v1.2.3 h1:a9vnzlIBPQBBkeaR9IuMUfmVOrQlkoC4YfPoFkX3T7A=
github.com/go-logr/zapr v1.2.3/go.mod h1:eIauM6P8qSvTw5o2ez6UEAfGjQKrxQTl5EoK+Qa2oG4=
github.com/go-openapi/jsonpointer v0.19.3/go.mod h1:Pl9vOtqEWErmShwVjC8pYs9cog34VGT37dQOVbmoatg=
Expand Down Expand Up @@ -248,6 +250,8 @@ github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e h1:fD57ERR4JtEqsWb
github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e/go.mod h1:zD1mROLANZcx1PVRCS0qkT7pwLkGfwJo4zjcN/Tysno=
github.com/onsi/ginkgo/v2 v2.9.2 h1:BA2GMJOtfGAfagzYtrAlufIP0lq6QERkFmHLMLPwFSU=
github.com/onsi/ginkgo/v2 v2.9.2/go.mod h1:WHcJJG2dIlcCqVfBAwUCrJxSPFb6v4azBwgxeMeDuts=
github.com/onsi/ginkgo/v2 v2.9.5 h1:+6Hr4uxzP4XIUyAkg61dWBw8lb/gc4/X5luuxN/EC+Q=
github.com/onsi/ginkgo/v2 v2.9.5/go.mod h1:tvAoo1QUJwNEU2ITftXTpR7R1RbCzoZUOs3RonqW57k=
github.com/onsi/gomega v1.27.6 h1:ENqfyGeS5AX/rlXDd/ETokDz93u0YufY1Pgxuy/PvWE=
github.com/onsi/gomega v1.27.6/go.mod h1:PIQNjfQwkP3aQAH7lf7j87O/5FiNr+ZR8+ipb+qQlhg=
github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
Expand Down Expand Up @@ -418,6 +422,8 @@ golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.6.0 h1:MVltZSvRTcU2ljQOhs94SXPftV6DCNnZViHeQps87pQ=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.8.0 h1:EBmGv8NaZBZTWvrbjNoL6HVt+IVy3QDQpJs7VRIw3tU=
golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/term v0.1.0/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.5.0 h1:n2a8QNdAb0sZNpU9R1ALUXBbY+w51fCQDN+7EdxNBsY=
golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
Expand All @@ -431,6 +437,8 @@ golang.org/x/text v0.4.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
golang.org/x/text v0.8.0 h1:57P1ETyNKtuIjB4SRd15iJxuhj8Gc416Y78H3qgMh68=
golang.org/x/text v0.8.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
golang.org/x/text v0.9.0 h1:2sjJmO8cDvYveuX97RDLsxlyUxLl+GHoLxBiRdHllBE=
golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
Expand Down Expand Up @@ -483,6 +491,8 @@ golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4f
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
golang.org/x/tools v0.7.0 h1:W4OVu8VVOaIO0yzWMNdepAulS7YfoS3Zabrm8DOXXU4=
golang.org/x/tools v0.7.0/go.mod h1:4pg6aUX35JBAogB10C9AtvVL+qowtN4pT3CGSQex14s=
golang.org/x/tools v0.9.1 h1:8WMNJAz3zrtPmnYC7ISf5dEn3MT0gY7jBJfw27yrrLo=
golang.org/x/tools v0.9.1/go.mod h1:owI94Op576fPu3cIGQeHs3joujW/2Oc6MtlxbF5dfNc=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
Expand Down
13 changes: 13 additions & 0 deletions kuttl/tests/helm/c3-with-apps-private-link/00-create-bucket.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
---
apiVersion: kuttl.dev/v1beta1
kind: TestStep
commands:
- script: aws s3 mb s3://helm-test-c3-with-apps-$NAMESPACE --region $AWS_DEFAULT_REGION
background: false
skipLogOutput: true
- script: aws s3 cp --recursive s3://$TEST_S3_BUCKET/appframework/v2apps s3://helm-test-c3-with-apps-$NAMESPACE/appframework --region $AWS_DEFAULT_REGION
background: false
skipLogOutput: true
- script: aws s3 mb s3://helm-test-c3-with-apps-smartstore-$NAMESPACE --region $AWS_DEFAULT_REGION
background: false
skipLogOutput: true
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
---
apiVersion: kuttl.dev/v1beta1
kind: TestStep
commands:
- script: pwd
background: false
- script: yq w -i c3_config.yaml "clusterManager.appRepo.defaults.volumeName" "volume_app_repo"
background: false
skipLogOutput: true
- script: yq w -i c3_config.yaml "clusterManager.appRepo.defaults.scope" "local"
background: false
skipLogOutput: true
- script: yq w -i c3_config.yaml "clusterManager.appRepo.appSources[0].name" "apps"
background: false
skipLogOutput: true
- script: yq w -i c3_config.yaml "clusterManager.appRepo.appSources[0].location" "appframework"
background: false
skipLogOutput: true
- script: yq w -i c3_config.yaml "clusterManager.appRepo.appSources[0].scope" "cluster"
background: false
skipLogOutput: true
- script: yq w -i c3_config.yaml "clusterManager.appRepo.volumes[0].name" "volume_app_repo"
background: false
skipLogOutput: true
- script: yq w -i c3_config.yaml "clusterManager.appRepo.volumes[0].storageType" "s3"
background: false
skipLogOutput: true
- script: yq w -i c3_config.yaml "clusterManager.appRepo.volumes[0].provider" "aws"
background: false
skipLogOutput: true
- script: yq w -i c3_config.yaml "clusterManager.appRepo.volumes[0].path" "helm-test-c3-with-apps-$NAMESPACE"
background: false
skipLogOutput: true
- script: yq w -i c3_config.yaml "clusterManager.appRepo.volumes[0].endpoint" $TEST_VPC_ENDPOINT_URL
background: false
skipLogOutput: true
- script: yq w -i c3_config.yaml "clusterManager.appRepo.volumes[0].region" $S3_REGION
background: false
skipLogOutput: true
- script: pwd
background: false
- script: yq w -i c3_config.yaml "searchHeadCluster.appRepo.defaults.volumeName" "volume_app_repo"
background: false
skipLogOutput: true
- script: yq w -i c3_config.yaml "searchHeadCluster.appRepo.defaults.scope" "local"
background: false
skipLogOutput: true
- script: yq w -i c3_config.yaml "searchHeadCluster.appRepo.appSources[0].name" "apps"
background: false
skipLogOutput: true
- script: yq w -i c3_config.yaml "searchHeadCluster.appRepo.appSources[0].location" "appframework"
background: false
skipLogOutput: true
- script: yq w -i c3_config.yaml "searchHeadCluster.appRepo.appSources[0].scope" "cluster"
background: false
skipLogOutput: true
- script: yq w -i c3_config.yaml "searchHeadCluster.appRepo.volumes[0].name" "volume_app_repo"
background: false
skipLogOutput: true
- script: yq w -i c3_config.yaml "searchHeadCluster.appRepo.volumes[0].storageType" "s3"
background: false
skipLogOutput: true
- script: yq w -i c3_config.yaml "searchHeadCluster.appRepo.volumes[0].provider" "aws"
background: false
skipLogOutput: true
- script: yq w -i c3_config.yaml "searchHeadCluster.appRepo.volumes[0].path" "helm-test-c3-with-apps-$NAMESPACE"
background: false
skipLogOutput: true
- script: yq w -i c3_config.yaml "searchHeadCluster.appRepo.volumes[0].endpoint" $TEST_VPC_ENDPOINT_URL
background: false
skipLogOutput: true
- script: yq w -i c3_config.yaml "searchHeadCluster.appRepo.volumes[0].region" $S3_REGION
background: false
skipLogOutput: true
- script: yq w -i c3_config.yaml "clusterManager.smartstore.defaults.volumeName" "helm-test-c3-with-apps-smartstore-$NAMESPACE"
background: false
skipLogOutput: true
- script: yq w -i c3_config.yaml "clusterManager.smartstore.indexes[0].volumeName" "helm-test-c3-with-apps-smartstore-$NAMESPACE"
background: false
skipLogOutput: true
- script: yq w -i c3_config.yaml "clusterManager.smartstore.indexes[1].volumeName" "helm-test-c3-with-apps-smartstore-$NAMESPACE"
background: false
skipLogOutput: true
- script: yq w -i c3_config.yaml "clusterManager.smartstore.volumes[0].name" "helm-test-c3-with-apps-smartstore-$NAMESPACE"
background: false
skipLogOutput: true
- script: yq w -i c3_config.yaml "clusterManager.smartstore.volumes[0].path" "helm-test-c3-with-apps-smartstore-$NAMESPACE"/smartstore
background: false
skipLogOutput: true
- script: yq w -i c3_config.yaml "clusterManager.smartstore.volumes[0].region" $S3_REGION
background: false
skipLogOutput: true
- script: yq w -i c3_config.yaml "clusterManager.smartstore.volumes[0].endpoint" $TEST_VPC_ENDPOINT_URL
background: false
skipLogOutput: true
5 changes: 5 additions & 0 deletions kuttl/tests/helm/c3-with-apps-private-link/02-assert.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
apiVersion: v1
kind: Secret
metadata:
name: s3-secret
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
#create new s3 secret to be used in app installation
---
apiVersion: kuttl.dev/v1beta1
kind: TestStep
commands:
- script: kubectl create secret generic s3-secret --from-literal=s3_access_key=$AWS_ACCESS_KEY_ID --from-literal=s3_secret_key=$AWS_SECRET_ACCESS_KEY --namespace $NAMESPACE
background: false
#namespaced: true
skipLogOutput: true
100 changes: 100 additions & 0 deletions kuttl/tests/helm/c3-with-apps-private-link/03-assert.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
---
# assert for splunk operator deployment to be ready
apiVersion: apps/v1
kind: Deployment
metadata:
name: splunk-operator-controller-manager
status:
readyReplicas: 1
availableReplicas: 1

---
# assert for cluster manager custom resource to be ready
apiVersion: enterprise.splunk.com/v4
kind: ClusterManager
metadata:
name: cm
status:
phase: Ready

---
# check if stateful sets are created
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: splunk-cm-cluster-manager
status:
replicas: 1

---
# check if secret object are created
apiVersion: v1
kind: Secret
metadata:
name: splunk-cm-cluster-manager-secret-v1

---
# assert for indexer cluster custom resource to be ready
apiVersion: enterprise.splunk.com/v4
kind: IndexerCluster
metadata:
name: idxc
status:
phase: Ready

---
# check for stateful set and replicas as configured
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: splunk-idxc-indexer
status:
replicas: 3

---
# check if secret object are created
apiVersion: v1
kind: Secret
metadata:
name: splunk-idxc-indexer-secret-v1

---
# assert for SearchHeadCluster custom resource to be ready
apiVersion: enterprise.splunk.com/v4
kind: SearchHeadCluster
metadata:
name: shc
status:
phase: Ready

---
# check if secret object are created
apiVersion: v1
kind: Secret
metadata:
name: splunk-shc-deployer-secret-v1

---
# check if secret object are created
apiVersion: v1
kind: Secret
metadata:
name: splunk-shc-search-head-secret-v1

---
# check for stateful set and replicas as configured
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: splunk-shc-search-head
status:
replicas: 3

---
# check for statefull set
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: splunk-shc-deployer
status:
replicas: 1
6 changes: 6 additions & 0 deletions kuttl/tests/helm/c3-with-apps-private-link/03-install-c3.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
---
apiVersion: kuttl.dev/v1beta1
kind: TestStep
commands:
- command: helm install splunk-c3 $HELM_REPO_PATH/splunk-enterprise -f c3_config.yaml --set splunk-operator.splunkOperator.image.repository=${KUTTL_SPLUNK_OPERATOR_IMAGE} --set splunk-operator.image.repository=${KUTTL_SPLUNK_ENTERPRISE_IMAGE}
namespaced: true
17 changes: 17 additions & 0 deletions kuttl/tests/helm/c3-with-apps-private-link/04-assert.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
---
# assert for indexer cluster custom resource to be ready
apiVersion: enterprise.splunk.com/v4
kind: IndexerCluster
metadata:
name: idxc
status:
phase: Ready

---
# check for stateful sets and replicas updated
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: splunk-idxc-indexer
status:
replicas: 4
5 changes: 5 additions & 0 deletions kuttl/tests/helm/c3-with-apps-private-link/04-scaleup-c3.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
apiVersion: kuttl.dev/v1beta1
kind: TestStep
commands:
- command: helm upgrade splunk-c3 $HELM_REPO_PATH/splunk-enterprise --reuse-values --set indexerCluster.replicaCount=4
namespaced: true
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
apiVersion: kuttl.dev/v1beta1
kind: TestStep
commands:
- command: helm uninstall splunk-c3
namespaced: true
Loading

0 comments on commit 48c7ec1

Please sign in to comment.