Skip to content

Commit

Permalink
kms: refine proposal
Browse files Browse the repository at this point in the history 
Signed-off-by: Damien Grisonnet <[email protected]>
  • Loading branch information
@dgrisonnet
dgrisonnet committed Nov 21, 2024
1 parent ecf7cbb commit fd0142d
Show file tree
Hide file tree
Showing 2 changed files with 111 additions and 77 deletions.
Binary file added enhancements/kube-apiserver/kms-design.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
188 changes: 111 additions & 77 deletions enhancements/kube-apiserver/kms-encryption-provider.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,12 @@ reviewers:
- "@rvanderp"
approvers:
- "@tkashem"
- "deads2k"
- "derekwaynecarr"
api-approvers:
- "@JoelSpeed"
creation-date: 2024-08-14
last-updated: 2024-08-14
last-updated: 2024-11-21
status: implementable
see-also:
- "/enhancements/kube-apiserver/encrypting-data-at-datastore-layer.md"
Expand All @@ -31,9 +33,9 @@ Provide a user-configurable interface to support encryption of data stored in et

## Motivation

Today, we support local AES encryption at the datastore layer which protects against etcd data leaks in the event of a etcd backup compromise. However, aescbc and aesgcm which are supported ecncryption technologies today available in OpenShift do not protect against online host compromise i.e. in such cases attacker can decrypt encrypted data from etcd using local keys, KMS managed keys protects against such scenarios.
Today, we support local AES encryption at the datastore layer. It protects against etcd data leaks in the event of a etcd backup compromise. However, aescbc and aesgcm, which are supported encryption technologies today available in OpenShift do not protect against online host compromise i.e. in such cases, attackers can decrypt encrypted data from etcd using local keys, KMS managed keys protects against such scenarios.

Users of OpenShift would like the encrypt secret data in etcd using self-managed KMS backed keys.
Users of OpenShift would like to encrypt secret data in etcd using self-managed KMS-backed keys.
- https://issues.redhat.com/browse/OCPSTRAT-108

### User Stories
Expand All @@ -43,8 +45,8 @@ As an OpenShift administrator, I want to encrypt secrets in my cluster at rest u
As an OpenShift administrator, I want to let the KMS provider manage the lifecycle of the encryption keys.

As an OpenShift user, I enable encryption by setting apiserver.spec.encryption.type to kms.
- After some time passes, user makes a backup of etcd.
- The user confirms that the secret values are encrypted by checking to see if they have the related kms prefix.
- After some time passes, a user makes a backup of etcd.
- The user confirms that the secret values are encrypted by checking to see if they have the related KMS prefix.

### Goals

Expand All @@ -58,13 +60,46 @@ As an OpenShift user, I enable encryption by setting apiserver.spec.encryption.t
### Non-Goals

1. Allow the user to force key rotation
2. Support for the complete lifecyle of KMS managed keys directly within OpenShift control plane
2. Support for the complete lifecycle of KMS-managed keys directly within OpenShift control-plane
3. Support for hardware security modules
4. The user has in-depth understanding of each phase of the encryption process
4. The user has an in-depth understanding of each phase of the encryption process
5. Completely recover the cluster in the event of the KMS instance itself going down or keys getting lost
6. Allow users to configure which resources will be encrypted

### Proposal
## Proposal

To support KMS encryption in OpenShift, we will be able to leverage the work that was done in [upstream Kubernetes](https://kubernetes.io/docs/tasks/administer-cluster/kms-provider/). However, we will need to extend and adapt the encryption workflow in OpenShift to support new constraints introduced by the externalization of encryption keys in a KMS. Because OpenShift will not own the keys from the KMS, we will also need to provide tools to the users to detect KMS-related failures and take action toward recovering their clusters whenever possible.

There exist two versions of the KMS API upstream today. In OpenShift, we specifically want to use KMS v2 only as it is an optimized version of the first API that is more production-ready than its predecessor, which was infamous for its impact on performance and the pressure it puts on the KMS.

In OpenShift, we will add a new encryption type to the list in the apiserver API as `kms`, similar to the list of possible encryption providers from the apiserver's [EncryptionConfig](https://github.com/kubernetes/apiserver/blob/cccad306d649184bf2a0e319ba830c53f65c445c/pkg/apis/apiserver/types_encryption.go#L89-L101).

Unlike the other providers, KMS will require additional configuration from the users. Because of that, we will add a new configuration to the API to allow users to configure the connection to their KMS. In this new API, users will be able to configure some options specific to the KMS provider they want to use. There will be dedicated APIs for each KMS supported by OpenShift.

From a UX perspective, these are the only changes the KMS feature will introduce. It is intentionally minimal to reduce the burden on the users and the potential for errors.

In practice, this feature will re-use as much of the existing encryption logic as possible. It will leverage the existing encryption and migration workflow introduced for AES-CBC and AES-GCM. However, unlike the aescbc and aesgcm providers, the encryption keys for KMS are not managed by the apiserver operators, so we have to extend the existing controllers to support that new workflow. On top of that, the operators used to be in charge of rotating the keys, but it will now be in the hands of the users to rotate the keys. We will need to introduce a new workflow to react when keys are rotated in the external KMS to make sure that the encrypted data is migrated to use the new key.

### KMS plugins

One aspect of the upstream feature that wasn't mentioned yet is that it requires a third-party application called a KMS plugin to bridge between the apiservers and the KMS. In OpenShift, these plugins will be configured and managed by the kube-apiserver-operator. There are multiple reasons behind this choice:

1. Reduces the complexity for the users that want to use the KMS feature
2. Simplifies key rotation when users manually rotate the key because it requires creating a second instance of the plugin that would use the new key while the old plugin would still allow using the old key as a read key
3. It is cheap to maintain the plugins as we can either leverage the upstream communities or the vendors
4. Have more trust and guarantees towards the plugins that will be running in the platform

Plugins available in the open will be forked and maintained downstream. The images for these plugins will be published and distributed on the official Red Hat registry in the same way as they are today for HyperShift.
Existing forks include:

* https://github.com/openshift/aws-encryption-provider/
* https://github.com/openshift/azure-kubernetes-kms/

For the plugins we can't distribute, an `image` field will be available in the relevant KMS API to allow users to configure the plugin.

In the future, we will also be able to think about ways to qualify new plugins to be distributed and supported by OCP.

### API Extensions

OpenShift would need to align closer with KMS evolution upstream with respect to the different Kubernetes Encryption Providers available today.

Expand All @@ -75,82 +110,88 @@ diff --git a/config/v1/types_apiserver.go b/config/v1/types_apiserver.go
index d815556d2..c9098024f 100644
--- a/config/v1/types_apiserver.go
+++ b/config/v1/types_apiserver.go
@@ -173,6 +173,9 @@ type APIServerNamedServingCert struct {
ServingCertificate SecretNameReference `json:"servingCertificate"`
}

+// APIServerEncryption is used to encrypt sensitive resources on the cluster.
+// +openshift:validation:FeatureGateAwareXValidation:featureGate=KMSEncryptionProvider,rule="has(self.type) && self.type == 'KMS' ? has(self.kms) : !has(self.kms)",message="kms config is required when encryption type is KMS, and forbidden otherwise"
+// +union
type APIServerEncryption struct {
// type defines what encryption type should be used to encrypt resources at the datastore layer.
// When this field is unset (i.e. when it is set to the empty string), identity is implied.
@@ -191,9 +194,23 @@ type APIServerEncryption struct {
// +unionDiscriminator
// +optional
Type EncryptionType `json:"type,omitempty"`
+
+ // kms defines the configuration for the external KMS instance that manages the encryption keys,
+ // when KMS encryption is enabled sensitive resources will be encrypted using keys managed by an
+ // externally configured KMS instance.
+ //
+ // The Key Management Service (KMS) instance provides symmetric encryption and is responsible for
+ // managing the lifecyle of the encryption keys outside of the control plane.
+ // This allows integration with an external provider to manage the data encryption keys securely.
+ //
+ // +openshift:enable:FeatureGate=KMSEncryptionProvider
+ // +unionMember
+ // +optional
+ KMS *KMSConfig `json:"kms,omitempty"`
}

-// +kubebuilder:validation:Enum="";identity;aescbc;aesgcm
+// +openshift:validation:FeatureGateAwareEnum:featureGate="",enum="";identity;aescbc;aesgcm
+// +openshift:validation:FeatureGateAwareEnum:featureGate=KMSEncryptionProvider,enum="";identity;aescbc;aesgcm;KMS
type EncryptionType string

const (
@@ -208,6 +225,11 @@ const (
// aesgcm refers to a type where AES-GCM with random nonce and a 32-byte key
// is used to perform encryption at the datastore layer.
EncryptionTypeAESGCM EncryptionType = "aesgcm"
// aesgcm refers to a type where AES-GCM with random nonce and a 32-byte key
// is used to perform encryption at the datastore layer.
EncryptionTypeAESGCM EncryptionType = "aesgcm"
+
+ // kms refers to a type of encryption where the encryption keys are managed
+ // outside the control plane in a Key Management Service instance,
+ // encryption is still performed at the datastore layer.
+ EncryptionTypeKMS EncryptionType = "KMS"
+ // kms refers to a type of encryption where the encryption keys are managed
+ // outside the control plane in a Key Management Service instance,
+ // encryption is still performed at the datastore layer.
+ EncryptionTypeKMS EncryptionType = "KMS"
)

type APIServerStatus struct {
```

The default value today is an empty string, which implies identity and that no encryption is used in the cluster by default. Other possible local encryption schemes include `aescbc` and `aesgcm`, which will remain as-is. Similar to how local AES encryption works, the apiserver operators will observe this config and apply the KMS EncryptionProvider to the EncryptionConfig.

```diff
@@ -191,9 +194,23 @@ type APIServerEncryption struct {
// +unionDiscriminator
// +optional
Type EncryptionType `json:"type,omitempty"`
+
+ // kms defines the configuration for the external KMS instance that manages the encryption keys,
+ // when KMS encryption is enabled sensitive resources will be encrypted using keys managed by an
+ // externally configured KMS instance.
+ //
+ // The Key Management Service (KMS) instance provides symmetric encryption and is responsible for
+ // managing the lifecyle of the encryption keys outside of the control plane.
+ // This allows integration with an external provider to manage the data encryption keys securely.
+ //
+ // +openshift:enable:FeatureGate=KMSEncryptionProvider
+ // +unionMember
+ // +optional
+ KMS *KMSConfig `json:"kms,omitempty"`
```

As mentioned before, the KMS encryption type will have a dedicated configuration.

```diff
diff --git a/config/v1/types_kmsencryption.go b/config/v1/types_kmsencryption.go
new file mode 100644
index 000000000..575affae6
index 000000000..8841cd749
--- /dev/null
+++ b/config/v1/types_kmsencryption.go
@@ -0,0 +1,50 @@
@@ -0,0 +1,49 @@
+package v1
+
+// KMSConfig defines the configuration for the KMS instance
+// that will be used with KMSEncryptionProvider encryption
+// +kubebuilder:validation:XValidation:rule="has(self.type) && self.type == 'AWS' ? has(self.aws) : !has(self.aws)",message="aws config is required when kms provider type is AWS, and forbidden otherwise"
+// +union
+type KMSConfig struct {
+ // type defines the kind of platform for the KMS provider.
+ // Available provider types are AWS only.
+ //
+ // +unionDiscriminator
+ // +kubebuilder:validation:Required
+ Type KMSProviderType `json:"type"`
+ // type defines the kind of platform for the KMS provider
+ //
+ // +unionDiscriminator
+ // +kubebuilder:validation:Required
+ Type KMSProviderType `json:"type"`
+
+ // aws defines the key config for using an AWS KMS instance
+ // for the encryption. The AWS KMS instance is managed
+ // by the user outside the purview of the control plane.
+ //
+ // +unionMember
+ // +optional
+ AWS *AWSKMSConfig `json:"aws,omitempty"`
+ // aws defines the key config for using an AWS KMS instance
+ // for the encryption. The AWS KMS instance is managed
+ // by the user outside the purview of the control plane.
+ //
+ // +unionMember
+ // +optional
+ AWS *AWSKMSConfig `json:"aws,omitempty"`
+}

+// KMSProviderType is a specific supported KMS provider
+// +kubebuilder:validation:Enum="";AWS
+type KMSProviderType string
+
+const (
+ // AWSKMSProvider represents a supported KMS provider for use with AWS KMS
+ AWSKMSProvider KMSProviderType = "AWS"
+)
```

This configuration will also include an enum of the various KMS supported by OCP. At first, it will only have the `AWS` type, but we will add more as we progress on the feature. This enum is essential to avoid potential ambiguities that might arise with future KMS.

Each KMS type will have a dedicated configuration that will be reflected on the plugin when installed. It will only be a partial representation of the plugin's configuration because most fields are irrelevant to the end users.

At first, this configuration will only include the `AWSKMSConfig`, but more KMS-specifc configs will be added as we include new KMS.

```diff
+// AWSKMSConfig defines the KMS config specific to AWS KMS provider
+type AWSKMSConfig struct {
+ // keyARN specifies the Amazon Resource Name (ARN) of the AWS KMS key used for encryption.
Expand All @@ -169,18 +210,11 @@ index 000000000..575affae6
+ // +kubebuilder:validation:XValidation:rule="self.matches('^[a-z]{2}-[a-z]+-[0-9]+$') && self.size() <= 64",message="region must be a valid AWS region"
+ Region string `json:"region"`
+}
+
+// KMSProviderType is a specific supported KMS provider
+// +kubebuilder:validation:Enum="";AWS
+type KMSProviderType string
+
+const (
+ // AWSKMSProvider represents a supported KMS provider for use with AWS KMS
+ AWSKMSProvider KMSProviderType = "AWS"
+)
```

The default value today is an empty string which implies identity and that no encryption is used in the cluster by default. Other possible local encryption schemes include `aescbc` and `aesgcm` which will remain as-is. Similar to how local AES encryption works, the kube-apiserver operator and openshift-apiserver operator will observe this config to apply the KMS Encryption Provider config onto kube-apiserver(s) and openshift-apiserver(s) respectively.
From a very high-level, the figure below shows how the new APIs will be used by the various components.

![KMS high-level design](./kms-design.png)

### Implementation Details/Notes/Constraints

Expand Down

0 comments on commit fd0142d

Please sign in to comment.