Skip to content

Commit

Permalink
Seperate general sections from architecture to partials
Browse files Browse the repository at this point in the history
  • Loading branch information
Stephan Feurer committed Dec 14, 2023
1 parent bb342d5 commit 9b46d52
Show file tree
Hide file tree
Showing 8 changed files with 182 additions and 165 deletions.
180 changes: 15 additions & 165 deletions docs/modules/ROOT/pages/references/vsphere/architecture.adoc
View file
Original file line number Diff line number Diff line change
@@ -1,17 +1,10 @@
:infra-type: VMWare vSphere
:infra-svg: ocp4-architecture-vsphere.svg
= APPUiO Managed OpenShift 4 on VMWare vSphere

== Architecture overview

The APPUiO Managed OpenShift 4 architecture on VMWare vSphere is based on the xref:references/architecture/index.adoc[generic APPUiO Managed OpenShift 4 architecture].
We expect that readers of this document are familiar with the xref:references/architecture/index.adoc[generic APPUiO Managed OpenShift 4 architecture] and the overall Kubernetes and OpenShift 4 architecture.

This architecture document extends upon the generic architecture by defining how the APPUiO Managed OpenShift 4 cluster is embedded into the VMWare vSphere environment.
The diagram below shows a detailed view of how an APPUiO Managed OpenShift 4 cluster is embedded into an existing VMWare vSphere environment.

.APPUiO Managed OpenShift4 on vSphere architecture
image::ocp4-architecture-vsphere.svg[alt=OCP4 vSphere architecture, width=900]

The following sections of this document provide detailed descriptions for the elements shown in the architecture diagram.
include::partial$architecture/overview.adoc[]

== vSphere requirements

Expand All @@ -28,13 +21,13 @@ NOTE: The installer also needs to be able to access at least one ESXi host to up

The upstream OpenShift 4 documentation has detailed informations about the https://docs.openshift.com/container-platform/latest/installing/installing_vsphere/installing-vsphere-installer-provisioned.html#installation-vsphere-installer-infra-requirements_installing-vsphere-installer-provisioned[required permissions on vSphere].

NOTE: Entries for "VMWare vSphere CSI Driver Operator" are required.
NOTE: Entries for "{infra-type} CSI Driver Operator" are required.

== Networking

=== Bastion host

To deploy an APPUiO Managed OpenShift 4 cluster on VMWare vSphere, a bastion host inside the customer's premise is required.
To deploy an APPUiO Managed OpenShift 4 cluster on {infra-type}, a bastion host inside the customer's premise is required.
The bastion host:

* must be accessible via SSH from a management system operated by VSHN
Expand All @@ -48,14 +41,7 @@ The bastion host must be provided by the vSphere infrastructure operator, but VS

=== Machine network

Each APPUiO Managed OpenShift 4 cluster is deployed into a /24 "cluster machine network" (sometimes also "cluster network" or "machine network")
This network must be provided by the vSphere infrastructure operator.
DHCP is mandatory for this network, but a number of IPs must be reserved to be used as Virtual IPs for the cluster.

Traffic inside this network shouldn't be restricted.

VMs in this network must be able to reach various services on the internet.
See below for a detailed list of external systems that must be reachable.
include::partial$architecture/networking-cluster.adoc[]

=== Virtual IPs

Expand All @@ -73,29 +59,11 @@ These additional VIPs will be managed by `keepalived` instances on the cluster.

=== Pod and service networks

APPUiO Managed Openshift 4 uses https://cilium.io/[Cilium] to provide in-cluster networking.
Cilium allocates two cluster-internal networks:

1. The pod network: every pod on the cluster will get an IP address from this network.
This network enables basic in-cluster connectivity.
APPUiO Managed OpenShift 4 uses `10.128.0.0/14` as the pod network.
Each node in the cluster is assigned a `/23` from this range.
Pods on a node are always assigned an IP from the range allocated to that node.
2. Service network: used for service discovery.
Traffic to IPs in this network is forwarded to the appropriate pods by Cilium.
APPUiO Managed OpenShift 4 uses `172.30.0.0/16` as the service network.

Both of these networks are interanl to the OpenShift 4 cluster.
Therefore, the IP CIDRs for these networks must not be routable from the outside.
Additionally, the same IP CIDRs can be reused for multiple OpenShift 4 clusters.

However, the chosen CIDRs shouldn't overlap with existing networks allocated by the customer.
If there are overlaps, external systems in the overlapping ranges won't be accessible from within the OpenShift 4 cluster.
The pod and service network CIDRs can be customized if and only if there are conflicts.
include::partial$architecture/networking-pod.adoc[]

=== Exposing the cluster

The vSphere infrastructure operator must provide some form of ingress and egress gateway for the cluster.
The {infra-type} infrastructure operator must provide some form of ingress and egress gateway for the cluster.
The ingress gateway must expose two public IPs:

1. A public IP for the API.
Expand All @@ -112,88 +80,15 @@ Additional DNS records can be pointed to this IP by the customer.

=== External services

APPUiO Managed OpenShift 4 requires various external services.

==== VSHN services

APPUiO Managed OpenShift 4 requires access to VSHN's https://syn.tools[Project Syn] infrastructure.
The Project Syn infrastructure components that must be reachable are

* the Project Syn API at `\https://api.syn.vshn.net`
* the Project Syn Vault at `\https://vault-prod.syn.vshn.net`
* VSHN's GitLab instance at `ssh://[email protected]`
* VSHN's acme-dns instance at `\https://acme-dns-api.vshn.net`

Additionally, APPUiO Managed OpenShift 4 requires access to VSHN's identity management:

* VSHN LDAP at `ldaps://ldap.vshn.net:636`
* VSHN SSO at `\https://id.vshn.net`

Finally, APPUiO Managed OpenShift 4 requires access to VSHN's central metrics storage at `\https://metrics-receive.appuio.net`

==== Red Hat services

See the https://docs.openshift.com/container-platform/4.14/installing/install_config/configuring-firewall.html#configuring-firewall_configuring-firewall[upstream documentation] for the full list of services.

The most important services for APPUiO Managed OpenShift 4 are

* the Red Hat container registries at `registry.redhat.io` and `registry.access.redhat.com`.
* the OpenShift Update Service (OSUS) at `\https://api.openshift.com`.

==== 3rd party services

Finally, APPUiO Managed OpenShift 4 requires access to a number of third party services:

* OpsGenie at `\https://api.opsgenie.com`
* Passbolt at `\https://cloud.passbolt.com/vshn`
* Let's Encrypt at `\https://acme-v02.api.letsencrypt.com` and `\https://acme-staging-v02.api.letsencrypt.com`
* Various container registries
** GitHub at `ghcr.io`
** Quay at `quay.io`
** DockerHub at `docker.io`
** Google container registry at `gcr.io`
** Kubernetes container registry at `registry.k8s.io`
include::partial$architecture/networking-external.adoc[]

== Storage

APPUiO managed OpenShift 4 requires 3 different types of storage:

1. Root disks
2. Persistent volumes
3. S3 compatible object storage

=== Root disks

Root disks are virtual block devices (100 GiB) which are attached to the VMs which make up the APPUiO Managed OpenShift 4 cluster.
The root disks are allocated and attached to the VM when the VM is created.
They hold the operating system and temporary data.
They're ephemeral (no application data is stored on them), and don't need to be backed up.
Finally, root disks are deleted when the VM to which they're attached is deleted.

=== Persistent volumes

Persistent volumes are virtual block devices with arbitrary sizes.
They're allocated dynamically based on requests from workloads (applications or infrastructure components) within the cluster.
These block devices are automatically attached to the VM hosting the application container.
They're deleted when the corresponding Kubernetes `PersistentVolume` resource is deleted.

The VMWare vSphere CSI driver is the in-cluster component which is responsible for allocating, attaching and deleting the persistent volume block devices.

These devices hold application data, but backups are usually done from within the cluster.

=== S3 compatible object storage

Various OpenShift components, such as the integrated image registry, the logging stack and backups, require S3 compatible object storage.
The customer or vSphere infrastructure operator must provide S3 compatible object storage.
Most modern storage solutions offer some object storage functionality.

If https://products.vshn.ch/appcat/index.html[VSHN's Application Catalog (AppCat)] offering is required on the cluster, the object storage must support automatic bucket creation via an AppCat-supported provisioner.

NOTE: If no object storage is available, we can use external object storage as a fallback.
include::partial$architecture/storage.adoc[]

== Glossary

=== Components
=== Components vSphere

[cols="1,3,1"]
|===
Expand All @@ -215,15 +110,6 @@ a|A simple Ubuntu VM which is used by VSHN to bootstrap the cluster(s) and for e

|vSphere infrastructure operator

|Installer
|A CLI tool that bootstraps an OpenShift 4 cluster based on a configuration file.
|VSHN / Red Hat

|Bootstrap Node
|A temporary VM in the cluster machine network which is provisioned by the installer to facilitate the initial setup of the cluster.
This VM is decommissioned by the installer once the cluster installation is completed.
| VSHN / Installer

|vSphere & vCenter
a|VMWare virtualization platform.

Expand All @@ -249,26 +135,6 @@ These two IPs are used for the Kubernetes API and the ingress router.

|vSphere infrastructure operator

|Pod network
a|A subnet that's internal to the Openshift 4 cluster.
This subnet shouldn't be routable from outside the cluster.

This subnet is managed by Cilium and is implemented with VXLAN traffic between the cluster VMs

APPUiO Managed OpenShift 4 uses `10.128.0.0/14` as the pod network.
If the pod network IP range conflicts with existing subnets, the pod network IP range can be adjusted.
| VSHN / Cilium

|Service network
a|A subnet that's internal to the OpenShift 4 cluster.
This subnet shouldn't be routable from outside the cluster.

This subnet is managed by Cilium and is implemented with eBPF rules on the cluster VMs.

APPUiO Managed OpenShift 4 uses `172.30.0.0/16` as the service network.
If the service network IP range conflicts with existing subnets, the service network IP range can be adjusted.
| VSHN / Cilium

|S3 compatible storage
a|Various OpenShift components require S3 compatible storage.
This storage must be provided by the customer.
Expand All @@ -289,28 +155,12 @@ The following forwarding is required:

|Customer / vSphere infrastructure provider

|DNS
|The APPUiO Managed OpenShift 4 cluster's base DNS records are defined and managed by VSHN.
All records must be publicly resolvable.
To expose applications under a customer domain, a CNAME target is provided.
| VSHN

|===

=== Other terms
=== Components General

[cols="1,4"]
|===
|Name|Description

|Node
|A virtual machine that's part of an OpenShift 4 cluster

|Control plane
a|A collection of components that
include::partial$architecture/glossary-general.adoc[]

* facilitate the management of the container platform
* manage the virtual hardware making up the cluster
* manage the applications running on the cluster
=== Other terms

|===
include::partial$architecture/glossary-others.adoc[]
40 changes: 40 additions & 0 deletions docs/modules/ROOT/partials/architecture/glossary-general.adoc
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
[cols="1,3,1"]
|===
|Name|Description|provided by

|Installer
|A CLI tool that bootstraps an OpenShift 4 cluster based on a configuration file.
|VSHN / Red Hat

|Bootstrap Node
|A temporary VM in the cluster machine network which is provisioned by the installer to facilitate the initial setup of the cluster.
This VM is decommissioned by the installer once the cluster installation is completed.
| VSHN / Installer

|Pod network
a|A subnet that's internal to the Openshift 4 cluster.
This subnet shouldn't be routable from outside the cluster.

This subnet is managed by Cilium and is implemented with VXLAN traffic between the cluster VMs

APPUiO Managed OpenShift 4 uses `10.128.0.0/14` as the pod network.
If the pod network IP range conflicts with existing subnets, the pod network IP range can be adjusted.
| VSHN / Cilium

|Service network
a|A subnet that's internal to the OpenShift 4 cluster.
This subnet shouldn't be routable from outside the cluster.

This subnet is managed by Cilium and is implemented with eBPF rules on the cluster VMs.

APPUiO Managed OpenShift 4 uses `172.30.0.0/16` as the service network.
If the service network IP range conflicts with existing subnets, the service network IP range can be adjusted.
| VSHN / Cilium

|DNS
|The APPUiO Managed OpenShift 4 cluster's base DNS records are defined and managed by VSHN.
All records must be publicly resolvable.
To expose applications under a customer domain, a CNAME target is provided.
| VSHN

|===
15 changes: 15 additions & 0 deletions docs/modules/ROOT/partials/architecture/glossary-others.adoc
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
[cols="1,4"]
|===
|Name|Description

|Node
|A virtual machine that's part of an OpenShift 4 cluster

|Control plane
a|A collection of components that

* facilitate the management of the container platform
* manage the virtual hardware making up the cluster
* manage the applications running on the cluster
|===
8 changes: 8 additions & 0 deletions docs/modules/ROOT/partials/architecture/networking-cluster.adoc
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
Each APPUiO Managed OpenShift 4 cluster is deployed into a /24 "cluster machine network" (sometimes also "cluster network" or "machine network")
This network must be provided by the {infra-type} infrastructure operator.
DHCP is mandatory for this network, but a number of IPs must be reserved to be used as Virtual IPs for the cluster.

Traffic inside this network shouldn't be restricted.

VMs in this network must be able to reach various services on the internet.
See below for a detailed list of external systems that must be reachable.
41 changes: 41 additions & 0 deletions docs/modules/ROOT/partials/architecture/networking-external.adoc
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
APPUiO Managed OpenShift 4 requires various external services.

==== VSHN services

APPUiO Managed OpenShift 4 requires access to VSHN's https://syn.tools[Project Syn] infrastructure.
The Project Syn infrastructure components that must be reachable are

* the Project Syn API at `\https://api.syn.vshn.net`
* the Project Syn Vault at `\https://vault-prod.syn.vshn.net`
* VSHN's GitLab instance at `ssh://[email protected]`
* VSHN's acme-dns instance at `\https://acme-dns-api.vshn.net`

Additionally, APPUiO Managed OpenShift 4 requires access to VSHN's identity management:

* VSHN LDAP at `ldaps://ldap.vshn.net:636`
* VSHN SSO at `\https://id.vshn.net`

Finally, APPUiO Managed OpenShift 4 requires access to VSHN's central metrics storage at `\https://metrics-receive.appuio.net`

==== Red Hat services

See the https://docs.openshift.com/container-platform/4.14/installing/install_config/configuring-firewall.html#configuring-firewall_configuring-firewall[upstream documentation] for the full list of services.

The most important services for APPUiO Managed OpenShift 4 are

* the Red Hat container registries at `registry.redhat.io` and `registry.access.redhat.com`.
* the OpenShift Update Service (OSUS) at `\https://api.openshift.com`.

==== 3rd party services

Finally, APPUiO Managed OpenShift 4 requires access to a number of third party services:

* OpsGenie at `\https://api.opsgenie.com`
* Passbolt at `\https://cloud.passbolt.com/vshn`
* Let's Encrypt at `\https://acme-v02.api.letsencrypt.com` and `\https://acme-staging-v02.api.letsencrypt.com`
* Various container registries
** GitHub at `ghcr.io`
** Quay at `quay.io`
** DockerHub at `docker.io`
** Google container registry at `gcr.io`
** Kubernetes container registry at `registry.k8s.io`
19 changes: 19 additions & 0 deletions docs/modules/ROOT/partials/architecture/networking-pods.adoc
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
APPUiO Managed Openshift 4 uses https://cilium.io/[Cilium] to provide in-cluster networking.
Cilium allocates two cluster-internal networks:

1. The pod network: every pod on the cluster will get an IP address from this network.
This network enables basic in-cluster connectivity.
APPUiO Managed OpenShift 4 uses `10.128.0.0/14` as the pod network.
Each node in the cluster is assigned a `/23` from this range.
Pods on a node are always assigned an IP from the range allocated to that node.
2. Service network: used for service discovery.
Traffic to IPs in this network is forwarded to the appropriate pods by Cilium.
APPUiO Managed OpenShift 4 uses `172.30.0.0/16` as the service network.
Both of these networks are interanl to the OpenShift 4 cluster.
Therefore, the IP CIDRs for these networks must not be routable from the outside.
Additionally, the same IP CIDRs can be reused for multiple OpenShift 4 clusters.

However, the chosen CIDRs shouldn't overlap with existing networks allocated by the customer.
If there are overlaps, external systems in the overlapping ranges won't be accessible from within the OpenShift 4 cluster.
The pod and service network CIDRs can be customized if and only if there are conflicts.
10 changes: 10 additions & 0 deletions docs/modules/ROOT/partials/architecture/overview.adoc
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
The APPUiO Managed OpenShift 4 architecture on {infra-type} is based on the xref:references/architecture/index.adoc[generic APPUiO Managed OpenShift 4 architecture].
We expect that readers of this document are familiar with the xref:references/architecture/index.adoc[generic APPUiO Managed OpenShift 4 architecture] and the overall Kubernetes and OpenShift 4 architecture.

This architecture document extends upon the generic architecture by defining how the APPUiO Managed OpenShift 4 cluster is embedded into the {infra-type} environment.
The diagram below shows a detailed view of how an APPUiO Managed OpenShift 4 cluster is embedded into an existing {infra-type} environment.

.APPUiO Managed OpenShift4 on {infra-type} architecture
image::{infra-svg}[alt=OCP4 {infra-type} architecture, width=900]

The following sections of this document provide detailed descriptions for the elements shown in the architecture diagram.
Loading

0 comments on commit 9b46d52

Please sign in to comment.