Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] docs: updates to README #93

Closed
wants to merge 1 commit into from
Closed

[WIP] docs: updates to README #93

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
166 changes: 133 additions & 33 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,55 +1,155 @@
# Cluster API k3s
# Cluster API Provider k3s

Cluster API bootstrap provider k3s (CABP3) is a component of [Cluster API](https://github.com/kubernetes-sigs/cluster-api/blob/master/README.md) that is responsible for generating a cloud-init script to turn a Machine into a Kubernetes Node; this implementation brings up [k3s](https://k3s.io/) clusters instead of full kubernetes clusters.
Cluster API Provider k3s provides the following [Cluster API](<https://github.com/kubernetes-sigs/cluster-api> (CAPI) providers:

CABP3 is the bootstrap component of Cluster API for k3s and brings in the following CRDS and controllers:
- k3s bootstrap provider (KThrees, KThreesTemplate)
- **Cluster API Bootstrap Provider k3s (CABP3)** is responsible for generating the instructions (and encoding them as cloud-init) to turn a Machine into a Kubernetes Node; this implementation brings up [k3s](https://k3s.io/) clusters instead of full kubernetes clusters.

Cluster API ControlPlane provider k3s (CACP3) is a component of [Cluster API](https://github.com/kubernetes-sigs/cluster-api/blob/master/README.md) that is responsible for managing the lifecycle of control plane machines for k3s; this implementation brings up [k3s](https://k3s.io/) clusters instead of full kubernetes clusters.
- **Cluster API ControlPlane Provider k3s (CACP3)** is responsible for managing the lifecycle of control plane machines for k3s; this implementation brings up [k3s](https://k3s.io/) clusters instead of full kubernetes clusters.

CACP3 is the controlplane component of Cluster API for k3s and brings in the following CRDS and controllers:
- k3s controlplane provider (KThreesControlPlane)
## Getting Started

Together these two components make up Cluster API k3s...
**Warning**: Project and documentation are in an early stage, there is an assumption that a user of this provider is already familiar with Cluster API. Please consider contributing.

## Testing it out.
### Prerequisites

**Warning**: Project and documentation are in an early stage, there is an assumption that an user of this provider is already familiar with ClusterAPI.
Check out the general [Cluster API Quickstart](https://cluster-api.sigs.k8s.io/user/quick-start.html) page to see the prerequisites for Cluster API.

### Prerequisites
Three main pieces are:

Check out the [ClusterAPI Quickstart](https://cluster-api.sigs.k8s.io/user/quick-start.html) page to see the prerequisites for ClusterAPI.
1. Management cluster. In the `samples/azure/azure-setup.sh` script, [k3d](https://k3d.io/) is used, but feel free to use [kind](https://kind.sigs.k8s.io/) as well .
2. clusterctl. Please check out [Cluster API Quickstart](https://cluster-api.sigs.k8s.io/user/quick-start.html) for instructions.
3. Infrastructure specific prerequisites:
- For more Azure information go to [CAPZ Getting Started](https://capz.sigs.k8s.io/topics/getting-started.html)
- For more AWS information go to [CAPA Getting Started](https://cluster-api-aws.sigs.k8s.io/)
- For more Nutanix information go to [CAPX Getting Started](https://opendocs.nutanix.com/capx/latest/getting_started/)
- For more OpenStack information go to [CAPO Getting Started](https://cluster-api.sigs.k8s.io/user/quick-start.html)
- For more Vsphere information go to [CAPV Getting Started](https://cluster-api.sigs.k8s.io/user/quick-start.html)

Three main pieces are
In this getting started guide we'll be using Docker as the infrastructure provider (CAPD).

1. Bootstrap cluster. In the `samples/azure/azure-setup.sh` script, I use [k3d](https://k3d.io/), but feel free to use [kind](https://kind.sigs.k8s.io/) as well.
2. clusterctl. Please check out [ClusterAPI Quickstart](https://cluster-api.sigs.k8s.io/user/quick-start.html) for instructions.
3. Infrastructure Specific Prerequisites:
### Create a management cluster

* For more Azure information go to [CAPZ Getting Started](https://capz.sigs.k8s.io/topics/getting-started.html)
* For more AWS information go to [CAPA Getting Started](https://cluster-api-aws.sigs.k8s.io/)
* For more Nutanix information go to [CAPX Getting Started](https://opendocs.nutanix.com/capx/latest/getting_started/)
* For more OpenStack information go to [CAPO Getting Started](https://cluster-api.sigs.k8s.io/user/quick-start.html)
* For more Vsphere information go to [CAPV Getting Started](https://cluster-api.sigs.k8s.io/user/quick-start.html)
1. Ensure kind is installed ([instructions](https://kind.sigs.k8s.io/docs/user/quick-start/#installation))
2. Create a kind configuration to expose the local docker socket:

Cluster API k3s has been tested on AWS, Azure, AzureStackHCI, Nutanix, OpenStack, and Vsphere environments.
```bash
cat > kind-cluster-with-extramounts.yaml <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: capi-quickstart
nodes:
- role: control-plane
extraMounts:
- hostPath: /var/run/docker.sock
containerPath: /var/run/docker.sock
EOF
```

To try out the Azure flow, fork the repo and look at `samples/azure/azure-setup.sh`.
3. Run the following command to create the cluster:

To try out the AWS flow, fork the repo and look at `samples/aws/aws-setup.sh`.
```bash
kind create cluster --config kind-cluster-with-extramounts.yaml
```

To try out the Nutanix flow, fork the repo and look at `samples/nutanix/nutanix-setup.sh`.
4. Check that you can connect to the cluster

To try out the OpenStack flow, fork the repo and look at `samples/openstack/setup.sh`.
```bash
kubectl get pods -A
```

To try out the Vsphere flow, fork the repo and look at `samples/vsphere-capv/setup.sh`.
### Install the providers

### Known Issues
1. Create a file called `clusterctl.yaml` in the `$HOME/.config/cluster-api` directory with the following content:

## Roadmap
```yaml
# NOTE: the following will be changed to use 'latest' in the future
providers:
- name: "k3s"
url: "https://github.com/k3s-io/cluster-api-k3s/releases/v0.1.9/bootstrap-components.yaml"
type: "BootstrapProvider"
- name: "k3s"
url: "https://github.com/k3s-io/cluster-api-k3s/releases/v0.1.9/control-plane-components.yaml"
type: "ControlPlaneProvider"
```

> This configuration tells clusterctl where to look for the provider manifests.

2. Install the providers:

```bash
clusterctl init --bootstrap k3s --control-plane k3s --infrastructure docker
```

3. Wait for the pods to start

### Create a workload cluster

There are a number of different cluster templates in the [samples](./samples/) directory.

1. Run the following command to generate your cluster definition:

```bash
clusterctl generate cluster --from samples/docker/quickstart.yaml test1 --kubernetes-version v1.28.6 --worker-machine-count 2 --control-plane-machine-count 1 > cluster.yaml
```

> NOTE: the kubernetes version specified with without the k3s suffix, this gets added in the template.

2. Check the contents of the generated cluster definition in **cluster.yaml**
3. Ensure the definition is valid by doing a dry run:

```bash
kubectl apply -f cluster.yaml --dry-run=server
```

4. When you are happy apply the definition:

```bash
kubectl apply -f cluster.yaml
```

* Support for External Databases
* Fix Token Logic
* Clean up Control Plane Provider Code
* Post an issue!
## Check the workload cluster

- Check the state of the CAPI machines:

```bash
kubectl get machine
```

- Get the kubeconfig for the cluster:

```bash
clusterctl get kubeconfig test1 > workload-kubeconfig.yaml
```

- Connect to the child cluster

```bash
kubectl --kubeconfig workload-kubeconfig.yaml get pods -A
```

### Deleting the workload cluster

When deleting a cluster created via CAPI you must delete the top level **Cluster** resource. DO NOT delete using the original file.

For the quick start:

```bash
kubectl delete cluster test1
```

### Additional Samples

Cluster API k3s has been tested on AWS, Azure, AzureStackHCI, Nutanix, OpenStack, Docker and Vsphere environments.

- To try out the Azure flow, fork the repo and look at `samples/azure/azure-setup.sh`.
- To try out the AWS flow, fork the repo and look at `samples/aws/aws-setup.sh`.
- To try out the Nutanix flow, fork the repo and look at `samples/nutanix/nutanix-setup.sh`.
- To try out the OpenStack flow, fork the repo and look at `samples/openstack/setup.sh`.
- To try out the Vsphere flow, fork the repo and look at `samples/vsphere-capv/setup.sh`.

## Roadmap

- Support for External Databases
- Fix Token Logic
- Clean up Control Plane Provider Code
- Post an issue!
91 changes: 91 additions & 0 deletions samples/docker/quickstart.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
name: ${CLUSTER_NAME}
spec:
clusterNetwork:
pods:
cidrBlocks:
- 10.45.0.0/16
services:
cidrBlocks:
- 10.46.0.0/16
serviceDomain: cluster.local
controlPlaneRef:
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KThreesControlPlane
name: ${CLUSTER_NAME}-control-plane
infrastructureRef:
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: DockerCluster
name: ${CLUSTER_NAME}
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: DockerCluster
metadata:
name: ${CLUSTER_NAME}
spec: {}
---
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KThreesControlPlane
metadata:
name: ${CLUSTER_NAME}-control-plane
namespace: default
spec:
infrastructureTemplate:
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: DockerMachineTemplate
name: ${CLUSTER_NAME}-control-plane
replicas: ${CONTROL_PLANE_MACHINE_COUNT}
version: ${KUBERNETES_VERSION}+k3s2
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: DockerMachineTemplate
metadata:
name: ${CLUSTER_NAME}-control-plane
spec:
template:
spec:
customImage: kindest/node:${KUBERNETES_VERSION}
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
name: worker-md-0
spec:
clusterName: ${CLUSTER_NAME}
replicas: ${WORKER_MACHINE_COUNT}
selector:
matchLabels:
cluster.x-k8s.io/cluster-name: ${CLUSTER_NAME}
template:
spec:
version: ${KUBERNETES_VERSION}
clusterName: ${CLUSTER_NAME}
bootstrap:
configRef:
apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KThreesConfigTemplate
name: ${CLUSTER_NAME}-md-0
infrastructureRef:
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: DockerMachineTemplate
name: ${CLUSTER_NAME}-md-0
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: DockerMachineTemplate
metadata:
name: ${CLUSTER_NAME}-md-0
spec:
template:
spec:
customImage: kindest/node:${KUBERNETES_VERSION}
---
apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KThreesConfigTemplate
metadata:
name: ${CLUSTER_NAME}-md-0
spec:
template:
spec:
version: ${KUBERNETES_VERSION}+k3s2
Loading