Finished k3s-ha implementation; Add three (3) cluster VIP methods implemented via ansible

README.md

@@ -1,60 +1,95 @@
-# Build a Kubernetes cluster using k3s via Ansible
+# Build a Kubernetes cluster using *k3s* with *ansible*
 
 Author: <https://github.com/itwars>
 
-## K3s Ansible Playbook
+## Introduction to *k3s-ansible*
 
-Build a Kubernetes cluster using Ansible with k3s. The goal is easily install a Kubernetes cluster on machines running:
+The goal of *k3s-ansible* is to easily install a Kubernetes cluster on a variety of
+operating systems running on machines with different architectures.
+In general, users of *k3s-ansible* should only need to edit two files:
+- `inventory/sample/group_vars/all.yml`
+- `inventory/sample/hosts.ini`
 
-- [X] Debian
-- [X] Ubuntu
-- [X] CentOS
+All you need to get started is a list of IP addresses for the hosts that you want to
+participate in the cluster and a username that has password-less *ssh* access to all
+those hosts.  That's it!
+No need to futz with lots of settings and variables (unless you like that sort of thing;
+then, have at it).
 
-on processor architecture:
+And, to setup an HA cluster, you need one more IP address - not of a host,
+but for your cluster virtual IP address.
+You don't need to know how to setup a clustering solution since *k3s-ansible* does it for you.
+But, for HA, you just need at least three hosts.
 
-- [X] x64
-- [X] arm64
-- [X] armhf
+The intention is for *k3s-ansible* to support what *k3s* supports.\
+Here is what has been tested (:heavy_check_mark:) with *k3s-ansible*.
+
+| Operating System | amd64 | arm64 | armhf |
+| :--------------- | :---: | :---: | :---: |
+| Debian           | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
+| Ubuntu           | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
+| CentOS           | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
 
 ## System requirements
 
-Deployment environment must have Ansible 2.4.0+
-Master and nodes must have passwordless SSH access
+- The deployment environment must have *ansible* v2.4.0+.
+- Hosts in the cluster must have password-less *ssh* access.
+- HA requires at least three hosts.
+
+##  Caveats
+
+- *k3s-ansible* will overwrite an existing *k3s* installation on the hosts.
+- *k3s-ansible* will overwrite the `.kube` directory of the `ansible_user` specified on each server.
+- An HA configuration using *keepalived* will overwrite an existing *keepalived* configuration.
 
 ## Usage
 
-First create a new directory based on the `sample` directory within the `inventory` directory:
+1. Create a new cluster definition based on the `inventory/sample` directory.
 
 ```bash
 cp -R inventory/sample inventory/my-cluster
 ```
 
-Second, edit `inventory/my-cluster/hosts.ini` to match the system information gathered above. For example:
+2. Edit `inventory/my-cluster/hosts.ini` to include the hosts that will make up your new cluster.\
+For example:
 
 ```bash
-[master]
+[k3s_server]
 192.16.35.12
 
-[node]
+[k3s_agent]
 192.16.35.[10:11]
 
 [k3s_cluster:children]
-master
-node
+k3s_server
+k3s_agent
 ```
 
-If needed, you can also edit `inventory/my-cluster/group_vars/all.yml` to match your environment.
+3. Edit `inventory/my-cluster/group_vars/all.yml` to best match your environment.\
+See, `inventory/sample/group_vars/README.md` for more details.
 
-Start provisioning of the cluster using the following command:
+4. Provision your new cluster.
 
 ```bash
-ansible-playbook site.yml -i inventory/my-cluster/hosts.ini
+ansible-playbook playbook/site.yml -i inventory/my-cluster/hosts.ini
 ```
 
 ## Kubeconfig
 
-To get access to your **Kubernetes** cluster just
+To get access to your new **Kubernetes** cluster, just use the generated kube configuration file.
 
 ```bash
-scp debian@master_ip:~/.kube/config ~/.kube/config
+kubectl --kubeconfig playbook/cluster.conf ...
 ```
+
+## High Availability
+*k3s-ansible* can now configure a high-availability (HA) cluster.
+If you enable HA (**ha_enabled**), the playbook will setup an embedded database using *etcd*.
+HA requires at least version **v1.19.5+k3s1** and an odd number of servers (minimum of three).
+See the [HA-embedded documentation](https://rancher.com/docs/k3s/latest/en/installation/ha-embedded/) for more details.
+
+HA expects that there is a virtual IP (**ha_cluster_vip**) in front of the *control-plane* servers.
+A few methods have been implemented to provide and manage this VIP.
+See `inventory/turingpi` for my example HA setup on my Turing Pi v1.
+See `inventory/sample/group_vars/README.md` for more details on variables.
+

TODO.md

@@ -0,0 +1,78 @@
+# Update
+
+Author: [https://github.com/jon-stumpf](https://github.com/jon-stumpf)
+
+I came across *k3s-ansible* looking for an easy why to create a highly-available cluster
+on my [Turing Pi 1](https://turingpi.com/v1/).
+I saw *k3s-ansible* could easily configure my hosts but was missing the HA component.
+Further research led me to the [k3s-ha](https://github.com/k3s-io/k3s-ansible/tree/k3s-ha)
+branch but found that it still was incomplete for my needs.
+In developing the needed additions, I discovered issues in the
+[master](https://github.com/k3s-io/k3s-ansible/tree/master) branch and spent a month
+reviewing the yaml files of *k3s-ansible* and the shell scripts from
+[https://get.k3s.io](https://get.k3s.io).
+In the end, I brought *k3s-ansible* to be at near parity with https://get.k3s.io and
+I believe I have addressed some open issues as referenced in
+[my pull requests](https://github.com/k3s-io/k3s-ansible/pulls/jon-stumpf).
+
+Once I completed my changes to the *master* branch, I got back to work on *k3s-ha*.
+Building on the
+[work of St0rmingBr4in](https://github.com/k3s-io/k3s-ansible/commits?author=St0rmingBr4in),
+I implemented the HA embedded database using *etcd* and three cluster VIP methods:
+1. **external**: uses an externally provided cluster VIP
+2. **kube-vip**: uses [kube-vip](https://kube-vip.io/) with arp arbitration
+3. **keepalived**: uses [keepalived](https://www.redhat.com/sysadmin/keepalived-basics) to implement VRRP
+
+I have reached out to
+[itwars](https://github.com/itwars) and
+[St0rmingBr4in](https://github.com/St0rmingBr4in) to get their feedback on this work and
+to collaborate on closing the open issues and pull requests.
+In the meantime, I would like others to provide feedback on my
+[k3s-ha](https://github.com/jon-stumpf/k3s-ansible/tree/k3s-ha) branch.
+This is now stable and incorporates all my previous work on *k3s-ansible* except for a few commits.
+Please, try it out.
+
+# TODO
+
+1. Make all roles *idempotent* and not report changes when none, in fact, are needed or material.
+2. Add *keepalived*' label to servers when using keepalived;  Add the following annotations:
+    - `keepalived/vrrp_instance=<name>`
+    - `keepalived/master=[true|false]`
+    - `keepalived/version=<version>`
+    - `keepalived/vip=<ipaddr>/<cidr>`
+3. Add the ability to download the latest version of *kube-vip*
+    - Currently, uses a static version (v0.4.0) that can be manually changed
+4. Make sure all roles have defaults defined
+5. Make HA not require `k3s_token` to be defined
+    - i.e., use the `node-token` from the first server
+6. Replace `command` and `shell` tasks with *ansible* equivalents (where appropriate)
+    - `ip`
+    - `kubectl`
+    - etc.
+7. Is the `raspberrypi` role a NO-OP?
+    - It does not appear to execute any tasks that induce change
+    - Should it be deleted?
+8. From where does *k3s-selinux* get installed?
+    - The `reset/download` role deletes it.
+9. Document lesser switches to control behavior of roles (e.g., remove_packages)
+10. Create playbooks for other common operations beyond install/uninstall
+
+# Progress Report
+
+| Role                  | Role Type  | Idempotent         | Only Real Changes  | Defaults           | commands | TODOs | BUGs  |
+| :-------------------- | :--------: | :---:              | :---:              | :---:              | :---:    | :---: | :---: |
+| cluster-config        | install    | :heavy_check_mark: | :heavy_check_mark: |                    | -        | -     | -     |
+| config-check          | install    | :heavy_check_mark: | :heavy_check_mark: | *under review*     | -        | -     | -     |
+| prereq                | install    | :heavy_check_mark: | :heavy_check_mark: | **n/a**            | -        | -     | -     |
+| download              | install    | :heavy_check_mark: | :heavy_check_mark: | *under review*     | -        | -     | -     |
+| raspberrypi           | install    | :heavy_check_mark: | :heavy_check_mark: | **n/a**            | 2        | 1     | -     |
+| ha/etcd               | HA-only    |                    | unknown            | *under review*     | 3        | 1     | -     |
+| ha/keepalived         | HA-only    | :heavy_check_mark: | :heavy_check_mark: | *under review*     | 1        | 1     | -     |
+| ha/kube-vip           | HA-only    | :heavy_check_mark: | :heavy_check_mark: | *under review*     | 2        | 3     | -     |
+| k3s/server            | install    | :heavy_check_mark: | :heavy_check_mark: | *under review*     | -        | -     | -     |
+| k3s/agent             | install    | :heavy_check_mark: | :heavy_check_mark: |                    | -        | -     | -     |
+| reset/download        | uninstall  | :heavy_check_mark: | :heavy_check_mark: | *under review*     | 2        | 1     | -     |
+| reset/ha/keepalived   | uninstall  | :heavy_check_mark: | :heavy_check_mark: | *by reference*     | -        | -     | -     |
+| reset/ha/kube-vip     | uninstall  | :heavy_check_mark: | :heavy_check_mark: | *by reference*     | 3        | 1     | -     |
+| reset/k3s             | uninstall  | :heavy_check_mark: | :heavy_check_mark: | *under review*     | 9        | 1     | 2     |
+

inventory/.gitignore

@@ -1,3 +1,6 @@
 *
 !.gitignore
-!sample/
+!sample/
+!sample/**
+!turingpi/
+!turingpi/**

inventory/sample/group_vars/README.md

@@ -0,0 +1,108 @@
+
+# Introduction
+
+`inventory/x/group_vars/all.yml` is meant to be modified appropriately for your environment.
+
+*ansible* variables that were previously here have moved to `playbook/group_vars/all.yml`.
+Those variables are used within the playbooks and roles are not meant to be changed by a user of *k3s-ansible*.
+When adding a new _install_ variable, a corresponding variable is added to `playbook/group_vars/all.yml`
+which is then used throughout *k3s-ansible*.
+
+## General Variables
+
+- **ansible_user**: specifies the username that has *ssh* password-less access to configure your hosts.
+The default is `debian`.
+
+- **cluster_config**: specifies the location of where to capture the kube configuration file for the new cluster.
+The default is `playbook/cluster.conf`.
+
+## High-Availability (HA) Variables
+
+- **ha_enabled**: specifies if the cluster will have an HA embedded database using *etcd*.
+The default is `false`.
+
+- **ha_cluster_vip**: specifies the virtual IP (VIP) address in front of the control-plane servers for
+agent configuration as well as cluster definition in `.kube/config`.
+Note: This is an IP address different than those of the cluster nodes.
+Today, this is a static IP address provided in this file.
+It is possible to get an IP address dynamically but that is not implemented here.
+
+- **ha_cluster_method**: specifies the method of clustering to use for the virtual IP.
+The methods implemented today are:
+    1. `external` - requires a load-balancer external to the cluster
+    2. `kube-vip` - [https://kube-vip.io](https://kube-vip.io), arp-based daemonset using leader election
+    3. `keepalived` - all *k3s* servers are configured with [keepalived](https://www.redhat.com/sysadmin/keepalived-basics) to manage a VRRP instance
+
+- **ha_k3s_token**: specifies k3s token used by hosts to join the cluster
+
+## Install Variables
+
+If you have installed *k3s* from [https://get.k3s.io](https://get.k3s.io), these variables will be familiar.
+*Install* variables are meant to duplicate the install flags and environment variables found in the install script
+(see [Installation Options](https://rancher.com/docs/k3s/latest/en/installation/install-options/#options-for-installation-with-script)).
+Each variable has a prefix of `install_` and implements, to the extent possible, the actions of the shell script as documented below.
+
+### Variables that control the version of *k3s* downloaded
+
+There are four (4) variables that control which version of *k3s* is installed on your hosts.
+
+- **install_k3s_commit**: specifies the commit of *k3s* to download from temporary cloud storage.
+The default is to leave this `undefined` as this variable is for developers and QA use. 
+
+- **install_k3s_version**: specifies the version of *k3s* to download from Github.
+If left `undefined` (the default), *ansible* will attempt to download from a channel.
+
+- **install_k3s_channel_url**: specifies the URL for the channels.
+The default is [https://update.k3s.io/v1-release/channels](https://update.k3s.io/v1-release/channels).
+It is not something typically changed but is implemented for completeness sake.
+
+- **install_k3s_channel**: specifies the channel from which to get the version.
+The default is the `stable` channel.  A typical channel used is `latest`.
+
+### Variables that change the location of binaries and data
+
+There are three (3) variables that change the default location of files.
+
+- **install_k3s_bin_dir**: specifies the directory to install the *k3s* binary and links.
+The default is `/usr/local/bin`.
+
+- **install_k3s_systemd_dir**: specifies the directory to install *systemd*
+service and environment files.  The default is `/etc/systemd/system`.
+
+- **install_k3s_data_dir**: specifies the data directory for the *k3s* service.
+This defaults to `/var/lib/rancher/k3s`.
+Note: this is not (yet) an option in *k3s-io/k3s*.
+
+### Variables for the *k3s* executable
+
+The install script from [https://get.k3s.io/](https://get.k3s.io/) has one flag (**INSTALL_K3S_EXEC**) to
+provide extra arguments to the *k3s* executable.  *k3s-ansible* uses two variables:
+one for servers and one for agents.  These are:
+
+- **install_k3s_server_args**: the default is `''`.
+- **install_k3s_agent_args**: the default is `''`.
+
+### Install Flags not yet implemented
+
+The install flags that have yet to be implemented are:
+
+| Install Flag | What it does |
+| :--- | :--- |
+| **INSTALL_K3S_SKIP_SELINUX_RPM** | If set to true, *ansible* will skip automatic installation of the *k3s* RPM.
+| **INSTALL_K3S_SELINUX_WARN** | If set to true, *ansible* will continue if the *k3s-selinux* policy is not found.
+| **INSTALL_K3S_NAME** | specifies the name of *systemd* service to create.
+| **INSTALL_K3S_TYPE** | specifies the type of *systemd* service to create.
+
+Currently, nothing will happen if these are set.
+
+### Install Flags that will not be implemented
+
+Lastly, some install flags did not make sense to implement with *k3s-ansible*:
+
+| Install Flag | What *k3s-ansible* does |
+| :--- | :--- |
+| **INSTALL_K3S_SKIP_DOWNLOAD** | *k3s-ansible* always downloads the *k3s* binary and its hash. |
+| **INSTALL_K3S_FORCE_RESTART** | *k3s-ansible* always restarts the service. |
+| **INSTALL_K3S_SKIP_ENABLE**   | *k3s-ansible* always enables the service. |
+| **INSTALL_K3S_SKIP_START**    | *k3s-ansible* always starts the service. |
+

inventory/sample/group_vars/all.yml

@@ -1,7 +1,14 @@
 ---
-k3s_version: v1.22.3+k3s1
+# See inventory/sample/group_vars/README.md for more options.
+# If this file is empty, default values will be used for all mandatory fields.
+
+# The user that has password-less ssh access to configure your hosts
 ansible_user: debian
-systemd_dir: /etc/systemd/system
-master_ip: "{{ hostvars[groups['master'][0]]['ansible_host'] | default(groups['master'][0]) }}"
-extra_server_args: ""
-extra_agent_args: ""
+
+# The location of where to capture the kube config of the new cluster
+# Relative paths are relative to the playbook directory.
+cluster_config: cluster.conf
+
+# Use the latest k3s version instead of 'stable'
+install_k3s_channel: 'latest'
+

inventory/sample/hosts.ini

@@ -1,12 +1,12 @@
-[master]
+[k3s_server]
 192.168.1.26
 
-[node]
+[k3s_agent]
 192.168.1.34
 192.168.1.39
 192.168.1.16
 192.168.1.32
 
 [k3s_cluster:children]
-master
-node
+k3s_server
+k3s_agent

inventory/turingpi/group_vars/README.md

@@ -0,0 +1 @@
+../../sample/group_vars/README.md

inventory/turingpi/group_vars/all.yml

@@ -0,0 +1,19 @@
+---
+
+# See inventory/sample/group_vars/README.md for more options.
+# If this file is empty, default values will be used for all mandatory fields.
+
+# The user that has password-less ssh access to configure your hosts
+ansible_user: pirate
+
+# The location of where to capture the kube config of the new cluster
+# Relative paths are relative to the playbook directory.
+cluster_config: cluster.conf
+
+ha_enabled: true
+ha_cluster_vip: 192.168.140.127
+ha_cluster_method: kube-vip
+
+# Use the latest k3s version instead of 'stable'
+install_k3s_channel: 'latest'
+

inventory/turingpi/hosts.ini

@@ -0,0 +1,14 @@
+[k3s_server]
+192.168.140.120
+192.168.140.123
+192.168.140.126
+
+[k3s_agent]
+192.168.140.121
+192.168.140.122
+192.168.140.124
+192.168.140.125
+
+[k3s_cluster:children]
+k3s_server
+k3s_agent

playbook/.gitignore

@@ -0,0 +1 @@
+cluster.conf