Single Machine Development Environment
Developing cluster software is complicated if you must actually run a whole cluster on a set of physical machines. This begs for a development environment that is self contained and can be run without any setup. This page documents a way to set up an entire Ciao cluster inside a single machine. This cluster-in-a-machine mode is ideal for developers that desire the ability to build Ciao from sources, make changes and perform quick end to end functional integration testing without requiring multiple machines/VM's or creating a custom networking environment or maintaining a bevy of physical machines and a physical network.
In this case Ciao is configured in a special all in one development mode where cluster nodes have dual roles (i.e launcher can be a Network Node and a Compute Node at the same time)
1. Controller
2. Scheduler
3. Compute+Network Node Agent (i.e. CN + NN Launcher)
4. Workloads (Containers and VMs)
5. WebUI
6. Mock Openstack Services
7. Machine Local DHCP Server
...
The one machine acts as the ciao compute node, network node, ciao-controller, ciao-scheduler and also hosts the ciao-webui as well as other openstack and dhcp services.
When the system is functioning the overall setup manifests as follows:
As you can see below the Cluster runs on a isolated virtual network resident on the host. Hence the cluster is invisible outside the host and completely self contained.
_____________________________________________________________________________________
| |
| |
| |
| [Tenant VMs] [CNCI VMs] |
| | | | || |
| Tenant Bridges ---------- || |
| | || |
| | || |
| [scheduler] [controller] [keystone] [CN+NN Launcher] | || |
| || || || || | || |
| || || || || | || |
| || || || || | || |
| || || || || | || |
| || || || || | || |
| || || || || [DHCP/DNS | || |
| || || || || Server] | || |
| || || || || || | || |
| -------------------------------------------------------------------------- |
| Host Local Network Bridge + macvlan (ciao_eth, ciaovlan) |
| |
| |
| ____________________________________________________________________________________|
Development Machine
The following instructions can be performed on a bare metal system or in a VM.
If you use a VM please ensure that the VM has host CPU pass through enabled.
<cpu mode='host-passthrough'/>
This ensures that VT-x and other properties required by Ciao are available to Ciao
if you desire to provide external network connectivity to the workloads then the host needs to act as gateway to the internet. The host needs to enable ipv4 forwarding and ensure all traffic exiting the cluster via the host is NATed.
This assumes the host has a single network interface. For multi homed systems, the setup is more complicated and needs appropriate routing setup which is outside the scope of this document. If you have a custom firewall configuration, you will need set things up appropriately.
Very simplistically this can be done by
iptables -t nat -A POSTROUTING -o $device -j MASQUERADE #$device is the network interface on the host
echo 1 > /proc/sys/net/ipv4/ip_forward
Install the latest release of go for your distribution Installing Go.
NOTE: Go version 1.7 or later is required for Ciao. Ciao will not work with older version of Go. Hence it is best you download and install the latest version of Go if you distro is not on Go 1.7.
Install latest docker for your distribution based on the instructions from Docker Installing Docker.
Install the following packages which are required:
1. qemu-system-x86_64 and qemu-img, to launch the VMs and create qcow images
2. gcc, required to build some of the ciao dependencies
3. dnsmasq, required to setup a test DHCP server
On clearlinux all of these dependencies can be satisfied by installing the following bundles:
swupd bundle-add cloud-control go-basic os-core-dev kvm-host os-installer
Setup passwordless sudo for the user who will be running the script below.
Download and build the ciao sources:
cd $GOPATH/src
go get -v -u -tags debug github.com/01org/ciao/...
You should see no errors.
You can now quickly verify that all aspects of Ciao including VM launch, container launch, and networking. TO do this simply run the following:
cd $GOPATH/src/github.com/01org/ciao/testutil/singlevm
#Cleanup any previous setup
./cleanup.sh
#Set up the test enviornment
./setup.sh
#Perform a full cluster test
./verify.sh
The verify.sh script will:
- Create multiple Instances of Tenant VMs and Containers
- Test network connectivity between containers
- Test for ssh reach ability into VMs
- Delete all the VM's and Container that were created
If the script reports success, it indicates to the developer that any changes made have not broken any functionality across all the Ciao components.
Meeting the goal originally outlined at the top of the page, build/setup/running your cluster all-in-one all transpires quickly and easily from the single script. The time needed for ./setup.sh and ./verify.sh to build ciao from source, configure it components into a virtual cluster, then launch and teardown containers and VMs is on the order of one minute total elapsed time.
Once it's finished, the setup.sh script leaves behind a virtual cluster which can be used to perform manual tests. These tests are performed using the ciao-cli tool. The ciao-cli tool requires that some environment variables be set up before it will work properly. These variables contain the URLs of the various ciao services and the credentials needed to access these services. The setup.sh script creates a shell source that contains valid values for the newly set up cluster. To initialise these variables you just need to source that file, e.g,
. ~/local/demo.sh
To check everything is working try the following command
ciao-cli workload list
The ciao project includes a set of acceptance tests that must pass before each release is made. The tests perform various tasks such as listing workloads, creating and deleting instances, etc. These tests can be run inside the Single Machine VM.
# Source the demo.sh file if you have not already done so
. ~/local/demo.sh
cd $GOPATH/src/github.com/01org/ciao/_release/bat
go test -v ./...
For more information on the BAT tests please see the README.
To cleanup and tear down the cluster:
cd $GOPATH/src/github.com/01org/ciao/testutil/singlevm
#Cleanup any previous setup
./cleanup.sh
- Does not work on Fedora due to default firewall rules. https://github.com/01org/ciao/issues/526
In order to allow the traffic required by the test cases you can add temporary rules like the ones show below
#!/bin/bash
iptables -I INPUT 1 -p tcp -m tcp --dport 8888 -j ACCEPT
iptables -I INPUT 1 -p 47 -j ACCEPT
iptables -I OUTPUT 1 -p 47 -j ACCEPT
iptables -I INPUT 1 -p tcp --dport 22 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
iptables -I OUTPUT 1 -p tcp --sport 22 -m conntrack --ctstate ESTABLISHED -j ACCEPT
iptables -I FORWARD 1 -p tcp --dport 22 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
iptables -I FORWARD 1 -p tcp --sport 22 -m conntrack --ctstate ESTABLISHED -j ACCEPT
iptables -I FORWARD 1 -p udp -m udp --dport 67:68 -j ACCEPT
iptables -I FORWARD 1 -p udp -m udp --dport 123 -j ACCEPT
iptables -I FORWARD 1 -p udp -m udp --dport 53 -j ACCEPT
iptables -I FORWARD 1 -p udp -m udp --dport 5355 -j ACCEPT
iptables -I FORWARD 1 -p icmp -j ACCEPT
And delete them after the tests using
#!/bin/bash
iptables -D INPUT -p tcp -m tcp --dport 8888 -j ACCEPT
iptables -D INPUT -p 47 -j ACCEPT
iptables -D OUTPUT -p 47 -j ACCEPT
iptables -D INPUT -p tcp --dport 22 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
iptables -D OUTPUT -p tcp --sport 22 -m conntrack --ctstate ESTABLISHED -j ACCEPT
iptables -D FORWARD -p tcp --dport 22 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
iptables -D FORWARD -p tcp --sport 22 -m conntrack --ctstate ESTABLISHED -j ACCEPT
iptables -D FORWARD -p udp -m udp --dport 67:68 -j ACCEPT
iptables -D FORWARD -p udp -m udp --dport 123 -j ACCEPT
iptables -D FORWARD -p udp -m udp --dport 53 -j ACCEPT
iptables -D FORWARD -p udp -m udp --dport 5355 -j ACCEPT
iptables -D FORWARD -p icmp -j ACCEPT