Cluster management

GIS.lab cluster is managed by decentralized cluster management tool called Serf, which is based on gossip protocol. Serf is responsible for automatic joining and removing machines to and from GIS.lab cluster and OWS load balancer management. It is also used as interface for running cluster events and queries.

Roles, events and queries

Machines belonging in to GIS.lab cluster are divided in to three roles:

• server: GIS.lab server
• client: GIS.lab Desktop clients
• worker: GIS.lab Worker clients

All machines are capable of running different set of cluster events and queries depending on their role membership. Events and queries can be send from any machine which is a member of GIS.lab cluster using gislab-cluster client (which is currently just symlink to serf binary), or programmatically using RPC mechanism. All machines in cluster will receive all events and queries and will decide to respond or not depending on existence of handler responsible for particular event or query.

The main difference between event and query is that while query is designed to send some query and receive response, the purpose of event is just to announce that something has happen or should happen without receiving any response. Response from query can be returned in two formats: text or JSON.

Public events and queries

Here is a list of publicly available events and queries designed for ordinary usage. This list doesn't contain system events and queries which are used for internal GIS.lab cluster management.

Get list of cluster members, theirs roles membership and list of supported events and queries

$ gislab-cluster members

Get list of cluster members in JSON format

$ gislab-cluster members -format json

{
  "members": [
    {
      "name": "server.gis.lab",
      "addr": "192.168.15.5:7946",
      "port": 7946,
      "tags": {
        "role": "server"
      },
      "status": "alive",
      "protocol": {
        "max": 4,
        "min": 2,
        "version": 4
      }
    },
    {
      "name": "w10",
      "addr": "192.168.15.10:7946",
      "port": 7946,
      "tags": {
        "events": "reboot,shutdown",
        "queries": "uptime,test",
        "role": "worker"
      },
      "status": "alive",
      "protocol": {
        "max": 4,
        "min": 2,
        "version": 4
      }
    },
    {
      "name": "c50",
      "addr": "192.168.15.50:7946",
      "port": 7946,
      "tags": {
        "events": "reboot,shutdown",
        "queries": "session-active,uptime,test",
        "role": "client"
      },
      "status": "alive",
      "protocol": {
        "max": 4,
        "min": 2,
        "version": 4
      }
    }
  ]
}

Reboot/shutdown all client and worker machines

$ gislab-cluster event reboot
$ gislab-cluster event shutdown

Reboot/shutdown particular client or worker machine

$ gislab-cluster event reboot "<hostname>,<hostname>,..."
$ gislab-cluster event shutdown "<hostname>,<hostname>,..."

Get information about actively running desktop client sessions in cluster

$ gislab-cluster query session-active

Running commands on whole cluster with parallel-ssh

Install paralell-ssh utility

$ sudo apt-get install pssh

Get list of currently running client machines

$ MACHINES="$(gislab-cluster members -tag role=client -status=alive | awk -F " " '{printf "%s ", $1}')"

Install gedit on all client machines

$ parallel-ssh -i -H "$MACHINES" sudo DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends gedit

[1] 23:02:57 [SUCCESS] c51
Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  gedit
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/827 kB of archives.
After this operation, 2,781 kB of additional disk space will be used.
Selecting previously unselected package gedit.
(Reading database ... 134642 files and directories currently installed.)
Unpacking gedit (from .../gedit_3.4.1-0ubuntu1_amd64.deb) ...
Processing triggers for desktop-file-utils ...
Setting up gedit (3.4.1-0ubuntu1) ...
update-alternatives: using /usr/bin/gedit to provide /usr/bin/gnome-text-editor (gnome-text-editor) in auto mode.
Processing triggers for libc-bin ...
ldconfig deferred processing now taking place

[2] 23:02:57 [SUCCESS] c50
Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  gedit
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/827 kB of archives.
After this operation, 2,781 kB of additional disk space will be used.
Selecting previously unselected package gedit.
(Reading database ... 134642 files and directories currently installed.)
Unpacking gedit (from .../gedit_3.4.1-0ubuntu1_amd64.deb) ...
Processing triggers for desktop-file-utils ...
Setting up gedit (3.4.1-0ubuntu1) ...
update-alternatives: using /usr/bin/gedit to provide /usr/bin/gnome-text-editor (gnome-text-editor) in auto mode.
Processing triggers for libc-bin ...
ldconfig deferred processing now taking place

Perform performance test of parallel write to network share

$ parallel-ssh -i -H "$MACHINES" 'dd if=/dev/zero of=/mnt/barrel/file-$(hostname).io bs=1M count=1000'

Perform performance test of parallel read from network share

$ parallel-ssh -i -H "$MACHINES" 'dd if=/mnt/barrel/file-$(hostname).io of=/dev/zero bs=1M'

Perform CPU performance test

$ parallel-ssh -i -H "$MACHINES" 'dd if=/dev/zero bs=1M count=1024 | md5sum'