Cluster management

GIS.lab cluster is managed by decentralized cluster management tool called Serf, which is based on gossip protocol. Serf is responsible for automatic joining and removing machines to and from GIS.lab cluster and OWS load balancer management. It is also used as interface for running cluster events and queries.

Roles, events and queries

Machines belonging in to GIS.lab cluster are divided in to three roles:

• server: GIS.lab server
• client: GIS.lab Desktop clients
• worker: GIS.lab Worker clients

All machines are capable of running different set of cluster events and queries depending on their role membership. Events and queries can be send from any machine which is a member of GIS.lab cluster using gislab-cluster client (which is currently just symlink to serf binary), or programmatically using RPC mechanism. All machines in cluster will receive all events and queries and will decide to respond or not depending on existence of handler responsible for particular event or query.

The main difference between event and query is that while query is designed to send some query and receive response, the purpose of event is just to announce that something has happen or should happen without receiving any response. Response from query can be returned in two formats: text or JSON.

Public events and queries

Here is a list of publicly available events and queries designed for ordinary usage. This list doesn't contain system events and queries which are used for internal GIS.lab cluster management.

Get list of cluster members, theirs roles membership and list of supported events and queries

$ gislab-cluster members

Get list of cluster members in JSON format

$ gislab-cluster members -format json

{
  "members": [
    {
      "name": "server.gis.lab",
      "addr": "192.168.15.5:7946",
      "port": 7946,
      "tags": {
        "role": "server"
      },
      "status": "alive",
      "protocol": {
        "max": 4,
        "min": 2,
        "version": 4
      }
    },
    {
      "name": "w10",
      "addr": "192.168.15.10:7946",
      "port": 7946,
      "tags": {
        "events": "reboot,shutdown",
        "queries": "script,client-session-active,uptime,test",
        "role": "worker"
      },
      "status": "alive",
      "protocol": {
        "max": 4,
        "min": 2,
        "version": 4
      }
    },
    {
      "name": "c50",
      "addr": "192.168.15.50:7946",
      "port": 7946,
      "tags": {
        "events": "reboot,shutdown",
        "queries": "script,client-session-active,uptime,test",
        "role": "client"
      },
      "status": "alive",
      "protocol": {
        "max": 4,
        "min": 2,
        "version": 4
      }
    }
  ]
}

Reboot/shutdown all client and worker machines

$ gislab-cluster event reboot
$ gislab-cluster event shutdown

Reboot/shutdown particular client or worker machine

$ gislab-cluster event reboot "<hostname>,<hostname>,..."
$ gislab-cluster event shutdown "<hostname>,<hostname>,..."

Get information about actively running desktop client sessions in cluster

$ gislab-cluster query client-session-active

Run script saved in '~/Publish' directory on all desktop client and worker machines (script is executed with user permissions)

$ gislab-cluster query script "<USERNAME>/<SCRIPT-PATH>"

Example of running performance test script saved in ~/Publish/gislab/scripts/performance-test.sh

#!/bin/bash

set -e

OUTFILE=/mnt/barrel/performance-test-$(hostname --fqdn).log

echo "* HOSTNAME: $(hostname)" > $OUTFILE
echo "* UPTIME: $(uptime)" >> $OUTFILE
echo "* WRITE TO SHARE: $(dd if=/dev/zero of=/mnt/barrel/file-$(hostname).io bs=1M count=1000 2>&1)" >> $OUTFILE
echo "* READ FROM SHARE: $(dd if=/mnt/barrel/file-$(hostname).io of=/dev/zero bs=1M 2>&1)" >> $OUTFILE
echo "* CPU PERFORMANCE: $(dd if=/dev/zero bs=1M count=1024 2>&1 | md5sum > /dev/null)" >> $OUTFILE

echo "DONE. Find results in '$OUTFILE'."

$ gislab-cluster query -timeout=60s script "gislab/scripts/performance-test.sh"