Cluster management

GIS.lab cluster is managed by decentralized cluster management tool called Serf, which is based on gossip protocol. Serf is responsible for automatic joining and removing machines to and from GIS.lab cluster and OWS load balancer management. It is also used as interface for running cluster events and queries.

Roles, events and queries

Machines belonging in to GIS.lab cluster are divided in to three roles:

• server: GIS.lab server
• client: GIS.lab Desktop clients
• worker: GIS.lab Worker clients

All machines are capable of running different set of cluster events and queries depending on their role membership. Events and queries can be send from any machine which is a member of GIS.lab cluster using serf client (serf binary), or programmatically using RPC mechanism. All machines in cluster will receive all events and queries and will decide to respond or not depending on existence of handler responsible for particular event or query.

The main difference between event and query is that while query is designed to send some query and receive response, the purpose of event is just to announce that something has happen or should happen without receiving any response. Response from query can be returned in two formats: text or JSON.

Public events and queries

Here is a list of publicly available events and queries designed for ordinary usage. This list doesn't contain system events and queries which are used for internal GIS.lab cluster management.

Get list of cluster members, theirs roles membership and list of supported events and queries

$ sudo serf members

Reboot/shutdown all client and worker machines

$ sudo serf event reboot
$ sudo serf event shutdown

Reboot/shutdown particular client or worker machine

$ sudo serf event reboot "<hostname>,<hostname>,..."
$ sudo serf event shutdown "<hostname>,<hostname>,..."

Get information about actively running desktop client sessions in cluster

$ sudo serf query client-session-active

Perform shell command on all desktop client and worker machines (command is executed with superuser permissions)

Response from command query is automatically truncated to last 512 bytes.

$ sudo serf query command "<COMMAND>"

Run script published in '~/Project' directory on all desktop client and worker machines (command is executer with user permissions)

$ sudo serf query script "<USERNAME>/<SCRIPT-PATH>"

Example queries

Get uptime of all client and worker machines

$ sudo serf query command uptime

Query 'command' dispatched
Ack from 'server.gis.lab'
Ack from 'c50'
Ack from 'w10'
Response from 'c50': 06:04:46 up 10 min, 1 user, load average: 0.00, 0.08, 0.10, EXIT CODE: 0
Response from 'w10': 06:04:22 up 2 min, 0 users, load average: 0.64, 0.24, 0.09, EXIT CODE: 0
Total Acks: 3
Total Responses: 2

Perform performance test of parallel write to network share

$ sudo serf query -timeout=60s command 'dd if=/dev/zero of=/mnt/barrel/file-$(hostname).io bs=1M count=1000'

Query 'command' dispatched
Ack from 'server.gis.lab'
Ack from 'w10'
Ack from 'c50'
Response from 'w10': 1000+0 records in 1000+0 records out 1048576000 bytes (1.0 GB) copied, 14.7506 s, 71.1 MB/s, EXIT CODE: 0
Response from 'c50': 1000+0 records in 1000+0 records out 1048576000 bytes (1.0 GB) copied, 15.6153 s, 67.2 MB/s, EXIT CODE: 0
Total Acks: 3
Total Responses: 2

Perform performance test of parallel read from network share

$ sudo serf query -timeout=60s command 'dd if=/mnt/barrel/file-$(hostname).io of=/dev/zero bs=1M'

Query 'command' dispatched
Ack from 'server.gis.lab'
Ack from 'w10'
Ack from 'c50'
Response from 'w10': 1000+0 records in 1000+0 records out 1048576000 bytes (1.0 GB) copied, 9.12073 s, 115 MB/s, EXIT CODE: 0
Response from 'c50': 1000+0 records in 1000+0 records out 1048576000 bytes (1.0 GB) copied, 9.81312 s, 107 MB/s, EXIT CODE: 0
Total Acks: 3
Total Responses: 2

Perform CPU performance test

$ sudo serf query -timeout=60s command 'dd if=/dev/zero bs=1M count=1024 | md5sum'

Query 'command' dispatched
Ack from 'server.gis.lab'
Ack from 'w10'
Ack from 'c50'
Response from 'w10': 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 2.3575 s, 455 MB/s cd573cfaace07e7949bc0c46028904ff -, EXIT CODE: 0
Response from 'c50': 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 2.72427 s, 394 MB/s cd573cfaace07e7949bc0c46028904ff -, EXIT CODE: 0
Total Acks: 3
Total Responses: 2

Install Emacs to all client and worker machines

$ sudo serf query -timeout=60s command 'DEBIAN_FRONTEND=noninteractive apt-get install -yq emacs23-nox'

Query 'command' dispatched
Ack from 'server.gis.lab'
Ack from 'c50'
Ack from 'w10'
Response from 'w10': tories currently installed.) Unpacking emacs23-nox (from .../emacs23-nox_23.3+1-1ubuntu9.2_amd64.deb) ... Processing triggers for man-db ... Setting up emacs23-nox (23.3+1-1ubuntu9.2) ... update-alternatives: using /usr/bin/emacs23-nox to provide /usr/bin/emacs (emacs) in auto mode. emacs-install emacs23 emacsen-common: Handling install of emacsen flavor emacs23 emacsen-common: byte-compiling for emacs23 Wrote /etc/emacs23/site-start.d/00debian-vars.elc Wrote /usr/share/emacs23/site-lisp/debian-startup.elc, EXIT CODE: 0
Response from 'c50': ode. emacs-install emacs23 install/dictionaries-common: Byte-compiling for emacsen flavour emacs23 Wrote /usr/share/emacs23/site-lisp/dictionaries-common/debian-ispell.elc Wrote /usr/share/emacs23/site-lisp/dictionaries-common/ispell.elc Wrote /usr/share/emacs23/site-lisp/dictionaries-common/flyspell.elc emacsen-common: Handling install of emacsen flavor emacs23 emacsen-common: byte-compiling for emacs23 Wrote /etc/emacs23/site-start.d/00debian-vars.elc Wrote /usr/share/emacs23/site-lisp/debian-startup.elc, EXIT CODE: 0
Total Acks: 3
Total Responses: 2

Verify that Emacs is installed

$ sudo serf query -timeout=60s command 'dpkg -l | grep emacs23-nox'

Query 'command' dispatched
Ack from 'server.gis.lab'
Ack from 'w10'
Ack from 'c50'
Response from 'w10': ii emacs23-nox 23.3+1-1ubuntu9.2 The GNU Emacs editor (without X support), EXIT CODE: 0
Response from 'c50': ii emacs23-nox 23.3+1-1ubuntu9.2 The GNU Emacs editor (without X support), EXIT CODE: 0
Total Acks: 3
Total Responses: 2