Whenever sysadmins get together, I've noticed we often end up trading interesting tools and tricks that others may not have heard of. There is just so much out there that you can't know about every available tool, so it's always fun to hear what others are using. In that spirit, I thought I'd start the occasional "tooltip" blog entry about tools I find useful at work or in playing around. (The name for the series being borrowed both from the little hover text, and from the similar segment on The Ship Show.)
ClusterShell
When you're helping to manage a large computing system, it's sometimes unavoidable that you will have to run some ad-hoc command on a large number of nodes. In theory, perhaps, your configuration management software or cluster manager should be the only tool used to manage most systems; in reality, these systems occasionally break, or become too cumbersome to make quick, transient actions. Sometimes you just need to give your system a kick, and the only tool you have left is ssh.
ClusterShell, from the
HPC Group at CEA, provides
a collection of tools and Python libraries for
executing commands across a large number of machines at the same time and
gathering the results. You can think of it as a parallel ssh command,
and it's very lightweight in both usability and
performance. It works a lot like LLNL's
pdsh which provides similar functionality,
but also provides a lot of nice bells and whistles which I think make it a
better tool for everyday system admin work.
clush
For everyday work, the primary ClusterShell tool you'll probably use is called
clush. This is the actual command-line "parallel ssh" tool, and works pretty
much exactly like pdsh. When you call it with a list of hosts and a command,
it ssh's to each of the hosts, executes the command, and displays the result:
[root@c01 ~]# clush -w c01,c02,c03,c04 uname -r
c01: 3.10.0-123.8.1.el7.centos.plus.x86_64
c03: 3.10.0-123.8.1.el7.centos.plus.x86_64
c04: 3.10.0-123.8.1.el7.centos.plus.x86_64
c02: 3.10.0-123.8.1.el7.centos.plus.x86_64
Because listing hostnames as comma-separated lists can be a pain when you have a lot of hosts, it supports folding hostnames which include numeric ranges:
[root@c01 ~]# clush -w c[01-04] uname -r
c01: 3.10.0-123.8.1.el7.centos.plus.x86_64
c02: 3.10.0-123.8.1.el7.centos.plus.x86_64
c03: 3.10.0-123.8.1.el7.centos.plus.x86_64
c04: 3.10.0-123.8.1.el7.centos.plus.x86_64
And because many hosts might return the same result, you can fold the results
as well using the option -b:
[root@c01 ~]# clush -bw c[01-04] uname -r
---------------
c[01-04] (4)
---------------
3.10.0-123.8.1.el7.centos.plus.x86_64
(If you were using pdsh, you'd have to pipe to dshbak to get a similar result:
i.e. pdsh -w c[01-04] uname -r | grep dshbak -c.)
You can also use clush interactively. For example:
[root@c01 ~]# clush -bw c0[1-4]
Enter 'quit' to leave this interactive mode
Working with nodes: c[01-04]
clush> touch /tmp/example
clush> ls -l /tmp/example
---------------
c[01-04] (4)
---------------
-rw-r--r-- 1 root root 0 Dec 13 19:07 /tmp/example
clush>
If you have differences in output, clush will split out each node individually:
[root@c01 ~]# ssh c03 rm /tmp/example
[root@c01 ~]# clush -bw c0[1-4] ls -l /tmp/example
c03: ls: cannot access /tmp/example: No such file or directory
---------------
c[01-02,04] (3)
---------------
-rw-r--r-- 1 root root 0 Dec 13 19:07 /tmp/example
clush: c03: exited with exit code 2
Or if you just want to see specific differences in output:
[root@c01 ~]# ssh c03 touch /tmp/example2
[root@c01 ~]# clush -bw c0[1-4] --diff "ls -l /tmp/ | grep example"
--- c[01-02,04] (3)
+++ c03
@@ -1 +1 @@
--rw-r--r-- 1 root root 0 Dec 13 19:07 example
+-rw-r--r-- 1 root root 0 Dec 13 19:14 example2
nodeset
The nodeset command provides a nice little interface for folding, expanding, and otherwise manipulating
groups of node names. I mostly use this command in shell scripts when I want to perform some action
for every node name in a group (i.e. create a log directory, or execute some ipmitool command)
and want to be able to pass that script a nicely-formatted nodeset instead of a long list.
For example, a list of comma-separated hosts can be folded into an easier-to-handle nodeset:
[root@c01 ~]# nodeset -f c01,c02,c03,c07,c10,c11,c12,c17,c18
c[01-03,07,10-12,17-18]
Expand a nodeset, for example to use in a bash loop:
[root@c01 ~]# nodeset -e c[00-50,60-99]
c00 c01 c02 c03 c04 c05 c06 c07 c08 c09 c10 c11 c12 c13 c14 c15 c16 c17 c18 c19 c20 c21 c22 c23 c24 c25 c26 c27 c28 c29 c30 c31 c32 c33 c34 c35 c36 c37 c38 c39 c40 c41 c42 c43 c44 c45 c46 c47 c48 c49 c50 c60 c61 c62 c63 c64 c65 c66 c67 c68 c69 c70 c71 c72 c73 c74 c75 c76 c77 c78 c79 c80 c81 c82 c83 c84 c85 c86 c87 c88 c89 c90 c91 c92 c93 c94 c95 c96 c97 c98 c99
Compute unions, intersections, and xors:
[root@c01 ~]# nodeset -f c[01-03],c[04-07]
c[01-07]
[root@c01 ~]# nodeset -f "c[01-07]&c[04-11]"
c[04-07]
[root@c01 ~]# nodeset -f "c[01-07]^c[04-11]"
c[01-03,08-11]
And several other operations I never use personally, but which you might. The man page has many more examples.
Python library
While I mostly use the command-line ClusterShell tools, clush and nodeset, one
of the big advantages over pdsh and others is that ClusterShell also provides
a set of good Python libraries which these tools are based on. I find this very
helpful, as I do occasionally find myself needing to execute a bunch of
parallel ssh tasks within a script and I prefer to stay within Python rather than
shelling out.
ClusterShell provides a decent programming guide
for these libraries, but here's a short example script which I've used as an
extremely basic sanity check in the past. It runs a command on all nodes and
looks for the expected output, with the default being to run uname and
look for "Linux":
#!/usr/bin/env python
import sys
from optparse import OptionParser
from ClusterShell.Task import task_self
from ClusterShell.Event import EventHandler
# Class to handle ClusterShell events
class SanityHandler(EventHandler):
# Get expected output
def __init__(self, expected):
self.expected = expected
# Handle successful read events
def ev_read(self, worker):
if worker.current_msg == self.expected:
print "%s ok" % worker.current_node
else:
print "%s returned value: '%s'" % (worker.current_node,\
worker.current_msg)
# Handle failed events
def ev_hup(self, worker):
if worker.current_rc != 0:
print "%s returned with error code %s" % (worker.current_node,\
worker.current_rc)
# Command-line options (just the nodeset, in this case)
def get_options():
parser = OptionParser()
parser.add_option("-c", "--command", dest="command", default="/bin/uname")
parser.add_option("-e", "--expected", dest="expected", default="Linux")
parser.add_option("-n", "--nodeset", dest="nodeset")
(options, _) = parser.parse_args()
if not options.nodeset:
print "You must specify a nodeset. Try running with --help"
sys.exit(1)
return options
def main():
opts = get_options()
task = task_self()
task.run(opts.command, nodes=opts.nodeset, handler=SanityHandler(opts.expected))
if __name__ == "__main__":
main()
And a sample run:
[root@c01 ~]# ./sanity.py -n c[01-04] -c "uname -r" -e "3.10.0-123.8.1.el7.centos.plus.x86_64"
c01 ok
c02 ok
c03 ok
c04 ok
[root@c01 ~]# ./sanity.py -n c[01-04] -c "uname -r" -e "2.0"
c01 returned value: '3.10.0-123.8.1.el7.centos.plus.x86_64'
c02 returned value: '3.10.0-123.8.1.el7.centos.plus.x86_64'
c03 returned value: '3.10.0-123.8.1.el7.centos.plus.x86_64'
c04 returned value: '3.10.0-123.8.1.el7.centos.plus.x86_64'