Tooltip: ClusterShell

Whenever sysadmins get together, I’ve noticed we often end up trading interesting tools and tricks that others may not have heard of. There is just so much out there that you can’t know about every available tool, so it’s always fun to hear what others are using. In that spirit, I thought I’d start the occasional “tooltip” blog entry about tools I find useful at work or in playing around. (The name for the series being borrowed both from the little hover text, and from the similar segment on The Ship Show.)

ClusterShell

When you’re helping to manage a large computing system, it’s sometimes unavoidable that you will have to run some ad-hoc command on a large number of nodes. In theory, perhaps, your configuration management software or cluster manager should be the only tool used to manage most systems; in reality, these systems occasionally break, or become too cumbersome to make quick, transient actions. Sometimes you just need to give your system a kick, and the only tool you have left is ssh.

ClusterShell, from the HPC Group at CEA, provides a collection of tools and Python libraries for executing commands across a large number of machines at the same time and gathering the results. You can think of it as a parallel ssh command, and it’s very lightweight in both usability and performance. It works a lot like LLNL’s pdsh which provides similar functionality, but also provides a lot of nice bells and whistles which I think make it a better tool for everyday system admin work.

clush

For everyday work, the primary ClusterShell tool you’ll probably use is called clush. This is the actual command-line “parallel ssh” tool, and works pretty much exactly like pdsh. When you call it with a list of hosts and a command, it ssh’s to each of the hosts, executes the command, and displays the result:

    [root@c01 ~]# clush -w c01,c02,c03,c04 uname -r
    c01: 3.10.0-123.8.1.el7.centos.plus.x86_64
    c03: 3.10.0-123.8.1.el7.centos.plus.x86_64
    c04: 3.10.0-123.8.1.el7.centos.plus.x86_64
    c02: 3.10.0-123.8.1.el7.centos.plus.x86_64

Because listing hostnames as comma-separated lists can be a pain when you have a lot of hosts, it supports folding hostnames which include numeric ranges:

    [root@c01 ~]# clush -w c[01-04] uname -r
    c01: 3.10.0-123.8.1.el7.centos.plus.x86_64
    c02: 3.10.0-123.8.1.el7.centos.plus.x86_64
    c03: 3.10.0-123.8.1.el7.centos.plus.x86_64
    c04: 3.10.0-123.8.1.el7.centos.plus.x86_64

And because many hosts might return the same result, you can fold the results as well using the option -b:

    [root@c01 ~]# clush -bw c[01-04] uname -r
    ---------------
    c[01-04] (4)
    ---------------
    3.10.0-123.8.1.el7.centos.plus.x86_64

(If you were using pdsh, you’d have to pipe to dshbak to get a similar result: i.e. pdsh -w c[01-04] uname -r | grep dshbak -c.)

You can also use clush interactively. For example:

    [root@c01 ~]# clush -bw c0[1-4]
    Enter 'quit' to leave this interactive mode
    Working with nodes: c[01-04]
    clush> touch /tmp/example
    clush> ls -l /tmp/example
    ---------------
    c[01-04] (4)
    ---------------
    -rw-r--r-- 1 root root 0 Dec 13 19:07 /tmp/example
    clush>

If you have differences in output, clush will split out each node individually:

    [root@c01 ~]# ssh c03 rm /tmp/example
    [root@c01 ~]# clush -bw c0[1-4] ls -l /tmp/example
    c03: ls: cannot access /tmp/example: No such file or directory
    ---------------
    c[01-02,04] (3)
    ---------------
    -rw-r--r-- 1 root root 0 Dec 13 19:07 /tmp/example
    clush: c03: exited with exit code 2

Or if you just want to see specific differences in output:

    [root@c01 ~]# ssh c03 touch /tmp/example2
    [root@c01 ~]# clush -bw c0[1-4]  --diff "ls -l /tmp/ | grep example"
    --- c[01-02,04] (3)
    +++ c03
    @@ -1 +1 @@
    --rw-r--r-- 1 root root     0 Dec 13 19:07 example
    +-rw-r--r-- 1 root root     0 Dec 13 19:14 example2

nodeset

The nodeset command provides a nice little interface for folding, expanding, and otherwise manipulating groups of node names. I mostly use this command in shell scripts when I want to perform some action for every node name in a group (i.e. create a log directory, or execute some ipmitool command) and want to be able to pass that script a nicely-formatted nodeset instead of a long list.

For example, a list of comma-separated hosts can be folded into an easier-to-handle nodeset:

    [root@c01 ~]# nodeset -f c01,c02,c03,c07,c10,c11,c12,c17,c18
    c[01-03,07,10-12,17-18]

Expand a nodeset, for example to use in a bash loop:

    [root@c01 ~]# nodeset -e c[00-50,60-99]
    c00 c01 c02 c03 c04 c05 c06 c07 c08 c09 c10 c11 c12 c13 c14 c15 c16 c17 c18 c19 c20 c21 c22 c23 c24 c25 c26 c27 c28 c29 c30 c31 c32 c33 c34 c35 c36 c37 c38 c39 c40 c41 c42 c43 c44 c45 c46 c47 c48 c49 c50 c60 c61 c62 c63 c64 c65 c66 c67 c68 c69 c70 c71 c72 c73 c74 c75 c76 c77 c78 c79 c80 c81 c82 c83 c84 c85 c86 c87 c88 c89 c90 c91 c92 c93 c94 c95 c96 c97 c98 c99

Compute unions, intersections, and xors:

    [root@c01 ~]# nodeset -f c[01-03],c[04-07]
    c[01-07]
    [root@c01 ~]# nodeset -f "c[01-07]&c[04-11]"
    c[04-07]
    [root@c01 ~]# nodeset -f "c[01-07]^c[04-11]"
    c[01-03,08-11]

And several other operations I never use personally, but which you might. The man page has many more examples.

Python library

While I mostly use the command-line ClusterShell tools, clush and nodeset, one of the big advantages over pdsh and others is that ClusterShell also provides a set of good Python libraries which these tools are based on. I find this very helpful, as I do occasionally find myself needing to execute a bunch of parallel ssh tasks within a script and I prefer to stay within Python rather than shelling out.

ClusterShell provides a decent programming guide for these libraries, but here’s a short example script which I’ve used as an extremely basic sanity check in the past. It runs a command on all nodes and looks for the expected output, with the default being to run uname and look for “Linux”:

    #!/usr/bin/env python

    import sys
    from optparse import OptionParser
    from ClusterShell.Task import task_self
    from ClusterShell.Event import EventHandler


    # Class to handle ClusterShell events
    class SanityHandler(EventHandler):
        # Get expected output
        def __init__(self, expected):
            self.expected = expected
        # Handle successful read events
        def ev_read(self, worker):
            if worker.current_msg == self.expected:
                print "%s ok" % worker.current_node
            else:
                print "%s returned value: '%s'" % (worker.current_node,\
                                                   worker.current_msg)
        # Handle failed events
        def ev_hup(self, worker):
            if worker.current_rc != 0:
                print "%s returned with error code %s" % (worker.current_node,\
                                                          worker.current_rc)


    # Command-line options (just the nodeset, in this case)
    def get_options():
        parser = OptionParser()
        parser.add_option("-c", "--command", dest="command", default="/bin/uname")
        parser.add_option("-e", "--expected", dest="expected", default="Linux")
        parser.add_option("-n", "--nodeset", dest="nodeset")
        (options, _) = parser.parse_args()
        if not options.nodeset:
            print "You must specify a nodeset. Try running with --help"
            sys.exit(1)
        return options


    def main():
        opts = get_options()
        task = task_self()
        task.run(opts.command, nodes=opts.nodeset, handler=SanityHandler(opts.expected))


    if __name__ == "__main__":
        main()

And a sample run:

    [root@c01 ~]# ./sanity.py -n c[01-04] -c "uname -r" -e "3.10.0-123.8.1.el7.centos.plus.x86_64"
    c01 ok
    c02 ok
    c03 ok
    c04 ok

    [root@c01 ~]# ./sanity.py -n c[01-04] -c "uname -r" -e "2.0"
    c01 returned value: '3.10.0-123.8.1.el7.centos.plus.x86_64'
    c02 returned value: '3.10.0-123.8.1.el7.centos.plus.x86_64'
    c03 returned value: '3.10.0-123.8.1.el7.centos.plus.x86_64'
    c04 returned value: '3.10.0-123.8.1.el7.centos.plus.x86_64'
Questions, comments, interesting anecdotes? Tweet to me at @ajdecon, or send me an email at ajdecon@ajdecon.org.