Published: Tue 29 December 2015
By Adam DeConinck
In misc .
tags: 'tooltip' 'hpc' 'python'
Whenever sysadmins get together, I've noticed we often end up trading interesting
tools and tricks that others may not have heard of. There is just so much out there
that you can't know about every available tool, so it's always fun to hear what others are using.
In that spirit, I thought I'd start the occasional "tooltip" blog entry about
tools I find useful at work or in playing around. (The name for the series being
borrowed both from the little hover text, and from the similar segment on
The Ship Show .)
ClusterShell
When you're helping to manage a large computing system, it's sometimes unavoidable
that you will have to run some ad-hoc command on a large number of nodes. In theory,
perhaps, your configuration management software or cluster manager should be the only
tool used to manage most systems; in reality, these systems occasionally break, or
become too cumbersome to make quick, transient actions. Sometimes you just need to
give your system a kick, and the only tool you have left is ssh.
ClusterShell , from the
HPC Group at CEA , provides
a collection of tools and Python libraries for
executing commands across a large number of machines at the same time and
gathering the results. You can think of it as a parallel ssh command,
and it's very lightweight in both usability and
performance. It works a lot like LLNL's
pdsh which provides similar functionality,
but also provides a lot of nice bells and whistles which I think make it a
better tool for everyday system admin work.
clush
For everyday work, the primary ClusterShell tool you'll probably use is called
clush. This is the actual command-line "parallel ssh" tool, and works pretty
much exactly like pdsh. When you call it with a list of hosts and a command,
it ssh's to each of the hosts, executes the command, and displays the result:
Because listing hostnames as comma-separated lists can be a pain when you have
a lot of hosts, it supports folding hostnames which include numeric ranges:
And because many hosts might return the same result, you can fold the results
as well using the option -b:
(If you were using pdsh, you'd have to pipe to dshbak to get a similar result:
i.e. pdsh -w c[01-04] uname -r | grep dshbak -c.)
You can also use clush interactively. For example:
If you have differences in output, clush will split out each node individually:
Or if you just want to see specific differences in output:
nodeset
The nodeset command provides a nice little interface for folding, expanding, and otherwise manipulating
groups of node names. I mostly use this command in shell scripts when I want to perform some action
for every node name in a group (i.e. create a log directory, or execute some ipmitool command)
and want to be able to pass that script a nicely-formatted nodeset instead of a long list.
For example, a list of comma-separated hosts can be
folded into an easier-to-handle nodeset:
Expand a nodeset, for example to use in a bash loop:
Compute unions, intersections, and xors:
And several other operations I never use personally, but which you might. The man page has many
more examples.
Python library
While I mostly use the command-line ClusterShell tools, clush and nodeset, one
of the big advantages over pdsh and others is that ClusterShell also provides
a set of good Python libraries which these tools are based on. I find this very
helpful, as I do occasionally find myself needing to execute a bunch of
parallel ssh tasks within a script and I prefer to stay within Python rather than
shelling out.
ClusterShell provides a decent programming guide
for these libraries, but here's a short example script which I've used as an
extremely basic sanity check in the past. It runs a command on all nodes and
looks for the expected output, with the default being to run uname and
look for "Linux":
#!/usr/bin/env python
import sys
from optparse import OptionParser
from ClusterShell.Task import task_self
from ClusterShell.Event import EventHandler
# Class to handle ClusterShell events
class SanityHandler ( EventHandler ):
# Get expected output
def __init__ ( self , expected ):
self . expected = expected
# Handle successful read events
def ev_read ( self , worker ):
if worker . current_msg == self . expected :
print " %s ok" % worker . current_node
else :
print " %s returned value: ' %s '" % ( worker . current_node , \
worker . current_msg )
# Handle failed events
def ev_hup ( self , worker ):
if worker . current_rc != 0 :
print " %s returned with error code %s " % ( worker . current_node , \
worker . current_rc )
# Command-line options (just the nodeset, in this case)
def get_options ():
parser = OptionParser ()
parser . add_option ( "-c" , "--command" , dest = "command" , default = "/bin/uname" )
parser . add_option ( "-e" , "--expected" , dest = "expected" , default = "Linux" )
parser . add_option ( "-n" , "--nodeset" , dest = "nodeset" )
( options , _ ) = parser . parse_args ()
if not options . nodeset :
print "You must specify a nodeset. Try running with --help"
sys . exit ( 1 )
return options
def main ():
opts = get_options ()
task = task_self ()
task . run ( opts . command , nodes = opts . nodeset , handler = SanityHandler ( opts . expected ))
if __name__ == "__main__" :
main ()
And a sample run: