Building Torque 4.1.0 and Maui 3.3.1 (using fpm) on EL 6.2

layout: post published: true Date: 2012-08-05 Tags:

posterous_url: posterous_slug: installing-the-maui-scheduler-with-torque-410

Torque is an open-source implementation of the popular PBS resource manager. A large proportion of academic HPC clusters use Torque, and quite a few commercial sites also choose to use it instead of a commercial solution like PBS Professional. If you've submitted an HPC job using the "qsub" command, there's a pretty good chance you were using Torque.

However, while Torque is a pretty sophisticated resource manager, its actual scheduler is pretty limited. One big thing it doesn't support is advance reservations, so there's no way to keep resources free for future jobs. It also doesn't support a lot of tuning parameters for scheduling, like "fair-share" scheduling. But it does support the option of letting another program do the scheduling for it, while it keeps track of the underlying resources. And that's where Maui comes in.

I recently had to get a Torque/Maui installation up and running, and with the most recent versions there were a few annoyances to deal with. This includes the fact that Maui doesn't include a spec file to build RPMs for Enterprise Linux derivatives, so rather just doing "make install" I used fpm to build an RPM myself. My notes on the process follow.

Download Torque and Maui

Torque and Maui are both distributed by Adaptive Computing, who also develop the commercial Moab scheduler. Annoyingly, you have to register an account on their website for download, and Maui is hidden under "Other Open Source Products": they really would rather you buy Moab. But once you've jumped through those hoops, you can download the software. 

Building Torque RPMs

This is the easy part:

[ajdecon@tyr-sl ~]$ rpmbuild -tb torque-4.1.0.tar.gz 
Wrote: /home/ajdecon/rpmbuild/RPMS/x86_64/torque-4.1.0-1.cri.x86_64.rpm
Wrote: /home/ajdecon/rpmbuild/RPMS/x86_64/torque-server-4.1.0-1.cri.x86_64.rpm
Wrote: /home/ajdecon/rpmbuild/RPMS/x86_64/torque-devel-4.1.0-1.cri.x86_64.rpm
Wrote: /home/ajdecon/rpmbuild/RPMS/x86_64/torque-scheduler-4.1.0-1.cri.x86_64.rpm
Wrote: /home/ajdecon/rpmbuild/RPMS/x86_64/torque-client-4.1.0-1.cri.x86_64.rpm
Wrote: /home/ajdecon/rpmbuild/RPMS/x86_64/torque-debuginfo-4.1.0-1.cri.x86_64.rpm
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.qg7cBZ
+ umask 022
+ cd /home/ajdecon/rpmbuild/BUILD
+ cd torque-4.1.0
+ test x/home/ajdecon/rpmbuild/BUILDROOT/torque-4.1.0-1.cri.x86_64 '!=' x/
+ rm -rf /home/ajdecon/rpmbuild/BUILDROOT/torque-4.1.0-1.cri.x86_64
+ exit 0

You'll need Torque installed to build Maui, so go ahead and install your new RPMs:

[ajdecon@tyr-sl ~]$ sudo yum localinstall ~/rpmbuild/RPMS/x86_64/torque-*rpm
  torque.x86_64 0:4.1.0-1.cri              torque-client.x86_64 0:4.1.0-1.cri   
  torque-debuginfo.x86_64 0:4.1.0-1.cri    torque-devel.x86_64 0:4.1.0-1.cri    
  torque-scheduler.x86_64 0:4.1.0-1.cri    torque-server.x86_64 0:4.1.0-1.cri

Patching Maui

Unfortunately, Maui has not yet been updated to build with newer versions of torque:

[ajdecon@tyr-sl maui-3.3.1]$ ./configure --prefix=/usr/local/maui
[ajdecon@tyr-sl maui-3.3.1]$ make
MPBSI.c:177: error: conflicting types for ‘get_svrport’
/usr/include/torque/pbs_ifl.h:684: note: previous declaration of ‘get_svrport’ was here
MPBSI.c:178: error: conflicting types for ‘openrm’
/usr/include/torque/pbs_ifl.h:685: note: previous declaration of ‘openrm’ was here
make[1]: *** [MPBSI.o] Error 1
make[1]: Leaving directory `/home/ajdecon/maui-3.3.1/src/moab'
make: *** [all] Error 2

Luckily this was noted on the torqueusers list, and it can be fixed by changing a couple of definitions in Maui's src/moab/MPBSI.c to match those in Torque:

[ajdecon@tyr-sl moab]$ diff MPBSI.c.orig MPBSI.c
< extern int get_svrport(const char *,char *,int);
< extern int openrm(char *,int);
> extern unsigned int get_svrport(char *,char *,unsigned int);
> extern int openrm(char *,unsigned int);

Building Maui

Once this patch is in place, you can build:

[ajdecon@tyr-sl maui-3.3.1]$ ./configure --prefix=/usr/local/maui
[ajdecon@tyr-sl maui-3.3.1]$ make

After this you could `make install` and have a functioning server... but as system administrators, don't we all hate software that doesn't work with our package managers? Much better to put together an RPM which we can work with more easily later. For this we'll use fpm ("Effing Package Management").

fpm supports building packages from "make install" if the makefiles support the "make install DESTDIR=/path/to/install" syntax, which Maui's Makefiles don't. But that's easy to fix with a little inspection of the Makefiles and a little sed:

[ajdecon@tyr-sl maui-3.3.1]$ sed -i'.bkp' 's/\$(INST_DIR)/\$(DESTDIR)\/\$(INST_DIR)/g' src/*/Makefile
[ajdecon@tyr-sl maui-3.3.1]$ sed -i'' 's/\$(MSCHED_HOME)/\$(DESTDIR)\/\$(MSCHED_HOME)/g' src/*/Makefile

Then we use DESTDIR to install everything into a location in /tmp, which we'll use as the basis for our RPM:

[ajdecon@tyr-sl maui-3.3.1]$ DESTDIR=/tmp/maui make install

Setting up the install

Unfortunately, we're not done yet: Maui's `make install` doesn't automatically set up init scripts, which it would be nice to do. So we've got a little bit yet to go before we can build our RPM.

First, we need to copy the etc/ files into their proper locations in the tmp install directory:

[ajdecon@tyr-sl maui-3.3.1]$ mkdir /tmp/maui/etc
[ajdecon@tyr-sl maui-3.3.1]$ mkdir /tmp/maui/etc/profile.d
[ajdecon@tyr-sl maui-3.3.1]$ mkdir /tmp/maui/etc/init.d
[ajdecon@tyr-sl maui-3.3.1]$ cp etc/maui.d /tmp/maui/etc/init.d/
[ajdecon@tyr-sl maui-3.3.1]$ cp etc/maui.{csh,sh} /tmp/maui/etc/profile.d/

Make sure to edit the /tmp/maui/etc/init.d/maui.d file so "MAUI_PREFIX=/usr/local/maui", since that's the prefix we're using.

Then, just to be fancy, we'll set up some post-install and pre-uninstall scripts to handle the maui.d service.

[ajdecon@tyr-sl maui-3.3.1]$ cat /tmp/maui/ 
chkconfig --add maiu.d
chkconfig --level 3456 maui.d on
[ajdecon@tyr-sl maui-3.3.1]$ cat /tmp/maui/ 
chkconfig --del maui.d

Building the RPM

Finally, we can build the rpm using fpm!

[ajdecon@tyr-sl maui-3.3.1]$ fpm -s dir -t rpm -n maui -v 3.3.1 -C /tmp/maui \
-p /tmp/maui-3.3.1-x86_64-fpmbuild.rpm --post-install /tmp/maui/ \
--pre-uninstall /tmp/maui/ etc usr
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.W8v7PU
Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.rGXX9h
Executing(%install): /bin/sh -e /var/tmp/rpm-tmp.wJAluF
Processing files: maui-3.3.1-1.x86_64
Wrote: /tmp/package-rpm-build20120805-26510-1z6ht3/RPMS/x86_64/maui-3.3.1-1.x86_64.rpm
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.nYdLBS
Created rpm {"path":"/tmp/maui-3.3.1-x86_64-fpmbuild.rpm"}

If we check the rpm, we'll see that all our files are present:

[ajdecon@tyr-sl maui-3.3.1]$ rpm -q --filesbypkg -p /tmp/maui-3.3.1-x86_64-fpmbuild.rpm 
maui                      /etc/init.d/maui.d
maui                      /etc/profile.d/maui.csh
maui                      /etc/profile.d/
maui                      /usr/local/maui/bin/canceljob

This RPM can be installed with the command:

sudo yum localinstall /tmp/maui-3.3.1-x86_64-fpmbuild.rpm

Of course, setting up and configuring Torque and Maui is a totally separate project. Check out the documentation on Adaptive Computing's website. And I may blog on that process another time...