Archive for February, 2011

Service as a Job: The Tomcat App Server

February 27, 2011

As seen previously, anything with a life-cycle to be managed can be turned into a job. This time around it’s the Tomcat application server.

A job description that you can give to condor_submit:

cmd =
args = conf.tar.gz webapps.tar.gz

transfer_input_files = conf.tar.gz, webapps.tar.gz
when_to_transfer_output = ON_EXIT_OR_EVICT

log = tomcat6.$(cluster).$(process).log

kill_sig = SIGTERM

+WantIOProxy = TRUE


The primary components are the same, a controlling script called is the program Condor will run, it will still accept SIGTERM to handle shutdown, and it wants to use chirp to publish information. However, the input parameters are different. The controller is taking two parameters, tarballs, and they need to be transferred along with the job, thus the transfer_input_files.

Here is the controlling script,


# Parameters -
#  $1 is a tar.gz of a conf/
#  $2 is a tar.gz of a webapps/
#  $3 is the Catalina service port, often 8080
#  $4 is the Catalina service ssl port, often 8443
#  $5 is the Shutdown port, often 8005
#  $6 is the AJP connector port, often 8009

# tomcat6 lives in /usr/sbin,
# condor_chirp in /usr/libexec/condor
export PATH=$PATH:/usr/sbin:/usr/libexec/condor

# Home configuration and install location
export CATALINA_HOME=/usr/share/tomcat6

# Base configuration and install location for this instance

# Pid file, needed to prevent exiting before tomcat6

# When we get SIGTERM, which Condor will send when
# we are kicked, kill off tomcat6.
function term {
    tomcat6 stop

function find_free_port {
    local skip=$(netstat -ntl | awk '/^tcp/ {gsub("(.)*:", "", $4); print $4}')
    local ground=10000
    local port=$(($ground + ${RANDOM:0:4}))
    while [ ! -z $(expr "$skip" : ".*\($port\).*") ]; do
	port=$(($ground + ${RANDOM:0:4}))
    echo $port

# Make sure ports are set
if [ -z "$CATALINA_PORT" ]; then
if [ -z "$CATALINA_SSL_PORT" ]; then
if [ -z "$SHUTDOWN_PORT" ]; then
if [ -z "$AJP_PORT" ]; then

# Need logs directory
mkdir -p logs

# Need a temp directory
mkdir -p temp

# Setup configuration
tar zxfv $CONF_TGZ

# Install webapps
tar zxfv $WEBAPPS_TGZ

echo "Catalina Port: $CATALINA_PORT"
echo "Catalina SSL Port: $CATALINA_SSL_PORT"
echo "Shutdown Port: $SHUTDOWN_PORT"
echo "AJP Port: $AJP_PORT"

# Configure ports
sed -e "s/8005/$SHUTDOWN_PORT/g" -e "s/8080/$CATALINA_PORT/g" \
    -e "s/8009/$AJP_PORT/g" -e "s/8443/$CATALINA_SSL_PORT/g" \
    -i ${CATALINA_BASE}/conf/server.xml

# Spawn tomcat6, and make sure we can shut it down cleanly
trap term SIGTERM
tomcat6 start

# We might have to wait for the pid
while [ ! -s $CATALINA_PID ]; do sleep 1; done

# Record port numbers where everyone can see them
# (debug with alias condor_chirp=echo)
condor_chirp set_job_attr CatalinaEndpoint \"$HOSTNAME:$CATALINA_PORT\"
condor_chirp set_job_attr CatalinaSSLEndpoint \"$HOSTNAME:$CATALINA_SSL_PORT\"
condor_chirp set_job_attr ShutdownEndpoint \"$HOSTNAME:$SHUTDOWN_PORT\"
condor_chirp set_job_attr AJPEndpoint \"$HOSTNAME:$AJP_PORT\"

# There are all sorts of useful things that could
# happen here, such as looping and using condor_chirp
# to publish statistics or monitoring for base state.
# The important thing is not to exit until ready to
# shutdown tomcat.
while [ true ]; do
   ps $PID
   if [ $? -eq 0 ]; then
       sleep 15
       echo "Tomcat exited, we are too"

It is hopefully straightforward enough. The important pieces to notice: 1) SIGTERM handler, used for shutting down cleanly; 2) pid file, used to make sure the controller does not exit before the controlled program, Tomcat, does; 3) a handful of setup instructions that are specific to Tomcat and make sure that multiple instances of Tomcat do not conflict with one another on a single machine.

You can test this out by installing tomcat6 on your machine. I did yum install tomcat6 on Fedora 13.

Once that is done, the only thing you have to do is tar up a conf/ directory and a webapps/ directory. They will serve as input to I used /usr/share/tomcat6/conf and /usr/share/tomcat6/webapps, which you can get from the tomcat6-webapps package.

$ tar ztf conf.tar.gz| head -n3
$ tar ztf webapps.tar.gz| head -n3 

Submit the app server with:

$ condor_submit tomcat6.sub 
Submitting job(s).
1 job(s) submitted to cluster 10434.
$ condor_submit tomcat6.sub
Submitting job(s).
1 job(s) submitted to cluster 10435.
$ condor_submit tomcat6.sub
Submitting job(s).
1 job(s) submitted to cluster 10436.

And watch them run, you won’t see any output until they start running:

$ condor_q -const 'CatalinaEndpoint =!= UNDEFINED' -format '%d.' ClusterId -format '%d\t' ProcId -format "%s\t" App -format 'http://%s\n' CatalinaEndpoint
10434.0	http://eeyore.local:18814
10435.0	http://eeyore.local:11304
10436.0	http://eeyore.local:12156

You should be able to click on the link to see the default Tomcat install pages.

Simple as that. When you want to take them down just condor_hold or condor_rm the jobs.

For some extra fun, look at how you can parametrize a submission file.

$ cat tomcat6-param.sub 
cmd =
args = conf.tar.gz $(APP)


transfer_input_files = conf.tar.gz, $(APP)
when_to_transfer_output = ON_EXIT_OR_EVICT

log = tomcat6.$(cluster).$(process).log
output = tomcat6.$(cluster).$(process).out
error = tomcat6.$(cluster).$(process).err

kill_sig = SIGTERM

+WantIOProxy = TRUE


I created a simple Hudson webapps tarball.

$ tar ztf hudson.tar.gz 

That you can submit with:

$ condor_submit -a APP=hudson.tar.gz tomcat6-param.sub
Submitting job(s).
1 job(s) submitted to cluster 10437.

Once it starts running you’ll see it in the condor_q output, where it is nicely labeled:

$ condor_q -const 'CatalinaEndpoint =!= UNDEFINED' -format '%d.' ClusterId -format '%d\t' ProcId -format "%s\t" App -format 'http://%s\n' CatalinaEndpoint
10434.0	http://eeyore.local:18814
10435.0	http://eeyore.local:11304
10436.0	http://eeyore.local:12156
10437.0	hudson.tar.gz	http://eeyore.local:11829

Going to the URL directly is not as interesting. It is a Hudson instance, so you must go to http://eeyore.local:11829/hudson.

%d bloggers like this: