Service as a Job: The Tomcat App Server

As seen previously, anything with a life-cycle to be managed can be turned into a job. This time around it’s the Tomcat application server.

A job description that you can give to condor_submit:

cmd = tomcat6.sh
args = conf.tar.gz webapps.tar.gz

transfer_input_files = conf.tar.gz, webapps.tar.gz
when_to_transfer_output = ON_EXIT_OR_EVICT

log = tomcat6.$(cluster).$(process).log

kill_sig = SIGTERM

+WantIOProxy = TRUE

queue

The primary components are the same, a controlling script called tomcat6.sh is the program Condor will run, it will still accept SIGTERM to handle shutdown, and it wants to use chirp to publish information. However, the input parameters are different. The tomcat6.sh controller is taking two parameters, tarballs, and they need to be transferred along with the job, thus the transfer_input_files.

Here is the controlling script, tomcat6.sh:

#!/bin/sh

# Parameters -
#  $1 is a tar.gz of a conf/
#  $2 is a tar.gz of a webapps/
#  $3 is the Catalina service port, often 8080
#  $4 is the Catalina service ssl port, often 8443
#  $5 is the Shutdown port, often 8005
#  $6 is the AJP connector port, often 8009
CONF_TGZ=$1
WEBAPPS_TGZ=$2
CATALINA_PORT=$3
CATALINA_SSL_PORT=$4
SHUTDOWN_PORT=$5
AJP_PORT=$6

# tomcat6 lives in /usr/sbin,
# condor_chirp in /usr/libexec/condor
export PATH=$PATH:/usr/sbin:/usr/libexec/condor

# Home configuration and install location
export CATALINA_HOME=/usr/share/tomcat6

# Base configuration and install location for this instance
export CATALINA_BASE=.

# Pid file, needed to prevent exiting before tomcat6
export CATALINA_PID=$CATALINA_BASE/pid

# When we get SIGTERM, which Condor will send when
# we are kicked, kill off tomcat6.
function term {
    tomcat6 stop
}

function find_free_port {
    local skip=$(netstat -ntl | awk '/^tcp/ {gsub("(.)*:", "", $4); print $4}')
    local ground=10000
    local port=$(($ground + ${RANDOM:0:4}))
    while [ ! -z $(expr "$skip" : ".*\($port\).*") ]; do
	port=$(($ground + ${RANDOM:0:4}))
    done
    echo $port
}

# Make sure ports are set
if [ -z "$CATALINA_PORT" ]; then
    CATALINA_PORT=$(find_free_port)
fi
if [ -z "$CATALINA_SSL_PORT" ]; then
    CATALINA_SSL_PORT=$(find_free_port)
fi
if [ -z "$SHUTDOWN_PORT" ]; then
    SHUTDOWN_PORT=$(find_free_port)
fi
if [ -z "$AJP_PORT" ]; then
    AJP_PORT=$(find_free_port)
fi

# Need logs directory
mkdir -p logs

# Need a temp directory
mkdir -p temp
export CATALINA_TMPDIR=$PWD/temp

# Setup configuration
tar zxfv $CONF_TGZ

# Install webapps
tar zxfv $WEBAPPS_TGZ

echo "Catalina Port: $CATALINA_PORT"
echo "Catalina SSL Port: $CATALINA_SSL_PORT"
echo "Shutdown Port: $SHUTDOWN_PORT"
echo "AJP Port: $AJP_PORT"

# Configure ports
sed -e "s/8005/$SHUTDOWN_PORT/g" -e "s/8080/$CATALINA_PORT/g" \
    -e "s/8009/$AJP_PORT/g" -e "s/8443/$CATALINA_SSL_PORT/g" \
    -i ${CATALINA_BASE}/conf/server.xml

# Spawn tomcat6, and make sure we can shut it down cleanly
trap term SIGTERM
tomcat6 start

# We might have to wait for the pid
while [ ! -s $CATALINA_PID ]; do sleep 1; done
PID=$(cat $CATALINA_PID)
rm -f $CATALINA_PID

# Record port numbers where everyone can see them
# (debug with alias condor_chirp=echo)
condor_chirp set_job_attr CatalinaEndpoint \"$HOSTNAME:$CATALINA_PORT\"
condor_chirp set_job_attr CatalinaSSLEndpoint \"$HOSTNAME:$CATALINA_SSL_PORT\"
condor_chirp set_job_attr ShutdownEndpoint \"$HOSTNAME:$SHUTDOWN_PORT\"
condor_chirp set_job_attr AJPEndpoint \"$HOSTNAME:$AJP_PORT\"

# There are all sorts of useful things that could
# happen here, such as looping and using condor_chirp
# to publish statistics or monitoring for base state.
# The important thing is not to exit until ready to
# shutdown tomcat.
while [ true ]; do
   ps $PID
   if [ $? -eq 0 ]; then
       sleep 15
   else
       echo "Tomcat exited, we are too"
       break
   fi
done

It is hopefully straightforward enough. The important pieces to notice: 1) SIGTERM handler, used for shutting down cleanly; 2) pid file, used to make sure the controller does not exit before the controlled program, Tomcat, does; 3) a handful of setup instructions that are specific to Tomcat and make sure that multiple instances of Tomcat do not conflict with one another on a single machine.

You can test this out by installing tomcat6 on your machine. I did yum install tomcat6 on Fedora 13.

Once that is done, the only thing you have to do is tar up a conf/ directory and a webapps/ directory. They will serve as input to tomcat6.sh. I used /usr/share/tomcat6/conf and /usr/share/tomcat6/webapps, which you can get from the tomcat6-webapps package.

$ tar ztf conf.tar.gz| head -n3
conf/
conf/Catalina/
conf/Catalina/localhost/
$ tar ztf webapps.tar.gz| head -n3
webapps/
webapps/sample/
webapps/sample/index.html

Submit the app server with:

$ condor_submit tomcat6.sub
Submitting job(s).
1 job(s) submitted to cluster 10434.
$ condor_submit tomcat6.sub
Submitting job(s).
1 job(s) submitted to cluster 10435.
$ condor_submit tomcat6.sub
Submitting job(s).
1 job(s) submitted to cluster 10436.

And watch them run, you won’t see any output until they start running:

$ condor_q -const 'CatalinaEndpoint =!= UNDEFINED' -format '%d.' ClusterId -format '%d\t' ProcId -format "%s\t" App -format 'http://%s\n' CatalinaEndpoint
10434.0	http://eeyore.local:18814
10435.0	http://eeyore.local:11304
10436.0	http://eeyore.local:12156

You should be able to click on the link to see the default Tomcat install pages.

Simple as that. When you want to take them down just condor_hold or condor_rm the jobs.

For some extra fun, look at how you can parametrize a submission file.

$ cat tomcat6-param.sub
cmd = tomcat6.sh
args = conf.tar.gz $(APP)

+App="$(APP)"

transfer_input_files = conf.tar.gz, $(APP)
when_to_transfer_output = ON_EXIT_OR_EVICT

log = tomcat6.$(cluster).$(process).log
output = tomcat6.$(cluster).$(process).out
error = tomcat6.$(cluster).$(process).err

kill_sig = SIGTERM

+WantIOProxy = TRUE

queue

I created a simple Hudson webapps tarball.

$ tar ztf hudson.tar.gz
webapps/
webapps/hudson.war

That you can submit with:

$ condor_submit -a APP=hudson.tar.gz tomcat6-param.sub
Submitting job(s).
1 job(s) submitted to cluster 10437.

Once it starts running you’ll see it in the condor_q output, where it is nicely labeled:

$ condor_q -const 'CatalinaEndpoint =!= UNDEFINED' -format '%d.' ClusterId -format '%d\t' ProcId -format "%s\t" App -format 'http://%s\n' CatalinaEndpoint
10434.0	http://eeyore.local:18814
10435.0	http://eeyore.local:11304
10436.0	http://eeyore.local:12156
10437.0	hudson.tar.gz	http://eeyore.local:11829

Going to the URL directly is not as interesting. It is a Hudson instance, so you must go to http://eeyore.local:11829/hudson.

Advertisement

Tags: App Server, Condor, Service as a Job, Tomcat