Posts Tagged ‘Service as a Job’

Service as a Job: Hadoop MapReduce TaskTracker

June 6, 2012

Continuing to build on other examples of services run as jobs, such as HDFS NameNode and HDFS DataNode, here is an example for the Hadoop MapReduce framework’s TaskTracker.

mapred_tasktracker.sh

#!/bin/sh -x

# condor_chirp in /usr/libexec/condor
export PATH=$PATH:/usr/libexec/condor

HADOOP_TARBALL=$1
JOBTRACKER_ENDPOINT=$2

# Note: bin/hadoop uses JAVA_HOME to find the runtime and tools.jar,
#       except tools.jar does not seem necessary therefore /usr works
#       (there's no /usr/lib/tools.jar, but there is /usr/bin/java)
export JAVA_HOME=/usr

# When we get SIGTERM, which Condor will send when
# we are kicked, kill off the datanode and gather logs
function term {
   ./bin/hadoop-daemon.sh stop tasktracker
# Useful if we can transfer data back
#   tar czf logs.tgz logs
#   cp logs.tgz $_CONDOR_SCRATCH_DIR
}

# Unpack
tar xzfv $HADOOP_TARBALL

# Move into tarball, inefficiently
cd $(tar tzf $HADOOP_TARBALL | head -n1)

# Configure,
#  . http.address must be set to port 0 (ephemeral)
cat > conf/mapred-site.xml <<EOF
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>mapred.job.tracker</name>
    <value>$JOBTRACKER_ENDPOINT</value>
  </property>
  <property>
    <name>mapred.task.tracker.http.address</name>
    <value>0.0.0.0:0</value>
  </property>
</configuration>
EOF

# Try to shutdown cleanly
trap term SIGTERM

export HADOOP_CONF_DIR=$PWD/conf
export HADOOP_PID_DIR=$PWD
export HADOOP_LOG_DIR=$_CONDOR_SCRATCH_DIR/logs
./bin/hadoop-daemon.sh start tasktracker

# Wait for pid file
PID_FILE=$(echo hadoop-*-tasktracker.pid)
while [ ! -s $PID_FILE ]; do sleep 1; done
PID=$(cat $PID_FILE)

# Report back some static data about the tasktracker
# e.g. condor_chirp set_job_attr SomeAttr SomeData
# None at the moment.

# While the tasktracker is running, collect and report back stats
while kill -0 $PID; do
   # Collect stats and chirp them back into the job ad
   # None at the moment.
   sleep 30
done

The job description below uses the same templating technique as the DataNode example. The description uses a variable JobTrackerAddress, provided on the command-line as an argument to condor_submit.

mapred_tasktracker.job

# Submit w/ condor_submit -a JobTrackerAddress=<address>
# e.g. <address> = $HOSTNAME:9001

cmd = mapred_tasktracker.sh
args = hadoop-1.0.1-bin.tar.gz $(JobTrackerAddress)

transfer_input_files = hadoop-1.0.1-bin.tar.gz

# RFE: Ability to get output files even when job is removed
#transfer_output_files = logs.tgz
#transfer_output_remaps = "logs.tgz logs.$(cluster).tgz"
output = tasktracker.$(cluster).out
error = tasktracker.$(cluster).err

log = tasktracker.$(cluster).log

kill_sig = SIGTERM

# Want chirp functionality
+WantIOProxy = TRUE

should_transfer_files = yes
when_to_transfer_output = on_exit

requirements = HasJava =?= TRUE

queue

From here you can run condor_submit -a JobTrackerAddress=$HOSTNAME:9001 mapred_tasktracker.job a few times and build up a MapReduce cluster. Assuming you are already running a JobTrackers. You can also hit the JobTracker’s web interface to see TaskTrackers check in.

Assuming you are also running an HDFS instance, you can run some jobs against your new cluster.

For fun, I ran time hadoop jar hadoop-test-1.0.1.jar mrbench -verbose -maps 100 -inputLines 100 three times against 4 TaskTrackers, 8 TaskTrackers and 12 TaskTrackers. The resulting run times were,

# nodes runtime
4 01:26
8 00:50
12 00:38
Advertisements

Service as a Job: HDFS NameNode

April 16, 2012

Scheduling an HDFS DataNode is a powerful function. However, an operational HDFS instance also requires a NameNode. Here is an example of how a NameNode can be scheduled, followed by scheduled DataNodes, to create an HDFS instance.

From here, HDFS instances can be dynamically created on shared resources. Workflows can be built to manage, grow and shrink HDFS instances. Multiple HDFS instances can be deployed on a single set of resources.

The control script is based on hdfs_datanode.sh. It discovers the NameNode’s endpoints and chirps them.

hdfs_namenode.sh

#!/bin/sh -x

# condor_chirp in /usr/libexec/condor
export PATH=$PATH:/usr/libexec/condor

HADOOP_TARBALL=$1

# Note: bin/hadoop uses JAVA_HOME to find the runtime and tools.jar,
#       except tools.jar does not seem necessary therefore /usr works
#       (there's no /usr/lib/tools.jar, but there is /usr/bin/java)
export JAVA_HOME=/usr

# When we get SIGTERM, which Condor will send when
# we are kicked, kill off the namenode
function term {
   ./bin/hadoop-daemon.sh stop namenode
}

# Unpack
tar xzfv $HADOOP_TARBALL

# Move into tarball, inefficiently
cd $(tar tzf $HADOOP_TARBALL | head -n1)

# Configure,
#  . fs.default.name,dfs.http.address must be set to port 0 (ephemeral)
#  . dfs.name.dir must be in _CONDOR_SCRATCH_DIR for cleanup
# FIX: Figure out why a hostname is required, instead of 0.0.0.0:0 for
#      fs.default.name
cat > conf/hdfs-site.xml <<EOF
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://$HOSTNAME:0</value>
  </property>
  <property>
    <name>dfs.name.dir</name>
    <value>$_CONDOR_SCRATCH_DIR/name</value>
  </property>
  <property>
    <name>dfs.http.address</name>
    <value>0.0.0.0:0</value>
  </property>
</configuration>
EOF

# Try to shutdown cleanly
trap term SIGTERM

export HADOOP_CONF_DIR=$PWD/conf
export HADOOP_LOG_DIR=$_CONDOR_SCRATCH_DIR/logs
export HADOOP_PID_DIR=$PWD

./bin/hadoop namenode -format
./bin/hadoop-daemon.sh start namenode

# Wait for pid file
PID_FILE=$(echo hadoop-*-namenode.pid)
while [ ! -s $PID_FILE ]; do sleep 1; done
PID=$(cat $PID_FILE)

# Wait for the log
LOG_FILE=$(echo $HADOOP_LOG_DIR/hadoop-*-namenode-*.log)
while [ ! -s $LOG_FILE ]; do sleep 1; done

# It would be nice if there were a way to get these without grepping logs
while [ ! $(grep "IPC Server listener on" $LOG_FILE) ]; do sleep 1; done
IPC_PORT=$(grep "IPC Server listener on" $LOG_FILE | sed 's/.* on \(.*\):.*/\1/')
while [ ! $(grep "Jetty bound to port" $LOG_FILE) ]; do sleep 1; done
HTTP_PORT=$(grep "Jetty bound to port" $LOG_FILE | sed 's/.* to port \(.*\)$/\1/')

# Record the port number where everyone can see it
condor_chirp set_job_attr NameNodeIPCAddress \"$HOSTNAME:$IPC_PORT\"
condor_chirp set_job_attr NameNodeHTTPAddress \"$HOSTNAME:$HTTP_PORT\"

# While namenode is running, collect and report back stats
while kill -0 $PID; do
   # Collect stats and chirp them back into the job ad
   # Nothing to do.
   sleep 30
done

The job description file is standard at this point.

hdfs_namenode.job

cmd = hdfs_namenode.sh
args = hadoop-1.0.1-bin.tar.gz

transfer_input_files = hadoop-1.0.1-bin.tar.gz

#output = namenode.$(cluster).out
#error = namenode.$(cluster).err

log = namenode.$(cluster).log

kill_sig = SIGTERM

# Want chirp functionality
+WantIOProxy = TRUE

should_transfer_files = yes
when_to_transfer_output = on_exit

requirements = HasJava =?= TRUE

queue

In operation –

Submit the namenode and find its endpoints,

$ condor_submit hdfs_namenode.job
Submitting job(s).
1 job(s) submitted to cluster 208.

$ condor_q -long 208| grep NameNode       
NameNodeHTTPAddress = "eeyore.local:60182"
NameNodeIPCAddress = "eeyore.local:38174"

Open a browser window to http://eeyore.local:60182 to find the cluster summary,

1 files and directories, 0 blocks = 1 total. Heap Size is 44.81 MB / 888.94 MB (5%)
  Configured Capacity                   :        0 KB
  DFS Used                              :        0 KB
  Non DFS Used                          :        0 KB
  DFS Remaining                         :        0 KB
  DFS Used%                             :       100 %
  DFS Remaining%                        :         0 %
 Live Nodes                             :           0
 Dead Nodes                             :           0
 Decommissioning Nodes                  :           0
  Number of Under-Replicated Blocks     :           0

Add a datanode,

$ condor_submit -a NameNodeAddress=hdfs://eeyore.local:38174 hdfs_datanode.job 
Submitting job(s).
1 job(s) submitted to cluster 209.

Refresh the cluster summary,

1 files and directories, 0 blocks = 1 total. Heap Size is 44.81 MB / 888.94 MB (5%)
  Configured Capacity                   :     9.84 GB
  DFS Used                              :       28 KB
  Non DFS Used                          :     9.54 GB
  DFS Remaining                         :   309.79 MB
  DFS Used%                             :         0 %
  DFS Remaining%                        :      3.07 %
 Live Nodes                             :           1
 Dead Nodes                             :           0
 Decommissioning Nodes                  :           0
  Number of Under-Replicated Blocks     :           0

And another,

$ condor_submit -a NameNodeAddress=hdfs://eeyore.local:38174 hdfs_datanode.job
Submitting job(s).
1 job(s) submitted to cluster 210.

Refresh,

1 files and directories, 0 blocks = 1 total. Heap Size is 44.81 MB / 888.94 MB (5%)
  Configured Capacity                   :    19.69 GB
  DFS Used                              :       56 KB
  Non DFS Used                          :    19.26 GB
  DFS Remaining                         :   435.51 MB
  DFS Used%                             :         0 %
  DFS Remaining%                        :      2.16 %
 Live Nodes                             :           2
 Dead Nodes                             :           0
 Decommissioning Nodes                  :           0
  Number of Under-Replicated Blocks     :           0

All the building blocks necessary to run HDFS on scheduled resources.

Service as a Job: HDFS DataNode

April 4, 2012

Building on other examples of services run as jobs, such as Tomcat, Qpidd and memcached, here is an example for the Hadoop Distributed File System‘s DataNode.

Below is the control script for the datanode. It mirrors the memcached’s script, but does not publish statistics. However, datanode statistic/metrics could be pulled and published.

hdfs_datanode.sh

#!/bin/sh -x

# condor_chirp in /usr/libexec/condor
export PATH=$PATH:/usr/libexec/condor

HADOOP_TARBALL=$1
NAMENODE_ENDPOINT=$2

# Note: bin/hadoop uses JAVA_HOME to find the runtime and tools.jar,
#       except tools.jar does not seem necessary therefore /usr works
#       (there's no /usr/lib/tools.jar, but there is /usr/bin/java)
export JAVA_HOME=/usr

# When we get SIGTERM, which Condor will send when
# we are kicked, kill off the datanode and gather logs
function term {
   ./bin/hadoop-daemon.sh stop datanode
# Useful if we can transfer data back
#   tar czf logs.tgz logs
#   cp logs.tgz $_CONDOR_SCRATCH_DIR
}

# Unpack
tar xzfv $HADOOP_TARBALL

# Move into tarball, inefficiently
cd $(tar tzf $HADOOP_TARBALL | head -n1)

# Configure,
#  . dfs.data.dir must be in _CONDOR_SCRATCH_DIR for cleanup
#  . address,http.address,ipc.address must be set to port 0 (ephemeral)
cat > conf/hdfs-site.xml <<EOF
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>fs.default.name</name>
    <value>$NAMENODE_ENDPOINT</value>
  </property>
  <property>
    <name>dfs.data.dir</name>
    <value>$_CONDOR_SCRATCH_DIR/data</value>
  </property>
  <property>
    <name>dfs.datanode.address</name>
    <value>0.0.0.0:0</value>
  </property>
  <property>
    <name>dfs.datanode.http.address</name>
    <value>0.0.0.0:0</value>
  </property>
  <property>
    <name>dfs.datanode.ipc.address</name>
    <value>0.0.0.0:0</value>
  </property>
</configuration>
EOF

# Try to shutdown cleanly
trap term SIGTERM

export HADOOP_CONF_DIR=$PWD/conf
export HADOOP_PID_DIR=$PWD
export HADOOP_LOG_DIR=$_CONDOR_SCRATCH_DIR/logs
./bin/hadoop-daemon.sh start datanode

# Wait for pid file
PID_FILE=$(echo hadoop-*-datanode.pid)
while [ ! -s $PID_FILE ]; do sleep 1; done
PID=$(cat $PID_FILE)

# Report back some static data about the datanode
# e.g. condor_chirp set_job_attr SomeAttr SomeData
# None at the moment.

# While the datanode is running, collect and report back stats
while kill -0 $PID; do
   # Collect stats and chirp them back into the job ad
   # None at the moment.
   sleep 30
done

The job description below uses a templating technique. The description uses a variable NameNodeAddress, which is not defined in the description. Instead, the value is provided as an argument to condor_submit. In fact, a complete job can be defined without a description file, e.g. echo queue | condor_submit -a executable=/bin/sleep -a args=1d, but more on that some other time.

hdfs_datanode.job

# Submit w/ condor_submit -a NameNodeAddress=<address>
# e.g. <address> = hdfs://$HOSTNAME:2007

cmd = hdfs_datanode.sh
args = hadoop-1.0.1-bin.tar.gz $(NameNodeAddress)

transfer_input_files = hadoop-1.0.1-bin.tar.gz

# RFE: Ability to get output files even when job is removed
#transfer_output_files = logs.tgz
#transfer_output_remaps = "logs.tgz logs.$(cluster).tgz"
output = datanode.$(cluster).out
error = datanode.$(cluster).err

log = datanode.$(cluster).log

kill_sig = SIGTERM

# Want chirp functionality
+WantIOProxy = TRUE

should_transfer_files = yes
when_to_transfer_output = on_exit

requirements = HasJava =?= TRUE

queue

hadoop-1.0.1-bin.tar.gz is available from http://archive.apache.org/dist/hadoop/core/hadoop-1.0.1/

Finally, here is a running example,

A namenode is already running –

$ ./bin/hadoop dfsadmin -conf conf/hdfs-site.xml -report
Configured Capacity: 0 (0 KB)
Present Capacity: 0 (0 KB)
DFS Remaining: 0 (0 KB)
DFS Used: 0 (0 KB)
DFS Used%: 0%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
-------------------------------------------------
Datanodes available: 0 (0 total, 0 dead)

Submit a datanode, knowing the namenode’s IPC port is 9000 –

$ condor_submit -a NameNodeAddress=hdfs://$HOSTNAME:9000 hdfs_datanode.job
Submitting job(s).
1 job(s) submitted to cluster 169.

$ condor_q
-- Submitter: eeyore.local : <127.0.0.1:59889> : eeyore.local
 ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD 
 169.0   matt            3/26 15:17   0+00:00:16 R  0   0.0 hdfs_datanode.sh h
1 jobs; 0 idle, 1 running, 0 held

Storage is now available, though not very much as my disk is almost full –

$ ./bin/hadoop dfsadmin -conf conf/hdfs-site.xml -report
Configured Capacity: 63810015232 (59.43 GB)
Present Capacity: 4907495424 (4.57 GB)
DFS Remaining: 4907466752 (4.57 GB)
DFS Used: 28672 (28 KB)
DFS Used%: 0%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
-------------------------------------------------
Datanodes available: 1 (1 total, 0 dead)

Submit 9 more datanodes –

$ condor_submit -a NameNodeAddress=hdfs://$HOSTNAME:9000 hdfs_datanode.job
Submitting job(s).
1 job(s) submitted to cluster 170.
...
1 job(s) submitted to cluster 178.

$ ./bin/hadoop dfsadmin -conf conf/hdfs-site.xml -report
Configured Capacity: 638100152320 (594.28 GB)
Present Capacity: 40399958016 (37.63 GB)
DFS Remaining: 40399671296 (37.63 GB)
DFS Used: 286720 (280 KB)
DFS Used%: 0%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
-------------------------------------------------
Datanodes available: 10 (10 total, 0 dead)

At this point you can run a workload against the storage, visit the namenode at http://localhost:50070, or simply use ./bin/hadoop fs to interact with the filesystem.

Remember, all the datanodes were dispatched by a scheduler, run along side existing workload on your pool, and are completely manageable by standard policies.

Service as a Job: Memcached

December 5, 2011

Running services such as Tomcat or Qpidd show how to schedule and manage a service’s life-cycle via Condor. It is also possible to gather and centralize statistics about a service as it runs. Here is an example of how with memcached.

As with tomcat and qpidd, there is a control script and a job description.

New in the control script for memcached will be a loop to monitor and chirp back statistic information.

memcached.sh

#!/bin/sh

# condor_chirp in /usr/libexec/condor
export PATH=$PATH:/usr/libexec/condor

PORT_FILE=$TMP/.ports

# When we get SIGTERM, which Condor will send when
# we are kicked, kill off memcached.
function term {
   rm -f $PORT_FILE
   kill %1
}

# Spawn memcached, and make sure we can shut it down cleanly.
trap term SIGTERM
# memcached will write port information to env(MEMCACHED_PORT_FILENAME)
env MEMCACHED_PORT_FILENAME=$PORT_FILE memcached -p -1 "$@" &

# We might have to wait for the port
while [ ! -s $PORT_FILE ]; do sleep 1; done

# The port file's format is:
#  TCP INET: 56697
#  TCP INET6: 47318
#  UDP INET: 34453
#  UDP INET6: 54891
sed -i -e 's/ /_/' -e 's/\(.*\): \(.*\)/\1=\2/' $PORT_FILE
source $PORT_FILE
rm -f $PORT_FILE

# Record the port number where everyone can see it
condor_chirp set_job_attr MemcachedEndpoint \"$HOSTNAME:$TCP_INET\"
condor_chirp set_job_attr TCP_INET $TCP_INET
condor_chirp set_job_attr TCP_INET6 $TCP_INET6
condor_chirp set_job_attr UDP_INET $UDP_INET
condor_chirp set_job_attr UDP_INET6 $UDP_INET6

# While memcached is running, collect and report back stats
while kill -0 %1; do
   # Collect stats and chirp them back into the job ad
   echo stats | nc localhost $TCP_INET | \
    grep -v -e END -e version | tr '\r' '\0' | \
     awk '{print "stat_"$2,$3}' | \
      while read -r stat; do
         condor_chirp set_job_attr $stat
      done
   sleep 30
done

A refresher about chirp. Jobs are stored in condor_schedd processes. They are described using the ClassAd language, extensible name value pairs. chirp is a tool a job can use while it runs to modify its classad stored in the schedd.

The job description, passed to condor_submit, is vanilla except for how arguments are passed to memcached.sh. The dollardollar use, see man condor_submit, allows memcached to use as much memory as is available on the slot where it gets scheduled. Slots may have different amounts of Memory available.

memcached.job

cmd = memcached.sh
args = -m $$(Memory)

log = memcached.log

kill_sig = SIGTERM

# Want chirp functionality
+WantIOProxy = TRUE

should_transfer_files = if_needed
when_to_transfer_output = on_exit

queue

An example, note that the set of memcached servers to use is generated from condor_q,

$ condor_submit -a "queue 4" memcached.job
Submitting job(s)....
4 job(s) submitted to cluster 80.

$ condor_q -format "%s\t" MemcachedEndpoint -format "total_items: %d\t" stat_total_items -format "memory: %d/" stat_bytes -format "%d\n" stat_limit_maxbytes
eeyore.local:50608	total_items: 0	memory: 0/985661440
eeyore.local:47766	total_items: 0	memory: 0/985661440
eeyore.local:39130	total_items: 0	memory: 0/985661440
eeyore.local:57410	total_items: 0	memory: 0/985661440

$ SERVERS=$(condor_q -format "%s," MemcachedEndpoint); for word in $(cat words); do echo $word > $word; memcp --servers=$SERVERS $word; \rm $word; done &
[1] 959

$ condor_q -format "%s\t" MemcachedEndpoint -format "total_items: %d\t" stat_total_items -format "memory: %d/" stat_bytes -format "%d\n" stat_limit_maxbytes
eeyore.local:50608	total_items: 480	memory: 47740/985661440
eeyore.local:47766	total_items: 446	memory: 44284/985661440
eeyore.local:39130	total_items: 504	memory: 50140/985661440
eeyore.local:57410	total_items: 490	memory: 48632/985661440

$ condor_q -format "%s\t" MemcachedEndpoint -format "total_items: %d\t" stat_total_items -format "memory: %d/" stat_bytes -format "%d\n" stat_limit_maxbytes
eeyore.local:50608	total_items: 1926	memory: 191264/985661440
eeyore.local:47766	total_items: 1980	memory: 196624/985661440
eeyore.local:39130	total_items: 2059	memory: 204847/985661440
eeyore.local:57410	total_items: 2053	memory: 203885/985661440

$ condor_q -format "%s\t" MemcachedEndpoint -format "total_items: %d\t" stat_total_items -format "memory: %d/" stat_bytes -format "%d\n" stat_limit_maxbytes
eeyore.local:50608	total_items: 3408	memory: 338522/985661440
eeyore.local:47766	total_items: 3542	memory: 351784/985661440
eeyore.local:39130	total_items: 3666	memory: 364552/985661440
eeyore.local:57410	total_items: 3600	memory: 357546/985661440

[1]  + done       for word in $(cat words); do; echo $word > $word; memcp --servers=$SERVERS ; 

Enjoy.

Service as a Job: The Tomcat App Server

February 27, 2011

As seen previously, anything with a life-cycle to be managed can be turned into a job. This time around it’s the Tomcat application server.

A job description that you can give to condor_submit:

cmd = tomcat6.sh
args = conf.tar.gz webapps.tar.gz

transfer_input_files = conf.tar.gz, webapps.tar.gz
when_to_transfer_output = ON_EXIT_OR_EVICT

log = tomcat6.$(cluster).$(process).log

kill_sig = SIGTERM

+WantIOProxy = TRUE

queue

The primary components are the same, a controlling script called tomcat6.sh is the program Condor will run, it will still accept SIGTERM to handle shutdown, and it wants to use chirp to publish information. However, the input parameters are different. The tomcat6.sh controller is taking two parameters, tarballs, and they need to be transferred along with the job, thus the transfer_input_files.

Here is the controlling script, tomcat6.sh:

#!/bin/sh

# Parameters -
#  $1 is a tar.gz of a conf/
#  $2 is a tar.gz of a webapps/
#  $3 is the Catalina service port, often 8080
#  $4 is the Catalina service ssl port, often 8443
#  $5 is the Shutdown port, often 8005
#  $6 is the AJP connector port, often 8009
CONF_TGZ=$1
WEBAPPS_TGZ=$2
CATALINA_PORT=$3
CATALINA_SSL_PORT=$4
SHUTDOWN_PORT=$5
AJP_PORT=$6

# tomcat6 lives in /usr/sbin,
# condor_chirp in /usr/libexec/condor
export PATH=$PATH:/usr/sbin:/usr/libexec/condor

# Home configuration and install location
export CATALINA_HOME=/usr/share/tomcat6

# Base configuration and install location for this instance
export CATALINA_BASE=.

# Pid file, needed to prevent exiting before tomcat6
export CATALINA_PID=$CATALINA_BASE/pid

# When we get SIGTERM, which Condor will send when
# we are kicked, kill off tomcat6.
function term {
    tomcat6 stop
}

function find_free_port {
    local skip=$(netstat -ntl | awk '/^tcp/ {gsub("(.)*:", "", $4); print $4}')
    local ground=10000
    local port=$(($ground + ${RANDOM:0:4}))
    while [ ! -z $(expr "$skip" : ".*\($port\).*") ]; do
	port=$(($ground + ${RANDOM:0:4}))
    done
    echo $port
}

# Make sure ports are set
if [ -z "$CATALINA_PORT" ]; then
    CATALINA_PORT=$(find_free_port)
fi
if [ -z "$CATALINA_SSL_PORT" ]; then
    CATALINA_SSL_PORT=$(find_free_port)
fi
if [ -z "$SHUTDOWN_PORT" ]; then
    SHUTDOWN_PORT=$(find_free_port)
fi
if [ -z "$AJP_PORT" ]; then
    AJP_PORT=$(find_free_port)
fi

# Need logs directory
mkdir -p logs

# Need a temp directory
mkdir -p temp
export CATALINA_TMPDIR=$PWD/temp

# Setup configuration
tar zxfv $CONF_TGZ

# Install webapps
tar zxfv $WEBAPPS_TGZ

echo "Catalina Port: $CATALINA_PORT"
echo "Catalina SSL Port: $CATALINA_SSL_PORT"
echo "Shutdown Port: $SHUTDOWN_PORT"
echo "AJP Port: $AJP_PORT"

# Configure ports
sed -e "s/8005/$SHUTDOWN_PORT/g" -e "s/8080/$CATALINA_PORT/g" \
    -e "s/8009/$AJP_PORT/g" -e "s/8443/$CATALINA_SSL_PORT/g" \
    -i ${CATALINA_BASE}/conf/server.xml

# Spawn tomcat6, and make sure we can shut it down cleanly
trap term SIGTERM
tomcat6 start

# We might have to wait for the pid
while [ ! -s $CATALINA_PID ]; do sleep 1; done
PID=$(cat $CATALINA_PID)
rm -f $CATALINA_PID

# Record port numbers where everyone can see them
# (debug with alias condor_chirp=echo)
condor_chirp set_job_attr CatalinaEndpoint \"$HOSTNAME:$CATALINA_PORT\"
condor_chirp set_job_attr CatalinaSSLEndpoint \"$HOSTNAME:$CATALINA_SSL_PORT\"
condor_chirp set_job_attr ShutdownEndpoint \"$HOSTNAME:$SHUTDOWN_PORT\"
condor_chirp set_job_attr AJPEndpoint \"$HOSTNAME:$AJP_PORT\"

# There are all sorts of useful things that could
# happen here, such as looping and using condor_chirp
# to publish statistics or monitoring for base state.
# The important thing is not to exit until ready to
# shutdown tomcat.
while [ true ]; do
   ps $PID
   if [ $? -eq 0 ]; then
       sleep 15
   else
       echo "Tomcat exited, we are too"
       break
   fi
done

It is hopefully straightforward enough. The important pieces to notice: 1) SIGTERM handler, used for shutting down cleanly; 2) pid file, used to make sure the controller does not exit before the controlled program, Tomcat, does; 3) a handful of setup instructions that are specific to Tomcat and make sure that multiple instances of Tomcat do not conflict with one another on a single machine.

You can test this out by installing tomcat6 on your machine. I did yum install tomcat6 on Fedora 13.

Once that is done, the only thing you have to do is tar up a conf/ directory and a webapps/ directory. They will serve as input to tomcat6.sh. I used /usr/share/tomcat6/conf and /usr/share/tomcat6/webapps, which you can get from the tomcat6-webapps package.

$ tar ztf conf.tar.gz| head -n3
conf/
conf/Catalina/
conf/Catalina/localhost/
$ tar ztf webapps.tar.gz| head -n3 
webapps/
webapps/sample/
webapps/sample/index.html

Submit the app server with:

$ condor_submit tomcat6.sub 
Submitting job(s).
1 job(s) submitted to cluster 10434.
$ condor_submit tomcat6.sub
Submitting job(s).
1 job(s) submitted to cluster 10435.
$ condor_submit tomcat6.sub
Submitting job(s).
1 job(s) submitted to cluster 10436.

And watch them run, you won’t see any output until they start running:

$ condor_q -const 'CatalinaEndpoint =!= UNDEFINED' -format '%d.' ClusterId -format '%d\t' ProcId -format "%s\t" App -format 'http://%s\n' CatalinaEndpoint
10434.0	http://eeyore.local:18814
10435.0	http://eeyore.local:11304
10436.0	http://eeyore.local:12156

You should be able to click on the link to see the default Tomcat install pages.

Simple as that. When you want to take them down just condor_hold or condor_rm the jobs.

For some extra fun, look at how you can parametrize a submission file.

$ cat tomcat6-param.sub 
cmd = tomcat6.sh
args = conf.tar.gz $(APP)

+App="$(APP)"

transfer_input_files = conf.tar.gz, $(APP)
when_to_transfer_output = ON_EXIT_OR_EVICT

log = tomcat6.$(cluster).$(process).log
output = tomcat6.$(cluster).$(process).out
error = tomcat6.$(cluster).$(process).err

kill_sig = SIGTERM

+WantIOProxy = TRUE

queue

I created a simple Hudson webapps tarball.

$ tar ztf hudson.tar.gz 
webapps/
webapps/hudson.war

That you can submit with:

$ condor_submit -a APP=hudson.tar.gz tomcat6-param.sub
Submitting job(s).
1 job(s) submitted to cluster 10437.

Once it starts running you’ll see it in the condor_q output, where it is nicely labeled:

$ condor_q -const 'CatalinaEndpoint =!= UNDEFINED' -format '%d.' ClusterId -format '%d\t' ProcId -format "%s\t" App -format 'http://%s\n' CatalinaEndpoint
10434.0	http://eeyore.local:18814
10435.0	http://eeyore.local:11304
10436.0	http://eeyore.local:12156
10437.0	hudson.tar.gz	http://eeyore.local:11829

Going to the URL directly is not as interesting. It is a Hudson instance, so you must go to http://eeyore.local:11829/hudson.

Service as a Job: The Qpid C++ Broker

May 18, 2010

Yes. A service as a job! Why? Three quick reasons: 1) dynamic, even on-demand/opportunistic, deployment of the service, 2) policy driven control of the service’s execution, 3) abstraction for interacting with service life-cycle

Condor provides strong management, deployment and policy features around what it calls jobs. Jobs come in all shapes and sizes – from those that run for less than a minute (probabilistic simulations) to those that run for months (VMs holding developer desktops) or those that use large amounts of disk or network I/O to those that use large amounts of CPU and memory.

Definitely in that spectrum you’ll find common services, be they full LAMP stacks in VMs or just the Apache HTTP server. Here’s an example of the Qpid broker (qpidd), a messaging service, as a job.

The job description is what you submit with condor_submit:

cmd = qpidd.sh
error = qpidd.log

kill_sig = SIGTERM

# Want chirp functionality
+WantIOProxy = TRUE

queue

It specifies the job, or in this case the service, to run is qpidd.sh, and that SIGTERM should be used to shut it down. qpidd.sh wraps the actual execution of qpidd for one important reason: advertising the qpidd’s endpoint. qpidd will start up on port 5672 by default. That’s all well and good, unless you want to run more than one qpidd on a single machine. qpidd.sh start qpidd up on an ephemeral port, which qpidd kindly prints to stdout, and then advertises the chosen port number back into the Schedd’s queue via condor_chirp, which is available when the job specifies WantIOProxy = TRUE.

#!/bin/sh

# qpidd lives in /usr/sbin,
# condor_chirp in /usr/libexec/condor
export PATH=$PATH:/usr/sbin:/usr/libexec/condor

# When we get SIGTERM, which Condor will send when
# we are kicked, kill off qpidd.
function term {
    rm -f port.out
    kill %1
}

# Spawn qpidd, and make sure we can shut it down cleanly.
rm -f port.out
trap term SIGTERM
# qpidd will print the port to stdout, capture it,
# no auth required, don't read /etc/qpidd.conf,
# log to stderr
qpidd --auth no \
      --config /dev/null \
      --log-to-stderr yes \
      --no-data-dir \
      --port 0 \
      1> port.out &

# We might have to wait for the port on stdout
while [ ! -s port.out ]; do sleep 1; done
PORT=$(cat port.out)
rm -f port.out

# There are all sorts of useful things that could
# happen here, such as setting up queues with
# qpid-config
#...

# Record the port number where everyone can see it
condor_chirp set_job_attr QpiddEndpoint \"$HOSTNAME:$PORT\"

# Nothing more to do, just wait on qpidd
wait %1

In action –

$ condor_submit qpidd.sub
Submitting job(s).
1 job(s) submitted to cluster 2.

$ condor_q
-- Submitter: woods :  : woods
 ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD               
   2.0   matt            5/18 14:21   0+00:00:03 R  0   0.0  qpidd.sh          
1 jobs; 0 idle, 1 running, 0 held

$ condor_q -format "qpidd running at %s\n" QpiddEndpoint
qpidd running at woods:58335

$ condor_hold 2
Cluster 2 held.
$ condor_release 2
Cluster 2 released.

$ condor_q
-- Submitter: woods :  : woods
 ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD               
   2.0   matt            5/18 14:21   0+00:00:33 I  0   73.2 qpidd.sh          
1 jobs; 1 idle, 0 running, 0 held

$ condor_reschedule 
Sent "Reschedule" command to local schedd

$ condor_q         
-- Submitter: woods :  : woods
 ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD               
   2.0   matt            5/18 14:21   0+00:00:38 R  0   73.2 qpidd.sh          
1 jobs; 0 idle, 1 running, 0 held

$ condor_q -format "qpidd running at %s\n" QpiddEndpoint
qpidd running at woods:54028

$ condor_rm -a
All jobs marked for removal.

$ condor_submit qpidd.sub
Submitting job(s).
1 job(s) submitted to cluster 9.

$ condor_submit qpidd.sub                               
Submitting job(s).
1 job(s) submitted to cluster 10.

$ condor_submit qpidd.sub                               
Submitting job(s).
1 job(s) submitted to cluster 11.

$ lsof -i | grep qpidd
qpidd     14231 matt    9u  IPv4 92241655       TCP *:50060 (LISTEN)
qpidd     14256 matt    9u  IPv4 92242129       TCP *:50810 (LISTEN)
qpidd     14278 matt    9u  IPv4 92242927       TCP *:34601 (LISTEN)

$ condor_q -format "qpidd running at %s\n" QpiddEndpoint
qpidd running at woods:34601
qpidd running at woods:50810
qpidd running at woods:50060

%d bloggers like this: