Custom ClassAd attributes in Condor: FreeMemoryMB via STARTD_CRON

ClassAds enable a lot of power in Condor. ClassAds are the data representation used for most everything in Condor. Generally, a ClassAd is a set of name-value pairs. Values are typed, and include the usual suspects plus expressions. In Condor, jobs, users, slots, daemons, etc. all have a ClassAd representation with their own set of attributes, e.g. a job has a Requirements attribute whose value is an expression such as FreeMemoryMB > 1024, and a slot has a Memory attribute whose value is the amount of memory allocated to the slot.

ClassAds are schema-free, which means they can be arbitrarily extended. Any given ad can have any attribute a user or administrator wants to put in it. Attributes are given meaning by when and where they are referenced.

On a slot there are different classes of attributes. Resource statistics, such as Disk and LoadAvg; resource properties, such as NumCpus and HasVM; policy expressions, such as Requirements or Rank; etc.

One attribute not provided by default on a slot is a statistic representing the total amount of free memory on the system. There are multiple interpretations of what free memory might really mean. On Linux, it could be the number in the free column on the Mem: row reported by the free program, e.g.

$ free -m
             total       used       free     shared    buffers     cached
Mem:          2016       1790        225          0         93        845
-/+ buffers/cache:        851       1165
Swap:         1983          0       1983

or it might be the value on the buffers/cache line. It might be some combination of totalram, freeram, sharedram, bufferedram as reported by sysinfo(2). It might also include information about totalswap and freeswap, also from sysinfo(2).

The meaning is often a function of what kinds of policies are desired in a Condor deployment.

Once you have picked a meaning for your deployment, Condor provides you with the STARTD_CRON mechanism to include a FreeMemoryMB attribute in your slot ads. From there the attribute can be referenced by policy on jobs, during negotiation, and slot policy.

You need two things: first, a way to calculate the value for FreeMemoryMB, we’ll use the simple bash script below that pulls FreeMem: out of /proc/meminfo; second, configuration available to the condor_startd to run the program.

free_memory_mb.sh:

#!/bin/sh

FREE_MEMORY_KB=$(grep ^MemFree < /proc/meminfo | awk '{print $2}')
echo "FreeMemoryMB = $((FREE_MEMORY_KB / 1024))"

condor_config.local:

STARTD_CRON_JOBLIST = FREE_MEMORY_MB
STARTD_CRON_FREE_MEMORY_MB_EXECUTABLE = $(LIBEXEC)/free_memory_mb.sh
STARTD_CRON_FREE_MEMORY_MB_PERIOD = $(UPDATE_INTERVAL)s

Notes: First, the documentation is out of sync with 7.4.0 and the units on _PERIOD must be specified; second, UPDATE_INTERVAL needs to be defined, it specifies how often the condor_startd will periodically send updates to the Collector.

After reconfiguring the Startd, or just restarting condor, you can view the new attribute with condor_status:

$ condor_status -long  | grep ^FreeMemoryMB | sort | uniq -c
     10 FreeMemoryMB = 174

$ free -m
             total       used       free     shared    buffers     cached
Mem:          2016       1842        173          0        101        874
-/+ buffers/cache:        866       1149
Swap:         1983          0       1983

Yes, the values are different between FreeMemoryMB and free. The amount of free memory is changing constantly and we are just sampling it. You can increase the sampling rate, but beware that means you will generate more frequent updates to the Collector. Maybe not a problem when you have 32 machines and 256 slots, but definitely something to consider when you have 3000 machines and 24000 slots.

Final note: A better name for the attribute representing free memory on a system is TotalFreeMemoryMB to remain consistent with other attributes. For instance, Disk is a slot’s share of the TotalDisk free on the system.

Submitting a workflow to Condor via SOAP using Java

In Condor, a workflow is called a DAG. DAG stand for directed acyclic graph. DAGs in Condor are managed by processes called condor_dagman. condor_dagman is a program that takes a description of a DAG as input and walks it; submitting jobs and monitoring their progress. The condor_dagman process itself is often submitted to Condor and managed by the Schedd like any other job. The only thing special about condor_dagman jobs is that they are run on the same machine as the Schedd. Typically in Condor’s Scheduler Universe. Historical note: the first job submitted to Condor via SOAP was a DAG.

Here’s a recipe for submitting a DAG to Condor via SOAP:

0) Get your favorite SOAP library. Here we’ll use Apache Axis: http://www.apache.org/dyn/closer.cgi/ws/axis/1_4

You’ll want to untar it:

$ tar zxf axis-bin-1_4.tar.gz

1) Generate and compile your SOAP library code

# To avoid passing -classpath to java and javac, set CLASSPATH in your env
$ export CLASSPATH=.:$(echo axis-1_4/lib/*.jar | tr ' ' ':')

# Have Axis generate Java stubs
$ java org.apache.axis.wsdl.WSDL2Java file:///usr/share/condor/webservice/condorSchedd.wsdl

# Compile the stubs
$ javac condor/*.java

2) Write a DAG. This one is pretty simple, it’s a diamond: D depends on B and C, B and C depend on A:

~/dagman/diamond.dag:
Job  A  A.submit 
Job  B  B.submit 
Job  C  C.submit        
Job  D  D.submit
PARENT A CHILD B C
PARENT B C CHILD D

~/dagman/A.submit:
executable = /bin/sleep
arguments = 112
output = A.out
log = diamond.log
queue

~/dagman/B.submit:
executable = /bin/sleep
arguments = 358
output = B.out
log = diamond.log
queue

~/dagman/C.submit:
executable = /bin/sleep
arguments = 132
output = C.out
log = diamond.log
queue

~/dagman/D.submit:
executable = /bin/sleep
arguments = 134
output = D.out
log = diamond.log
queue

3) Write a program that will submit your DAG via SOAP.

The Schedd’s Submit() function takes a ClassAd representing a job. What condor_submit does is take a submit file and convert it into a ClassAd. To look at a submit file for a DAG run condor_submit_dag -no_submit diamond.dag and read diamond.dag.condor.sub. That will give you a hint at what kind of environment condor_dagman wants to run in. Note: remove_kill_sig, arguments, environment and on_exit_remove.

However, diamond.dag.condor.sub is not a ClassAd. To see the ClassAd you can run diamond.dag.condor.sub through condor_submit. Do so with condor_submit -dump diamond.dag.condor.sub.ad diamond.dag.condor.sub. Have a look at diamond.dag.condor.sub.ad and note RemoveKillSig, Arguments, Env and OnExitRemove.

Now you have the basis for what a DAG job needs to run. In the example code I used a little extra knowledge to generate the ClassAd, for brevity. You can use the Schedd’s CreateJobTemplate function to help generate the ClassAd for you instead. Extending arrays is kinda annoying in Java, but cake in python.

Start with this example: CondorSubmitDAG.java

4) Configure your condor_schedd to accept SOAP requests. The basic configuration you will need is:

ENABLE_SOAP = TRUE
ALLOW_SOAP = *
QUEUE_ALL_USERS_TRUSTED = TRUE
SCHEDD_ARGS = -p 1984

This configuration lets anyone talk to your Schedd and submit jobs as any user. For deployment, you should restrict this access by narrowing the ALLOW_SOAP and by setting QUEUE_ALL_USERS_TRUSTED to FALSE. Note: changing QUEUE_ALL_USERS_TRUSTED requires that clients can authenticate themselves via SSL.

This configuration also gives you a fixed port, 1984, for the condor_schedd. Otherwise the port is ephemeral, and you’ll have to query the Collector to find it.

5) Compile your submission program and submit your DAG.

$ javac CondorSubmitDAG.java

$ java CondorSubmitDAG http://localhost:1984 soapmonkey /some/shared/space/where/you/put/dagman/diamond.dag

You can watch the DAG run with condor_q, and condor_q -dag. Don’t be afraid of the —????— for the DAG itself, that’s just because I pruned the job ad to the bare minimum to run. condor_q expects a few extra attributes to be present. If you use CreateJobTemplate you’ll get all the attributes condor_q wants.