Read the condor_submit manual page for the full gory details.
Submitting jobs to Condor primarily happens through a file being passed to condor_submit. The file is properly called a submit description file. The submit file language is a simplified scripting language. The language outputs a ClassAd, called a job ad, that fully specifies the job — it’s inputs, requirements, policy, how it runs, its data, how its data is handled, credentials required, execution time, etc.
The submit file language consists of a set of commands, macro substitutions and implicit iteration. Nearly all commands in the language can be omitted, and mostly sane defaults will be provided. Conditionals can either be handled while writing the submit file (by a human or script), or for evaluation during scheduling, matching and executing of the job. condor_submit does not evaluate conditionals during ClassAd generation.
The most basic submit file is,
executable = myprogram queue
It consists of just two commands: executable, which defines and performs some basic checks on the program to run; and, queue, which states that a job should be queued, based off the current job ad.
Based off the current job ad? Yes, a submit file is read top to bottom, each command helps to form the job ad that will be queued. For instance,
executable = myprogram queue arguments = a b c queue
This submit file will queue two jobs, one that executes “myprogram” and one that executes “myprogram a b c”. It is pretty simple.
The basic commands you will care about 98% of the time are: executable, arguments, output, error, input, log, and, unfortunately, should_transfer_files and when_to_transfer_output.
executable you already know, it defines the executable to be run as part of the job. arguments is already clear, it defines the arguments given to the executable. output specifies the file where the executable’s stdout is written. error specifies the file where the executable’s stderr is written. log specifies a file where condor writes information about the execution of the job, i.e. where/when it was submitted, where/when it ran, how it exited, etc. log is optional like perl’s -w.
Sane defaults, that are not actually defaults in 7.6, for should_transfer_files and when_to_transfer_output are IF_NEEDED and ON_EXIT, respectively.
The macro substitutions are also available top to bottom and must be defined before the first queue command where they are used. Macro definitions look like commands and commands can be treated as macros. Macro substitution happens with $().
mybaseargs = base executable = myprogram arguments = $(mybaseargs) a queue arguments = $(mybaseargs) $(executable) b queue
This will submit two jobs executing “myprogram base a” and “myprogram base myprogram b”. “mybaseargs = base” defines the macro “mybaseargs” and allows for the substitution with “$(mybaseargs)”. executable is the familiar command, but also allows the substitution “$(executable)”.
executable = myprogram arguments = $(mybaseargs) a mybaseargs = base queue
Also works, but the following will not.
executable = myprogram arguments = $(mybaseargs) a queue mybaseargs = base
The macro must be defined before the queue command where it is used.
Implicit iteration is driven by the queue command. The queue command actually takes an argument, a number. This is the number of times the current job ad will be queued.
executable = myprogram queue 2
This creates two jobs, both running “myprogram”. Boring.
This might not be very useful, many copies of the same command, were there no way to make each queued job slightly different. condor_submit maintains two special macros, think of them as loop indices, and queue performs the iteration. $(Process) and $(Cluster) are the indices. $(Process) is zero-based. It starts at 0 and proceeds to one minus the argument to queue. $(Cluster) is the next available identifier available from the Schedd. It could be any 32 bit number, and may be incremented during the submission (mostly if you have multiple executable commands). Typically $(Process) is generally helpful. $(Cluster) is useful if you plan to submit the same file multiple times.
executable = myprogram arguments = $(Process) queue 2
This creates two jobs, “myprogram 0” and “myprogram 1”. Pulling it all together,
$ cat > example.sub command = echo executable = /bin/$(command) arguments = $(Process) log = $(command).log output = $(command).$(Cluster).$(Process).out error = $(command).$(Cluster).$(Process).out queue 2 ^D $ condor_submit example.sub Submitting job(s).. 2 job(s) submitted to cluster 88. $ cat echo.88.0.out 0 $ cat echo.88.1.out 1 $ cat echo.log 000 (088.000.000) 07/04 08:00:80 Job submitted from host: ... 000 (088.001.000) 07/04 08:00:80 Job submitted from host: ... 001 (088.000.000) 07/04 08:00:80 Job executing on host: ... 001 (088.001.000) 07/04 08:00:80 Job executing on host: ... 005 (088.000.000) 07/04 08:00:80 Job terminated. (1) Normal termination (return value 0) Usr 0 00:00:00, Sys 0 00:00:00 - Run Remote Usage Usr 0 00:00:00, Sys 0 00:00:00 - Run Local Usage Usr 0 00:00:00, Sys 0 00:00:00 - Total Remote Usage Usr 0 00:00:00, Sys 0 00:00:00 - Total Local Usage 0 - Run Bytes Sent By Job 0 - Run Bytes Received By Job 0 - Total Bytes Sent By Job 0 - Total Bytes Received By Job ... 005 (088.001.000) 07/04 08:00:80 Job terminated. (1) Normal termination (return value 0) Usr 0 00:00:00, Sys 0 00:00:00 - Run Remote Usage Usr 0 00:00:00, Sys 0 00:00:00 - Run Local Usage Usr 0 00:00:00, Sys 0 00:00:00 - Total Remote Usage Usr 0 00:00:00, Sys 0 00:00:00 - Total Local Usage 0 - Run Bytes Sent By Job 0 - Run Bytes Received By Job 0 - Total Bytes Sent By Job 0 - Total Bytes Received By Job ...
One more advanced feature is the ability to output directly into the job ad. Remember each command performs some action and incrementally constructs the job ad. The executable command does some file checks and then inserts the Cmd attribute into the job ad.
$ condor_q -format "Cmd = %s\n" Cmd 88.0 Cmd = /bin/echo
The direct insertion works with a +. It starts to demonstrate the descriptive power underlying Condor, and also requires some knowledge of the underlying ClassAd language. For instance, jobs can be arbitrarily attributed, even with attributes Condor will never use directly. The types of the values are important, mostly, strings must be quoted.
executable = myprogram +Submission = "Submission A" queue +Submission = "Submission B" queue 2
This submits three jobs, all with an attribute named Submission, two with a value “Submission B” and one with “Submission A”.
$ condor_q -format "%d " ProcId -format "%s\n" Submission 0 Submission A 1 Submission B 2 Submission B $ condor_q -constraint 'Submission == "Submission A"' -- Submitter: eeyore.local : : eeyore.local ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 89.0 matt 7/04 08:08 0+00:00:00 I 0 0.0 myprogram
Notice, the $(Process) macro actually corresponds to the ProcId attribute on the job ad.
Tags: Condor, Submission, Tutorial
Spinning 
September 16, 2011 at 5:29 am |
[…] Submitting individual jobs through Condor’s various interfaces is, unsurprisingly, the first thing people do. A quick second is submitting DAGs. I have previously discussed this in Java with BirdBath. […]