Ordering Jobs: Job Priority and Submitters

Ordering jobs. The order jobs are put in for execution is scoped within a Schedd and then within a Submitter. A Submitter is the job’s Owner or AccountingGroup. This means that no order is specified between jobs of different submitters on a single Schedd, and jobs of even the same submitter on different Schedds are not ordered with respect to one another. Adjustments in the order does not impact already running jobs.

The default order operates similarly to FCFS. Jobs that are submitted earlier than others get a chance to run before later jobs. Specifically, jobs with a lower QDate run before those with a higher. If QDates are equal, jobs with a lower ClusterId run before those with a higher. If ClusterIds are equal, jobs with a lower ProcId run before those with a higher.

The order is similar to FCFS instead of exactly FCFS. A job that is not runnable, because it is on Hold or Requirements are not satisfied, will be skipped until it becomes runnable.

A job’s priority, its JobPrio attribute, provides a way to alter the default ordering. It is considered first, i.e. before QDate. If JobPrios are equal, order continues base on QDate etc. JobPrio works in the opposite direction from other ordering attribute. Jobs with a larger value run first, e.g. JobPrio = 1 runs before 0 which in turn runs before -1.

The JobPrio attribute of a job is controlled by the priority command in a submit file, and defaults to a value of 0. It can also be adjusted using the condor_prio command-line utility. If using a UserLog, the log command in a submit file, which you should be doing, you can see priority adjustments as they happen. Keep in mind that the JobPrio only helps order jobs to the point that they start running. Once running, adjustment to a lower priority will not cause them to stop and higher priority jobs to run.

Let’s get concrete. Observing execution order via a UserLog, using a single slot (NUM_CPUS=1), stopping the Negotiator, submitting jobs, then starting the Negotiation, let’s consider: 0) a single user submitting with priority 1) then adjusting with condor_prio, 2) multiple users submitting with priority, 3) a single user submitting with multiple AccountingGroups adjusting priority with condor_prio, 4) multiple users submitting with a single AccountingGroup adjusting priority with condor_prio.

The Negotiator stopping and starting is to make sure all jobs get considered at the same time. Without it, you may not get the same results.

NOTE: Endpoints, dates and hours anonymized to protect the guilty.


Zero: a single user submitting with priority

$ cat > zero.job << EOF
cmd = /bin/sleep
args = 1
log = zero.log
priority = -1
queue
priority = 0
queue
priority = 1
queue
EOF

$ condor_submit zero.job
Submitting job(s)...
3 job(s) submitted to cluster 1.

$ grep -e ^000 -e ^001 zero.log
000 (001.000.000) 03/17 01:40:55 Job submitted from host: 
000 (001.001.000) 03/17 01:40:55 Job submitted from host: 
000 (001.002.000) 03/17 01:40:55 Job submitted from host: 
001 (001.002.000) 03/17 01:41:08 Job executing on host: 
001 (001.001.000) 03/17 01:41:10 Job executing on host: 
001 (001.000.000) 03/17 01:41:13 Job executing on host: 

Success, execution of 1.2 then 1.1 then 1.0.


One: a single user submitting with priority, then adjusting with condor_prio

$ condor_off -negotiator

$ condor_submit -a log=one.log zero.job
Submitting job(s)...
3 job(s) submitted to cluster 1.

$ condor_prio 1.1 -p 3

$ condor_on -negotiator

$ grep -e ^000 -e ^001 -e ^033 one.log
000 (001.000.000) 03/17 01:46:40 Job submitted from host: 
000 (001.001.000) 03/17 01:46:40 Job submitted from host: 
000 (001.002.000) 03/17 01:46:40 Job submitted from host: 
033 (001.001.000) 03/17 01:47:02 Changing job attribute JobPrio from 0 to 3
001 (001.001.000) 03/17 01:47:50 Job executing on host: 
001 (001.002.000) 03/17 01:47:53 Job executing on host: 
001 (001.000.000) 03/17 01:47:55 Job executing on host: 

Success, priority change is visible for 1.1, and execution follows 1.1 then 1.2 then 1.0. Also, the change is priority for 1.1 is visible.


Two: multiple users submitting with priority

$ cat > two.job << EOF
cmd = /bin/sleep
args = 1
log = two.log
priority = -1
queue
priority = 0
queue
priority = 1
queue
EOF

$ condor_off -negotiator

userA $ condor_submit two.job
Submitting job(s)...
3 job(s) submitted to cluster 1.

userB $ condor_submit two.job
Submitting job(s)...
3 job(s) submitted to cluster 2.

$ condor_on -negotiator

userA $ grep -e ^000 -e ^001 two.log 
000 (001.000.000) 03/17 01:34:20 Job submitted from host: 
000 (001.001.000) 03/17 01:34:20 Job submitted from host: 
000 (001.002.000) 03/17 01:34:20 Job submitted from host: 
001 (001.002.000) 03/17 01:35:15 Job executing on host: 
001 (001.001.000) 03/17 01:35:17 Job executing on host: 
001 (001.000.000) 03/17 01:35:19 Job executing on host: 

userB $ grep -e ^000 -e ^001 two.log 
000 (002.000.000) 03/17 01:34:21 Job submitted from host: 
000 (002.001.000) 03/17 01:34:21 Job submitted from host: 
000 (002.002.000) 03/17 01:34:21 Job submitted from host: 
001 (002.002.000) 03/17 01:35:35 Job executing on host: 
001 (002.001.000) 03/17 01:35:38 Job executing on host: 
001 (002.000.000) 03/17 01:35:40 Job executing on host: 

Success, jobs ran in priority order and the priority is scoped to the user – you did not see 1.2 -> 2.2 -> 1.1 -> 2.1 -> 2.0 -> 1.0. Why though did 1.x run before 2.x? Because of a different type of priority: user priority.

$ condor_userprio
Last Priority Update:  3/17 01:36
                                    Effective
User Name                           Priority 
------------------------------      ---------
userA@localhost                          0.50
userB@localhost                          0.50
------------------------------      ---------
Number of users shown: 2                           

That means they can run in any order decided by the Negotiator. A topic for another time is how user priority works, but a hint is,

$ condor_userprio -setprio userA@localhost 100
The priority of userA@localhost was set to 100.000000

$ condor_userprio
Last Priority Update:  3/17 01:36
                                    Effective
User Name                           Priority 
------------------------------      ---------
userB@localhost                          0.50
userA@localhost                        100.00
------------------------------      ---------
Number of users shown: 2

$ condor_off -negotiator

userA $ condor_submit two.job
Submitting job(s)...
3 job(s) submitted to cluster 3.

userB $ condor_submit two.job
Submitting job(s)...
3 job(s) submitted to cluster 4.

$ condor_on -negotiator

userA $ grep -e ^000 -e ^001 two.log 
000 (003.000.000) 03/17 01:36:49 Job submitted from host: 
000 (003.001.000) 03/17 01:36:49 Job submitted from host: 
000 (003.002.000) 03/17 01:36:49 Job submitted from host: 
001 (003.002.000) 03/17 01:37:22 Job executing on host: 
001 (003.001.000) 03/17 01:37:24 Job executing on host: 
001 (003.000.000) 03/17 01:37:26 Job executing on host: 

userB $ grep -e ^000 -e ^001 two.log 
000 (004.000.000) 03/17 01:36:51 Job submitted from host: 
000 (004.001.000) 03/17 01:36:51 Job submitted from host: 
000 (004.002.000) 03/17 01:36:51 Job submitted from host: 
001 (004.002.000) 03/17 01:37:02 Job executing on host: 
001 (004.001.000) 03/17 01:37:04 Job executing on host: 
001 (004.000.000) 03/17 01:37:06 Job executing on host: 

Success, userB’s jobs ran in priority order before userA’s jobs.


Three: a single user submitting with multiple AccountingGroups adjusting priority with condor_prio

$ cat > three-a.job < three-b.job << EOF
cmd = /bin/sleep
args = 1
log = three.log
+AccountingGroup = "three.b"
priority = -3
queue
priority = 0
queue
priority = 3
queue
EOF

$ condor_off -negotiator

$ condor_submit three-a.job
Submitting job(s)...
3 job(s) submitted to cluster 1.

$ condor_submit three-b.job
Submitting job(s)...
3 job(s) submitted to cluster 2.

$ condor_prio 2.1 -p 1

$ condor_on -negotiator

$ grep -e ^000 -e ^001 -e ^033 three.log
000 (001.000.000) 03/17 01:56:15 Job submitted from host: 
000 (001.001.000) 03/17 01:56:15 Job submitted from host: 
000 (001.002.000) 03/17 01:56:15 Job submitted from host: 
000 (002.000.000) 03/17 01:56:17 Job submitted from host: 
000 (002.001.000) 03/17 01:56:17 Job submitted from host: 
000 (002.002.000) 03/17 01:56:17 Job submitted from host: 
033 (002.001.000) 03/17 01:56:57 Changing job attribute JobPrio from 0 to 1
001 (001.002.000) 03/17 01:57:08 Job executing on host: 
001 (001.001.000) 03/17 01:57:11 Job executing on host: 
001 (001.000.000) 03/17 01:57:13 Job executing on host: 
001 (002.002.000) 03/17 01:57:29 Job executing on host: 
001 (002.001.000) 03/17 01:57:31 Job executing on host: 
001 (002.000.000) 03/17 01:57:33 Job executing on host: 

Success, jobs ran in priority order and the priority is scoped to the AccountingGroup even though they are from the same user. Why though did 1.x run before 2.x? Because of group priority, same thing as user priority. Groups three.a and three.b have the same global priority, stored with the Negotiator.

$ condor_userprio 
Last Priority Update:  3/17  02:00
                                    Effective
User Name                           Priority 
------------------------------      ---------
three.a@localhost                     0.50
three.b@localhost                     0.50
------------------------------      ---------
Number of users shown: 2

That means they can run in any order decided by the Negotiator.

$ condor_userprio -setprio three.a@localhost 100
The priority of three.a@localhost was set to 100.000000

$ condor_userprio
Last Priority Update:  3/17  02:01
                                    Effective
User Name                           Priority
------------------------------      ---------
three.b@localhost                     0.50
three.a@localhost                   100.00
------------------------------      ---------
Number of users shown: 2

$ condor_off -negotiator

$ condor_submit three-a.job
Submitting job(s)...
3 job(s) submitted to cluster 3.

$ condor_submit three-b.job
Submitting job(s)...
3 job(s) submitted to cluster 4.

$ condor_on -negotiator

$ grep -e ^000 -e ^001 three.log
000 (003.000.000) 03/17 02:01:50 Job submitted from host: 
000 (003.001.000) 03/17 02:01:50 Job submitted from host: 
000 (003.002.000) 03/17 02:01:50 Job submitted from host: 
000 (004.000.000) 03/17 02:01:51 Job submitted from host: 
000 (004.001.000) 03/17 02:01:51 Job submitted from host: 
000 (004.002.000) 03/17 02:01:51 Job submitted from host: 
001 (004.002.000) 03/17 02:01:58 Job executing on host: 
001 (004.001.000) 03/17 02:02:00 Job executing on host: 
001 (004.000.000) 03/17 02:02:03 Job executing on host: 
001 (003.002.000) 03/17 02:02:18 Job executing on host: 
001 (003.001.000) 03/17 02:02:20 Job executing on host: 
001 (003.000.000) 03/17 02:02:23 Job executing on host: 

Success, three.b’s jobs ran in priority order before three.a’s jobs.


Four: multiple users submitting with a single AccountingGroup adjusting priority with condor_prio

$ cat > four.job << EOF
cmd = /bin/sleep
args = 1
log = four.log
+AccountingGroup = "four.user"
priority = -1
queue
priority = 0
queue
priority = 1
queue
EOF

$ condor_off -negotiator

userA $ condor_submit four.job
Submitting job(s)...
3 job(s) submitted to cluster 1.

userB $ condor_submit four.job
Submitting job(s)...
3 job(s) submitted to cluster 2.

$ condor_q
-- Submitter: localhost :  : localhost
 ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD               
   1.0   userA           3/17  02:05   0+00:00:00 I  -1  0.0  sleep 1
   1.1   userA           3/17  02:05   0+00:00:00 I  0   0.0  sleep 1
   1.2   userA           3/17  02:05   0+00:00:00 I  1   0.0  sleep 1
   2.0   userB           3/17  02:05   0+00:00:00 I  -1  0.0  sleep 1
   2.1   userB           3/17  02:05   0+00:00:00 I  0   0.0  sleep 1
   2.2   userB           3/17  02:05   0+00:00:00 I  1   0.0  sleep 1
6 jobs; 6 idle, 0 running, 0 held

userB $ condor_prio 2.1 -p 2
userB $ condor_prio 2.2 -p 3
userA $ condor_prio 1.0 -p -2

$ condor_q
-- Submitter: localhost :  : localhost
 ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD               
   1.0   userA           3/17  02:05   0+00:00:00 I  -2  0.0  sleep 1           
   1.1   userA           3/17  02:05   0+00:00:00 I  0   0.0  sleep 1           
   1.2   userA           3/17  02:05   0+00:00:00 I  1   0.0  sleep 1           
   2.0   userB           3/17  02:05   0+00:00:00 I  -1  0.0  sleep 1           
   2.1   userB           3/17  02:05   0+00:00:00 I  2   0.0  sleep 1           
   2.2   userB           3/17  02:05   0+00:00:00 I  3   0.0  sleep 1           
6 jobs; 6 idle, 0 running, 0 held

$ condor_on -negotiator

userA $ grep -e ^000 -e ^001 -e ^033 four.log
000 (001.000.000) 03/17 02:05:42 Job submitted from host: 
000 (001.001.000) 03/17 02:05:42 Job submitted from host: 
000 (001.002.000) 03/17 02:05:42 Job submitted from host: 
033 (001.000.000) 03/17 02:06:14 Changing job attribute JobPrio from -1 to -2
001 (001.002.000) 03/17 02:06:24 Job executing on host: 
001 (001.001.000) 03/17 02:06:26 Job executing on host: 
001 (001.000.000) 03/17 02:06:29 Job executing on host: 

userB $ grep -e ^000 -e ^001 -e ^033 four.log
000 (002.000.000) 03/17 02:05:44 Job submitted from host: 
000 (002.001.000) 03/17 02:05:44 Job submitted from host: 
000 (002.002.000) 03/17 02:05:44 Job submitted from host: 
033 (002.001.000) 03/17 02:05:50 Changing job attribute JobPrio from 0 to 2
033 (002.002.000) 03/17 02:05:57 Changing job attribute JobPrio from 1 to 3
001 (002.002.000) 03/17 02:06:21 Job executing on host: 
001 (002.001.000) 03/17 02:06:23 Job executing on host: 
001 (002.000.000) 03/17 02:06:28 Job executing on host: 

Success, execution went 2.2 (06:21) -> 2.1 (06:23) -> 1.2 (06:24) -> 1.1 (06:26) -> 2.0 (06:28) -> 1.0 (06:29).

Advertisements

Tags: , , ,

3 Responses to “Ordering Jobs: Job Priority and Submitters”

  1. Sivarama Raju Says:

    Hello,

    I have a condor pool with 3 fedora linux pc’s. I can run jobs on individual machines using condor_submit. But I cant run jobs remotely. Whenever I run condor_submit , it runs locally. Can you please let me know how can I do that.

    I removed stard daemon from the machine, on which I am submitting job, and I was guessing that would send the job to another machine which has startd, in the pool. But that is not happening, the job just goes idle.

    Also can you please let me know how to install and configure a checkpoint server.

    Thanks,
    Siva

  2. spinningmatt Says:

    You should follow https://spinningmatt.wordpress.com/2011/06/12/getting-started-creating-a-multiple-node-condor-pool/

  3. Sivarama Raju Says:

    Thanks a lot Spinningmatt. Its working now :-)

    Can you also help me with checkpoint server install and configuration, i dont have checkpoint server binary with me. Where can I download that ? I searched Condor website, but could not find the same.

    Thanks in advance,
    Siva

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: