Often people will think of Condor’s Negotiator as strictly matching a job to a resource. This is only conceptually true, and gives a 1:1 job:resource impression that is easy to reason about. In reality the match is N:1. The Negotiator actually creates matches between a job prototype (sometimes called a resource request or an auto cluster) with a resource. The difference may be subtle, but it is powerful. There are two obvious benefits: increased performance in generating matches; and, decreased latency in starting jobs.
The matchmaking algorithms the Negotiator uses to create matches are on the order of N*M in the worst case, where N is the number of jobs in the system and M is the number of resources in the system. N can be a very large number, often in the millions, and M is often not small, though it tends to be in the thousands to tens of thousands range. Drop me a note if you run a pool with >100K cores.
As you know, resources and jobs have requirements on each other. Those requirements (and preferences) reference only a subset of attributes that may be on a job. While there are N jobs in the system, when only looking at attributes that play a role in matchmaking policies, it is very rare for all N jobs to be unique.
For instance, submitting 1,000 jobs in a parameter sweep will result in 1,000 unique jobs when you consider all attributes on the jobs, but maybe only 1 job prototype when you throw out the Args attribute. And, if you don’t reference Args within requirements expressions, which you surely do not, it is safe to throw it out. All of a sudden the N=1000 becomes N=1.
It is often the case that in a system with millions of jobs there may only be thousands of job prototypes.
To take advantage of this compression, when a match is generated, the Negotiator gives a claim to a Schedd. That claim is for a resource and embodies the type of job that can be run on the resource. As we now know, there may be many such jobs at the Schedd. The Schedd is free to reuse the claim, within reason.
It is through the reuse that the second benefit becomes clear. A job on the Schedd does not have to wait for a negotiation cycle before it can be run. It only has to wait for a compatible claim to free up. Of course, if all the claims are used, then the job may wait for a negotiation cycle to provide more claims.
As a side note, claim reuse provides dramatic wins for very short running jobs.
Hold on, “within reason”? Yes, as for anything in a distributed system that provides ownership, a claim must have strong semantics. It would not be desirable to give a Schedd a claim and never get it back, which might happen if a Schedd was always busy or if it simply goes away (crashes, gets powered off, etc). Condor allows for an arbitrary number of Schedds to exist in the system. Sometimes one for each user. If all resources are claimed by existing Schedds and a new Schedd appears, well, you can imagine.
Enter the lease. To combat this issue, a claim comes with a lease, which specifies the amount of time the Schedd should expect to have exclusive access to the claimed resource. Ultimately, the the resource controls access. After any job completes, the resource can opt to terminate the lease, and certainly will if the lease has expired. Termination means the Schedd should not try to use the claim anymore. All attempts to use it will be rejected.
Tags: Claims, Condor, Distributed Systems, Leases
Spinning 
Leave a comment