NFS and Job Initial Working Directory (Iwd)

Condor deployments tend to include a network file system, such as NFS, AFS or SMB, which allows users easy access to their files across many machines. The presence of such file systems also means that a user can skip using Condor’s file transfer mechanisms and have their jobs write output or read input directly from the networked locations, often the user’s home directory. Condor is more than happy to do this, as long as the user’s credentials are available to access the home directory, which is often the case. Condor will even go one step further.

Sometime in the past, a user was automating their job submission to Condor, similar to what DAGMan does, and ran into a problem when their files were written to NFS. Their meta-scheduler, as they’re called, was reading job output files and getting stale cached data. This meant the job may have completed but the machine on which the meta-scheduler was running only saw part of the output. To get around this issue the condor_schedd, which in this case was managing jobs for the meta-scheduler, was changed to try and flush the NFS cache for the job’s Iwd. When a job completes the Schedd checks to see if the Iwd is on NFS, and if so creates a temporary file that is immediately deleted. The Schedd’s log reports “Forcing NFS sync of Iwd” and a .condor_nfs_sync_XXXXXX file briefly lives in the Iwd. This of course has pros and cons.

On the plus side, this is helpful to meta-schedulers because now they never have to bother making sure data sources aren’t stale. Arguably the meta-scheduler should be fixed in this situation. On the negative side, all jobs that have an Iwd in NFS now incur a penalty in the form of some NFS round trips when they complete. This penalty can actually be very dramatic, even halving the number of jobs a single Schedd can complete in a second.

To address the performance hit, in Condor 7.4, the IwdFlushNFSCache job attribute was introduced. It defaults to True, and can be changed in a submit file with +IwdFlushNFSCache = False or for all new jobs with IwdFlushNFSCache = False followed by SUBMIT_EXPRS = IwdFlushNFSCache in configuration. As expected, IwdFlushNFSCache works as a guard to the code in the condor_schedd that flushes the Iwd on job completion.

Maybe in future versions of Condor (7.5+) the default will become False and those who need the cache flushing functionality will place +IwdFlushNFSCache = True in their submit files.

Tags: Cache, Condor, Iwd, NFS, Performance