Posts Tagged ‘Clock skew’

Detecting clock skew with Condor

June 8, 2011

Anyone who works with distributed systems knows the value of maintaining clock synchronization. Within reason of course. If you do not work with such systems, imagine debugging something by reading a trace log. The messages will be in an order you can reason about, chronological. Now imagine if the order were jumbled, or worse you did not know the order was jumbled. In a distributed system, you often have to correlate logs between systems. If the clocks are skewed it is difficult to reconstruct a timeline.

This is not a new trick, but it is one I just found useful again and thought worth sharing:

$ condor_status -master -constraint '((MyCurrentTime-LastHeardFrom) > 60) || ((MyCurrentTime-LastHeardFrom) < -60)' \
   -format "%s\t" Name \
   -format "%d\n" '(MyCurrentTime - LastHeardFrom)'
eeyore.local	75

Turns out that system has a clock about 75 seconds in the future, and did not have ntpd installed.

See also GT1671.

Advertisements

%d bloggers like this: