Hadoop JobTracker and NameNode configuration error: FileNotFoundException jobToken

FYI for anyone running into java.io.FileNotFoundException: File file:/tmp/hadoop-USER/mapred/system/JOBID/jobToken does not exist. when running Hadoop MapReduce jobs.

Your JobTracker needs access to an HDFS NameNode to share information with TaskTrackers. If the JobTracker is misconfigured and cannot connect to a NameNode it will write to local disk instead. Or at least it will in hadoop-1.0.1. The result is TaskTrackers fail to find the jobToken in HDFS, or on their local disks, and will throw a FileNotFoundException.

I ran into this because I did not have a fs.default.name property in my JobTracker’s mapred-site.xml. I was putting it in hdfs-site.xml instead.

I have not found messages in the JobTracker (or TaskTracker) logs that explicitly indicate this misconfiguration.

Oops.

Advertisements

Tags: , ,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: