Hadoop FileSystem closed exception when doing BufferedReader.close()

Question

From within the Reduce setup method,I am trying to close a BufferedReader object and getting a FileSystem closed exception. It does not happen all

Answer 1

There is a little-known gotcha with the hadoop filesystem API: FileSystem.get returns the same object for every invocation with the same filesystem. So if one is closed anywhere, they are all closed. You could debate the merits of this decision, but that's the way it is.

So, if you attempt to close your BufferedReader, and it tries to flush out some data it has buffered, but the underlying stream is connected to a FileSystem that is already closed, you'll get this error. Check your code for any other places you are closing a FileSystem object, and look for race conditions. Also, I believe Hadoop itself will at some point close the FileSystem, so to be safe, you should probably only be accessing it from within the Reducer's setup, reduce, or cleanup methods (or configure, reduce, and close, depending on which API you're using).

Answer 2

You have to use FileSystem.newInstance to avoid using a shared connection (as described by Joe K). It will give you a unique, non-shared instance.