Hadoop Directory with Spaces

Question

I'm running into a problem while providing Hadoop a directory that contains spaces.

e.g

inputDir = /abc/xyz/folder name/abc.txt

Hadoop somehow doesn't know about the "folder name" being the name of the folder with spaces between words.

I get the below error while doing that

java.io.FileNotFoundException: File does not exist: /abc/xyz/folder

Also, I tried providing with URL encoded.

java.io.FileNotFoundException: File does not exist: /abc/xyz/folder%20name/abc.txt

But still throws me the same error.

Does anybody know the workaround for this ?

Any help is appreciated.

Answer 1

Replacing the space with %20 works for Hadoop shell. As in

sed 's/ /\%20/g'

And in the actual put command

hadoop fs -put "$inputDir" $putDest

Without the %20 you get a URI exception. (Which gave me my clue to use %20 over an escape character \ .)

I realize you're doing via Java. The fact that you're getting a java.io.FileNotFoundException makes me wonder if the code is doing something else with inputDir as opposed to being just the argument to the hadoop put, or an equivalent command of put. If it does any kind of checking of inputDir outside of Hadoop commands it will fail. Java sees it as a path. Hadoop sees it as a URI.

Answer 2

Hadoop does not support empty spaces in input directory paths.

Replace space with _ or your preferred separator character in your directory paths.

Answer 3

Try setting using set("path", "/abc/xyz/folder\\ name/abc.txt"); Kindly, note the double back slash.