Transfer file out from HDFS

Question

I want to transfer files out from HDFS to local filesystem of a different server which is not in hadoop cluster but in the network.

I could have done:

Answer 1

I was trying to do this too (I was using Kerberos security). This helped me after small update: https://hadoop.apache.org/docs/r1.0.4/webhdfs.html#OPEN

Run directly curl -L -i --negotiate "http://:/webhdfs/v1/?op=OPEN" didn't worked for me, I'll explain why.

This command will do two steps:

1. find a file you want to download and create a temporary link - return 307 Temporary Redirect

2. from this link he will download a data - return HTTP 200 OK.

The switcher -L is saying that he take a file and continue with sawing directly. If you add to curl command -v, it'll log to output; if so, you'll see described two steps in command line, as I said. BUT - because due to older version curl (which I cannot udpate) it won't work.

SOLUTION FOR THIS (in Shell):

LOCATION=`curl -i --negotiate -u : "${FILE_PATH_FOR_DOWNLOAD}?op=OPEN" | /usr/bin/perl -n -e '/^Location: (.*)$/ && print "$1\n"'`

This will get temporary link and save it to $LOCATION variable.

RESULT=`curl -v -L --negotiate -u : "${LOCATION}" -o ${LOCAL_FILE_PATH_FOR_DOWNLOAD}`

And this will save it to your local file, if you add -o .

I hope it helped.

J.