问题
i am in a big confusion with the below two statements. 1) where exactly LOAD statement stores this relation(student), is it on hdfs/PIG internal storage/local machine ???
example : student = LOAD 'HDFS:/student' using PigStorage(',');
2) if i try to DUMP student; then it takes almost 30-40 sec to display result where as LOAD statement takes 1-2 sec.... if we are trying to retrieve data from pig internal storage then why is this delay ??
would be grateful if anyone can clear this doubts(preferably the flow of execution). thanks in adv.
my env: i am using VM for learning purpose.
回答1:
The LOAD does not store the data but it is just a pointer to the file.
When LOAD statement is executed, no MapReduce task is executed.
It is only after the DUMP or STORE statement that a MapReduce job is initiated.
We see our data in the output and we can confirm that the data has been loaded successfully.
DUMP take time as it disables multi-query execution and and slows down execution. (If you have included DUMP statements in your scripts for debugging purposes, you should remove them.)
If you want to store any data then can use the STORE command.
来源：https://stackoverflow.com/questions/36353195/where-exactly-pig-stores-its-relations