Export from pig to CSV

Question

I\'m having a lot of trouble getting data out of pig and into a CSV that I can use in Excel or SQL (or R or SPSS etc etc) without a lot of manipulation ...

I\'ve tri

Answer 1

if you will store your data as PigStorage on HDFS and then merge it using -getmerge -nl:

STORE pig_object INTO '/user/hadoop/csvoutput/pig_object'
    using PigStorage('\t','-schema');
fs -getmerge -nl /user/hadoop/csvoutput/pig_object  /Users/Name/Folder/pig_object.csv;

Docs:

Optionally -nl can be set to enable adding a newline character (LF) at the end of each file.

you will have a single TSV/CSV file with the following structure:

1 - header
2 - empty line
3 - pig schema
4 - empty line
5 - 1st line of DATA
6 - 2nd line of DATA
...

so we can simply remove lines [2,3,4] using AWK:

awk 'NR==1 || NR>4 {print}' /Users/Name/Folder/pig_object.csv > /Users/Name/Folder/pig_object_clean.csv