Apache Pig, Suppress “Output Location Validation Failed” “Output directory … already exists”

有些话、适合烂在心里 提交于 2020-01-14 19:14:11

问题


After getting help from orangeoctopus with this question, I now need to suppress the message "Output Location Validation Failed" "Output directory ... already exists". I know the directory exists, I want it that way. I am pretty sure this will be a matter of overriding something in my Storage UDF, but I am having trouble figuring out what. Totally new to Java so bear with me. Thanks in advance.


回答1:


as far as i know, you cannot reuse a direct output directory. Hadoop prevents it. if i understand correctly, you're dealing with daily logs, therefore, i suggest you set a parent output directory say called output, and set your output directory in the script to be output/daily_date.




回答2:


Delete your output directory before the store operation:

rmf $outpath;


来源:https://stackoverflow.com/questions/10441755/apache-pig-suppress-output-location-validation-failed-output-directory-a

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!