scalars can only be used with projection in PIG

穿精又带淫゛_ 提交于 2019-12-12 02:49:05

问题


scalars can only be used with projection i am getting this error while using foreach.How can i resolved this error ? how can i use LIMIT within foreach ? please suggest some thanks in advance..

Edit (Tichdroma): Copied code from comment

A = LOAD 'part-r-00000';
G = Group A by ($0,$2 );
Y = foreach G generate FLATTEN(group), FLATTEN($1);
sorted = order Y by $0 ASC, $1 DESC;
X = foreach Y {
  lim = LIMIT sorted 3;
  generate lim;
};
Dump x;

回答1:


LIMIT is available in Pig 0.9 in the FOREACH nested_op.

If you want the top N element of each group, you might want to try to iterate on each one and individually sort and limit them:

A = LOAD 'part-r-00000';
G = GROUP A by ($0, $2);
X = FOREACH G {
  sorted = ORDER A by $0 ASC, $1 DESC;
  lim = LIMIT sorted 3;
  GENERATE lim;
};
DUMP X;

Notice that TOP can be efficient when you just have a column of comparable values (not in this case).



来源:https://stackoverflow.com/questions/9108206/scalars-can-only-be-used-with-projection-in-pig

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!