sparksql比hivesql优化的点（窗口函数）

有时候，一个 select 语句中包含多个窗口函数，它们的窗口定义（OVER 子句）可能相同、也可能不同。

对于相同的窗口，完全没必要再做一次分区和排序，我们可以将它们合并成一个 Window 算子。

 select     id,    sq,    cell_type,    rank,    row_number() over(partition by id  order by rank ) naturl_rank,    rank() over(partition by id order by rank) as r,    dense_rank() over(partition by  cell_type order by id) as dr   from window_test_table  group by id,sq,cell_type,rank;

row_number() rank() 的窗口一样，可以放在一次分区和排序中完成，这一块hive sql与spark sql的表现是一致的。

但对于另外一种情况：

 select    id,    rank,    row_number() over(partition by id  order by rank ) naturl_rank,    sum(rank) over(partition by id) as snum from window_test_table

虽然这 2 个窗口并非完全一致，但是 sum(rank) 不关心分区内的顺序，完全可以复用 row_number() 的窗口。

从下面执行计划可以看出，spark sql sum(rank) 和row_number() 复用了同一个窗口，而hive sql没有。

spark sql的执行计划：

spark-sql>  explain select  id,rank,row_number() over(partition by id  order by rank ) naturl_rank,sum(rank) over(partition by id) as snum from window_test_table;      == Physical Plan ==*(3) Project [id#13, rank#16, naturl_rank#8, snum#9L]+- Window [row_number() windowspecdefinition(id#13, rank#16 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS naturl_rank#8], [id#13], [rank#16 ASC NULLS FIRST]   +- *(2) Sort [id#13 ASC NULLS FIRST, rank#16 ASC NULLS FIRST], false, 0      +- Window [sum(cast(rank#16 as bigint)) windowspecdefinition(id#13, specifiedwindowframe(RowFrame, unboundedpreceding$(), unboundedfollowing$())) AS snum#9L], [id#13]         +- *(1) Sort [id#13 ASC NULLS FIRST], false, 0            +- Exchange hashpartitioning(id#13, 200)               +- Scan hive tmp.window_test_table [id#13, rank#16], HiveTableRelation `tmp`.`window_test_table`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, [id#13, sq#14, cell_type#15, rank#16]Time taken: 0.278 seconds, Fetched 1 row(s)

hive sql执行计划：

hive> explain select  id,rank,row_number() over(partition by id  order by rank ) naturl_rank,sum(rank) over(partition by id) as snum from window_test_table;OKSTAGE DEPENDENCIES:  Stage-1 is a root stage  Stage-2 depends on stages: Stage-1  Stage-0 depends on stages: Stage-2STAGE PLANS:  Stage: Stage-1    Map Reduce      Map Operator Tree:          TableScan            alias: window_test_table            Statistics: Num rows: 13 Data size: 104 Basic stats: COMPLETE Column stats: NONE            Reduce Output Operator              key expressions: id (type: int), rank (type: int)              sort order: ++              Map-reduce partition columns: id (type: int)              Statistics: Num rows: 13 Data size: 104 Basic stats: COMPLETE Column stats: NONE      Reduce Operator Tree:        Select Operator          expressions: KEY.reducesinkkey0 (type: int), KEY.reducesinkkey1 (type: int)          outputColumnNames: _col0, _col3          Statistics: Num rows: 13 Data size: 104 Basic stats: COMPLETE Column stats: NONE          PTF Operator            Function definitions:                Input definition                  input alias: ptf_0                  output shape: _col0: int, _col3: int                  type: WINDOWING                Windowing table definition                  input alias: ptf_1                  name: windowingtablefunction                  order by: _col3 ASC NULLS FIRST                  partition by: _col0                  raw input shape:                  window functions:                      window function definition                        alias: row_number_window_0                        name: row_number                        window function: GenericUDAFRowNumberEvaluator                        window frame: PRECEDING(MAX)~FOLLOWING(MAX)                        isPivotResult: true            Statistics: Num rows: 13 Data size: 104 Basic stats: COMPLETE Column stats: NONE            Select Operator              expressions: _col0 (type: int), _col3 (type: int), row_number_window_0 (type: int)              outputColumnNames: _col0, _col3, row_number_window_0              Statistics: Num rows: 13 Data size: 104 Basic stats: COMPLETE Column stats: NONE              File Output Operator                compressed: false                table:                    input format: org.apache.hadoop.mapred.SequenceFileInputFormat                    output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat                    serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe  Stage: Stage-2    Map Reduce      Map Operator Tree:          TableScan            Reduce Output Operator              key expressions: _col0 (type: int)              sort order: +              Map-reduce partition columns: _col0 (type: int)              Statistics: Num rows: 13 Data size: 104 Basic stats: COMPLETE Column stats: NONE              value expressions: row_number_window_0 (type: int), _col3 (type: int)      Reduce Operator Tree:        Select Operator          expressions: VALUE._col0 (type: int), KEY.reducesinkkey0 (type: int), VALUE._col3 (type: int)          outputColumnNames: _col0, _col1, _col4          Statistics: Num rows: 13 Data size: 104 Basic stats: COMPLETE Column stats: NONE          PTF Operator            Function definitions:                Input definition                  input alias: ptf_0                  output shape: _col0: int, _col1: int, _col4: int                  type: WINDOWING                Windowing table definition                  input alias: ptf_1                  name: windowingtablefunction                  order by: _col1 ASC NULLS FIRST                  partition by: _col1                  raw input shape:                  window functions:                      window function definition                        alias: sum_window_1                        arguments: _col4                        name: sum                        window function: GenericUDAFSumLong                        window frame: PRECEDING(MAX)~FOLLOWING(MAX)            Statistics: Num rows: 13 Data size: 104 Basic stats: COMPLETE Column stats: NONE            Select Operator              expressions: _col1 (type: int), _col4 (type: int), _col0 (type: int), sum_window_1 (type: bigint)              outputColumnNames: _col0, _col1, _col2, _col3              Statistics: Num rows: 13 Data size: 104 Basic stats: COMPLETE Column stats: NONE              File Output Operator                compressed: false                Statistics: Num rows: 13 Data size: 104 Basic stats: COMPLETE Column stats: NONE                table:                    input format: org.apache.hadoop.mapred.SequenceFileInputFormat                    output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat                    serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe  Stage: Stage-0    Fetch Operator      limit: -1      Processor Tree:        ListSinkTime taken: 0.244 seconds, Fetched: 106 row(s)

来源：oschina

链接：https://my.oschina.net/xiaoluobutou/blog/3223236

标签

Hive

partition

stage

input

num

table