Apache beam windowing: consider late data but emit only one pane

后端 未结 2 512
南旧
南旧 2020-12-21 09:18

I would like to emit a single pane when the watermark reaches x minutes past the end of the window. This let\'s me ensure I handle some late data, but still only emit one pa

2条回答
  •  伪装坚强ぢ
    2020-12-21 09:39

    Thanks Guillem, in the end I used your answer to find this very useful link with lots of apache beam examples. From this I came up with the following solution:

     // We first specify to never emit any panes
     .triggering(Never.ever())
    
     // We then specify to fire always when closing the window. This will emit a
     // single final pane at the end of allowedLateness
     .withAllowedLateness(allowedLateness, Window.ClosingBehavior.FIRE_ALWAYS)
     .discardingFiredPanes())
    

提交回复
热议问题