flink calculate median on stream

六月ゝ 毕业季﹏ 提交于 2019-12-12 17:36:29

问题


I'm required to calculate median of many parameters received from a kafka stream for 15 min time window.

i couldn't find any built in function for that, but I have found a way using custom WindowFunction.

my questions are:

  1. is it a difficult task for flink? the data can be very large.
  2. if the data gets to giga bytes, will flink store everything in memory until the end of the time window? (one of the arguments of apply WindowFunction implementation is Iterable - a collection of all data which came during the time window )

thanks

来源:https://stackoverflow.com/questions/46451604/flink-calculate-median-on-stream

标签
易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!