Spark job with Async HTTP call

后端 未结 4 436
遥遥无期
遥遥无期 2020-12-05 16:10

I build a RDD from a list of urls, and then try to fetch datas with some async http call. I need all the results before doing other calculs. Ideally, I need to make the http

4条回答
  •  遥遥无期
    2020-12-05 16:34

    I finally made it using scalaj-http instead of Dispatch. Call are synchronous, but this match my use case.

    I think the Spark Job never finish using Dispatch because the Http connection was not closed properly.

    Best Regards

提交回复
热议问题