How to create multiple SparkContexts in a console

后端 未结 4 736
温柔的废话
温柔的废话 2020-12-10 07:24

I want to create more than one SparkContext in a console. According to a post in mailing list, I need to do SparkConf.set( \'spark.driver.allowMultipleContexts\' , true), it

4条回答
  •  孤街浪徒
    2020-12-10 07:59

    My way:

    from pyspark import SparkContext
    try:
        sc.stop()
    except:
        pass
    sc=SparkContext('local','pyspark')
    '''
    your code
    '''
    sc.stop()
    

提交回复
热议问题