Apache Spark: pyspark crash for large dataset

Question

I am new to Spark. and I have input file with training data 4000x1800. When I try to train this data (written python) get following error:

1. 14/11/15 22:39:13

Answer 1

It's so simple.

conf = SparkConf().setMaster("local").setAppName("RatingsHistogram") 
sc = SparkContext(conf = conf) 
lines = sc.textFile("file:///SparkCourse/filter_1.csv",2000) 
print lines.first()

while using sc.textfile add one more parameters for the number of divisions to a large value. The bigger the data the larger the value.