spark :12: error: not found: value sc

Question

I wrote the following:

val a = 1 to 10000
val b = sc.parallelize(a)

and it shows error saying:

:12: error: n         


        

Answer 1

As stated in this thread, one solution may be to switch off permissions checking.

In cloudera manager, go to hdfs configuration under advanced and put the following code in "HDFS Service Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml" :

dfs.permissions
false

After that, it is necessary to restart the HDFS component.

It worked for me. It might not be appropriate for a production environment, however.