pyspark

问题 I have trained a model in python using sklearn. How we can use same model to load in Spark and generate predictions on a spark RDD ? 回答1: Well, I will show an example of linear regression in Sklearn and show you how to use that to predict elements in Spark RDD. First training the model with sklearn example: # Create linear regression object regr = linear_model.LinearRegression() # Train the model using the training sets regr.fit(diabetes_X_train, diabetes_y_train) Here we just have the fit,

问题 I have recently started getting a bunch of errors on a number of pyspark jobs running on EMR clusters. The erros are java.lang.IllegalArgumentException at java.nio.ByteBuffer.allocate(ByteBuffer.java:334) at org.apache.arrow.vector.ipc.message.MessageSerializer.readMessage(MessageSerializer.java:543) at org.apache.arrow.vector.ipc.message.MessageChannelReader.readNext(MessageChannelReader.java:58) at org.apache.arrow.vector.ipc.ArrowStreamReader.readSchema(ArrowStreamReader.java:132) at org

问题 programming with pyspark on a Spark cluster, the data is large and in pieces so can not be loaded into the memory or check the sanity of the data easily basically it looks like af.b Current%20events 1 996 af.b Kategorie:Musiek 1 4468 af.b Spesiaal:RecentChangesLinked/Gebruikerbespreking:Freakazoid 1 5209 af.b Spesiaal:RecentChangesLinked/Sir_Arthur_Conan_Doyle 1 5214 wikipedia data: I read it from aws S3 and then try to construct spark Dataframe with the following python code in pyspark

问题 programming with pyspark on a Spark cluster, the data is large and in pieces so can not be loaded into the memory or check the sanity of the data easily basically it looks like af.b Current%20events 1 996 af.b Kategorie:Musiek 1 4468 af.b Spesiaal:RecentChangesLinked/Gebruikerbespreking:Freakazoid 1 5209 af.b Spesiaal:RecentChangesLinked/Sir_Arthur_Conan_Doyle 1 5214 wikipedia data: I read it from aws S3 and then try to construct spark Dataframe with the following python code in pyspark

问题 For a dataframe, before it is like: +----+----+----+ | ID|TYPE|CODE| +----+----+----+ | 1| B| X1| |null|null|null| |null| B| X1| +----+----+----+ After I hope it's like: +----+----+----+ | ID|TYPE|CODE| +----+----+----+ | 1| B| X1| |null| B| X1| +----+----+----+ I prefer a general method such that it can apply when df.columns is very long. Thanks! 回答1: One option is to use functools.reduce to construct the conditions: from functools import reduce df.filter(~reduce(lambda x, y: x & y, [df[c]

问题 I want to launch some pyspark jobs on YARN . I have 2 nodes, with 10 GB each. I am able to open up the pyspark shell like so: pyspark Now when I have a very simple example that I try to launch: import random NUM_SAMPLES=1000 def inside(p): x, y = random.random(), random.random() return x*x + y*y < 1 count = sc.parallelize(xrange(0, NUM_SAMPLES)) \ .filter(inside).count() print "Pi is roughly %f" % (4.0 * count / NUM_SAMPLES) I get as a result a very long spark log with the error output. The

问题 I have been trying to use sqlContext.read.format("jdbc").options(driver="org.apache.hive.jdbc.HiveDriver") to get Hive table into Spark without any success. I have done research and read below: How to connect to remote hive server from spark Spark 1.5.1 not working with hive jdbc 1.2.0 http://belablotski.blogspot.in/2016/01/access-hive-tables-from-spark-using.html I used the latest Hortonworks Sandbox 2.6 and asked the community there the same question: https://community.hortonworks.com

问题 This is the structure of my dataframe using df.columns . ['LastName', 'FirstName', 'Stud. ID', '10 Relations', 'Related to Politics', '3NF', 'Documentation & Scripts', 'SQL', 'Data (CSV, etc.)', '20 Relations', 'Google News', 'Cheated', 'Sum', 'Delay Factor', 'Grade (out of 2)'] I have transformed this dataframe in pyspark using assembler = VectorAssembler(inputCols=['10 Relations', 'Related to Politics', '3NF'],outputCol='features') and output = assembler.transform(df) . Now it contains some

问题 I installed Zeppelin on Windows using this tutorial and this. I also installed java 8 to avoid problems. I'm now able to start the Zeppelin server, and I'm trying to run this code - %pyspark a=5*4 print("value = %i" % (a)) sc.version I'm getting this error, related to py4j . I had other problems with this library before (same as here), and to avoid them I replaced the library of py4j in the Zeppelin and Spark on my computer with the latest version- py4j 0.10.7 . This is the error I get-

问题 I installed Zeppelin on Windows using this tutorial and this. I also installed java 8 to avoid problems. I'm now able to start the Zeppelin server, and I'm trying to run this code - %pyspark a=5*4 print("value = %i" % (a)) sc.version I'm getting this error, related to py4j . I had other problems with this library before (same as here), and to avoid them I replaced the library of py4j in the Zeppelin and Spark on my computer with the latest version- py4j 0.10.7 . This is the error I get-