pyspark-sql

I'm evaluating Mongo Spark connector for a project and I'm getting the inconsistent results. I use MongoDB server version 3.4.5, Spark (via PySpark) version 2.2.0, Mongo Spark Connector version 2.11;2.2.0 locally on my laptop. For my test DB I use the Enron dataset http://mongodb-enron-email.s3-website-us-east-1.amazonaws.com/ I'm interested in Spark SQL queries and when I started to run simple test queries for count I received different counts for each run. Here is output from my mongo shell: > db.messages.count({'headers.To': 'eric.bass@enron.com'}) 203 Here are some output from my PySpark

Can someone tell me how to convert a list containing strings to a Dataframe in pyspark. I am using python 3.6 with spark 2.2.1. I am just started learning spark environment and my data looks like below my_data =[['apple','ball','ballon'],['cat','camel','james'],['none','focus','cake']] Now, i want to create a Dataframe as follows --------------------------------- |ID | words | --------------------------------- 1 | ['apple','ball','ballon'] | 2 | ['cat','camel','james'] | I even want to add ID column which is not associated in the data You can convert the list to a list of Row objects, then use

I am new to pyspark. I want to plot the result using matplotlib, but not sure which function to use. I searched for a way to convert sql result to pandas and then use plot. Hi Team I have found the solution for this. I converted sql dataframe to pandas dataframe and then I was able to plot the graphs. below is the sample code.from pyspark.sql import Row from pyspark.sql import HiveContext import pyspark from IPython.display import display import matplotlib import matplotlib.pyplot as plt %matplotlib inline sc = pyspark.SparkContext() sqlContext = HiveContext(sc) test_list = [(1, 'hasan'),(2,

I am using the following code to get the average age of people whose salary is greater than some threshold. dataframe.filter(df['salary'] > 100000).agg({"avg": "age"}) the column age is numeric (float) but still I am getting this error. py4j.protocol.Py4JJavaError: An error occurred while calling o86.agg. : scala.MatchError: age (of class java.lang.String) Do you know any other way to obtain the avg etc. without using groupBy function and SQL queries. zero323 Aggregation function should be a value and a column name a key: dataframe.filter(df['salary'] > 100000).agg({"age": "avg"})