spark-dataframe

Normally all rows in a group are passed to an aggregate function. I would like to filter rows using a condition so that only some rows within a group are passed to an aggregate function. Such operation is possible with PostgreSQL . I would like to do the same thing with Spark SQL DataFrame (Spark 2.0.0). The code could probably look like this: val df = ... // some data frame df.groupBy("A").agg( max("B").where("B").less(10), // there is no such method as `where` :( max("C").where("C").less(5) ) So for a data frame like this: | A | B | C | | 1| 14| 4| | 1| 9| 3| | 2| 5| 6| The result would be:

Is there a way to calculate the size in bytes of an Apache spark Data Frame using pyspark? why don't you just cache the df and then look in the spark UI under storage and convert the units to bytes df.cache() 来源： https://stackoverflow.com/questions/38180140/how-can-you-calculate-the-size-of-an-apache-spark-data-frame-using-pyspark

I am currently pulling data from SQL Server using PyODBC and trying to insert into a table in Hive in a Near Real Time (NRT) manner. I got a single row from source and converted into List[Strings] and creating schema programatically but while creating a DataFrame, Spark is throwing StructType error. >>> cnxn = pyodbc.connect(con_string) >>> aj = cnxn.cursor() >>> >>> aj.execute("select * from tjob") <pyodbc.Cursor object at 0x257b2d0> >>> row = aj.fetchone() >>> row (1127, u'', u'8196660', u'', u'', 0, u'', u'', None, 35, None, 0, None, None, None, None, None, None, None, None, None, None,

I am reading parquet data and I see that it is listing all the directories on driver side Listing s3://xxxx/defloc/warehouse/products_parquet_151/month=2016-01 on driver Listing s3://xxxx/defloc/warehouse/products_parquet_151/month=2014-12 on driver I have specified month=2014-12 in my where clause. I have tried using spark sql and data frame API, and looks like both aren't pruning partitions. Using Dataframe API df.filter("month='2014-12'").show() Using Spark SQL sqlContext.sql("select name, price from products_parquet_151 where month = '2014-12'") I have tried the above on versions 1.5.1, 1

I have a pySpark dataframe that looks like this: +-------------+----------+ | sku| date| +-------------+----------+ |MLA-603526656|02/09/2016| |MLA-603526656|01/09/2016| |MLA-604172009|02/10/2016| |MLA-605470584|02/09/2016| |MLA-605502281|02/10/2016| |MLA-605502281|02/09/2016| +-------------+----------+ I want to group by sku, and then calculate the min and max dates. If I do this: df_testing.groupBy('sku') \ .agg({'date': 'min', 'date':'max'}) \ .limit(10) \ .show() the behavior is the same as Pandas, where I only get the sku and max(date) columns. In Pandas I would normally do the following

I thought that with the integration of project Tungesten, spark would automatically use off heap memory. What for are spark.memory.offheap.size and spark.memory.offheap.enabled? Do I manually need to specify the amount of off heap memory for Tungsten here? Spark/Tungsten use Encoders/Decoders to represent JVM objects as a highly specialized Spark SQL Types objects which then can be serialized and operated on in a highly performant way. Internal format representation is highly efficient and friendly to GC memory utilization. Thus, even operating in the default on-heap mode Tungsten alleviates