window-functions

What is the equivalent of row_number() over partition in hql I have the following query in hql: select s.Companyname, p.Productname, sum(od.Unitprice * od.Quantity - od.Discount) as SalesAmount FROM OrderDetails as od inner join od.Orders as o inner join od.Products as p " + "inner join p.Suppliers as s" + " where o.Orderdate between '2010/01/01' and '2014/01/01' GROUP BY s.Companyname,p.Productname" I want to do partition by s.Companyname where RowNumber <= n . As far as I know you cannot use row_number() neither in HQL nor in JPQL . I propose to use a native SQL query in this case:

I must apply a certain transformation fn(argument) . Here argument is equal to value , but not when it is negative. When you get a first negative value , then you "wait" until it sums up with consecutive values and this sum becomes positive. Then you do fn(argument) . See the table I want to get: value argument --------------------- 2 2 3 3 -10 0 4 0 3 0 10 7 1 1 I could have summed all values and apply fn to the sum, but fn can be different for different rows and it is essential to know the row number to choose a concrete fn. As want a Postgres SQL solution, looks like window functions fit,

I am implementing the Cumulative Sum in Spark using Window Function. But the order of records input is not maintained while applying the window partition function Input data: val base = List(List("10", "MILLER", "1300", "2017-11-03"), List("10", "Clark", "2450", "2017-12-9"), List("10", "King", "5000", "2018-01-28"), List("30", "James", "950", "2017-10-18"), List("30", "Martin", "1250", "2017-11-21"), List("30", "Ward", "1250", "2018-02-05")) .map(row => (row(0), row(1), row(2), row(3))) val DS1 = base.toDF("dept_no", "emp_name", "sal", "date") DS1.show() +-------+--------+----+----------+

I've noticed that after I use a Window function over a DataFrame if I call a map() with a function, Spark returns a "Task not serializable" Exception This is my code: val hc:org.apache.spark.sql.hive.HiveContext = new org.apache.spark.sql.hive.HiveContext(sc) import hc.implicits._ import org.apache.spark.sql.expressions.Window import org.apache.spark.sql.functions._ def f():String = "test" case class P(name:String,surname:String) val lag_result:org.apache.spark.sql.Column = lag($"name",1).over(Window.partitionBy($"surname")) val lista:List[P] = List(P("N1","S1"),P("N2","S2"),P("N2","S2")) val

I'm not sure this is a bug (or just incorrect syntax). I searched around and didn't see this mentioned elsewhere so I'm asking here before filing a bug report. I'm trying to use a Window function partitioned on a nested column. I've created a small example below demonstrating the problem. import sqlContext.implicits._ import org.apache.spark.sql.functions._ import org.apache.spark.sql.expressions.Window val data = Seq(("a", "b", "c", 3), ("c", "b", "a", 3)).toDF("A", "B", "C", "num") .withColumn("Data", struct("A", "B", "C")).drop("A").drop("B").drop("C") val winSpec = Window.partitionBy("Data