spark-dataframe

I have a Spark DataFrame as shown below: #Create DataFrame df <- data.frame(name = c("Thomas", "William", "Bill", "John"), dates = c('2017-01-05', '2017-02-23', '2017-03-16', '2017-04-08')) df <- createDataFrame(df) #Make sure df$dates column is in 'date' format df <- withColumn(df, 'dates', cast(df$dates, 'date')) name | dates -------------------- Thomas |2017-01-05 William |2017-02-23 Bill |2017-03-16 John |2017-04-08 I want to change dates to the end of month date, so they would look like shown below. How do I do this? Either SparkR or PySpark code is fine. name | dates --------------------

I have a StructField in a dataframe that is not nullable. Simple example: import pyspark.sql.functions as F from pyspark.sql.types import * l = [('Alice', 1)] df = sqlContext.createDataFrame(l, ['name', 'age']) df = df.withColumn('foo', F.when(df['name'].isNull(),False).otherwise(True)) df.schema.fields which returns: [StructField(name,StringType,true), StructField(age,LongType,true), StructField(foo,BooleanType,false)] Notice that the field foo is not nullable. Problem is that (for reasons I won't go into) I want it to be nullable. I found this post Change nullable property of column in spark

I have a numpy matrix: arr = np.array([[2,3], [2,8], [2,3],[4,5]]) I need to create a PySpark Dataframe from arr . I can not manually input the values because the length/values of arr will be changing dynamically so I need to convert arr into a dataframe. I tried the following code to no success. df= sqlContext.createDataFrame(arr,["A", "B"]) However, I get the following error. TypeError: Can not infer schema for type: <type 'numpy.ndarray'> Hope this helps! import numpy as np #sample data arr = np.array([[2,3], [2,8], [2,3],[4,5]]) rdd1 = sc.parallelize(arr) rdd2 = rdd1.map(lambda x: [int(i)

I'm new to Spark 2.0 and using datasets in our code base. I'm kinda noticing that I need to import spark.implicits._ everywhere in our code. For example: File A class A { def job(spark: SparkSession) = { import spark.implcits._ //create dataset ds val b = new B(spark) b.doSomething(ds) doSomething(ds) } private def doSomething(ds: Dataset[Foo], spark: SparkSession) = { import spark.implicits._ ds.map(e => 1) } } File B class B(spark: SparkSession) { def doSomething(ds: Dataset[Foo]) = { import spark.implicits._ ds.map(e => "SomeString") } } What I wanted to ask is if there's a cleaner way to

I want to control the reading and writing speed to an RDB by Spark directly, yet the related parameters as the title already revealed seemingly were not working. Can I conclude that fetchsize and batchsize didn't work with my testing method? Or they do affect on the facet of reading and writing since the measure result is reasonable based on scale. Stats of betchsize , fetchsize and data set /*Dataset*/ +--------------+-----------+ | Observations | Dataframe | +--------------+-----------+ | 109,077 | Initial | | 345,732 | Ultimate | +--------------+-----------+ /*fetchsize*/ +-----------+-----

I have the following two JSON files: { "name" : "Agent1", "age" : "32", "details" : [{ "d1" : 1, "d2" : 2 } ] } { "name" : "Agent2", "age" : "42", "details" : [] } I read them with spark: val jsonDf1 = spark.read.json(pathToJson1) val jsonDf2 = spark.read.json(pathToJson2) two dataframes are created with the following schemas: root |-- age: string (nullable = true) |-- details: array (nullable = true) | |-- element: struct (containsNull = true) | | |-- d1: long (nullable = true) | | |-- d2: long (nullable = true) |-- name: string (nullable = true) root |-- age: string (nullable = true) |--

I have a list with more than 30 strings. how to convert list into dataframe . what i tried: eg Val list=List("a","b","v","b").toDS().toDF() Output : +-------+ | value| +-------+ |a | |b | |v | |b | +-------+ Expected Output is +---+---+---+---+ | _1| _2| _3| _4| +---+---+---+---+ | a| b| v| a| +---+---+---+---+ any help on this . List("a","b","c","d") represents a record with one field and so the resultset displays one element in each row. To get the expected output, the row should have four fields/elements in it. So, we wrap around the list as List(("a","b","c","d")) which represents one row,