databricks

I am trying to retrieve data from oracle using spark-sql-2.4.1 version. I tried to set the JdbcOptions as below : .option("lowerBound", "31-MAR-02"); .option("upperBound", "01-MAY-19"); .option("partitionColumn", "data_date"); .option("numPartitions", 240); But gives error : java.lang.IllegalArgumentException: Timestamp format must be yyyy-mm-dd hh:mm:ss[.fffffffff] at java.sql.Timestamp.valueOf(Timestamp.java:204) at org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation$.toInternalBoundValue(JDBCRelation.scala:179) Then tried as below .option("lowerBound", "2002-03-31"); //changed the

I have a DataFrame ( df ) which consists of more than 50 columns and different types of data types, such as df3.printSchema() CtpJobId: string (nullable = true) |-- TransformJobStateId: string (nullable = true) |-- LastError: string (nullable = true) |-- PriorityDate: string (nullable = true) |-- QueuedTime: string (nullable = true) |-- AccurateAsOf: string (nullable = true) |-- SentToDevice: string (nullable = true) |-- StartedAtDevice: string (nullable = true) |-- ProcessStart: string (nullable = true) |-- LastProgressAt: string (nullable = true) |-- ProcessEnd: string (nullable = true) |--

I have a dataframe like: ID Notes 2345 Checked by John 2398 Verified by Stacy 3983 Double Checked on 2/23/17 by Marsha Let's say for example there are only 3 employees to check: John, Stacy, or Marsha. I'd like to make a new column like so: ID Notes Employee 2345 Checked by John John 2398 Verified by Stacy Stacy 3983 Double Checked on 2/23/17 by Marsha Marsha Is regex or grep better here? What kind of function should I try? Thanks! EDIT: I've been trying a bunch of solutions, but nothing seems to work. Should I give up and instead create columns for each employee, with a binary value? IE: ID

I am using Spark SQL where I want to fetch whole data everyday from a Oracle table(consist of more than 1800k records). The application is hanging up when I read from Oracle hence I used concept of partitionColumn,lowerBound & upperBound . But,the problem is how can I get l owerBound & upperBound value of primary key column dynamically ?? Every day value of lowerBound & upperBound will be changing.Hence how can I get the boundary values of primary key column dynamically?? Can anyone guide me an sample example for my problem? Just fetch required values from the database: url = ... properties =

This question already has an answer here: How to save RDD data into json files, not folders 2 answers I have dataframe and i want to save in single file on hdfs location. i found the solution here Write single CSV file using spark-csv df.coalesce(1) .write.format("com.databricks.spark.csv") .option("header", "true") .save("mydata.csv") But all data will be written to mydata.csv/part-00000 and i wanted to be mydata.csv file. is that possible? any help appreciate It's not possible using standard spark library, but you can use Hadoop API for managing filesystem - save output in temporary