databricks

I need to generate Azure Databricks token using Powershell script. I am done with creation of Azure Databricks using ARM template , now i am looking to generate Databricks token using powershell script . Kindly let me know how to create Databricks token using Powershell script The only way to generate a new token is via the api which requires you to have a token in the first place. Or use the Web ui manually. There is no official powershell commands for databricks, there are some unofficial ones but they still require you to generate a token manually first. https://github.com/DataThirstLtd

I am new to Spark SQL. We are migrating data from SQL server to Databricks. I am using SPARK SQL . Can you please suggest how to achieve below functionality in SPARK sql for the below datefunctions. I can see datediff gives only days in spark sql. DATEDIFF(YEAR,StartDate,EndDate) DATEDIFF(Month,StartDate,EndDate) DATEDIFF(Quarter,StartDate,EndDate) As you have mentioned SparkSQL does support DATEDIFF but for days only. I would also be careful as it seems the parameters are the opposite way round for Spark, ie --SQL Server DATEDIFF ( datepart , startdate , enddate ) --Spark DATEDIFF ( enddate ,

Our Spark environment: DataBricks 4.2 (includes Apache Spark 2.3.1, Scala 2.11) What we try to achieve: We want to enrich streaming data with some reference data, which is updated regularly. The enrichment is done by joining the stream with the reference data. What we implemented: We implemented two spark jobs (jars): The first one is updating a Spark table TEST_TABLE every hour (let’s call it ‘reference data’) by using .write.mode(SaveMode.Overwrite).saveAsTable("TEST_TABLE") And afterwards calling spark.catalog.refreshTable("TEST_TABLE") The second job (let’s call it streaming data) is using

Presently, I have all my data files in Azure Data Lake Store. I need to process these files which are mostly in csv format. The processing would be running jobs on these files to extract various information for e.g.Data for certain periods of dates or certain events related to a scenario or adding data from multiple tables/files. These jobs run everyday through u-sql jobs in data factory(v1 or v2) and then sent to powerBI for visualization. Using ADLA for all this processing, I feel it takes a lot of time to process and seems very expensive. I got a suggestion that I should use Azure

I'm working through the Databricks example . The schema for the dataframe looks like: > parquetDF.printSchema root |-- department: struct (nullable = true) | |-- id: string (nullable = true) | |-- name: string (nullable = true) |-- employees: array (nullable = true) | |-- element: struct (containsNull = true) | | |-- firstName: string (nullable = true) | | |-- lastName: string (nullable = true) | | |-- email: string (nullable = true) | | |-- salary: integer (nullable = true) In the example, they show how to explode the employees column into 4 additional columns: val explodeDF = parquetDF

We are reading data from MongoDB Collection . Collection column has two different values (e.g.: (bson.Int64,int) (int,float) ). I am trying to get a datatype using pyspark. My problem is some columns have different datatype. Assume quantity and weight are the columns quantity weight --------- -------- 12300 656 123566000000 789.6767 1238 56.22 345 23 345566677777789 21 Actually we didn't defined data type for any column of mongo collection. When I query to the count from pyspark dataframe dataframe.count() I got exception like this "Cannot cast STRING into a DoubleType (value: BsonString{value