databricks

Writing large DataFrame from PySpark to Kafka runs into timeout

由 旧巷老猫 提交于 2019-12-07 07:49:42
问题 I'm trying to write a data frame which has about 230 million records to a Kafka. More specifically to a Kafka-enable Azure Event Hub, but I'm not sure if that's actually the source of my issue. EH_SASL = 'kafkashaded.org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="Endpoint=sb://myeventhub.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=****";' dfKafka \ .write \ .format("kafka") \ .option("kafka.sasl

Cannot create Dataframe in PySpark

由 隐身守侯 提交于 2019-12-06 13:39:35
I want to create a Dataframe in PySpark with the following code from pyspark.sql import * from pyspark.sql.types import * temp = Row("DESC", "ID") temp1 = temp('Description1323', 123) print temp1 schema = StructType([StructField("DESC", StringType(), False), StructField("ID", IntegerType(), False)]) df = spark.createDataFrame(temp1, schema) But i am receiving the following error: TypeError: StructType can not accept object 'Description1323' in type type 'str' Whats wrong with my code? The problem is that you are passing a Row where you should be passing a list of Row s. Try this: from pyspark

Generate Azure Databricks Token using Powershell script

由 旧街凉风 提交于 2019-12-06 05:41:20
问题 I need to generate Azure Databricks token using Powershell script. I am done with creation of Azure Databricks using ARM template , now i am looking to generate Databricks token using powershell script . Kindly let me know how to create Databricks token using Powershell script 回答1: The only way to generate a new token is via the api which requires you to have a token in the first place. Or use the Web ui manually. There is no official powershell commands for databricks, there are some

Consume events from EventHub In Azure Databricks using pySpark

由 主宰稳场 提交于 2019-12-06 05:10:31
I could see spark connectors & guidelines for consuming events from Event Hub using Scala in Azure Databricks. But, How can we consume events in event Hub from azure databricks using pySpark? any suggestions/documentation details would help. thanks Below is the snippet for reading events from event hub from pyspark on azure data-bricks. // With an entity path val with = "Endpoint=sb://SAMPLE;SharedAccessKeyName=KEY_NAME;SharedAccessKey=KEY;EntityPath=EVENTHUB_NAME" # Source with default settings connectionString = "Valid EventHubs connection string." ehConf = { 'eventhubs.connectionString' :

Create External table in Azure databricks

由 萝らか妹 提交于 2019-12-06 04:46:05
I am new to azure databricks and trying to create an external table, pointing to Azure Data Lake Storage (ADLS) Gen-2 location. From databricks notebook i have tried to set the spark configuration for ADLS access. Still i am unable to execute the DDL created. Note : One solution working for me is mounting the ADLS account to cluster and then use the mount location in external table's DDL. But i needed to check if it is possible to create a external table DDL with ADLS path without mount location. # Using Principal credentials spark.conf.set("dfs.azure.account.auth.type", "OAuth") spark.conf

Spark 2.4.0 - unable to parse ISO8601 string into TimestampType preserving ms

由 一个人想着一个人 提交于 2019-12-06 00:52:27
问题 When trying to convert ISO8601 strings with time zone information into a TimestampType using a cast(TimestampType) only strings using the time zone format +01:00 is accepted. If the time zone is defined in the ISO8601 legal way +0100 (without the colon) the parse fails and returns null. I need to convert the string to a TimestampType while preserving the ms part. 2019-02-05T14:06:31.556+0100 Returns null 2019-02-05T14:06:31.556+01:00 Returns a correctly parsed TimestampType I have tried to

DATEDIFF in SPARK SQl

由 冷暖自知 提交于 2019-12-05 21:32:52
问题 I am new to Spark SQL. We are migrating data from SQL server to Databricks. I am using SPARK SQL . Can you please suggest how to achieve below functionality in SPARK sql for the below datefunctions. I can see datediff gives only days in spark sql. DATEDIFF(YEAR,StartDate,EndDate) DATEDIFF(Month,StartDate,EndDate) DATEDIFF(Quarter,StartDate,EndDate) 回答1: As you have mentioned SparkSQL does support DATEDIFF but for days only. I would also be careful as it seems the parameters are the opposite

Azure Databricks vs ADLA for processing

由 夙愿已清 提交于 2019-12-04 22:27:00
问题 Presently, I have all my data files in Azure Data Lake Store. I need to process these files which are mostly in csv format. The processing would be running jobs on these files to extract various information for e.g.Data for certain periods of dates or certain events related to a scenario or adding data from multiple tables/files. These jobs run everyday through u-sql jobs in data factory(v1 or v2) and then sent to powerBI for visualization. Using ADLA for all this processing, I feel it takes

Possible to put records that aren't same length as header records to bad_record directory

由 巧了我就是萌 提交于 2019-12-04 20:53:49
I am reading a file into a dataframe like this val df = spark.read .option("sep", props.inputSeperator) .option("header", "true") .option("badRecordsPath", "/mnt/adls/udf_databricks/error") .csv(inputLoc) The file is setup like this col_a|col_b|col_c|col_d 1|first|last| 2|this|is|data 3|ok 4|more||stuff 5||| Now, spark will read all of this as acceptable data. However, I want 3|ok to be marked as a bad record because it's size does not match the header size. Is this possible? val a = spark.sparkContext.textFile(pathOfYourFile) val size = a.first.split("\\|").length a.filter(i => i.split("\\|",

Exploding nested Struct in Spark dataframe

由 两盒软妹~` 提交于 2019-12-04 15:46:29
问题 I'm working through the Databricks example. The schema for the dataframe looks like: > parquetDF.printSchema root |-- department: struct (nullable = true) | |-- id: string (nullable = true) | |-- name: string (nullable = true) |-- employees: array (nullable = true) | |-- element: struct (containsNull = true) | | |-- firstName: string (nullable = true) | | |-- lastName: string (nullable = true) | | |-- email: string (nullable = true) | | |-- salary: integer (nullable = true) In the example,
订阅 databricks