databricks

来源： https://stackoverflow.com/questions/62758094/i-o-operations-with-azure-databricks-rest-jobs-api

来源： https://stackoverflow.com/questions/63580115/how-to-read-parquet-files-under-a-directory-using-pyspark

来源： https://stackoverflow.com/questions/63580115/how-to-read-parquet-files-under-a-directory-using-pyspark

问题 I'm using SparkSQL in a Java application to do some processing on CSV files using Databricks for parsing. The data I am processing comes from different sources (Remote URL, local file, Google Cloud Storage), and I'm in the habit of turning everything into an InputStream so that I can parse and process data without knowing where it came from. All the documentation I've seen on Spark reads files from a path, e.g. SparkConf conf = new SparkConf().setAppName("spark-sandbox").setMaster("local");

问题 I am creating an UDF which will find the first day of the week for me. The inputs to the UDF will be a String Column from the Dataframe storing datetime in yyyy-MM-dd hh:MM:ss . I agree that the same can be established without an UDF but I want to explore , all options of doing this. As of now , I am stuck with the implementation via UDF. Important Note - The Week Start Day is MONDAY. Code - import org.apache.spark.sql.functions._ import java.time.format.DateTimeFormatter import java.time

问题 I am studying pyspark in databricks. I want to generate a correlation heatmap. Let's say this is my data: myGraph=spark.createDataFrame([(1.3,2.1,3.0), (2.5,4.6,3.1), (6.5,7.2,10.0)], ['col1','col2','col3']) And this is my code: import pyspark from pyspark.sql import SparkSession import matplotlib.pyplot as plt import pandas as pd import numpy as np from ggplot import * from pyspark.ml.feature import VectorAssembler from pyspark.ml.stat import Correlation from pyspark.mllib.stat import

问题 I would really love some help with parsing nested JSON data using PySpark-SQL. The data has the following schema (blank spaces are edits for confidentiality purposes...) Schema root |-- location_info: array (nullable = true) | |-- element: struct (containsNull = true) | | |-- restaurant_type: string (nullable = true) | | | | | | | | |-- other_data: array (nullable = true) | | | |-- element: struct (containsNull = true) | | | | |-- other_data_1 string (nullable = true) | | | | |-- other_data_2

问题 I have a csv of size 6GB. So far I was using the following line which when I check its size on dbfs after this copy using java io, it still shows as 6GB so I assume it was right. But when I do a spark.read.csv(samplePath) it reads only 18mn rows instead of 66mn. Files.copy(Paths.get(_outputFile), Paths.get("/dbfs" + _outputFile)) So I tried dbutils to copy as shown below but it gives error. I have updated maven dbutil dependency and imported the same in this object where I am calling this

问题 I want to read data from a confluent cloud topic and then write in another topic. At localhost, I haven't had any major problems. But the schema registry of confluent cloud requires to pass some authentication data that I don't know how to enter them: basic.auth.credentials.source=USER_INFO schema.registry.basic.auth.user.info=: schema.registry.url=https://xxxxxxxxxx.confluent.cloudBlockquote Below is the current code: import com.databricks.spark.avro.SchemaConverters import io.confluent

问题 I want to read data from a confluent cloud topic and then write in another topic. At localhost, I haven't had any major problems. But the schema registry of confluent cloud requires to pass some authentication data that I don't know how to enter them: basic.auth.credentials.source=USER_INFO schema.registry.basic.auth.user.info=: schema.registry.url=https://xxxxxxxxxx.confluent.cloudBlockquote Below is the current code: import com.databricks.spark.avro.SchemaConverters import io.confluent