databricks

问题 I have created an Azure DataBricks cluster, which is in running state. But I cannot see shared, users and newly created notebooks under workspace. In fact, I can see a connecting... image on top left corner. Please help. 回答1: For us the problem was Websocket traffic being blocked on our proxy servers by McAfee Web Gateway. The solution was to selctively allow Websocket traffic as described here: https://kc.mcafee.com/corporate/index?page=content&id=KB84052&actp=null&showDraft=false&platinum

问题 I want to save or copy my file from the dbfs to my desktop (local). I use this command but get an error: dbutils.fs.cp('/dbfs/username/test.txt', 'C:\Users\username\Desktop') Error: SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 2-3: truncated \UXXXXXXXX escape When I lookup the dbutils.fs.help() for my case, I followed the instructions: dbutils.fs provides utilities for working with FileSystems. Most methods in this package can take either a DBFS path (e.g.

问题 I have a Data frame as below: id time type day ___ _____ _____ ____ 1 2016-10-12 01:45:01 1 3 1 2016-10-12 01:48:01 0 3 1 2016-10-12 01:50:01 1 3 1 2016-10-12 01:52:01 1 3 2 2016-10-12 01:53:01 1 3 2 2016-10-12 02:10:01 1 3 3 2016-10-12 01:45:01 1 3 3 2016-10-12 01:48:01 1 3 From this data frame I want to calculate to the occurences of type 1 in that id before half hour for each row. For example if we take the first row 1 2016-10-12 01:45:01 1 3 From this I want to count the type 1 occurences

问题 This question already has answers here : How to use JDBC source to write and read data in (Py)Spark? (3 answers) Closed last year . I have a dataframe in Databricks called customerDetails. +--------------------+-----------+ | customerName| customerId| +--------------------+-----------+ |John Smith | 0001| |Jane Burns | 0002| |Frank Jones | 0003| +--------------------+-----------+ I would like to be able to copy this from Databricks to a table within Postgres. I found this post which used

问题 I want to continuously elaborate rows of a dataset stream (originally initiated by a Kafka): based on a condition I want to update a Radis hash. This is my code snippet ( lastContacts is the result of a previous command, which is a stream of this type: org.apache.spark.sql.DataFrame = [serialNumber: string, lastModified: long] . This expands to org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] ): class MyStreamProcessor extends ForeachWriter[Row] { override def open(partitionId: Long,

问题 I have an Azure Databricks cluster that processes various tables and then as a final step I push these table into an Azure SQL Server to be used by some other processes. I have a cell in databricks that looks something like this: def generate_connection(): jdbcUsername = dbutils.secrets.get(scope = "Azure-Key-Vault-Scope", key = "AzureSqlUserName") jdbcPassword = dbutils.secrets.get(scope = "Azure-Key-Vault-Scope", key = "AzureSqlPassword") connectionProperties = { "user" : jdbcUsername,

问题 I have created two data frames. df_stg_raw data frame is holding duplicate records. df_qualify data frame is holding meta-information such as partition & order is based on which column. I want to remove duplicate records using window function available in PySpark. df_stg_raw ================================================== ACCNT_ID NAME SomeRandomID TABLE_NM ================================================== 1 A 123 TblA 1 A 123 TblA 2 B 124 TblA 2 B 124 TblA 3 C 125 TblA 3 C 125 TblA df

问题 I am facing an issue when reading and parsing a CSV file. Some records have a newline symbol, "escaped" by a \, and that record not being quoted. The file might look like this: Line1field1;Line1field2.1 \ Line1field2.2;Line1field3; Line2FIeld1;Line2field2;Line2field3; I've tried to read it using sc.textFile("file.csv") and using sqlContext.read.format("..databricks..").option("escape/delimiter/...").load("file.csv") However doesn't matter how I read it, a record/line/row is created when "\ \n

问题 I am using saveAsTextFile() to store the results of a Spark job in the folder dbfs:/FileStore/my_result. I can access to the different "part-xxxxx" files using the web browser, but I would like to automate the process of downloading all files to my local machine. I have tried to use cURL, but I can't find the RestAPI command to download a dbfs:/FileStore file. Question: How can I download a dbfs:/FileStore file to my Local Machine? I am using Databricks Community Edition to teach an

问题 I am trying to solve the following error (I am using the databricks platform and spark 2.0) tweets_cleaned.createOrReplaceTempView("tweets_cleanedSQL") def Occ(keyword): occurences = spark.sql("SELECT * \ FROM tweets_cleanedSQL \ WHERE LOWER(text) LIKE '%" + keyword + "%' \ ") return occurences.count() occurences_udf = udf(Occ) If I run this code, I receive the following error: py4j.Py4JException: Method getnewargs ([]) does not exist ==> error only occurs when trying to define the udf. 回答1: