databricks

I want to convert an org.apache.spark.sql.DataFrame to org.apache.spark.rdd.RDD[(String, String)] in Databricks. Can anyone help? Background (and a better solution is also welcome): I have a Kafka stream which (after some steps) becomes a 2 column data frame. I would like to put this into a Redis cache, first column as a key and second column as a value. More specifically the type of the input is this: lastContacts: org.apache.spark.sql.DataFrame = [serialNumber: string, lastModified: bigint] . I try to put into Redis as follows: sc.toRedisKV(lastContacts)(redisConfig) The error message looks

I have a DataFrame stream in Databricks, and I want to perform an action on each element. On the net I found specific purpose methods, like writing it to the console or dumping into memory, but I want to add some business logic, and put some results into Redis. To be more specific, this is how it would look like in non-stream case: val someDataFrame = Seq( ("key1", "value1"), ("key2", "value2"), ("key3", "value3"), ("key4", "value4") ).toDF() def someFunction(keyValuePair: (String, String)) = { println(keyValuePair) } someDataFrame.collect.foreach(r => someFunction((r(0).toString, r(1)

Trying to use a .dat file for ip lookup. File is on Databricks file store from Scala code: def getCountryCode(ip: String) { val filePath = "FileStore/maxmind/GeoIPCountry.dat" val ipLookups = new IpLookups(geoFile = Option(new File(filePath)), ispFile = None, orgFile = None, domainFile = None, memCache = false, lruCache = 0) val location = ipLookups.performLookups(ip)._1.head println(location.countryCode) } I am getting an exception: java.io.FileNotFoundException: FileStore/maxmind/GeoIPCountry.dat (No such file or directory) Method works on local environment with relative/absolute paths Use

I am able to create a UDF function and register to spark using spark.UDF method. However, this is per session only. How to register python UDF functions automatically when the Cluster starts?. These functions should be available to all users. Example use case is to convert time from UTC to local time zone. This is not possible; this is not like UDFs in Hive. Code the UDF as part of the package / program you submit or in the jar included in the Spark App, if using spark-submit. However, spark.udf.register.udf("... is required to be done as well. This applies to Databrick notebooks, etc. The

I am trying to push my Spark processed data to 3 node cluster of C*. I am pushing 200 million records to Cassandra it is failing with below err. Below it the my spark cluster configuration Nodes : 12 vCores Total : 112 Total memory : 1.5 TB. Below are my spark-sumbit parameters: $SPARK_HOME/bin/spark-submit --master yarn --deploy-mode cluster --name app --class Driver --executor-cores 3 --executor-memory 8g --num-executors 10 --driver-cores 2 --driver-memory 10g --conf spark.shuffle.service.enabled=true --conf spark.dynamicAllocation.enabled=false --conf spark.task.maxFailures=8 --conf spark

When I implement this part of this python code in Azure Databricks: class clustomTransformations(Transformer): <code> custom_transformer = customTransformations() .... pipeline = Pipeline(stages=[custom_transformer, assembler, scaler, rf]) pipeline_model = pipeline.fit(sample_data) pipeline_model.save(<your path>) When I attempt to save the pipeline, I get this: AttributeError: 'customTransformations' object has no attribute '_to_java' Any work arounds? dportman It seems like there is no easy workaround but to try and implement the _to_java method, as is suggested here for StopWordsRemover: