apache-spark

问题 I am trying to create a dataframe using random uniform distribution in Spark. I couldn't find anything on how to create a dataframe but when I read the documentation I found that pyspark.mllib.random has a RandomRDDs object which has a uniformRDD method which can create rdds from random uniform distribution. But the problem is that it doesn't create two dimensional rdds. Is there a way I can create a two-dimensional rdd or (preferably) dataframe? I can create a few rdds and use them to create

问题 I am trying to create a dataframe using random uniform distribution in Spark. I couldn't find anything on how to create a dataframe but when I read the documentation I found that pyspark.mllib.random has a RandomRDDs object which has a uniformRDD method which can create rdds from random uniform distribution. But the problem is that it doesn't create two dimensional rdds. Is there a way I can create a two-dimensional rdd or (preferably) dataframe? I can create a few rdds and use them to create

问题 I am trying to create a dataframe using random uniform distribution in Spark. I couldn't find anything on how to create a dataframe but when I read the documentation I found that pyspark.mllib.random has a RandomRDDs object which has a uniformRDD method which can create rdds from random uniform distribution. But the problem is that it doesn't create two dimensional rdds. Is there a way I can create a two-dimensional rdd or (preferably) dataframe? I can create a few rdds and use them to create

问题 I'm running a job in pyspark where I at one point use a grouped aggregate Pandas UDF. This results in the following (here abbreviate) error: org.apache.arrow.vector.util.OversizedAllocationException: Unable to expand the buffer I'm fairly sure this is because one of the groups the pandas UDF receives is huge, and if I reduce the dataset and removes enough rows I can run my UDF with no problems. However, I want to run with my original dataset and even if I run this spark job on a machine with

问题 I'm running a job in pyspark where I at one point use a grouped aggregate Pandas UDF. This results in the following (here abbreviate) error: org.apache.arrow.vector.util.OversizedAllocationException: Unable to expand the buffer I'm fairly sure this is because one of the groups the pandas UDF receives is huge, and if I reduce the dataset and removes enough rows I can run my UDF with no problems. However, I want to run with my original dataset and even if I run this spark job on a machine with

问题 I'm running a job in pyspark where I at one point use a grouped aggregate Pandas UDF. This results in the following (here abbreviate) error: org.apache.arrow.vector.util.OversizedAllocationException: Unable to expand the buffer I'm fairly sure this is because one of the groups the pandas UDF receives is huge, and if I reduce the dataset and removes enough rows I can run my UDF with no problems. However, I want to run with my original dataset and even if I run this spark job on a machine with

问题 For a given DataFrame just before being save 'd to parquet here is the schema: notice that the centroid0 is the first column and is StringType : However when saving the file using: df.write.partitionBy(dfHolder.metadata.partitionCols: _*).format("parquet").mode("overwrite").save(fpath) and with the partitionCols as centroid0 : then there is a (to me) surprising result: the centroid0 partition column has been moved to the end of the Row the data type has been changed to Integer I confirmed the

问题 I'm trying to understand databricks delta and thinking to do a POC using Kafka. Basically the plan is to consume data from Kafka and insert it to the databricks delta table. These are the steps that I did: Create a delta table on databricks. %sql CREATE TABLE hazriq_delta_trial2 ( value STRING ) USING delta LOCATION '/delta/hazriq_delta_trial2' Consume data from Kafka. import org.apache.spark.sql.types._ val kafkaBrokers = "broker1:port,broker2:port,broker3:port" val kafkaTopic = "kafkapoc"

问题 I am writing a Spark structured streaming application in PySpark to read data from Kafka in Confluent Cloud. The documentation for the spark readstream() function is too shallow and didn't specify much on the optional parameter part especially on the auth mechanism part. I am not sure what parameter goes wrong and crash the connectivity. Can anyone have experience in Spark help me to start this connection? Required Parameter > Consumer({'bootstrap.servers': > 'cluster.gcp.confluent.cloud:9092

问题 I want to write Structure Streaming Data into Cassandra using PySpark Structured Streaming API. My data flow is like below: REST API -> Kafka -> Spark Structured Streaming (PySpark) -> Cassandra Source and Version in below: Spark version: 2.4.3 DataStax DSE: 6.7.6-1 initialize spark: spark = SparkSession.builder\ .master("local[*]")\ .appName("Analytics")\ .config("kafka.bootstrap.servers", "localhost:9092")\ .config("spark.cassandra.connection.host","localhost:9042")\ .getOrCreate()