partitioning

问题 So I'm working on a homework assignment and I need to create recursive functions for partitions, Stirling numbers(first and second kind), and Chebyshev polynomials of the first. My program should be able to have a user input a positive integer n, and then create files named Partitions.txt, Stirling1.txt, Stirling2.txt, and Chebyshev.txt, that creates a table of all values f(k,m) for 1<=k<=n and 1<=m<=n. I'm struggling just to start off the assignment and feel like I have no understanding of

问题 I have the following dataframe (df_parquet): DataFrame[id: bigint, date: timestamp, consumption: decimal(38,18)] I intend to get sorted lists of dates and consumptions using collect_list, just as stated in this post: collect_list by preserving order based on another variable I am following the last approach (https://stackoverflow.com/a/49246162/11841618), which is the one i think its more efficient. So instead of just calling repartition with the default number of partitions (200) i call it

问题 I want to create a table like this. create table some_data ( id number(19,0), group_id number(19,0), value float, timestamp timestamp ); For this table i would like to have the data stored like group_id=1 jan-2015 feb-2015 ... group_id=2 jan-2015 feb-2015 ... and so on. So I assume i have to create a partition by range for the group_id and then a subpartition also by range with the timestamp column, right? So it should look like this: create table some_data ( id number(19,0), group_id number

问题 I want to create a table like this. create table some_data ( id number(19,0), group_id number(19,0), value float, timestamp timestamp ); For this table i would like to have the data stored like group_id=1 jan-2015 feb-2015 ... group_id=2 jan-2015 feb-2015 ... and so on. So I assume i have to create a partition by range for the group_id and then a subpartition also by range with the timestamp column, right? So it should look like this: create table some_data ( id number(19,0), group_id number

问题 I have a folder which has 14 files in it. I run the spark-submit with 10 executors on a cluster, which has resource manager as yarn. I create my first RDD as this: JavaPairRDD<String,String> files = sc.wholeTextFiles(folderPath.toString(), 10); However, files.getNumPartitions() gives me 7 or 8, randomly. Then I do not use coalesce/repartition anywhere and I finish my DAG with 7-8 partitions. As I know, we gave argument as the "minimum" number of partitions, so that why Spark divide my RDD to

问题 I have a table with over a billion records. In order to improve performance, I partitioned it to 30 partitions. The most frequent queries have (id = ...) in their where clause, so I decided to partition the table on the id column. Basically, the partitions were created in this way: CREATE TABLE foo_0 (CHECK (id % 30 = 0)) INHERITS (foo); CREATE TABLE foo_1 (CHECK (id % 30 = 1)) INHERITS (foo); CREATE TABLE foo_2 (CHECK (id % 30 = 2)) INHERITS (foo); CREATE TABLE foo_3 (CHECK (id % 30 = 3))

问题 Spark 1.6.0 Hive 1.1.0-cdh5.8.0 I have some problems saving my dataframe into parquet-backed partitioned Hive table from Spark. here is my code: val df = sqlContext.createDataFrame(rowRDD, schema) df.write .mode(SaveMode.Append) .format("parquet") .partitionBy("year") .saveAsTable(output) nothing special, actually, but I can't read any data from the table when it's generated. Key point is in partitioning - without it everything works fine. Here are my steps to fix the problem: At first, on

问题 Scenario I have a 10 million row table. I partition it into 10 partitions, which results in 1 million rows per partition but I do not do anything else (like move the partitions to different file groups or spindles) Will I see a performance increase? Is this in effect like creating 10 smaller tables? If I have queries that perform key lookups or scans, will the performance increase as if they were operating against a much smaller table? I'm trying to understand how partitioning is different

问题 Say I have a table with a large number of rows and one of the columns which I want to index can have one of 20 values. If I were to put an index on the column would it be large? If so, why? If I were to partition the data into the data into 20 tables, one for each value of the column, the index size would be trivial but the indexing effect would be the same. 回答1: It's not the indexes that will suck. It's putting indexes on the wrong columns that will suck. Seriously though, why would you need

问题 Say I have a table with a large number of rows and one of the columns which I want to index can have one of 20 values. If I were to put an index on the column would it be large? If so, why? If I were to partition the data into the data into 20 tables, one for each value of the column, the index size would be trivial but the indexing effect would be the same. 回答1: It's not the indexes that will suck. It's putting indexes on the wrong columns that will suck. Seriously though, why would you need