partitioning

I have parquet data partitioned by date & hour , folder structure: events_v3 -- event_date=2015-01-01 -- event_hour=2015-01-1 -- part10000.parquet.gz -- event_date=2015-01-02 -- event_hour=5 -- part10000.parquet.gz I have created a table raw_events via spark but when I try to query, it scans all the directories for footer and that slows down the initial query, even if I am querying only one day worth of data. query: select * from raw_events where event_date='2016-01-01' similar problem : http://mail-archives.apache.org/mod_mbox/spark-user/201508.mbox/%3CCAAswR-7Qbd2tdLSsO76zyw9tvs

Is it possible to partition a table using 2 columns instead of only 1 for the partition function? Consider a table with 3 columns ID (int, primary key, Date (datetime), Num (int) I want to partition this table by 2 columns: Date and Num . This is what I do to partition a table using 1 column (date): create PARTITION FUNCTION PFN_MonthRange (datetime) AS RANGE left FOR VALUES ('2009-11-30 23:59:59:997', '2009-12-31 23:59:59:997', '2010-01-31 23:59:59:997', '2010-28-02 23:59:59:997', '2010-03-31 23:59:59:997') go Bad News: The partition function has to be defined on a single column. Good News:

Kind of edge case, when saving parquet table in Spark SQL with partition, #schema definitioin final StructType schema = DataTypes.createStructType(Arrays.asList( DataTypes.createStructField("time", DataTypes.StringType, true), DataTypes.createStructField("accountId", DataTypes.StringType, true), ... DataFrame df = hiveContext.read().schema(schema).json(stringJavaRDD); df.coalesce(1) .write() .mode(SaveMode.Append) .format("parquet") .partitionBy("year") .saveAsTable("tblclick8partitioned"); Spark warns: Persisting partitioned data source relation into Hive metastore in Spark SQL specific

I want to check how can we get information about each partition such as total no. of records in each partition on driver side when Spark job is submitted with deploy mode as a yarn cluster in order to log or print on the console. You can get the number of records per partition like this : df .rdd .mapPartitionsWithIndex{case (i,rows) => Iterator((i,rows.size))} .toDF("partition_number","number_of_records") .show But this will also launch a Spark Job by itself (because the file must be read by spark to get the number of records). Spark could may also read hive table statistics, but I don't know

After reading the tips from this great Nettuts+ article I've come up with a table schema that would separate highly volatile data from other tables subjected to heavy reads and at the same time lower the number of tables needed in the whole database schema, however I'm not sure if this is a good idea since it doesn't follow the rules of normalization and I would like to hear your advice, here is the general idea: I've four types of users modeled in a Class Table Inheritance structure, in the main "user" table I store data common to all the users ( id , username , password , several flags , ...

I know that horizontal partitioning...you can create many tables. How can you do this with multiple servers? This will allow Mysql to scale. Create X tables on X servers? Does anyone care to explain, or have a good beginner's tutorial (step-by-step) that teaches you how to partition across multiple servers? With MySQL, people generally do what is called application based sharding . In a nutshell, you will have the same database structure on multiple database servers. But it won't contain the same data. So for example: Users 1 - 10000: server A Users 10001 - 20000: server B Sharding (of course)

In Kafka, I would like to use only a single broker, single topic and a single partition having one producer and multiple consumers (each consumer getting its own copy of data from the broker). Given this, I do not want the overhead of using Zookeeper; Can I not just use the broker only? Why is a Zookeeper must? John Petrone Yes, Zookeeper is required for running Kafka. From the Kafka Getting Started documentation: Step 2: Start the server Kafka uses zookeeper so you need to first start a zookeeper server if you don't already have one. You can use the convenience script packaged with kafka to