Hive external table optimal partition size

Question

What is the optimal size for external table partition? I am planning to partition table by year/month/day and we are getting about 2GB of data daily.

Answer 1

Hive partitions definition will be stored in the metastore, therefore too many partitions will take much space in the metastore.

Partitions will be stored as directories in the HDFS, therefore many partitions keys will produce hirarchical directories which make their scanning slower.

Your query will be executed as a MapReduce job, therefore it's useless to make too tiny partitions.

It's case depending, think how your data will be queried. For your case I prefer one key defined as 'yyyymmdd', hence we will get 365 partitions / year, only one level in the table directory and 2G data / partition which is nice for a MapReduce job.

For the completness of the answer, if you use Hive < 0.12, make your partition key string typed, see here.

Usefull blog here.