Hive external table optimal partition size

Question

What is the optimal size for external table partition? I am planning to partition table by year/month/day and we are getting about 2GB of data daily.

Answer 1

Hive partitioning is most effective in cases where the data is sparse. By sparse I mean that the data internally has visible partitions such as by year, month or day.

In your case, partitioning by date doesn't make much sense as each day will have 2 Gb of data which is not too big to handle. Partitioning by week or month makes more sense as it will optimize the query time and will not create too many small partition files.