Hive Create Multi small files for each insert in HDFS

Question

following is already been achieved

1. Kafka Producer pulling data from twitter using Spark Streaming.
2. Kafka Consumer ingesting data into Hive External t

Answer 1

you can use these options together.

1. turn on acid
2. create orc table K with transactional property.
3. insert many times into K. by streaming or just use insert dml.
4. hive will automatically create small delta files
5. minor ir major compactions will happen
6. small files will be merged to large file.