Flume HDFS Sink generates lots of tiny files on HDFS

Question

I have a toy setup sending log4j messages to hdfs using flume. I\'m not able to configure the hdfs sink to avoid many small files. I thought I could configure the hdfs sink to

Answer 1

It is your typo in conf.

#sink config
a1.sinks.i1.type=hdfs
a1.sinks.i1.hdfs.path=hdfs://localhost:8020/user/myName/flume/events
#never roll-based on time
a1.sinks.i1.hdfs.rollInterval=0
#10MB=10485760
a1.sinks.il.hdfs.rollSize=10485760
#never roll base on number of events
a1.sinks.il.hdfs.rollCount=0

where in the line 'rollSize' and 'rollCount', you put il as i1. Please try to use DEBUG, then you will find like:

[SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.hdfs.BucketWriter.shouldRotate:465)  - rolling: rollSize: 1024, bytes: 1024

Due to il, default value of rollSize 1024 is being used .