Flume HDFS sink keeps rolling small files
I'm trying to stream twitter data into hdfs using flume and this: https://github.com/cloudera/cdh-twitter-example/ Whatever I try here, it keeps creating files in HDFS that range in size from 1.5kB to 15kB where I would like to see large files (64Mb). Here is the agent configuration: TwitterAgent.sources = Twitter TwitterAgent.channels = MemChannel TwitterAgent.sinks = HDFS TwitterAgent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource TwitterAgent.sources.Twitter.channels = MemChannel TwitterAgent.sources.Twitter.consumerKey = xxxxxx TwitterAgent.sources.Twitter.consumerSecret =