Flume-ng tail a file

Question

I am trying to understand how to tail a file with flume-ng so that I can push the data into HDFS. In the first instance I have setup a simple conf file:

tail         


        

Answer 1

I suppose you can use taildir source if you are using Flume 1.7.0

The following is what I used in my project:

a1.sources.r1.type = TAILDIR
a1.sources.r1.positionFile = /xxx/env/flume/taildir_position.json
a1.sources.r1.filegroups = f1
a1.sources.r1.filegroups.f1 = /xxx/logs/file_name.*.log
a1.sources.r1.headers.f1.headerKey1 = yyy
a1.sources.r1.fileHeader = true