How can Apache Camel be used to monitor file changes?

Question

I would like to monitor all of the files in a given directory for changes, ie an updated timestamp. This use case seems natural for Camel using the file component, but I can

Answer 1

Setting noop to true will result in Camel setting idempotent=true as well, despite the fact that idempotent is false by default.

Simplest solution to monitor files would be:

.from("file:path?noop=true&idempotent=false&delay=60s")

This will monitor changes to all files in the given directory every one minute.

This can be found in the Camel documentation at: http://camel.apache.org/file2.html.

Answer 2

You can do this by setting up the idempotentKey to tell Camel how a file is considered changed. For example if the file size changes, or its timestamp changes etc.

See more details at the Camel file documentation at: https://camel.apache.org/components/latest/file-component.html

See the section Avoiding reading the same file more than once (idempotent consumer). And read about idempotent and idempotentKey.

So something alike

from("file:/somedir?noop=true&idempotentKey=${file:name}-${file:size}")

Or

from("file:/somedir?noop=true&idempotentKey=${file:name}-${file:modified}")

You can read here about the various ${file:xxx} tokens you can use: http://camel.apache.org/file-language.html

Answer 3

If you want monitor file changes in camel, use file-watch component.

Example -> RECURSIVE WATCH ALL EVENTS (FILE CREATION, FILE DELETION, FILE MODIFICATION):

from("file-watch://some-directory")
.log("File event: ${header.CamelFileEventType} occurred on file ${header.CamelFileName} at ${header.CamelFileLastModified}");

You can see the complete documentation here: Camel file-watch component

Answer 4

I faced the same problem i.e. wanted to copy updated files also (along with new files). Below is my configuration,

public static void main(String[] a) throws Exception {

    CamelContext cc = new DefaultCamelContext();

    cc.addRoutes(createRouteBuilder());

    cc.start();

    Thread.sleep(10 * 60 * 1000);

    cc.stop();
}


protected static RouteBuilder createRouteBuilder() {
    return new RouteBuilder() {
        public void configure() {
            from("file://D:/Production"
                    + "?idempotent=true"
                    + "&idempotentKey=${file:name}-${file:size}"
                    + "&include=.*.log"
                    + "&noop=true"
                    + "&readLock=changed")

            .to("file://D:/LogRepository");
        }
    };
}

My testing steps:

1. Run the program and it copies few .log files from D:/Production to D:/LogRepository and then continues to poll D:/Production directory
2. I opened a already copied log say A.log from D:/Production (since noop=true nothing is moved) and edited it with some editor tool. This doubled the file size and save it.

At this point I think Camel is supposed to copy that particular file again since its size is modified and in my route definition I used "idempotent=true&idempotentKey=${file:name}-${file:size}&readLock=changed". But camel ignores the file. When I use TRACE for logging it says "Skipping as file is already in progress...", but I did not find any lock file in D:/Production directory when I editted and saved the file.

I also checked that camel still ignores the file if I replace A.log (with same name but bigger size) in D:/Production directory from outside.

But I found, everything is working as expected if I remove noop=true option.

Am I missing something?

Answer 5

I don't think Camel supports that specific feature but with the existent options you can come up with a similar solution of monitoring a directory.

What you need to do is set a small delay value to check the directory and maintain a repository of the already read files. Depending on how you configure the repository (by size, by filename, by a mix of them...) this solution would be able to provide you information about news files and modified files. As a caveat it would be consuming the files in the directory very often.

Maybe you could use other solutions different from Camel like Apache Commons VFS2 (I wrote a explanation about how to use it for this scenario: WatchService locks some files?