How can Apache Camel be used to monitor file changes?

问题

I would like to monitor all of the files in a given directory for changes, ie an updated timestamp. This use case seems natural for Camel using the file component, but I can't seem to find a way to configure this behavior.

A uri like:

file:/some/directory

will consume the files in the provided directory but will delete them.

A uri like:

file:/some/directory?noop=true

consumes each file once when it is added or when the route is started.

It's surprising that there isn't an option along the lines of

consumeOnChange=true

Is there a straightforward way to monitor file changes and not delete the file after consuming?

回答1:

You can do this by setting up the idempotentKey to tell Camel how a file is considered changed. For example if the file size changes, or its timestamp changes etc.

See more details at the Camel file documentation at: http://camel.apache.org/file2

See the section Avoiding reading the same file more than once (idempotent consumer). And read about idempotent and idempotentKey.

So something alike

from("file:/somedir?noop=true&idempotentKey=${file:name}-${file:size}")

from("file:/somedir?noop=true&idempotentKey=${file:name}-${file:modified}")

You can read here about the various ${file:xxx} tokens you can use: http://camel.apache.org/file-language.html

回答2:

I don't think Camel supports that specific feature but with the existent options you can come up with a similar solution of monitoring a directory.

What you need to do is set a small delay value to check the directory and maintain a repository of the already read files. Depending on how you configure the repository (by size, by filename, by a mix of them...) this solution would be able to provide you information about news files and modified files. As a caveat it would be consuming the files in the directory very often.

Maybe you could use other solutions different from Camel like Apache Commons VFS2 (I wrote a explanation about how to use it for this scenario: WatchService locks some files?

回答3:

I faced the same problem i.e. wanted to copy updated files also (along with new files). Below is my configuration,

public static void main(String[] a) throws Exception {

    CamelContext cc = new DefaultCamelContext();

    cc.addRoutes(createRouteBuilder());

    cc.start();

    Thread.sleep(10 * 60 * 1000);

    cc.stop();
}


protected static RouteBuilder createRouteBuilder() {
    return new RouteBuilder() {
        public void configure() {
            from("file://D:/Production"
                    + "?idempotent=true"
                    + "&idempotentKey=${file:name}-${file:size}"
                    + "&include=.*.log"
                    + "&noop=true"
                    + "&readLock=changed")

            .to("file://D:/LogRepository");
        }
    };
}

My testing steps:

Run the program and it copies few .log files from D:/Production to D:/LogRepository and then continues to poll D:/Production directory
I opened a already copied log say A.log from D:/Production (since noop=true nothing is moved) and edited it with some editor tool. This doubled the file size and save it.

At this point I think Camel is supposed to copy that particular file again since its size is modified and in my route definition I used "idempotent=true&idempotentKey=${file:name}-${file:size}&readLock=changed". But camel ignores the file. When I use TRACE for logging it says "Skipping as file is already in progress...", but I did not find any lock file in D:/Production directory when I editted and saved the file.

I also checked that camel still ignores the file if I replace A.log (with same name but bigger size) in D:/Production directory from outside.

But I found, everything is working as expected if I remove noop=true option.

Am I missing something?

回答4:

Setting noop to true will result in Camel setting idempotent=true as well, despite the fact that idempotent is false by default.

Simplest solution to monitor files would be:

.from("file:path?noop=true&idempotent=false&delay=60s")

This will monitor changes to all files in the given directory every one minute.

This can be found in the Camel documentation at: http://camel.apache.org/file2.html.

回答5:

If you want monitor file changes in camel, use file-watch component.

Example -> RECURSIVE WATCH ALL EVENTS (FILE CREATION, FILE DELETION, FILE MODIFICATION):

from("file-watch://some-directory")
.log("File event: ${header.CamelFileEventType} occurred on file ${header.CamelFileName} at ${header.CamelFileLastModified}");

You can see the complete documentation here: Camel file-watch component

来源：https://stackoverflow.com/questions/20086532/how-can-apache-camel-be-used-to-monitor-file-changes

标签

file

apache-camel