问题
I am exploring ways to incrementally copy S3 blobs to azure blobs. I tried azure data factory and I was not able to find this option.
S3 has millions of objects and without an incremental option it takes hours to complete.
I am open to explore other tools/options.
回答1:
if you need to do daily/hourly incrementally copy, the scheduled/tumbling window trigger should be the option. Here is an example to reference. ADFv2 also supports compression in copy, you could specify the type and level of compression for the s3 object, click here for more information.
来源：https://stackoverflow.com/questions/53074516/incrementally-copy-s3-to-azure-blob