readdirectorychangesw

Why is GetOverlappedResult giving 0 bytes result for ReadDirectoryChangesW?

荒凉一梦 — Wed, 04 Mar 2020 21:08:06 +0000

Why is GetOverlappedResult giving 0 bytes result for ReadDirectoryChangesW? 荒凉一梦 2020-03-05 05:08:06

问题

I have written a file system watcher for our project. Suddenly, it stopped getting events properly. I found out, that after GetOverlappedResult returns true, the result data are empty and so is bytes returned.

This is how I create file handle for watching a directory:

_directoryHandle = ::CreateFileA("some path", FILE_LIST_DIRECTORY, FILE_SHARE_READ | FILE_SHARE_DELETE | FILE_SHARE_WRITE,
  NULL, OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OVERLAPPED, NULL);

This is how I start watching:

BOOL _watchRequestResult = false;
OVERLAPPED _ovl = { 0 };
static constexpr DWORD ResultDataSize = 20;
FILE_NOTIFY_INFORMATION _resultData[ResultDataSize] = { 0 };

_watchRequestResult = ::ReadDirectoryChangesW(
  _directoryHandle,
  (LPVOID)_resultData,
  ResultDataSize,
  TRUE,
  FILE_NOTIFY_CHANGE_FILE_NAME,
  NULL,
  &_ovl,
  NULL
);

After I use WaitForMultipleObjects to wait for the event (there's more than one), this is how I try to fetch the results:

DWORD _ovlBytesReturned;
if (::GetOverlappedResult(GetDirectoryHandle(), &_ovl, &_ovlBytesReturned, FALSE))
{
  // Read results
}

But suddenly when I copy file to watched directory the event fires - but I can see in the debugger that _ovlBytesReturned is 0 and _resultData also is just zeroes.

Is there any flag I could try changing to fix this? I'm quite sure it used to work, I have no idea what could have changed.

I already tried to change false to true in GetOverlappedResult(GetDirectoryHandle(), &_ovl, &_ovlBytesReturned, FALSE), in case there was additional need for waiting. It did not have any effect.

回答1:

FILE_NOTIFY_INFORMATION is at least 16 bytes (for 0 wchar_ts long filenames) and you tell ReadDirectoryChangesW that you only have 20 bytes in the buffer (nBufferLength) - so overlapped results will have problems to fit. Use sizeof(_resultData) instead of ResultDataSize for the nBufferLength - but I think you should increase the size of the buffer a lot. 16*20 bytes isn't much when stuff starts happening.

Also note that you can't use _resultData[ index+1 ] to get to the next result. FILE_NOTIFY_INFORMATION is variable length, the next FILE_NOTIFY_INFORMATION is NextEntryOffset bytes ahead (with 0 meaning that you're at the last overlapped result).

You also need to create and assign an event handle (hEvent) in your OVERLAPPED structure in order for GetOverlappedResult() to work unless you use a completion routine instead - and the directory handle must be open all the time or you'll miss events.

Pseudo code:

handle = CreateFileW(...FILE_FLAG_OVERLAPPED...);
while(read_directory_changes) {
  ReadDirectoryChangesW();
  WaitForSingleObject() / WaitForMultipleObjects();
  GetOverlappedResult();
}
CloseHandle(handle);

Here's an example with those things in place.

#include <Windows.h>

#include <iomanip>
#include <iostream>
#include <memory>
#include <string>
#include <stdexcept>
#include <tuple>
#include <utility>
#include <vector>

// A base class for handles with different invalid values.
template<std::uintptr_t hInvalid>
class Handle {
public:
    Handle(const Handle&) = delete;
    Handle(Handle&& rhs) :
        hHandle(std::exchange(rhs.hHandle, hInvalid))
    {}
    Handle& operator=(const Handle&) = delete;
    Handle& operator=(Handle&& rhs) {
        std::swap(hHandle, rhs.hHandle);
        return *this;
    }

    // converting to a normal HANDLE
    operator HANDLE () { return hHandle; }

protected:
    Handle(HANDLE v) : hHandle(v) {
        // throw if we got an invalid handle
        if (hHandle == reinterpret_cast<HANDLE>(hInvalid))
            throw std::runtime_error("invalid handle");
    }
    ~Handle() {
        if (hHandle != reinterpret_cast<HANDLE>(hInvalid)) CloseHandle(hHandle);
    }
private:
    HANDLE hHandle;
};

using InvalidNullptrHandle = Handle<reinterpret_cast<std::uintptr_t>(nullptr)>;
using InvalidHandleValueHandle =
                Handle<reinterpret_cast<std::uintptr_t>(INVALID_HANDLE_VALUE)>;

// A class for directory handles
class DirectoryHandleW : public InvalidHandleValueHandle {
public:
    DirectoryHandleW(const std::wstring& dir) :
        Handle(
            ::CreateFileW(
                dir.c_str(), FILE_LIST_DIRECTORY,
                FILE_SHARE_READ | FILE_SHARE_DELETE | FILE_SHARE_WRITE,
                NULL, OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS |
                FILE_FLAG_OVERLAPPED, NULL)
        )
    {}
};

// A class for event handles
class EventHandle : public InvalidNullptrHandle {
public:
    EventHandle() :
        Handle(::CreateEvent(nullptr, true, false, nullptr))
    {}
};

// FILE_NOTIFY_INFORMATION action names
wchar_t const* get_action(DWORD a) {
    static wchar_t const* const Actions[FILE_ACTION_RENAMED_NEW_NAME + 1] = {
        L"Unknown action",
        L"ADDED",
        L"REMOVED",
        L"MODIFIED",
        L"RENAMED_OLD_NAME",
        L"RENAMED_NEW_NAME"
    };

    if (a > FILE_ACTION_RENAMED_NEW_NAME) a = 0;
    return Actions[a];
}

// A stepping function for FILE_NOTIFY_INFORMATION*
bool StepToNextNotifyInformation(FILE_NOTIFY_INFORMATION*& cur) {
    if (cur->NextEntryOffset == 0) return false;
    cur = reinterpret_cast<FILE_NOTIFY_INFORMATION*>(
        reinterpret_cast<char*>(cur) + cur->NextEntryOffset
    );
    return true;
}

// A ReadDirectoryChanges support class
template<size_t Handles=1, size_t BufByteSize = 4096>
class DirectoryChangesReader {
public:
    static_assert(Handles > 0, "There must be room for at least 1 HANDLE");
    static_assert(BufByteSize >= sizeof(FILE_NOTIFY_INFORMATION) + MAX_PATH, "BufByteSize too small");
    static_assert(BufByteSize % sizeof(DWORD) == 0, "BufByteSize must be a multiple of sizeof(DWORD)");

    DirectoryChangesReader(const std::wstring& dirname) :
        hDir(dirname),
        ovl{},
        hEv{},
        handles{hEv},
        buffer{std::make_unique<DWORD[]>(BufByteSize/sizeof(DWORD))}
    {}

    // A function to fill in data to use with ReadDirectoryChangesW
    void EnqueueReadDirectoryChanges() {
        ovl = OVERLAPPED{};
        ovl.hEvent = hEv;;
        BOOL rdc = ::ReadDirectoryChangesW(
            hDir,
            buffer.get(),
            BufByteSize,
            TRUE,
            FILE_NOTIFY_CHANGE_FILE_NAME | FILE_NOTIFY_CHANGE_DIR_NAME |
            FILE_NOTIFY_CHANGE_ATTRIBUTES | FILE_NOTIFY_CHANGE_SIZE |
            FILE_NOTIFY_CHANGE_LAST_WRITE | FILE_NOTIFY_CHANGE_LAST_ACCESS |
            FILE_NOTIFY_CHANGE_CREATION | FILE_NOTIFY_CHANGE_SECURITY,
            NULL,
            &ovl,
            NULL
        );
        if (rdc == 0) throw std::runtime_error("EnqueueReadDirectoryChanges failed");
    }

    // A function to get a vector of <Action>, <Filename> pairs
    std::vector<std::pair<wchar_t const*, std::wstring>>
    GetDirectoryChangesResultW() {
        std::vector<std::pair<wchar_t const*, std::wstring>> retval;

        FILE_NOTIFY_INFORMATION* fni = reinterpret_cast<FILE_NOTIFY_INFORMATION*>(buffer.get());

        DWORD ovlBytesReturned;
        if (::GetOverlappedResult(hDir, &ovl, &ovlBytesReturned, TRUE)) {
            do {
                retval.emplace_back(
                    get_action(fni->Action),
                    std::wstring{fni->FileName,
                                 fni->FileName + fni->FileNameLength / sizeof(wchar_t)}
                );
            } while (StepToNextNotifyInformation(fni));
        }
        return retval;
    }

    // wait for the handles in the handles array
    DWORD WaitForHandles() {
        return ::WaitForMultipleObjects(Handles, handles, false, INFINITE);
    }

    // access to the handles array
    HANDLE& operator[](size_t idx) { return handles[idx]; }
    constexpr size_t handles_count() const { return Handles; }
private:
    DirectoryHandleW hDir;
    OVERLAPPED ovl;
    EventHandle hEv;
    HANDLE handles[Handles];
    std::unique_ptr<DWORD[]> buffer; // DWORD-aligned
};

int main()
{
    try {
        DirectoryChangesReader dcr(L"C:\\Users\\Ted\\Testing");

        while (true) {
            dcr.EnqueueReadDirectoryChanges();

            DWORD rv = dcr.WaitForHandles();
            if (rv == WAIT_OBJECT_0) {
                auto res = dcr.GetDirectoryChangesResultW();

                std::wcout << L"Got " << res.size() << L" changes\n";
                for (auto const& [action, filename] : res) {
                    std::wcout << action << L" " << filename << L"\n";
                }
            }
            else if (rv > WAIT_OBJECT_0 && rv < WAIT_OBJECT_0 + dcr.handles_count()) {
                // some other event you waited on
                auto event_idx = rv - WAIT_OBJECT_0;
            }
            else {
                std::wcerr << L"Some kind of problem\n";
                break;
            }
        }
    }
    catch (const std::exception& ex) {
        std::cout << ex.what() << "\n";
    }
}

来源：https://stackoverflow.com/questions/58508771/why-is-getoverlappedresult-giving-0-bytes-result-for-readdirectorychangesw

标签

c++

winapi

readdirectorychangesw

getoverlappedresult

ReadDirectoryChangesW only places single event in the FILE_NOTIFY_INFORMATION buffer

血红的双手。 — Fri, 24 Jan 2020 22:56:06 +0000

ReadDirectoryChangesW only places single event in the FILE_NOTIFY_INFORMATION buffer 血红的双手。 2020-01-25 06:56:06

问题

I have a problem that ReadDirectoryChangesW keeps missing events.

I did a lot of googling and the bellow function arguments seem correct according to my searches, but nobody knows for certain. I start watching like this.

BOOL _watchRequestResult = false;
OVERLAPPED _ovl = { 0 };
_ovl.hEvent = ::CreateEventA(NULL, TRUE, FALSE, NULL);

_directoryHandle = ::CreateFileA("some path here", FILE_LIST_DIRECTORY, FILE_SHARE_READ | FILE_SHARE_DELETE | FILE_SHARE_WRITE,
  NULL, OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OVERLAPPED, NULL);

// This should be quite enough to fit multiple file events
static constexpr DWORD ResultDataLength = 10000;
// Byte size used for winapi calls and memcpy during move operation
static constexpr DWORD ResultDataByteSize = ResultDataLength * sizeof(FILE_NOTIFY_INFORMATION);
FILE_NOTIFY_INFORMATION _resultData[ResultDataLength] = { 0 };

_watchRequestResult = ::ReadDirectoryChangesW(
  _directoryHandle,
  (LPVOID)_resultData,
  ResultDataByteSize,
  TRUE,
  FILE_NOTIFY_CHANGE_FILE_NAME,
  NULL,
  &_ovl,
  NULL
);

After the above, I wait for the _ovl.hEvent using WaitForMultipleObjects. I use multiple objects, because there's always also the event I ise to tell the watch thread to quit.

If the ovl.hEvent is notified, I do this:

DWORD _ovlBytesReturned = 0;
// Imagine some struct that I use to pass the file info, not important how it looks
std::vector<MyFileInfoStruct> results;
if (::GetOverlappedResult(_directoryHandle, &_ovl, &_ovlBytesReturned, TRUE))
{
  int byteIndex = 0;
  bool previousWasRename = false;
  const int minSize = min(ResultDataLength, _ovlBytesReturned);
  while (byteIndex < minSize)
  {
    FILE_NOTIFY_INFORMATION* info = reinterpret_cast<FILE_NOTIFY_INFORMATION*>(reinterpret_cast<char*>(&_resultData[0]) + byteIndex);
    byteIndex += info->NextEntryOffset;

    // read the stuff in the info
    results.push_back(MyFileInfoStruct::FromFileInfo(info));

    // If next entry index is 0, it means there is no next entry
    if (info->NextEntryOffset == 0)
    {
      break;
    }
  }
}
// if file is renamed, merge new name and old name to same result. However rename works to give me two FILE_NOTIFY_INFORMATION that both contain expected data
MergeResultRename(results)
// results is always 1 item long

I should note at this point, that info->NextEntryOffset is not always 0 - if I rename a file, I correctly get two entries in _resultData, one for new filename and one for old.

What never get is multiple file changes per event. This is a problem, the whole code looks like this (pseudocode)

Let EVENTS be an array of HANDLE event objects
Let FILE_CHANGES be a buffer of file changes 
while(shouldBeWatching) 
{
  Wait for events from previous iteration stored in EVENTS array. Skip on first iteration.
  if(event has fired) 
  {
    if(event that fired is ovl.hEvent)
    {
      Put file changes from the event that fired into FILE_CHANGES array (seen in 2nd code sample above)
      Delete and close all handles related to the event:
         Close directory handle
         Close ovl.hEvent
    }
    else 
    {
      Close everything and quit thread.
    }
  }
  Start new request (seen above in 1st code sample)
  if(FILE_CHANGES is not empty) 
  {
    Process all info from FILE_CHANGES
  }
}

Now you can see that I restart the request for ReadDirectoryChangesW before I process the MyFileInfoStruct array. But the problem is, if more than two files are copied, the second file is registered by the event while I am processing the previous one, but the subsequent changes are ignored until I "pick up" the last change and restart the event.

I could fist this partially by having second thread to do the Process all info from FILE_CHANGES part. But that only reduces chance of missing the events by making the whole start request -> wait -> pick up -> restart event routine a bit faster. It does not actually provide 100% coverage, there is still a moment where no ReadDirectoryChangesW request is pending.

I've been reading a lot on the internet and found two solutions being mentioned often:

Use separate thread for processing file changes (I already did that)
Increase the size of the FILE_NOTIFY_INFORMATION[]. this doesn't work for me, windows only puts one event there

Thus the question is: How do I get ReadDirectoryChangesW and GetOverlappedResult to keep adding file changes in the FILE_NOTIFY_INFORMATION[] buffer until I "pick up" the results by calling GetOverlappedResult? is this even possible? Has anyone managed to get multiple results into one buffer?

回答1:

For renaming a file there are two actions happened: FILE_ACTION_RENAMED_OLD_NAME and FILE_ACTION_RENAMED_NEW_NAME. These two action events you can retrieve via once calling of ReadDirectoryChanges and GetOverlappedResult as you already have done.

I have a problem that ReadDirectoryChangesW keeps missing events.

To capture events like copy two files and delete two files, for example, firstly I copy TESTA.txt and TESTB.txt to directory D:\testFolder, then delete them both. I can get all events by calling ReadDirectoryChanges and GetOverlappedResult in a while loop. Both functions called by four times for fours events.

Test code as follows:

#include <windows.h>
#include <vector>

using namespace std;

typedef struct TEST_INFO {
    DWORD NextEntryOffset;
    DWORD Action;
    DWORD FileNameLength;
    WCHAR FileName[100];
}_TEST_INFO;

int main()
{
    BOOL _watchRequestResult = false;
    OVERLAPPED _ovl = { 0 };
    _ovl.hEvent = ::CreateEventA(NULL, TRUE, FALSE, NULL);

    HANDLE _directoryHandle = ::CreateFileA("d:\\testFolder", FILE_LIST_DIRECTORY, FILE_SHARE_READ | FILE_SHARE_DELETE | FILE_SHARE_WRITE,
        NULL, OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OVERLAPPED, NULL);

    // This should be quite enough to fit multiple file events
    static constexpr DWORD ResultDataSize = 100;
    _TEST_INFO _resultData[ResultDataSize] = { 0 };

    while (true)
    {

        _watchRequestResult = ::ReadDirectoryChangesW(
            _directoryHandle,
            (LPVOID)_resultData,
            ResultDataSize * sizeof(_TEST_INFO),
            TRUE,
            FILE_NOTIFY_CHANGE_FILE_NAME,
            NULL,
            &_ovl,
            NULL
        );

        DWORD _ovlBytesReturned = 0;

        if (::GetOverlappedResult(_directoryHandle, &_ovl, &_ovlBytesReturned, TRUE))
        {
            int byteIndex = 0;

            while (TRUE)
            {
                _TEST_INFO* info = reinterpret_cast<_TEST_INFO*>(reinterpret_cast<char*>(&_resultData[0]) + byteIndex);
                byteIndex += info->NextEntryOffset;

                wprintf(L"File name: %s, ", info->FileName);
                printf("Action: ");
                switch (info->Action)
                {
                case FILE_ACTION_ADDED:
                    printf("Added \n");
                    break;
                case FILE_ACTION_REMOVED:
                    printf("Removed \n");
                    break;
                case FILE_ACTION_MODIFIED:
                    printf("Modified \n");
                    break;
                case FILE_ACTION_RENAMED_OLD_NAME:
                    printf("Rename old name \n");
                    break;
                case FILE_ACTION_RENAMED_NEW_NAME:
                    printf("Rename new name \n");
                    break;
                }

                // If next entry index is 0, it means there is no next entry
                if (info->NextEntryOffset == 0)
                {
                    break;
                }
            }
        }

    }

    getchar();
}

Summary:

GetOverlappedResult function:

The results reported by the GetOverlappedResult function are those of the specified handle's last overlapped operation.

So renaming a file is an overlapped (rename) operation and copying a file is an overlapped (copy) operation. However copying two files are two overlapped (copy) operations. So it requires calling GetOverlappedResult two times instead of one.

来源：https://stackoverflow.com/questions/59884174/readdirectorychangesw-only-places-single-event-in-the-file-notify-information-bu

标签

c++

winapi

readdirectorychangesw

getoverlappedresult

Using ReadDirectoryChangesW asynchronously in a loop

我的未来我决定 — Sun, 22 Dec 2019 18:16:39 +0000

Using ReadDirectoryChangesW asynchronously in a loop 我的未来我决定 2019-12-23 02:16:39

问题

INTRODUCTION:

I am trying to use ReadDirectoryChangesW asynchronously in a loop.

Below snippet illustrates what I am trying to achieve:

DWORD example()
{
    DWORD error = 0;

    OVERLAPPED ovl = { 0 };
    ovl.hEvent = ::CreateEvent(NULL, TRUE, FALSE, NULL);

    if (NULL == ovl.hEvent) return ::GetLastError();

    char buffer[1024];

    while(1)
    {
        process_list_of_existing_files();

        error = ::ReadDirectoryChangesW(
            m_hDirectory, // I have added FILE_FLAG_OVERLAPPED in CreateFile
            buffer, sizeof(buffer), FALSE,
            FILE_NOTIFY_CHANGE_FILE_NAME,
            NULL, &ovl, NULL);

        // we have new files, append them to the list
        if(error) append_new_files_to_the_list(buffer);
        // just continue with the loop
        else if(::GetLastError() == ERROR_IO_PENDING) continue;
        // RDCW error, this is critical -> exit
        else return ::GetLastError(); 
    }
}

PROBLEM:

I do not know how to handle the case when ReadDirectoryChangesW returns FALSE with GetLastError() code being ERROR_IO_PENDING.

In that case I should just continue with the loop and keep looping until ReadDirectoryChangesW returns buffer I can process.

MY EFFORTS TO SOLVE THIS:

I have tried using WaitForSingleObject(ovl.hEvent, 1000) but it crashes with error 1450 ERROR_NO_SYSTEM_RESOURCES. Below is the MVCE that reproduces this behavior:

#include <iostream>
#include <Windows.h>

DWORD processDirectoryChanges(const char *buffer)
{
    DWORD offset = 0;
    char fileName[MAX_PATH] = "";
    FILE_NOTIFY_INFORMATION *fni = NULL;

    do
    {
        fni = (FILE_NOTIFY_INFORMATION*)(&buffer[offset]);
        // since we do not use UNICODE, 
        // we must convert fni->FileName from UNICODE to multibyte
        int ret = ::WideCharToMultiByte(CP_ACP, 0, fni->FileName,
            fni->FileNameLength / sizeof(WCHAR),
            fileName, sizeof(fileName), NULL, NULL);

        switch (fni->Action)
        {
        case FILE_ACTION_ADDED:     
        {
            std::cout << fileName << std::endl;
        }
        break;
        default:
            break;
        }

        ::memset(fileName, '\0', sizeof(fileName));
        offset += fni->NextEntryOffset;

    } while (fni->NextEntryOffset != 0);

    return 0;
}

int main()
{
    HANDLE hDir = ::CreateFile("C:\\Users\\nenad.smiljkovic\\Desktop\\test", 
        FILE_LIST_DIRECTORY,
        FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE,
        NULL, OPEN_EXISTING, 
        FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OVERLAPPED, NULL);

    if (INVALID_HANDLE_VALUE == hDir) return ::GetLastError();

    OVERLAPPED ovl = { 0 };
    ovl.hEvent = ::CreateEvent(NULL, TRUE, FALSE, NULL);

    if (NULL == ovl.hEvent) return ::GetLastError();

    DWORD error = 0, br;
    char buffer[1024];

    while (1)
    {
        error = ::ReadDirectoryChangesW(hDir,
            buffer, sizeof(buffer), FALSE,
            FILE_NOTIFY_CHANGE_FILE_NAME,
            NULL, &ovl, NULL);

        if (0 == error)
        {
            error = ::GetLastError();

            if (ERROR_IO_PENDING != error)
            {
                ::CloseHandle(ovl.hEvent);
                ::CloseHandle(hDir);
                return error;
            }
        }

        error = ::WaitForSingleObject(ovl.hEvent, 0);

        switch (error)
        {
        case WAIT_TIMEOUT:
            break;
        case WAIT_OBJECT_0:
        {
            error = processDirectoryChanges(buffer);

            if (error > 0)
            {
                ::CloseHandle(ovl.hEvent);
                ::CloseHandle(hDir);
                return error;
            }

            if (0 == ::ResetEvent(ovl.hEvent))
            {
                error = ::GetLastError();
                ::CloseHandle(ovl.hEvent);
                ::CloseHandle(hDir);
                return error;
            }
        }
        break;
        default:
            error = ::GetLastError();
            ::CloseHandle(ovl.hEvent);
            ::CloseHandle(hDir);
            return error;
            break;
        }
    }

    return 0;
}

Reading through the documentation, it seems that I need GetOverlappedResult with last parameter set to FALSE but I do not know how to use this API properly.

QUESTION:

Since the MVCE illustrates very well what I am trying to do (print the names of the newly added files), can you show me what must be fixed in the while loop in order for it to work?

Again, the point is to use ReadDirectoryChangesW asynchronously, in a loop, as shown in the snippet from the INTRODUCTION.

回答1:

The basic structure of your program looks more or less OK, you're just using the asynchronous I/O calls incorrectly. Whenever there are no new files, the wait on the event handle times out immediately, which is fine, but you then issue a brand new I/O request, which isn't.

That's why you're running out of system resources; you're issuing I/O requests full tilt without waiting for any of them to complete. You should only issue a new request after the existing request has completed.

(Also, you should be calling GetOverlappedResult to check whether the I/O was successful or not.)

So your loop should look more like this:

    ::ReadDirectoryChangesW(hDir,
        buffer, sizeof(buffer), FALSE,
        FILE_NOTIFY_CHANGE_FILE_NAME,
        NULL, &ovl, NULL);

    while (1)
    {
        DWORD dw;
        DWORD result = ::WaitForSingleObject(ovl.hEvent, 0);

        switch (result)
        {
        case WAIT_TIMEOUT:

            processBackgroundTasks();

            break;

        case WAIT_OBJECT_0:

            ::GetOverlappedResult(hDir, &ovl, &dw, FALSE);

            processDirectoryChanges(buffer);

            ::ResetEvent(ovl.hEvent);

            ::ReadDirectoryChangesW(hDir,
                buffer, sizeof(buffer), FALSE,
                FILE_NOTIFY_CHANGE_FILE_NAME,
                NULL, &ovl, NULL);

            break;
        }
    }

Notes:

The error handling has been elided for simplicity; I have not done any testing or checked your code for any other problems.
If there might not be any background tasks to perform, you should test for that case and set the timeout to INFINITE rather than 0 when it occurs, otherwise you will be spinning.
I wanted to only show the minimal changes necessary to make it work, but calling WaitForSingleObject followed by GetOverlappedResult is redundant; a single call to GetOverlappedResult can both check whether the I/O is complete and retrieve the results if it is.

As requested, the modified version using only GetOverlappedResult and with minimal error checking. I've also added an example of how you might deal with the case where you've run out of work to do; if whatever processing you're doing on the files really does run forever, you don't need that bit.

    ::ResetEvent(ovl.hEvent);

    if (!::ReadDirectoryChangesW(hDir,
        buffer, sizeof(buffer), FALSE,
        FILE_NOTIFY_CHANGE_FILE_NAME,
        NULL, &ovl, NULL))
    {
       error = GetLastError();
       if (error != ERROR_IO_PENDING) fail();
    }

    while (1)
    {
        BOOL wait;

        result = process_list_of_existing_files();

        if (result == MORE_WORK_PENDING)
        {
           wait = FALSE;
        } 
        else if (result == NO_MORE_WORK_PENDING)
        {
           wait = TRUE;
        } 

        if (!::GetOverlappedResult(hDir, &ovl, &dw, wait))
        {
           error = GetLastError();
           if (error == ERROR_IO_INCOMPLETE) continue;
           fail();
        }

        processDirectoryChanges(buffer);

        ::ResetEvent(ovl.hEvent);

        if (!::ReadDirectoryChangesW(hDir,
            buffer, sizeof(buffer), FALSE,
            FILE_NOTIFY_CHANGE_FILE_NAME,
            NULL, &ovl, NULL))
        {
           error = GetLastError();
           if (error != ERROR_IO_PENDING) fail();
        } 
    }

回答2:

variant of indirect using IOCP

create class/struct inherited (containing) OVERLAPPED (or IO_STATUS_BLOCK), reference count, directory handle and data which you need
call BindIoCompletionCallback (RtlSetIoCompletionCallback) for directory handle, for setup your callback
have DoRead() routine, which we first time call from main thread and then from callback
in DoRead() before every call to ReadDirectoryChangesW call AddRef(); because we pass reference (across OVERLAPPED) to our struct to kernel
main (say GUI thread) can continue do own task after initial call DoRead(), unlike APC variant he not need wait in alertable state
in callback we got pointer to our struct from inherited (containing) OVERLAPPED. do any tasks (processDirectoryChanges) ,if need continue spy - call DoRead() and finally call Release()
if ReadDirectoryChangesW from DoRead() fail (as result will be no callback) - we need direct call callback with error code
for stop we can simply close directory handle - as result we got STATUS_NOTIFY_CLEANUP in callback

==================================

//#define _USE_NT_VERSION_

class SPYDATA : 
#ifdef _USE_NT_VERSION_
    IO_STATUS_BLOCK
#else
    OVERLAPPED 
#endif
{
    HANDLE _hFile;
    LONG _dwRef;
    union {
        FILE_NOTIFY_INFORMATION _fni;
        UCHAR _buf[PAGE_SIZE];
    };

    void DumpDirectoryChanges()
    {
        union {
            PVOID buf;
            PBYTE pb;
            PFILE_NOTIFY_INFORMATION pfni;
        };

        buf = _buf;

        for (;;)
        {
            DbgPrint("%x <%.*S>\n", pfni->Action, pfni->FileNameLength >> 1, pfni->FileName);

            ULONG NextEntryOffset = pfni->NextEntryOffset;

            if (!NextEntryOffset)
            {
                break;
            }

            pb += NextEntryOffset;
        }
    }

#ifdef _USE_NT_VERSION_
    static VOID WINAPI _OvCompRoutine(
        _In_    NTSTATUS dwErrorCode,
        _In_    ULONG_PTR dwNumberOfBytesTransfered,
        _Inout_ PIO_STATUS_BLOCK Iosb
        )
    {
        static_cast<SPYDATA*>(Iosb)->OvCompRoutine(dwErrorCode, (ULONG)dwNumberOfBytesTransfered);
    }
#else
    static VOID WINAPI _OvCompRoutine(
        _In_    DWORD dwErrorCode, // really this is NTSTATUS
        _In_    DWORD dwNumberOfBytesTransfered,
        _Inout_ LPOVERLAPPED lpOverlapped
        )
    {
        static_cast<SPYDATA*>(lpOverlapped)->OvCompRoutine(dwErrorCode, dwNumberOfBytesTransfered);
    }
#endif

    VOID OvCompRoutine(NTSTATUS status, DWORD dwNumberOfBytesTransfered)
    {
        DbgPrint("[%x,%x]\n", status, dwNumberOfBytesTransfered);

        if (0 <= status) 
        {
            if (status != STATUS_NOTIFY_CLEANUP)
            {
                if (dwNumberOfBytesTransfered) DumpDirectoryChanges();
                process_list_of_existing_files();// so hard do this here ?!?
                DoRead();
            }
            else
            {
                DbgPrint("\n---- NOTIFY_CLEANUP -----\n");
            }
        }

        Release();
        MyReleaseRundownProtection();
    }

    ~SPYDATA()
    {
        Cancel();
    }

public:

    void DoRead()
    {
        if (MyAcquireRundownProtection())
        {
            AddRef();
#ifdef _USE_NT_VERSION_
            NTSTATUS status = ZwNotifyChangeDirectoryFile(_hFile, 0, 0, this, this, &_fni, sizeof(_buf), FILE_NOTIFY_VALID_MASK, TRUE);
            if (NT_ERROR(status))
            {
                OvCompRoutine(status, 0);
            }
#else
            if (!ReadDirectoryChangesW(_hFile, _buf, sizeof(_buf), TRUE, FILE_NOTIFY_VALID_MASK, (PDWORD)&InternalHigh, this, 0))
            {
                OvCompRoutine(RtlGetLastNtStatus(), 0);
            }
#endif
        }
    }

    SPYDATA()
    {
        _hFile = 0;// ! not INVALID_HANDLE_VALUE because use ntapi for open file
        _dwRef = 1;
#ifndef _USE_NT_VERSION_
        RtlZeroMemory(static_cast<OVERLAPPED*>(this), sizeof(OVERLAPPED));
#endif
    }

    void AddRef()
    {
        InterlockedIncrement(&_dwRef);
    }

    void Release()
    {
        if (!InterlockedDecrement(&_dwRef))
        {
            delete this;
        }
    }

    BOOL Create(POBJECT_ATTRIBUTES poa)
    {
        IO_STATUS_BLOCK iosb;
        NTSTATUS status = ZwOpenFile(&_hFile, FILE_GENERIC_READ, poa, &iosb, FILE_SHARE_VALID_FLAGS, FILE_DIRECTORY_FILE);
        if (0 <= status)
        {
            return
#ifdef _USE_NT_VERSION_
                0 <= RtlSetIoCompletionCallback(_hFile, _OvCompRoutine, 0);
#else
                BindIoCompletionCallback(_hFile, _OvCompRoutine, 0);
#endif
        }
        return FALSE;
    }

    void Cancel()
    {
        if (HANDLE hFile = InterlockedExchangePointer(&_hFile, 0))
        {
            NtClose(hFile);
        }
    }
};

void DemoF()
{
    if (MyInitializeRundownProtection())
    {
        STATIC_OBJECT_ATTRIBUTES(oa, "<SOME_DIRECTORY>");

        if (SPYDATA* p = new SPYDATA)
        {
            if (p->Create(&oa))
            {
                p->DoRead();
            }

            //++ GUI thread run
            MessageBoxW(0, L"wait close program...", L"", MB_OK);
            //-- GUI thread end

            p->Cancel();

            p->Release();
        }

        MyWaitForRundownProtectionRelease();
    }
}

来源：https://stackoverflow.com/questions/40593500/using-readdirectorychangesw-asynchronously-in-a-loop

标签

c++

winapi

asynchronous

readdirectorychangesw

ReadDirectoryChangesW blocks deleting the watched directory

你离开我真会死。 — Sun, 22 Dec 2019 00:16:29 +0000

ReadDirectoryChangesW blocks deleting the watched directory 你离开我真会死。 2019-12-22 08:16:29

问题

I am trying to watch a directory for create/delete/rename changes on windows with python using the ReadDirectoryChangesW API. This is my code and it is working fine:

results = win32file.ReadDirectoryChangesW(self.hDir, 8192, True, self.type, None,
                                           None)
for action, file in results:
    full_filename = os.path.join (self.source_path, file)
    if   action == 1:                                    # Created
        self.fileCreated(full_filename)
    elif action == 2:                                    # Deleted
        self.fileDeleted(full_filename)
    elif action == 3:                                    # Updated
        self.fileUpdated(full_filename)
    elif action == 4:                                    # Renamed from something
        renamed_file = full_filename
    elif action == 5:                                    # Renamed to something
        self.fileRenamed(renamed_file, full_filename)

However, when I try to delete the watched folder from python or from windows explorer, I get:

WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'c:\users\user\appdata\local\temp\new_dir'

I believe this makes sense, but how should I solve this? Because my application should allow the user to remove a watched folder. I tried the solution for the asynchronous method http://www.themacaque.com/?p=859, but it didn't help.

Thanks in advance!

回答1:

From this blog post:

Another potential pitfall of [ReadDirectoryChangesW] is that the referenced directory itself is now "in use" and so can't be deleted. To monitor files in a directory and still allow the directory to be deleted, you would have to monitor the parent directory and its children.

The post also provides some more details on proper use of ReadDirectoryChangesW

回答2:

Deleting the watched folder IS possible under ReadDirectoryChangesW

"Understanding ReadDirectoryChangesW - Part 2" by Jim Beveridge is (as Artomegus mentioned) a very good background for this problem, but the statement explaining FILE_SHARE_DELETE usage is misleading.

I my tests, use of FILE_SHARE_DELETE actually allows deletion / rename of the watched folder. (In other words, you don't need to "watch parent folder" as the only option.)

Here is the working snippet (edited and borrowed heavily from this otherwise excellent "Watch a Directory for Changes" by Tim Golden

# License is same as snippets on this page
# http://timgolden.me.uk/python/win32_how_do_i/watch_directory_for_changes.html
# In other words, bug Tim Golden to publish a license for his snippets
def windows_watch_path(watched_path):
    import win32file
    import win32con

    ACTIONS = {
        1 : "Created",
        2 : "Deleted",
        3 : "Updated",
        4 : "RenamedFrom",
        5 : "RenamedTo"
    }
    # Thanks to Claudio Grondi for the correct set of numbers
    FILE_LIST_DIRECTORY = 0x0001

    try:
        hDir = win32file.CreateFile (
            watched_path
            , FILE_LIST_DIRECTORY
            , win32con.FILE_SHARE_READ | 
              win32con.FILE_SHARE_WRITE | 
              win32con.FILE_SHARE_DELETE
            , None
            , win32con.OPEN_EXISTING
            , win32con.FILE_FLAG_BACKUP_SEMANTICS
            , None
        )
    except:
        # either it does not exist by this time, or some other issue... blah.
        # we'll just say "it 'changed' from 'some other expected state'"
        return [[watched_path, '', ACTIONS[2]]]

    results = win32file.ReadDirectoryChangesW (
        hDir,
        1024,
        True,
        win32con.FILE_NOTIFY_CHANGE_FILE_NAME |
        win32con.FILE_NOTIFY_CHANGE_DIR_NAME |
        win32con.FILE_NOTIFY_CHANGE_ATTRIBUTES |
        win32con.FILE_NOTIFY_CHANGE_SIZE |
        win32con.FILE_NOTIFY_CHANGE_LAST_WRITE |
        win32con.FILE_NOTIFY_CHANGE_SECURITY,
        None,
        None
    )

    files_changed = []
    for action, fn in results:
        files_changed.append(
            [
                watched_path
                , fn
                , ACTIONS[action]
            ]
        )
        # print fullfn, ACTIONS.get(action, "Unknown")
    return files_changed

回答3:

Ok, to this is not simple to solve... In my case (http://www.themacaque.com/?p=859) I ignored the fact of allowing to rename or remove the directory.

What you could do to allow the user to rename the watch folder is to use the ReadDirectoryChangesW on the ancestor of the path to watch and filter the events according to the paths you are watching. I have implemented a new way to perform the watching using twisted to perform the processing of the events. With that solution you could be wathching the ancestors if:

You folder does not have too many brothers to ignore. You do not want to be performing lots and lots of operations to filter events you are no interested in.
There is no problem if the user cannot remove the ancestor.

In the code of Ubuntu One on windows we have been dealing with this problem and we have implemented a nice solution that you can take a look at. It follows a little the implementation of pyinotify on linux with a processor that will allow you to hook an object with callbacks that will be called according to the event in the twisted reactors main loop. Take a look at that code, it might help you.

Any problem late me know either in my blog or in irc (in freenode at #ubuntuone or #pyar) my nickname is mandel ;)

来源：https://stackoverflow.com/questions/6828544/readdirectorychangesw-blocks-deleting-the-watched-directory

标签

python

winapi

readdirectorychangesw

How to use ReadDirectoryChangesW() method with completion routine?

会有一股神秘感。 — Wed, 18 Dec 2019 07:29:22 +0000

How to use ReadDirectoryChangesW() method with completion routine? 会有一股神秘感。 2019-12-18 15:29:22

问题

I want to use function ReadDirectoryChangesW() in asynchronous mode with I/O completion routine supplied.

The question is I don't know how to retrieve the exact information about the change in the completion routine (a CALLBACK function). Completion routine is defined like this:

VOID CALLBACK FileIOCompletionRoutine(
  [in]                 DWORD dwErrorCode,
  [in]                 DWORD dwNumberOfBytesTransfered,
  [in]                 LPOVERLAPPED lpOverlapped
);

I wonder the information is included in the LPOVERLAPPED structure. But I don't know how to get it.

回答1:

Excellent question! It's 7 years late, but here's somewhat of an answer, or, more importantly, how to find it. So, the documentation for ReadDirectoryChangesW:

https://msdn.microsoft.com/en-us/library/windows/desktop/aa365465%28v=vs.85%29.aspx

in the Parameters section gives a link to FileIOCompletionRoutine:

https://msdn.microsoft.com/en-us/library/windows/desktop/aa364052%28v=vs.85%29.aspx

which in the Examples section gives a link to Named Pipe Server Using Completion Routines:

https://msdn.microsoft.com/en-us/library/windows/desktop/aa365601%28v=vs.85%29.aspx

which believe it or not doesn't even use ReadDirectoryChangesW but actually gives an example using ReadFileEx, which also uses FileIOCompletionRoutine:

https://msdn.microsoft.com/en-us/library/windows/desktop/aa365468%28v=vs.85%29.aspx

and you can see an example of them using these two functions (ReadFileEx and CompletedReadRoutine (which is an implementation of the application-defined callback function FileIOCompletionRoutine)) in this piece of code:

// CompletedWriteRoutine(DWORD, DWORD, LPOVERLAPPED) 
// This routine is called as a completion routine after writing to 
// the pipe, or when a new client has connected to a pipe instance.
// It starts another read operation. 

VOID WINAPI CompletedWriteRoutine(DWORD dwErr, DWORD cbWritten, 
   LPOVERLAPPED lpOverLap) 
{ 
   LPPIPEINST lpPipeInst; 
   BOOL fRead = FALSE; 

// lpOverlap points to storage for this instance. 

   lpPipeInst = (LPPIPEINST) lpOverLap; 

// The write operation has finished, so read the next request (if 
// there is no error). 

   if ((dwErr == 0) && (cbWritten == lpPipeInst->cbToWrite)) 
      fRead = ReadFileEx( 
         lpPipeInst->hPipeInst, 
         lpPipeInst->chRequest, 
         BUFSIZE*sizeof(TCHAR), 
         (LPOVERLAPPED) lpPipeInst, 
         (LPOVERLAPPED_COMPLETION_ROUTINE) CompletedReadRoutine); 

// Disconnect if an error occurred. 

   if (! fRead) 
      DisconnectAndClose(lpPipeInst); 
} 

// CompletedReadRoutine(DWORD, DWORD, LPOVERLAPPED) 
// This routine is called as an I/O completion routine after reading 
// a request from the client. It gets data and writes it to the pipe. 

VOID WINAPI CompletedReadRoutine(DWORD dwErr, DWORD cbBytesRead, 
    LPOVERLAPPED lpOverLap) 
{ 
   LPPIPEINST lpPipeInst; 
   BOOL fWrite = FALSE; 

// lpOverlap points to storage for this instance. 

   lpPipeInst = (LPPIPEINST) lpOverLap; 

// The read operation has finished, so write a response (if no 
// error occurred). 

   if ((dwErr == 0) && (cbBytesRead != 0)) 
   { 
      GetAnswerToRequest(lpPipeInst); 

      fWrite = WriteFileEx( 
         lpPipeInst->hPipeInst, 
         lpPipeInst->chReply, 
         lpPipeInst->cbToWrite, 
         (LPOVERLAPPED) lpPipeInst, 
         (LPOVERLAPPED_COMPLETION_ROUTINE) CompletedWriteRoutine); 
   } 

// Disconnect if an error occurred. 

   if (! fWrite) 
      DisconnectAndClose(lpPipeInst); 
}

It's not a great answer (I was only exploring whether I even wanted to use these functions, myself), but it should help people get started.

How to keep ReadDirectoryChangesW from missing file changes

烈酒焚心 — Wed, 18 Dec 2019 03:46:02 +0000

How to keep ReadDirectoryChangesW from missing file changes 烈酒焚心 2019-12-18 11:46:02

问题

There are many posts on the internet about the ReadDirectoryChangesW API function missing files when there is a lot of file activity. Most blame the speed at which the ReadDirectoryChangesW function loop is called. This is an incorrect assumption. The best explanation I have seen is in the following post, the comment on Monday, April 14, 2008 2:15:27 PM

http://social.msdn.microsoft.com/forums/en-US/netfxbcl/thread/4465cafb-f4ed-434f-89d8-c85ced6ffaa8/

The summary is that the ReadDirectoryChangesW function reports file changes as they leave the file-write-behind queue, not as they are added. And if too many are added before being committed, you lose notice on some of them. You can see this with your implementation, if you just write a program to generate a 1000+ files in a directory real quick. Just count how many file event notices you get and you will see there are times when you will not receive all of them.

The question is, has anyone found a reliable method to use the ReadDirectoryChangesW function without having to flush the volume each time? This is not allowed if the user is not an Administrator and can also take some time to complete.

回答1:

If the API is unreliable, then a workaround may be your only option. That of course likely involves keeping track of lastmodified and filenames. ~~What this doesn't mean is that you need to poll when looking for changes, rather, you can use the FileSystemWatcher as a means to trigger checking.~~

So if you keep track of the last 50-100 times the ReadDirectoryChangesW~~/FSW~~ event happened, and you see that it is being called rapidly, you can detect this and trigger the special condition to get all the files that have been changed (and set a flag to prevent future bogus FSW events temporarily) in a few seconds.

Since some people are confused in the comments about this solution, I am proposing that you should monitor how fast events are arriving from the ReadDirectoryChangesW and when they are arriving too fast, try to attempt a workaround (usually a manual sweep of a directory).

回答2:

We've never seen ReadDirectoryChangesW to be 100% reliable. But, the best way to handle it is separate the "reporting" from the "handling".

My implementation has a thread which has only one job, to re-queue all events. Then a second thread to process my intermediate queue. You basically, want to impede the reporting of events as little as possible.

Under high CPU situations, you can also impede the reporting of watcher events.

回答3:

I met same problem. But, I didn't find a solution that guarantee to get all of events. In several tests, I could know that ReadDirectoryChangesW function should be called again as fast as possible after GetQueuedCompletionStatus function returned. I guess if a processing speed of filesystem is very faster than my application processing speed, the application might be able to lose some events.

Anyway, I separated a parsing logic from a monitoring logic and placed a parsing logic on a thread.

来源：https://stackoverflow.com/questions/57254/how-to-keep-readdirectorychangesw-from-missing-file-changes

标签

.net

windows

readdirectorychangesw

CreateFile ,ReadDirectoryChanges issue

纵饮孤独 — Tue, 10 Dec 2019 21:56:23 +0000

CreateFile ,ReadDirectoryChanges issue 纵饮孤独 2019-12-11 05:56:23

问题

I’m using ReadDirectoryChangesW to spy a folder that I opened with CreateFile, once a file was added I call a function (OnFileChanged) that read the size and open it for reading, my application works fine for small sized file but the problem occurs when I try to copy a big file into my folder(7,24 M), and I get Permission denied error after calling fopen to read it.

the watching process is based on this

    void Init(const QString FullPath)
    {    ...
         hDir = CreateFile(
                dirPath, // pointer to the directory containing the tex files
                FILE_LIST_DIRECTORY|GENERIC_READ,                // access (read-write) mode
                FILE_SHARE_READ|FILE_SHARE_DELETE|FILE_SHARE_WRITE,  // share mode
                NULL, // security descriptor
                OPEN_EXISTING, // how to create
                FILE_FLAG_BACKUP_SEMANTICS  | FILE_FLAG_OVERLAPPED , // file attributes
                NULL); // file with attributes to copy


            SecureZeroMemory (&overl, sizeof(overl));
            SecureZeroMemory(buffer, sizeof(buffer));
            overl.hEvent = CreateEvent(NULL, FALSE, FALSE, NULL);

            // watch the directory
           BOOL res= ReadDirectoryChangesW(
                 hDir, /* handle to directory */
                 &buffer[curBuffer], /* read results buffer */
                 sizeof(buffer[curBuffer]), /* length of buffer */
                 FALSE, /* monitoring option */
                 FILE_NOTIFY_CHANGE_FILE_NAME ,
                 //FILE_NOTIFY_CHANGE_LAST_WRITE, /* filter conditions */
                 NULL, /* bytes returned */
                 &overl, /* overlapped buffer */
                 NULL); /* completion routine */

    }

void StartWatchThread()
{
    // if the thread already exists then stop it
    if (IsThreadRunning())
        SynchronousAbort();

    //CrashIf(!hDir);
    if(!hDir)
       { qDebug()<<" handle "<<hDir<<" last error"<<GetLastError();
        exit(-1);}

    else
    {// reset the hEvtStopWatching event so that it can be set if
    // some thread requires the watching thread to stop
    ResetEvent(hEvtStopWatching);

    DWORD watchingthreadID;
    qDebug()<<"befrore creating thread";
    hWatchingThread = CreateThread(NULL, 0, WatchingThread, this, 0, &watchingthreadID);
    qDebug()<<"watchingthreadID"<<watchingthreadID;
    }
}


DWORD WINAPI WatchingThread(void *param)
{
    //qDebug()<<"in WatchingThread";
    FileWatcher *fw = (FileWatcher *)param;

    HANDLE hp[2] = { fw->hEvtStopWatching, fw->overl.hEvent };
    for (;;)
    {

        DWORD dwObj = WaitForMultipleObjects((sizeof(hp)/(sizeof(hp[0])))
                                             , hp, FALSE, INFINITE);
        if (dwObj == WAIT_OBJECT_0) // the user asked to quit the program
        {
            qDebug()<<"in WatchingThread the user asked to quit the program";
            //exit(-1);
            break;
        }
        if (dwObj != WAIT_OBJECT_0 + 1)
        {
            // BUG!
            //assert(0);
            qDebug()<<"dwObj "<<dwObj<<" last error "<<GetLastError();
            break;
        }
        //qDebug()<<"WatchingThread fw->NotifyChange() ";

        //if (fw->wakeup)
            fw->NotifyChange();

    }
    return 0;
} 


bool NotifyChange()
{
    //qDebug()<<"in NotifyChange";

    GetOverlappedResult(hDir, &overl, &dwNumberbytes, FALSE);

    FILE_NOTIFY_INFORMATION *pFileNotify = (FILE_NOTIFY_INFORMATION *)buffer[curBuffer];
    // Switch the 2 buffers
    curBuffer = (curBuffer + 1) % (sizeof(buffer)/(sizeof(buffer[0])));
    SecureZeroMemory(buffer[curBuffer], sizeof(buffer[curBuffer]));
    // start a new asynchronous call to ReadDirectory in the alternate buffer
    ReadDirectoryChangesW(
         hDir, /* handle to directory */
         &buffer[curBuffer], /* read results buffer */
         sizeof(buffer[curBuffer]), /* length of buffer */
         FALSE, /* monitoring option */
         FILE_NOTIFY_CHANGE_FILE_NAME  ,
         //FILE_NOTIFY_CHANGE_LAST_WRITE, /* filter conditions */
         NULL, /* bytes returned */
        &overl, /* overlapped buffer */
         NULL); /* completion routine */

    // Note: the ReadDirectoryChangesW API fills the buffer with WCHAR strings.
    for (;;) {

        if (pFileNotify->Action == FILE_ACTION_ADDED)
        {
            qDebug()<<"in NotifyChange if ";
                char szAction[42];
                char szFilename[MAX_PATH] ;
                memset(szFilename,'\0',sizeof( szFilename));
                strcpy(szAction,"added");
                wcstombs( szFilename, pFileNotify->FileName, MAX_PATH);

                OnFileChanged(szFilename,szAction);
                qDebug()<<"in NotifyChange after OnFileChanged ";

        }

        // step to the next entry if there is one
        if (!pFileNotify->NextEntryOffset)
            return false;
        pFileNotify = (FILE_NOTIFY_INFORMATION *)((PBYTE)pFileNotify + pFileNotify->NextEntryOffset);
    }
    pFileNotify=NULL;
    return true;
}

I couldn't figure out how is handling the file and don't let me read it? I tried to call Sleep() after calling OnFileChanged and wake up when it finish but in vain. Any idea please .

回答1:

I call a function (OnFileChanged) that read the size and open it for reading

That is pretty much guaranteed to fail sooner or later. File-changed notifications are generated while a process is writing to the file. Which prevents you from opening the file if the process hasn't completed writing yet or is otherwise keeping the file open. And didn't permit read sharing, very common for programs that write a file. It is certainly related to file size, the larger the file, the longer it takes to write so the greater the odds you are trying to open it while the other process didn't close it yet.

You'll need to deal with this possible mishap. Nothing you can do but keep the path in a queue so you can try later.

来源：https://stackoverflow.com/questions/13063710/createfile-readdirectorychanges-issue

标签

readdirectorychangesw

ReadDirectoryChangesW blocks deleting the watched directory

二次信任 — Thu, 05 Dec 2019 09:51:27 +0000

ReadDirectoryChangesW blocks deleting the watched directory 二次信任 2019-12-05 17:51:27

I am trying to watch a directory for create/delete/rename changes on windows with python using the ReadDirectoryChangesW API. This is my code and it is working fine:

results = win32file.ReadDirectoryChangesW(self.hDir, 8192, True, self.type, None,
                                           None)
for action, file in results:
    full_filename = os.path.join (self.source_path, file)
    if   action == 1:                                    # Created
        self.fileCreated(full_filename)
    elif action == 2:                                    # Deleted
        self.fileDeleted(full_filename)
    elif action == 3:                                    # Updated
        self.fileUpdated(full_filename)
    elif action == 4:                                    # Renamed from something
        renamed_file = full_filename
    elif action == 5:                                    # Renamed to something
        self.fileRenamed(renamed_file, full_filename)

However, when I try to delete the watched folder from python or from windows explorer, I get:

WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'c:\users\user\appdata\local\temp\new_dir'

Thanks in advance!

From this blog post:

Another potential pitfall of [ReadDirectoryChangesW] is that the referenced directory itself is now "in use" and so can't be deleted. To monitor files in a directory and still allow the directory to be deleted, you would have to monitor the parent directory and its children.

The post also provides some more details on proper use of ReadDirectoryChangesW

Deleting the watched folder IS possible under ReadDirectoryChangesW

I my tests, use of FILE_SHARE_DELETE actually allows deletion / rename of the watched folder. (In other words, you don't need to "watch parent folder" as the only option.)

Here is the working snippet (edited and borrowed heavily from this otherwise excellent "Watch a Directory for Changes" by Tim Golden

# License is same as snippets on this page
# http://timgolden.me.uk/python/win32_how_do_i/watch_directory_for_changes.html
# In other words, bug Tim Golden to publish a license for his snippets
def windows_watch_path(watched_path):
    import win32file
    import win32con

    ACTIONS = {
        1 : "Created",
        2 : "Deleted",
        3 : "Updated",
        4 : "RenamedFrom",
        5 : "RenamedTo"
    }
    # Thanks to Claudio Grondi for the correct set of numbers
    FILE_LIST_DIRECTORY = 0x0001

    try:
        hDir = win32file.CreateFile (
            watched_path
            , FILE_LIST_DIRECTORY
            , win32con.FILE_SHARE_READ | 
              win32con.FILE_SHARE_WRITE | 
              win32con.FILE_SHARE_DELETE
            , None
            , win32con.OPEN_EXISTING
            , win32con.FILE_FLAG_BACKUP_SEMANTICS
            , None
        )
    except:
        # either it does not exist by this time, or some other issue... blah.
        # we'll just say "it 'changed' from 'some other expected state'"
        return [[watched_path, '', ACTIONS[2]]]

    results = win32file.ReadDirectoryChangesW (
        hDir,
        1024,
        True,
        win32con.FILE_NOTIFY_CHANGE_FILE_NAME |
        win32con.FILE_NOTIFY_CHANGE_DIR_NAME |
        win32con.FILE_NOTIFY_CHANGE_ATTRIBUTES |
        win32con.FILE_NOTIFY_CHANGE_SIZE |
        win32con.FILE_NOTIFY_CHANGE_LAST_WRITE |
        win32con.FILE_NOTIFY_CHANGE_SECURITY,
        None,
        None
    )

    files_changed = []
    for action, fn in results:
        files_changed.append(
            [
                watched_path
                , fn
                , ACTIONS[action]
            ]
        )
        # print fullfn, ACTIONS.get(action, "Unknown")
    return files_changed

Ok, to this is not simple to solve... In my case (http://www.themacaque.com/?p=859) I ignored the fact of allowing to rename or remove the directory.

You folder does not have too many brothers to ignore. You do not want to be performing lots and lots of operations to filter events you are no interested in.
There is no problem if the user cannot remove the ancestor.

Any problem late me know either in my blog or in irc (in freenode at #ubuntuone or #pyar) my nickname is mandel ;)

来源：https://stackoverflow.com/questions/6828544/readdirectorychangesw-blocks-deleting-the-watched-directory

标签

python

winapi

readdirectorychangesw

File-level filesystem change notification in Mac OS X

青春壹個敷衍的年華 — Mon, 02 Dec 2019 20:48:14 +0000

File-level filesystem change notification in Mac OS X 青春壹個敷衍的年華 2019-12-03 04:48:14

问题

I want my code to be notified when any file under (either directly or indirectly) a given directory is modified. By "modified", I mean I want my code to be notified whenever a file's contents are altered, it's renamed, or it's deleted; or if a new file is added. For my application, there can be thousands of files.

I looked as FSEvents, but its Technology Overview says, in part:

The important point to take away is that the granularity of notifications is at a directory level. It tells you only that something in the directory has changed, but does not tell you what changed.

It also says:

The file system events API is also not designed for finding out when a particular file changes. For such purposes, the kqueues mechanism is more appropriate.

However, in order to use kqueue on a given file, one has to open the file to obtain a file descriptor. It's impractical to manage thousands of file descriptors (and would probably exceed the maximum allowable number of open file descriptors anyway).

Curiously, under Windows, I can use the ReadDirectoryChangesW() function and it does precisely what I want.

So how can one do what I want under Mac OS X? Or, asked another way: how would one go about writing the equivalent of ReadDirectoryChangesW() for Mac OS X in user-space (and do so very efficiently)?

回答1:

I haven't tried this myself, but it seems like FSEvents is able to provide file-level notifications as of 10.7 (Lion). From the description of FSEventStreamCreateFlags:

kFSEventStreamCreateFlagFileEvents

Request file-level notifications. Your stream will receive events about individual files in the hierarchy you're watching instead of only receiving directory level notifications. Use this flag with care as it will generate significantly more events than without it.

Available in OS X v10.7 and later.

回答2:

EDIT: Not verified, but Konstantin indicates below that this code sample is obsolete as of 2012.

I don't believe there's a specific API for what you're looking for. Apple provides sample code for a similar problem called Watcher. It is not what you are looking for, but it is about the best you can do at this point. You have to take snapshots of the directory, and rescan it when you find out something changed. Modification time is the best thing to check of course, if you can trust modification time.

You are probably correct that trying to register for an unbounded number of kqueues would likely be unworkable.

回答3:

The closest utility (that I know of) that matches your needs on Mac OS X is fslogger. See the link for a description, dmg and source code: OSXBook - fslogger

回答4:

You might want to check out man fs_usage, though it's not specific to a directory and requires root privileges.

来源：https://stackoverflow.com/questions/1772209/file-level-filesystem-change-notification-in-mac-os-x

标签

macos

fsevents

readdirectorychangesw

kqueue

File-level filesystem change notification in Mac OS X

自闭症网瘾萝莉.ら — Mon, 02 Dec 2019 11:08:44 +0000

File-level filesystem change notification in Mac OS X 自闭症网瘾萝莉.ら 2019-12-02 19:08:44

I looked as FSEvents, but its Technology Overview says, in part:

The important point to take away is that the granularity of notifications is at a directory level. It tells you only that something in the directory has changed, but does not tell you what changed.

It also says:

The file system events API is also not designed for finding out when a particular file changes. For such purposes, the kqueues mechanism is more appropriate.

Curiously, under Windows, I can use the ReadDirectoryChangesW() function and it does precisely what I want.

I haven't tried this myself, but it seems like FSEvents is able to provide file-level notifications as of 10.7 (Lion). From the description of FSEventStreamCreateFlags:

kFSEventStreamCreateFlagFileEvents

Request file-level notifications. Your stream will receive events about individual files in the hierarchy you're watching instead of only receiving directory level notifications. Use this flag with care as it will generate significantly more events than without it.

Available in OS X v10.7 and later.

EDIT: Not verified, but Konstantin indicates below that this code sample is obsolete as of 2012.

You are probably correct that trying to register for an unbounded number of kqueues would likely be unworkable.

The closest utility (that I know of) that matches your needs on Mac OS X is fslogger. See the link for a description, dmg and source code: OSXBook - fslogger

You might want to check out man fs_usage, though it's not specific to a directory and requires root privileges.

来源：https://stackoverflow.com/questions/1772209/file-level-filesystem-change-notification-in-mac-os-x

标签

macos

fsevents

readdirectorychangesw

kqueue