How does a Windows antivirus hook into the file access process?

Question

The subject says it all. A normal antivirus has to intercept all file accesses, scan the files and then optionally deny access to the file (possibly even displaying a prompt

Answer 1

As you already noted, hooking is a key to what of-the-shelf AV software with "realtime" protection does.

You could have a look on the (widely discussed) winpooch, which already does API Hooking, but there are some major flaws in this software. Sourceforge of Winpooch

There is also an article on Codeproject on API hooking, providing some library to do hooking "in three layers". Dll Injection is somewhat hard, as you can image. CodeProject: EasyHook, reinvention of API Hooking

As you are probably interested in Antivirus strategies, i also suggest having a look at ClamAV, or WinClam, which is opensource (under GPL) ClamAV for windows

But i do not have a clue how to do API hooking with C#, i have to admit. In C / C++ this is (quite) easy...

ADD ON You may be interested in the sources of FileMon, a widely known FileSystem Monitor that was once by SysInternals and now by Microsoft: It uses Driver-Filter API by Microsoft, which is at least known as fragile.

Link may be found here in Sysinternals forum