How does a Windows antivirus hook into the file access process?

Question

The subject says it all. A normal antivirus has to intercept all file accesses, scan the files and then optionally deny access to the file (possibly even displaying a prompt

Answer 1

In general, these products intercept functions to get a HANDLE to a process like OpenProcess or NtOpenProcess. They also, hook CreateRemoteThread functions and memory allocation in a remote process: VirtualAlloc and VirtualProcect. Some AVs also hook SetWindowsHookEx function to detect global hooks to avoid key loggers.

Hooking these APIs they can control which modules (or dlls) can access remote processes and allow only those that the user know what they are doing.

You can use HookShark to see what user-mode functions are intercepted by each AV product.

To make your own user hooks you can use detours library but you have to develop an agent to run your in-process hooks and then communicate with an agent server. You can also use Deviare API Hook which is a framework that makes all the complex staff so you can code your hooks in your own process using any programming language.