Solved:
According to [1], you could use NtDeleteFile to avoid the asynchronous nature of DeleteFile. Also [1] gives some details on how DeleteFile works.
Unfortunately the official documentation on NtDeleteFile [2] doesn't mention any particular details on this issue.
[1] Undocumented functions of NTDLL
[2] ZwDeleteFile function