Solved:
I actually had the same issue while using the LoadLibrary(path). I couldn't delete the file in path.
The solution was to "close the handle" or use the FreeLibrary(path) method.
NOTE: Please read the "Remarks" on MSDN regarding the FreeLibrary().