From the documentation of File.Move:
Note that if you attempt to replace a file by moving a file of the same name into that directory, you get an IOEx
This is safe. File.Copy will either succeed entirely or throw. Of course, the delete could fail leaving the source file behind as garbage.
If your computer crashes, though, there is no guarantee that the copy oepration has hardened the data yet. You might loose data in that case.
During normal operations this is safe.