Does the .net function Parallel.ForEach block the calling thread? My guess as to the behavior is one of these:
Number 1 is correct; Parallel.ForEach does not return until the loop has completed. If you don't want that behavior, you can simply execute your loop as a Task and run it on another thread.
Parallel.ForEach
Task