I know I can start a process in code with Process.Start().
Is it also possible to attach the debugger to that process?
Not from code per se ,
You can attach to a running process using Tools | Attach to Process. If it's a Web Application, you can attach to it by attaching to aspnet_wp.exe or w3wp.exe.
To answer your question on how to attach to a process programmatically:
Here are other Stack Overflow questions that deal with that: