In Visual Studio 2010 for a C# project, if you go to Project Properties > Build > Advanced > Debug Info you have three options: none, full, or pdb-only. Based on the answer
You'll want PDB only, but you won't want to give the PDB files to users. Having them for yourself though, alongside your binaries, gives you the ability to load crash dumps into a debugger like WinDbg and see where your program actually failed. This can be rather useful when your code is crashing on a machine you don't have access to.
Full debug adds the [Debuggable] attribute to your code. This has a huge impact on speed. For example, some loop optimizations may be disabled to make single stepping easier. In addition, it has a small effect on the JIT process, as it turns on tracking.