Much of our C# release code is built with the \'Optimize code\' option turned off. I believe this is to allow code built in Release mode to be debugged more easily.
In fact, there is a difference, sometimes quite significant. What can really affect the performance (as it is something that JIT does not fully take care of):
Unnecessary local variables (i.e., bigger stack frames for each call)
Too generic conditional instructions, JIT translates them in quite a straightforward manner.
Unnecessary branching (also not served well by a JIT - after all, it does not have too much time to do all the smart optimisations)
So, if you're doing something numerical - turn on the optimisation. Otherwise you won't see any difference at all.