There might be a slight difference in the compiler optimization, but I'd say there is no noticeable difference. Both C# and VB.NET compile to Common Intermediate Language. In some cases, you may be able to get a significant performance gain in C# by using unsafe code, but under most circumstances I wouldn't recommend doing so. If you need something that performance critical, you shouldn't use C# either.
The myth probably started because of the huge difference in Visual Basic 6 performance compared to the average C++ application.
