I'd have to say the people you were talking to simply don't know what they're talking about. Plain and simple.
Many enterprise level applications are built on top of C# and other .Net languages. There is nothing inherently slow about them. Yes the tend to have slower startup times but that's pretty much where it ends.
I noticed you mentioned Java in the list of languages. If you're comfortable with the speed of Java, C# will not present any issues. Generally speaking C# performs at least as well as Java on many different types of bench marks.
