Lets imagine that we have two assemblies:
Unless you use explicitly use SxS, your .NET 2 assembly will be loaded into the .NET 4 CLR. There shouldn't be any performance problems at all... the code which originally targeted .NET 2 will just run using the .NET 4 libraries and CLR, etc. And yes, you can use an assembly compiled with .NET 1.1 in a .NET 2 or higher app, too.
There is a problem if you try to load a mixed-mode assembly (some native code, some IL) in .NET 4, IIRC... but if it's "just" a .NET class library, you should be fine.