I have used the following code in a number of applications to load .DLL assemblies that expose plugins.
However, I previously was always concerned with functionality
Take a look at AddIn framework from Microsoft as it provides some of the capabilities that you're looking for such as security and isolation.
I would also recommend that you use reflection only methods such as ReflectionOnlyLoad and ReflectionOnlyLoadFrom to ensure nothing from the assembly you're querying metadata from gets executed as you examine it. Once you determine that the assembly has what you're looking for only then you can load it.