I have used the following code in a number of applications to load .DLL assemblies that expose plugins.
However, I previously was always concerned with functionality
If you are concerned with the security of your assemblies, you should Strongly Name them. This provides a high level of security that the assembly is indeed the one you intend for it to be.
Exceptions you might encounter during load are as follows. Add try/catch around your load attempt at Assembly.Load() and react according to the error type:
Assemblies you load dynamically should have the same rights as the user account which loaded them unless this assembly is in the GAC. Create a service account with the rights you desire, and run your application using this account to control access.