I am developing a large application which consists of many smaller plugins/applications.
They are not big enough to be a full process, but are too small to be run in
AppDomain is more oft used for being able to unload assemblies (like your suggesting) and for controlling startup parameters like .NET access levels, configurations, etc. If you really want 'isolation' then the best bet is always a worker process; however, it's a lot more work.
I do a fair amount of this in several projects. Just to give a broad-stroke picture, we use Google ProtoBuffers (Jon Skeet's port) over a managed Windows LRPC Library for most of the communications. For worker-process management we rely heavily on named events, I recently published an inter-process event library here just for this purpose.