You can start a new Matlab instance from Matlab itself with a system call and the Matlab command line options. Then you can test your .NET libraries within the new instance. When the new instance is closed after your test, the .NET libraries are released. With this workaround, you don't need to reopen Matlab completely. Do not load the .NET assembly in the main Matlab instance.
Open a new Matlab instance (whole IDE GUI) on a Windows PC:
log_str = '-logfile "C:\Temp\logfile.txt"';
system_str = [ 'matlab.exe -nosplash ' log_str ];
system(system_str);
Automatically execute an m-File in the new Matlab instance via -r, without the IDE GUI with -nodesktop and have the new instance close itself after your test:
log_str = '-logfile "C:\Temp\logfile.txt"';
my_file = 'C:\Temp\test.m';
system_str = [ 'matlab.exe -nosplash -nodesktop -r "run(''' my_file '''), pause(2), exit" ' log_str ];
system(system_str);
The logfile logs the console output of the Matlab instance. It is optional, but helpful for testing. If you want your child instances to be asynchronous (i.e. returning the handle to the main Matlab instance), you can add ' &' at the end of the system_str.