I have created a small application to recursively load assemblies in a provided directory and read their custom attributes collection. Mainly just to read the DebuggableAttr
It isn't possible to ever unload an Assembly in the current AppDomain, that is just the way .NET is designed to work unfortunately. This is even the case with ReflectionOnly loads. There is also a slightly wrinkle with doing that as well as you then need to use the GetCustomAttributesData method instead of the normal GetCustomAttributes as the latter needs to run code in the attribute constructor. This can make life more difficult.
An alternative which should work is using Cecil which allows you to inspect the assembly without actually loading it in the normal sense. But that is a lot of extra work.