I have rephrased this question.
When .net objects are exposed to COM Clients through COM iterop, a CCW (COM Callable Wrapper) is created, this sits between
As far as I'm aware, the best coverage of this subject is in the book The .NET and COM Interoperability Handbook By Alan Gordon, and that link should go to the relevant page in Google Books. (Unfortunately I don't have it, I went for the Troelsen book instead.)
The guidance there implies that there isn't a well-defined way of hooking into the Release/reference counting in the CCW. Instead the suggestion is that you make your C# class disposable, and encourage your COM clients (in your case the VBScript authors) to call Dispose when they want deterministic finalisation to occur.
But happily there is a loophole for you because your clients are late-binding COM clients, because VBScript uses IDispatch to make all calls to objects.
Suppose your C# classes were exposed via COM. Get that working first.
Now in ATL/C++ create a wrapper class, using the ATL Simple Object wizard, and in the options page choose Interface: Custom instead of Dual. This stops the wizard putting in its own IDispatch support.
In the class's constructor, use CoCreateInstance to magic up an instance of your C# class. Query it for IDispatch and hold onto that pointer in a member.
Add IDispatch to the wrapper class's inheritance list, and forward all four methods of IDispatch straight through to the pointer you stashed away in the constructor.
In the FinalRelease of the wrapper, use the late binding technique (Invoke) to call the Dispose method of the C# object, as described in the Alan Gordon book (on the pages I linked to above).
So now your VBScript clients are talking via the CCW to the C# class, but you get to intercept the final release and forward it to the Dispose method.
Make your ATL library expose a separate wrapper for each "real" C# class. You'll probably want to use inheritance or templates to get good code reuse here. Each C# class you support should only require a couple of lines in the ATL wrapping code.