I have custom compiled directshow filter - filter.DLL - but how to use, or install this filter in system?
Like Cristian said, people typically register the filter on the system. But I prefer not to do this for a few reasons. In particular, see the section on "Registering Filters with the Operating System."
I think nine times out of ten, you're better off instantiating with new. See AppFilter on Geraint Davis' page for an example.