I have a working application using Json.NET (newtonsoft) as a custom serializer. Currently I\'m adding this derivative of WebHttpBehavior in a custom WebServiceHostFactory.
If you open this app.config with svcconfigeditor, it should start asking to select the assembly associated with the NewtonsoftJsonBehavior. If you select that, and save the app.config again, does that resolve your issue?
BTW, we also use a custom binding extension in our service configurations. The xml in the config always underlines the extension with a blue wave indicating that the element is not know to the schema as you describe it. However, when we start the service, both the service and the behavior function as expected.