On Visual Studio I added application insight to a project which creates a ApplicationInsights.config and not sure what other files were added to the project.
Thing i
If you use the NuGet Package Manager for the solution (Tools -> NuGet Package Manager -> Manage NuGet Packages for Solution) you can search for ApplicationInsights and uninstall the package, and there is an option to remove dependencies. There may be several. This is the easiest way to cleanup ALL the dependencies, not just some.