On Visual Studio I added application insight to a project which creates a ApplicationInsights.config and not sure what other files were added to the project.
Thing i
Unless I'm misunderstanding the question, you should only have to remove one extension and possibly one nuget package.
Uninstall the Application Insights Tools for Visual Studio extension and remove the Application Telemetry SDK for Services nuget package. The telemetry package is installed along with Application Insights but must be removed separately.
In my experience the telemetry package is not required if you wish to keep using Application Insights' other features. Removing the telemetry package will stop all telemetry logging but Application Insights will continue to report non-telemetry information just fine.