Clarified question (tl;dr)
After reading and profiling with all the results covered below, the problem seems to boil down to the GC not collecting G
You can try to enable gcTrimCommitOnLowMemory setting in Aspnet.config file in the .NET Framework directory:
When the gcTrimCommitOnLowMemory setting is enabled, the garbage collector evaluates the system memory load and enters a trimming mode when the load reaches 90%. It maintains the trimming mode until the load drops under 85%.
https://msdn.microsoft.com/en-us/library/bb384209(v=vs.110).aspx
Another option (since .net v4.5) is to set performanceScenario to "HighDensityWebHosting" in the same Aspnet.config file. This is useful for shared hosting scenarios as it will "tune garbage collection to optimize for memory": http://www.asp.net/aspnet/overview/aspnet-and-visual-studio-2012/whats-new#_Toc_perf_5
As you can see from CoreCLR sources the HighDensityWebHosting option mostly disables gcServer and gcConcurrent settings, but enables gcTrimCommitOnLowMemory: https://github.com/dotnet/coreclr/blob/cbf46fb0b6a0b209ed1caf4a680910b383e68cba/src/vm/perfdefaults.cpp