I have an ASP.NET website that will hit about 2gb physical memory used within about 3-4 days, which to me sounds really bad. At the moment, I have configured IIS to restart
Whilst it is true that you don't have to explicitly free memory and the garbage collector will do this for you it is possible for code in inadvertently hold a reference to an object to prevent it being collected - this is essentially a memory leak.
In ASP.NET objects remain "alive" whilst they are referenced by the Application and Session cache. Without knowing anything about your application I would suggest you get hold of a memory profiler and look more closely at exactly what is in memory and whether the Application or Session cache is holding onto something it shouldn't.