Here are the biggest differences between the two for me, and I've used both:
1) TFS is rather tightly coupled to the "Visual Studio way" of doing development. That's not to say that TFS is tightly coupled to the VS IDE, it means that TFS struggles to keep the familiar "check in"/"check out" paradigm of Visual SourceSafe, even when it really isn't an appropriate model anymore. Subversion's concept of "commit"/"update" is a lot more realistic when you have developers which might spend time disconnected from the network. TFS expects developers to always be connected to the server. That's a big minus. I personally find TFS to be less than transparent about how files are organized on the server and on your local disk because of the tight Visual Studio integration. Even TFS's bigger proponents concede that its connected check-in/check-out model is not a compelling option for developers who work disconnected. In a climate where people are starting to look at DVCS options like git and Mercurial over SVN, TFS's "check out" model seems a bit like a dinosaur.
2) Cost. Those who say that TFS isn't expensive are either probably very small shops, or are not in compliance with TFS's licensing terms. You need a Client Access License for darn near everything you do. Are you a manager who just manages the bugs? You need a ~$250 CAL (There are 5 included with a retail TFS License). A business user who just wants to report on their issues? A $250 CAL. A developer? $250 (Unless they have MSDN in which case it is included). The server? $500 (Included if you have MSDN). Of course, someone selling you a copy of TFS will tell you that work item tracking is free for additional users, but these additional users can only see the work items which they themselves create, and not the whole team's work items, which isn't too useful in an team-oriented, agile environment. All of this adds up when you have a mid-sized organization, and becomes tough to justify when so many best-of-breed products like SVN and CruiseControl.net's incremental cost is $0. (In fairness to TFS, though, I'm still waiting for a really good OSS issue tracker)
3) Project structure. In large teams with a smaller number of projects, TFS will probably work well. If you're a number of small, unconnected or loosely connected line-of-business apps in-house, TFS's structure can start to become overbearing. For one thing, it's not possible to define a taxonomy of projects themselves -- you can set up "Areas" within a project, but all issues and documents are tracked together within the basic context of a "project". Creating new "projects" is often time consuming, and is overkill for small efforts. Of course, SVN has nothing of the sort since it focuses only on source code control, but if you need good small-project flexibility, SVN and another issue tracking tool might be a better choice.
My opinion, for what it's worth:
For large teams with big, well-budgeted projects, in a Microsoft shop where developers work almost exclusively within the IDE, TFS is the winner. TFS also wins when you need to centrally enforce policy with your projects.
For a number of small teams, with many varied, smaller projects, or shops where cost is an issue, or teams who have developers who work disconnected from source control, go with SVN.
