Well, I believe the set of keys will determine which item is first, but not in a well-defined (or easy to predict) way. In other words, don't assume that it will always work the same way - it's as unsafe as relying on a hash code implementation staying the same between runs.
EDIT: I believe that in fact, the ordering of insertion does matter, contrary to my previous ideas. However, this is implementation-specific (so could easily change in the next version). I believe that with the current implementation, the first entry added will be the first one returned if it hasn't been removed. If the first entry added is ever removed, the ordering is broken - it's not that the earliest entry is removed. Here's an example:
using System;
using System.Collections.Generic;
class Test
{
static void Main(string[] args)
{
var dict = new Dictionary();
dict.Add(0, 0);
dict.Add(1, 1);
dict.Add(2, 2);
dict.Remove(0);
dict.Add(10, 10);
foreach (var entry in dict)
{
Console.WriteLine(entry.Key);
}
Console.WriteLine("First key: " + dict.First().Key);
}
}
The results are 10, 1, 2, and "First key: 10" - showing that the latest added entry ends up being returned first.
However, I'd like to stress again that everything can change between versions of the framework.