It's not magic the DbContext won't download entities from the DB until you not tirgger that. When you clicked on the context.Groups Results View in debugging, the DbContext gets all the Groups from the DB so after that it can associate to the users.
I think you are searching for the Include method where you can get the users with the associated groups in one query like this:
var allUsers = context.Users.Include(u => u.Groups).ToList();