By graph here I mean something resembling these images:
I have used MsAgl and it is pretty good. It also has viewers for both WinForms and WPF which may or may not be important to you. One does not have to use the viewers as they are quite loosely coupled to the graph layout engine.
It does not support everything that Graphviz does (notably in terms of some shape types). Also the documentation could be better but it is pretty solid if not the most polished MS product ever.
It does tick all of the items you mention as requirements, though the "on viewer" re-organization merely supports drag-moving nodes, re-routing edges and re-positioning labels. The graph does not respond to the movements of these elements by repositioning any other items.