JUNG — the Java Universal Network/Graph Framework - is one of the fastest available (implementing the CERN math library COLT), especially if you want to do graph analysis, centrality measurements (betweennes, pagerank, etc.). Furthermore it allows for network visualization.
