Filing this under the either the I Can\'t Believe No One Noticed This Before or the I Must Be Missing Something categories:
It appears that if you do a simple
I found a hack that kinda works. It turns out that if you change the hash right after history.pushState the location bar gets updated. Like:
window.history.pushState(data, title, 'a/new/url');
window.location.hash = 'new';
changes the location bar to http://example.com/a/new/url#new. Which introduces another problem because the hash becomes it's own history entry. So you'll need to listen to onHashChange anyway.
It's a bit complicated, but there are some people who really, really hate hashbang urls and are very vocal about it. So it's worth it.