We have been using JSunit for a while to do unit tests... it may not be the same kinds of tests you are talking about, but it is great for ensuring your JavaScript works as you expect.
You run it in the browser, and it can be set in an Ant build to be automatically run against a bunch of browsers on a bunch of platforms remotely (so you can ensure your code is cross-browser as well as ensure the logic is correct).
I don't think it replaces Selenium, but it complements it well.
