I don't know a solution for textarea but it sure works for a div with contenteditable.
You can use the Range API. Like so: (yes, you really only need just these 3 lines of code)
// get active selection
var selection = window.getSelection();
// get the range (you might want to check selection.rangeCount
// to see if it's popuplated)
var range = selection.getRangeAt(0);
// will give you top, left, width, height
console.log(range.getBoundingClientRect());
I'm not sure about browser compatibility but I've found it works in the latest Chrome, Firefox and even IE7 (I think I tested 7, otherwise it was 9).
You can even do 'crazy' things like this: if you're typing "#hash" and the cursor is at the last h, you can look in the current range for the # character, move the range back by n characters and get the bounding-rect of that range, this will make the popup-div seem to 'stick' to the word.
One minor drawback is that contenteditable can be a bit buggy sometimes. The cursor likes to go to impossible places and you now have to deal with HTML input. But I'm sure browser vendors will address these problems are more sites starting using them.
Another tip I can give is: look at the rangy library. It attempts to be a fully featured cross-compatible range library. You don't need it, but if you're dealing with old browsers it might be worth you while.