StringTokenizer? Convert the String to a char[] and iterate over that? Something else?
I agree that StringTokenizer is overkill here. Actually I tried out the suggestions above and took the time.
My test was fairly simple: create a StringBuilder with about a million characters, convert it to a String, and traverse each of them with charAt() / after converting to a char array / with a CharacterIterator a thousand times (of course making sure to do something on the string so the compiler can't optimize away the whole loop :-) ).
The result on my 2.6 GHz Powerbook (that's a mac :-) ) and JDK 1.5:
As the results are significantly different, the most straightforward way also seems to be the fastest one. Interestingly, charAt() of a StringBuilder seems to be slightly slower than the one of String.
BTW I suggest not to use CharacterIterator as I consider its abuse of the '\uFFFF' character as "end of iteration" a really awful hack. In big projects there's always two guys that use the same kind of hack for two different purposes and the code crashes really mysteriously.
Here's one of the tests:
int count = 1000;
...
System.out.println("Test 1: charAt + String");
long t = System.currentTimeMillis();
int sum=0;
for (int i=0; i