Pythonic way to implement a tokenizer

Question

I\'m going to implement a tokenizer in Python and I was wondering if you could offer some style advice?

I\'ve implemented a tokenizer before in C and in Java so I\'m

Answer 1

"Is there a better alternative to just simply returning a list of tuples?"

Nope. It works really well.