Ruby 1.9 has removed the current directory from the load path, and so you will need to do a relative require on this file, as David Grayson says:
require_relative 'tokenizer'
There's no need to suffix it with .rb, as Ruby's smart enough to know that's what you mean anyway.