What exactly does the Standard tokenfilter do in Elasticsearch?

Question

There are no examples in the documentation, and I was just wondering what to expect from input I give it.

Answer 1

Here's an excerpt of the source code of StandardFilter from Lucene trunk. It really does nothing.

public final boolean incrementToken() throws IOException {
    return input.incrementToken(); // TODO: add some niceties for the new grammar
}