Difference between Lucene stemmers: EnglishStemmer, PorterStemmer, LovinsStemmer

佐手、 提交于 2019-11-30 15:38:56

The Lovins stemmer is a very old algorithm that is not of much practical use, since the Porter stemmer is much stronger. Based on some quick skimming of the source code, it seems PorterStemmer implements Porter's original (1980) algorithm, while EnglishStemmer implements his updated version, which should be better.

A stronger stemming algorithm (actually a lemmatizer) is available in the Stanford NLP tools. A Lucene-Stanford NLP by yours truly bridge is available here (API docs).

See also Manning, Raghavan & Schütze for general info about stemming and lemmatization.

I've tested the 3 Lucene stemmers available from org.apache.lucene.analysis.en version 4.4.0, which are EnglishMinimalStemFilter, KStemFilter and PorterStemFilter, in a document classification problem I'm working on. My results corroborate the claims made by the authors of Introduction to Information Retrieval that for small training corpora in document classification settings stemming is harmful, and for large corpora stemming makes no difference.

For search and indexing, stemming can be more useful (see, e.g., Jenkins & Smith), but even there the answer to your question depends on the details of what you're doing. There is no free lunch!

At the end of the day, nothing beats empirical tests of real code on real data. The only way you'll really know which is better is by running the stemmers for yourself in your application.

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!