Is there any good open-source or freely available Chinese segmentation algorithm available? [closed]

问题

As phrased in the question, I'm looking for a free and/or open-source text-segmentation algorithm for Chinese, I do understand it is a very difficult task to solve, as there are many ambiguities involed. I know there's google's API, but well it is rather a black-box, i.e. not many information of what it is doing are passing through.

回答1:

The keyword text-segmentation for Chinese should be 中文分词 in Chinese.

Good and active open-source text-segmentation algorithm :

盘古分词(Pan Gu Segment) : C#, Snapshot
ik-analyzer : Java
ICTCLAS : C/C++, Java, C#, Demo
NlpBamboo : C, PHP, PostgreSQL
HTTPCWS : based on ICTCLAS, Demo
mmseg4j : Java
fudannlp : Java, Demo
smallseg : Python, Java, Demo
nseg : NodeJS
mini-segmenter: python

Other

Google Code : http://code.google.com/query/#q=中文分词
OSChina (Open Source China)

Sample

Google Chrome (Chromium) : src, cc_cedict.txt (73,145 Chinese words/pharases)
- In text field or textarea of Google Chrome with Chinese sentences, press Ctrl+← or Ctrl+→
- Double click on 中文分词指的是将一个汉字序列切分成一个一个单独的词

回答2:

Stanford segment using CRF algorithmn.

It's under GPL

link page is : http://nlp.stanford.edu/software/segmenter.shtml

回答3:

ICU has details on universal text segmentation - http://userguide.icu-project.org/boundaryanalysis

回答4:

Cursory Googling for "text segmentation chinese open source" reveals this library, which may or may not be what you're looking for...:

http://sourceforge.net/projects/ktdictseg/

The results hint at a few alternative venues to look for an open-source library, too:

Searching for an open-source search implementation that might work with Chinese.
Searching for an open-source plagiarism detection implementation that might with Chinese.

来源：https://stackoverflow.com/questions/5834371/is-there-any-good-open-source-or-freely-available-chinese-segmentation-algorithm

标签

algorithm

open-source

cjk

text-segmentation