n-gram

I want to perform both exact word match and partial word/substring match. For example if I search for "men's shaver" then I should be able to find "men's shaver" in the result. But in case case I search for "en's shaver" then also I should be able to find "men's shaver" in the result. I using following settings and mappings: Index settings: PUT /my_index { "settings": { "number_of_shards": 1, "analysis": { "filter": { "autocomplete_filter": { "type": "edge_ngram", "min_gram": 1, "max_gram": 20 } }, "analyzer": { "autocomplete": { "type": "custom", "tokenizer": "standard", "filter": [

I have some problems with my Hibernate Search analyzer configuration. One of my indexed entities ("Hospital") has a String field ("name") that could contain values with lengths from 1-40. I want to be able to find a entity by searching for just one character (because it could be possible, that a hospital has single character name). @Indexed(index = "HospitalIndex") @AnalyzerDef(name = "ngram", tokenizer = @TokenizerDef(factory = StandardTokenizerFactory.class), filters = { @TokenFilterDef(factory = StandardFilterFactory.class), @TokenFilterDef(factory = LowerCaseFilterFactory.class),

自动文档摘要评价方法大致分为两类： （1） 内部评价方法(Intrinsic Methods) ：提供参考摘要，以参考摘要为基准评价系统摘要的质量。系统摘要与参考摘要越吻合， 质量越高。 （2） 外部评价方法(Extrinsic Methods) ：不提供参考摘要，利用文档摘要代替原文档执行某个文档相关的应用。例如：文档检索、文档聚类、文档分类等， 能够提高应用性能的摘要被认为是质量好的摘要。 一、Edmundson： 　　Edmundson评价方法比较简单，可以客观评估，就是通过比较机械文摘（自动文摘系统得到的文摘）与目标文摘的句子重合率(coselection rate)的高低来对系统摘要进行评价。也可以主观评估，就是由专家比较机械文摘与目标文摘所含的信息，然后给机械文摘一个等级评分。 类如等级可以分为：完全不相似，基本相似，很相似，完全相似等。 　　Edmundson比较的基本单位是句子，通过句子级标号分隔开的文本单元，句子级标号包括“。”“：”“；”“！”“？”，并且只允许专家从原文中抽取句子，而不允许专家根据自己对原文的理解重新生成句子，专家文摘和机械文摘的句子都按照在原文中出现的先后顺序给出。 　　 计算公式为： \[ \text{重合率p}=\text{匹配句子数}/\text{专家文摘句子数}\times \] 每一个机械文摘的重合率为按三个专家给出的

I'm using NLTK to search for n-grams in a corpus but it's taking a very long time in some cases. I've noticed calculating n-grams isn't an uncommon feature in other packages (apparently Haystack has some functionality for it). Does this mean there's a potentially faster way of finding n-grams in my corpus if I abandon NLTK? If so, what can I use to speed things up? Fred Foo Since you didn't indicate whether you want word or character-level n-grams, I'm just going to assume the former, without loss of generality. I also assume you start with a list of tokens, represented by strings. What you

I wrote the following code for computing character bigrams and the output is right below. My question is, how do I get an output that excludes the last character (ie t)? and is there a quicker and more efficient method for computing character n-grams? b='student' >>> y=[] >>> for x in range(len(b)): n=b[x:x+2] y.append(n) >>> y ['st', 'tu', 'ud', 'de', 'en', 'nt', 't'] Here is the result I would like to get: ['st','tu','ud','de','nt] Thanks in advance for your suggestions. To generate bigrams: In [8]: b='student' In [9]: [b[i:i+2] for i in range(len(b)-1)] Out[9]: ['st', 'tu', 'ud', 'de', 'en'