spacy

what is difference between spacy.load('en_core_web_sm') and spacy.load('en') ? This link explains different model sizes. But i am still not clear how spacy.load('en_core_web_sm') and spacy.load('en') differ spacy.load('en') runs fine for me. But the spacy.load('en_core_web_sm') throws error i have installed spacy as below. when i go to jupyter notebook and run command nlp = spacy.load('en_core_web_sm') I get the below error --------------------------------------------------------------------------- OSError Traceback (most recent call last) <ipython-input-4-b472bef03043> in <module>() 1 #

一、中文分词工具 （1）Jieba （2）snowNLP分词工具 （3）thulac分词工具 （4）pynlpir 分词工具 （5）StanfordCoreNLP分词工具 1. from stanfordcorenlp import StanfordCoreNLP 2. with StanfordCoreNLP(r'E:\Users\Eternal Sun\PycharmProjects\1\venv\Lib\stanford-corenlp-full-2018-10-05', lang='zh') as nlp: 3. print("stanfordcorenlp分词：\n",nlp.word_tokenize(Chinese)) （6）Hanlp分词工具 分词结果如下： 二、英文分词工具 1. NLTK： 二者之间的区别在于，如果先分句再分词，那么将保留句子的独立性，即生成结果是一个二维列表，而对于直接分词来说，生成的是一个直接的一维列表，结果如下： 2. SpaCy： 3. StanfordCoreNLP： 分词结果 来源： oschina 链接： https://my.oschina.net/u/3793864/blog/3056365

SpaCy Version: 2.0.11 Python Version: 3.6.5 OS: Ubuntu 16.04 My Sentence Samples: Marketing-Representative- won't die in car accident. or Out-of-box implementation Expected Tokens: ["Marketing-Representative", "-", "wo", "n't", "die", "in", "car", "accident", "."] ["Out-of-box", "implementation"] SpaCy Tokens(Default Tokenizer): ["Marketing", "-", "Representative-", "wo", "n't", "die", "in", "car", "accident", "."] ["Out", "-", "of", "-", "box", "implementation"] I tried creating custom tokenizer but it won't handle all edge cases as handled by spaCy using tokenizer_exceptions(Code below):