How to initialize a `Doc` in textacy 0.6.2?

问题

Trying to follow the simple Doc initialization in the docs in Python 2 doesn't work:

>>> import textacy
>>> content = '''
...     The apparent symmetry between the quark and lepton families of
...     the Standard Model (SM) are, at the very least, suggestive of
...     a more fundamental relationship between them. In some Beyond the
...     Standard Model theories, such interactions are mediated by
...     leptoquarks (LQs): hypothetical color-triplet bosons with both
...     lepton and baryon number and fractional electric charge.'''
>>> metadata = {
...     'title': 'A Search for 2nd-generation Leptoquarks at √s = 7 TeV',
...     'author': 'Burton DeWilde',
...     'pub_date': '2012-08-01'}
>>> doc = textacy.Doc(content, metadata=metadata)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/a/anaconda/envs/env1/lib/python2.7/site-packages/textacy/doc.py", line 120, in __init__
    {compat.unicode_, SpacyDoc}, type(content)))
ValueError: `Doc` must be initialized with set([<type 'unicode'>, <type 'spacy.tokens.doc.Doc'>]) content, not "<type 'str'>"

What should that simple intialization look like for a string or a sequence of strings?

UPDATE:

Passing unicode(content) to textacy.Doc() spits out

ImportError: 'cld2-cffi' must be installed to use textacy's automatic language detection; you may do so via 'pip install cld2-cffi' or 'pip install textacy[lang]'.

which would've been nice to have from the moment when textacy was installed, imo.

Even after instaliing cld2-cffi, attempting the code above throws out

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/a/anaconda/envs/env1/lib/python2.7/site-packages/textacy/doc.py", line 114, in __init__
    self._init_from_text(content, metadata, lang)
  File "/Users/a/anaconda/envs/env1/lib/python2.7/site-packages/textacy/doc.py", line 136, in _init_from_text
    spacy_lang = cache.load_spacy(langstr)
  File "/Users/a/anaconda/envs/env1/lib/python2.7/site-packages/cachetools/__init__.py", line 46, in wrapper
    v = func(*args, **kwargs)
  File "/Users/a/anaconda/envs/env1/lib/python2.7/site-packages/textacy/cache.py", line 99, in load_spacy
    return spacy.load(name, disable=disable)
  File "/Users/a/anaconda/envs/env1/lib/python2.7/site-packages/spacy/__init__.py", line 21, in load
    return util.load_model(name, **overrides)
  File "/Users/a/anaconda/envs/env1/lib/python2.7/site-packages/spacy/util.py", line 120, in load_model
    raise IOError("Can't find model '%s'" % name)
IOError: Can't find model 'en'

回答1:

The issue, as shown in the traceback, is at textacy/doc.py in the _init_from_text() function, which tries to detect language and calls it with the string 'en' in line 136. (The spacy repo touches on this in this issue comment.)

I solved this by providing a valid lang (unicode) string of u'en_core_web_sm' and by using unicode in the content and lang argument strings.

import textacy

content = u'''
    The apparent symmetry between the quark and lepton families of
    the Standard Model (SM) are, at the very least, suggestive of
    a more fundamental relationship between them. In some Beyond the
    Standard Model theories, such interactions are mediated by
    leptoquarks (LQs): hypothetical color-triplet bosons with both
    lepton and baryon number and fractional electric charge.'''

metadata = {
    'title': 'A Search for 2nd-generation Leptoquarks at √s = 7 TeV',
    'author': 'Burton DeWilde',
    'pub_date': '2012-08-01'}

doc = textacy.Doc(content, metadata=metadata, lang=u'en_core_web_sm')

That a string instead of a unicode string (with a cryptic error message) changes behaviour, the fact there's a missing package, and the perhaps-outdated/perhaps-non-comprehensive way of using spacy language strings all seem like bugs to me. 🤷‍♂️

回答2:

It appears you are using Python 2 and got a unicode error. In the textacy docs there is a note about some unicode nuances when using Python 2:

Note: In almost all cases, textacy (as well as spacy) expects to be working with unicode text data. Throughout the code, this is indicated as str to be consistent with Python 3’s default string type; users of Python 2, however, must be mindful to use unicode, and convert from the default (bytes) string type as needed.

Therefore I would give this a shot (note the u'''):

content = u'''
          The apparent symmetry between the quark and lepton families of
          the Standard Model (SM) are, at the very least, suggestive of
          a more fundamental relationship between them. In some Beyond the
          Standard Model theories, such interactions are mediated by
          leptoquarks (LQs): hypothetical color-triplet bosons with both
          lepton and baryon number and fractional electric charge.'''

This produced a Doc object as expected for me (on Python 3 though).

来源：https://stackoverflow.com/questions/51431112/how-to-initialize-a-doc-in-textacy-0-6-2

标签

python

nlp

textacy