is there a way with spaCy's NER to calculate metrics per entity type?

Question

is there a way in the NER model in spaCy to extract the metrics (precision, recall, f1 score) per entity type?

Something that will look like this:

         precision    recall  f1-score   support

  B-LOC      0.810     0.784     0.797      1084
  I-LOC      0.690     0.637     0.662       325
 B-MISC      0.731     0.569     0.640       339
 I-MISC      0.699     0.589     0.639       557
  B-ORG      0.807     0.832     0.820      1400
  I-ORG      0.852     0.786     0.818      1104
  B-PER      0.850     0.884     0.867       735
  I-PER      0.893     0.943     0.917       634

avg / total 0.809 0.787 0.796 6178

taken from: http://www.davidsbatista.net/blog/2018/05/09/Named_Entity_Evaluation/

Thank you!

Answer 1

Nice question.

First, we should clarify that spaCy uses the BILUO annotation scheme instead of the BIO annotation scheme you are referring to. From the spacy documentation the letters denote the following:

• B: The first token of a multi-token entity.
• I: An inner token of a multi-token entity.
• L: The final token of a multi-token entity.
• U: A single-token entity.
• O: A non-entity token.

Then, some definitions:

Spacy has a built-in class to evaluate NER. It's called scorer. Scorer uses exact matching to evaluate NER. The precision score is returned as ents_p, the recall as ents_r and the F1 score as ents_f.

The only problem with that is that it returns the score for all the tags together in the document. However, we can call the function only with the TAG we want and get the desired result.

All together, the code should look like this:

import spacy
from spacy.gold import GoldParse
from spacy.scorer import Scorer

def evaluate(nlp, examples, ent='PERSON'):
    scorer = Scorer()
    for input_, annot in examples:
        text_entities = []
        for entity in annot.get('entities'):
            if ent in entity:
                text_entities.append(entity)
        doc_gold_text = nlp.make_doc(input_)
        gold = GoldParse(doc_gold_text, entities=text_entities)
        pred_value = nlp(input_)
        scorer.score(pred_value, gold)
    return scorer.scores


examples = [
    ("Trump says he's answered Mueller's Russia inquiry questions \u2013 live",{"entities":[[0,5,"PERSON"],[25,32,"PERSON"],[35,41,"GPE"]]}),
    ("Alexander Zverev reaches ATP Finals semis then reminds Lendl who is boss",{"entities":[[0,16,"PERSON"],[55,60,"PERSON"]]}),
    ("Britain's worst landlord to take nine years to pay off string of fines",{"entities":[[0,7,"GPE"]]}),
    ("Tom Watson: people's vote more likely given weakness of May's position",{"entities":[[0,10,"PERSON"],[56,59,"PERSON"]]}),
]

nlp = spacy.load('en_core_web_sm')
results = evaluate(nlp, examples)
print(results)

Call the evaluate function with the proper ent parameter to get the results for each tag.

Hope it helps :)

Answer 2

@gdaras 's answer is not right. The first comment gives the idea why. You should filter entities of

pred_value = nlp(input_)

I did it like this

pred_value.ents = [e for e in pred_value.ents if e.label_ == ent]

Answer 3

I have been working on this, and now its integrated withing spacy by this Pull Request.

Now you just need to call Scorer().scores and it will return the usual dict with an additional key, ents_per_type, that will contains the metrics Precision, Recall and F1-Score for each entity.

Hope it helps!