evaluate word2vec with SimLex-999 and wordsim353

问题

I have evaluated my model with SimLex-999 and wordsim353 but i don't know if the result is ok or not?

wordsim353 result

 Pearson correlation coefficient against C:\ProgramData\Anaconda3\lib\site-packages\gensim\test\test_data\wordsim353.tsv: 0.4895
2019-08-27 08:30:06,655 : INFO : Spearman rank-order correlation coefficient against C:\ProgramData\Anaconda3\lib\site-packages\gensim\test\test_data\wordsim353.tsv: 0.4799
2019-08-27 08:30:06,656 : INFO : Pairs with unknown words ratio: 7.1%

((0.4894983099817645, 3.6324947252392034e-21), SpearmanrResult(correlation=0.4798812637344527, pvalue=2.6991867797169835e-20), 7.0821529745042495)

SimLex-999 result

 2019-08-27 15:43:13,000 : INFO : Pearson correlation coefficient against C:\ProgramData\Anaconda3\lib\site-packages\gensim\test\test_data\simlex999.txt: 0.3138
    2019-08-27 15:43:13,001 : INFO : Spearman rank-order correlation coefficient against C:\ProgramData\Anaconda3\lib\site-packages\gensim\test\test_data\simlex999.txt: 0.2992
    2019-08-27 15:43:13,002 : INFO : Pairs with unknown words ratio: 1.2%
    ((0.31381174440491943, 5.375150591505246e-24), SpearmanrResult(correlation=0.29915866880742126, pvalue=7.433265418805336e-22), 1.2012012012012012)

回答1:

Only you can decide whether a result is adequate for your purposes. These kinds of scores are most meaningful when comparing one model against another, as a rough guide as to whether other changes – new parameters, new preprocessing, more/different data – are helping or hurting.

You could look at the paper introducing the evaluation dataset you're using to see how to interpret the scores:

https://arxiv.org/abs/1408.3456v1

You could also download some off-the-shelf word-vector sets, checking their evaluation scores, to compare with yours.

来源：https://stackoverflow.com/questions/57674722/evaluate-word2vec-with-simlex-999-and-wordsim353