问题
I am running a scrape and process routine in Python3 - but some of the sentences I get are garbage. I would like to reject these but cant figure out how to do it.
I am using POS tagging and chunking with NLTK but that doesn't seem to help me identify non-valid sentences. The number of NNs, VBs etc. doesn't seem to be any different in a garbage "sentence" than a good one.
I guess I am just looking for a simple method to score the grammar of a sentence and reject ones with too many "errors". I tried to use grammar_check but AWS Lambda doesn't like running it. I immediately get "connection refused" error as soon as I initialise it. (NLTK also needs to be 'spoofed' in order to run on AWS Lambda but I found how to do that).
EXAMPLES: GOOD:Manchester united boss jose mourinho has told his players to 'grow up' in order to stop conceding early on in games following their comeback wins over newcastle bournemouth and juventus
GARBAGE: [latest results brought to you by played 42 draws 8 etihad stadium manchester old trafford manchester etihad stadium manchester old trafford manchester etihad stadium manchester no content available city return to training after manchester derby win external link city draw fc basel in the ucl round of 16 external link report: united 1-2 city external link city win thrilling derby to move 11 point
回答1:
I have the beginnings of an answer to this, but it's not a simple or straightforward "do it like this" recipe.
The method I am attempting to work out in more detail amounts to articulating a number of constraints on well-formed sentences. There is no way you can enumerate all the possible kinds of noise in a corpus, but you can remove certain kinds of noise with certain kinds of filters, many of them ideally simple to understand and implement.
For example:
- Discard samples with a very low entropy
- Discard samples with characters or character sequences outside of the normal repertoire of English
- Discard samples with many repeated words
- Discard samples with many finite verbs
- I cooked up the last two just looking at your single example, but of course, it's impossible to tell whether this will work in the general case without access to more samples, or your entire corpus.
A prototype of this method was published in the LREC 2016 proceedings (helpfully, the proceedings are published under a lenient CC BY-NC 4.0 license): abstract but the submissions were restricted to a maximum of four pages, so the article is by necessity a very brief overview. My actual materials and scripts are on Github: https://github.com/rcv2/rcv2r1; but the corpus I used is not redistributable, so there's a piece missing.
Appendix A outlines a brief catalog of proposed constraints.
来源：https://stackoverflow.com/questions/53246564/how-to-rate-quality-of-a-scraped-sentence