Remove file in dir where grep 'pattern' | wc -l < 2

Question

How can I remove all *.txt files within a directory, where grep \'>\' | wc -l is less than 2?

Example:

file1.txt
>BB
&