I just facing this kind of problem. And if my interpreation for the problem and expected solution is correct, this is my working solution:
Publisher.objects.annotate(num_books=Count('book')).filter(book__rating__gt=3.0)
Just swap filter & annotate position. This is done in Django version 1.9