Django 1.11 Annotating a Subquery Aggregate

Question

This is a bleeding-edge feature that I\'m currently skewered upon and quickly bleeding out. I want to annotate a subquery-aggregate onto an existing queryset. Doing this bef

Answer 1

I just bumped into a VERY similar case, where I had to get seat reservations for events where the reservation status is not cancelled. After trying to figure the problem out for hours, here's what I've seen as the root cause of the problem:

Preface: this is MariaDB, Django 1.11.

When you annotate a query, it gets a GROUP BY clause with the fields you select (basically what's in your values() query selection). After investigating with the MariaDB command line tool why I'm getting NULLs or Nones on the query results, I've came to the conclusion that the GROUP BY clause will cause the COUNT() to return NULLs.

Then, I started diving into the QuerySet interface to see how can I manually, forcibly remove the GROUP BY from the DB queries, and came up with the following code:

from django.db.models.fields import PositiveIntegerField

reserved_seats_qs = SeatReservation.objects.filter(
        performance=OuterRef(name='pk'), status__in=TAKEN_TYPES
    ).values('id').annotate(
        count=Count('id')).values('count')
# Query workaround: remove GROUP BY from subquery. Test this
# vigorously!
reserved_seats_qs.query.group_by = []

performances_qs = Performance.objects.annotate(
    reserved_seats=Subquery(
        queryset=reserved_seats_qs,
        output_field=PositiveIntegerField()))

print(performances_qs[0].reserved_seats)

So basically, you have to manually remove/update the group_by field on the subquery's queryset in order for it to not have a GROUP BY appended on it on execution time. Also, you'll have to specify what output field the subquery will have, as it seems that Django fails to recognize it automatically, and raises exceptions on the first evaluation of the queryset. Interestingly, the second evaluation succeeds without it.

I believe this is a Django bug, or an inefficiency in subqueries. I'll create a bug report about it.

Edit: the bug report is here.