django's .extra(where= clauses are clobbered by table-renaming .filter(foo__in=… subselects

Question

The short of it is, the table names of all queries that are inside a filter get renamed to u0, u1, ..., so my extra where clauses won't know what table to point to. I would love to not have to hand-make all the queries for every way I might subselect on this data, and my current workaround is to turn my extra'd queries into pk values_lists, but those are really slow and something of an abomination.

Here's what this all looks like. You can mostly ignore the details of what goes in the extra of this manager method, except the first sql line which points to products_product.id:

def by_status(self, *statii):
    return self.extra(where=["""products_product.id IN                                                                                                                                                  
        (SELECT recent.product_id                                                                                                                                                                          
          FROM (                                                                                                                                                                                           
            SELECT product_id, MAX(start_date) AS latest                                                                                                                                                   
            FROM products_productstatus                                                                                                                                                                    
            GROUP BY product_id                                                                                                                                                                            
          ) AS recent                                                                                                                                                                                      
          JOIN products_productstatus AS ps ON ps.product_id = recent.product_id                                                                                                                           
          WHERE ps.start_date = recent.latest                                                                                                                                                              
            AND ps.status IN (%s))""" % (', '.join([str(stat) for stat in statii]),)])

Which works wonderfully for all the situations involving only the products_product table.

When I want these products as a subselect, i do:

Piece.objects.filter(
    product__in=Product.objects.filter(
        pk__in=list(
            Product.objects.by_status(FEATURED).values_list('id', flat=True))))

How can I keep the generalized abilities of a query set, yet still use an extra where clause?

Answer 1

At first: the issue is not totally clear to me. Is the second code block in your question the actual code you want to execute? If this is the case the query should work as expected since there is no subselect performed.

I assume so that you want to use the second code block without the list() around the subselect to prevent a second query being performed.

---

The django documentation refers to this issue in the documentation about the extra method. However its not very easy to overcome this issue.

The easiest but most "hakish" solution is to observe which table alias is produced by django for the table you want to query in the extra method. You can rely on the persistent naming of this alias as long as you construct the query always in the same fashion (you don't change the order of multiple extra methods or filter calls that cause a join).

You can inspect a query that will be execute in the DB queryset by using:

print Model.objects.filter(...).query

This will reveal the aliases that are used for the tables you want to query.