“Too many SQL variables” error in django with sqlite3

Question

I\'m getting this error using sqlite3 in django:

Exception Value: too many SQL variables

And I think the answer to it is this, from

Answer 1

I hava found the same error when I run the line:

    Entry.objects.all().delete()

I solved the problem by

    while Entry.objects.count():
          Entry.objects.all()[0].delete()

I think it's a better idea than others.

Answer 2

actually those limits are given here: https://www.sqlite.org/c3ref/c_limit_attached.html#sqlitelimitvariablenumber

with an explanation:

Maximum Number Of Host Parameters In A Single SQL Statement

A host parameter is a place-holder in an SQL statement that is filled in using one of the sqlite3_bind_XXXX() interfaces. Many SQL programmers are familiar with using a question mark ("?") as a host parameter. SQLite also supports named host parameters prefaced by ":", "$", or "@" and numbered host parameters of the form "?123".

Each host parameter in an SQLite statement is assigned a number. The numbers normally begin with 1 and increase by one with each new parameter. However, when the "?123" form is used, the host parameter number is the number that follows the question mark.

SQLite allocates space to hold all host parameters between 1 and the largest host parameter number used. Hence, an SQL statement that contains a host parameter like ?1000000000 would require gigabytes of storage. This could easily overwhelm the resources of the host machine. To prevent excessive memory allocations, the maximum value of a host parameter number is SQLITE_MAX_VARIABLE_NUMBER, which defaults to 999.

The maximum host parameter number can be lowered at run-time using the sqlite3_limit(db,SQLITE_LIMIT_VARIABLE_NUMBER,size) interface.

https://www.sqlite.org/limits.html

Answer 3

Note: I am not Django expert

The problem is that in your query

vals = Company.objects.filter(id__in=comp_ids)... 

you pass a bloated set of values and the underlaying engine cannot execute the query. Fortunately sqlalchemy, that you execute in that line, is lazy. You an execute a SELECT and then filter the entries "manually" without big performance penalties.

It would look more-less like that:

# here you should do a proper session.query(CompanyTable)
vals_all = Company.objects

vals_filtered = (item for item in vals_all if item.id in comp_ids)
vals_ordered = sorted(vals_filtered, key=attrgetter('name'))
vals_final = [(v.id, v.name) for v in vals_ordered]

Answer 4

Googling this error message brought me here so adding my solution.

In my case this error was caused by line:

Event.objects.all().delete()

Probably Django was trying to "DELETE WHERE id IN (?,?,?,..." and list of ids was to long.

Performance was not an issue for me so i solved this by:

while Event.objects.count():
    ids = Event.objects.values_list('pk', flat=True)[:100]
    Event.objects.filter(pk__in = ids).delete()

Answer 5

I had a similar problem, but when deleting just single row that had a large number of related rows in another table. Django was calling a cascading delete causing the problem, but since my code was only calling a single object delete, the solutions described in other answers could not be applied (unless I modified the Django library code, which I didn't want to do).

My solution was to first delete the related objects via raw sql query, then use the normal Django delete() on the parent object, all wrapped in a transaction.

Before:

object.delete() # Django tries to delete a large number of related object and fails

After:

from django.db import connection

Class MyObject:
    def delete_related(self):
        cur = connection.cursor()
        sql = 'delete from my_table where ...'
        cur.execute(sql, params)

...

with transaction.atomic():
    object.delete_related() # so that Django won't try (and fail) on its own
    object.delete()

Answer 6

There is no way to bypass this, it's an SQLITE limitation. Your query will run fine on MySQL however. But if you have to do this kind of query, you are most probably doing it wrong. Your mysql schema should be redone. And if you can't, then you may want to break this query into 100 of small queries.