Efficient way to bulk insert with get_or_create() in Django (SQL, Python, Django)

Question

Is there a more efficient way for doing this?

for item in item_list:
    e, new = Entry.objects.get_or_create(
        field1 = item.field1,
        field2 =         


        

Answer 1

If you're not sure whether the things in your item_list already exist in your DB, and you need the model objects, then get_or_create is definitely the way to go.

If you know the items are NOT in your DB, you would be much better doing:

for item in item_list:
    new = Entry.objects.create(
        field1 = item.field1,
        field2 = item.field2,
    )

And if you don't need the objects, then just ignore the return from the function call. It won't speed the DB stuff, but it will help with memory management if that's an issue.

If you're not sure whether the data is already in the DB, but either field has a unique=True flag on it, then the DB will enforce the uniqueness, and you can just catch the exception and move on. This will prevent an extra DB hit by avoiding the attempt to select the existing object.

from django.db import IntegrityError

for item in item_list:
    try:
        new = Entry.objects.create(
            field1 = item.field1,
            field2 = item.field2,
        )
    except IntegrityError:
        continue

You could increase speed in either case by manually managing the transactions. Django will automatically create and commit a transaction for every save, but provides some decorators that will greatly increase efficiency if you know you'll be doing a lot of DB saves in a particular function. The Django docs do a better job of explaining all of this than I can here, but you'll probably want to pay particular attention to django.db.transaction.commit_on_success