Django making a list of a field grouping by another field in model

Question

I have a model called MyModel which has some dummy data as follows:

     item    date     value
------------------------------
      ab    8/10/12   124
      ab    7/10/12   433
      ab    6/10/12    99
      abc   8/10/12    23
      abc   7/10/12    80

I would like to query this Model in such a way that i get an output as follows:

[{'item': 'ab', 'values': [ 124, 433, 99]},
 {'item': 'abc', 'values': [ 23, 80]}]

How would i be able to do this using the django ORM?

Answer 1

(Apr 4 '16) UPDATE: This is a working solution for Django <= 1.7. For newer versions please read Creating your own Aggregate Functions from the docs.

Using a custom Concat aggregate taken from here (an article about the topic)

Define this:

class Concat(models.Aggregate):
    def add_to_query(self, query, alias, col, source, is_summary):
        #we send source=CharField to prevent Django from casting string to int
        aggregate = SQLConcat(col, source=models.CharField(), is_summary=is_summary, **self.extra)
        query.aggregates[alias] = aggregate

#for mysql
class SQLConcat(models.sql.aggregates.Aggregate):
    sql_function = 'group_concat'

    @property
    def sql_template(self):
        if self.extra.get('separator'):
            return '%(function)s(%(field)s SEPARATOR "%(separator)s")'
        else:
            return '%(function)s(%(field)s)'

#For PostgreSQL >= 9.0
#Aways use with separator, e.g. .annotate(values=Concat('value', separator=','))     
class SQLConcat(models.sql.aggregates.Aggregate):
    sql_function = 'string_agg'

    @property
    def sql_template(self):
        #the ::text cast is a hardcoded hack to work with integer columns
        return "%(function)s(%(field)s::text, '%(separator)s')"

#For PostgreSQL >= 8.4 and < 9.0
#Aways use with separator, e.g. .annotate(values=Concat('value', separator=','))     
class SQLConcat(models.sql.aggregates.Aggregate):
    sql_function = 'array_to_string'

    @property
    def sql_template(self):
        return "%(function)s(array_agg(%(field)s), '%(separator)s')"

#For PostgreSQL < 8.4 you should define array_agg before using it:
#CREATE AGGREGATE array_agg (anyelement)
#(
#    sfunc = array_append,
#    stype = anyarray,
#    initcond = '{}'
#);

class MyModel(models.Model):
    item = models.CharField(max_length = 255)
    date = models.DateTimeField()
    value = models.IntegerField()

so now you can do:

>>> from my_app.models import MyModel, Concat
>>> MyModel.objects.values('item').annotate(values=Concat('value'))
[{'item': u'ab', 'values': u'124,433,99'}, {'item': u'abc', 'values': u'23,80'}]

to get values as a list of integers you need to manually .split and cast to int. Something like:

>>> my_list = MyModel.objects.values('item').annotate(values=Concat('value'))
>>> for i in my_list:
...     i['values'] = [int(v) for v in i['values'].split(',')]
...
>>> my_list
[{'item': u'ab', 'values': [124, 433, 99]}, {'item': u'abc', 'values': [23, 80]}]

Answer 2

In case your Django is backed by PostgreSQL, you can use extra in combination with string_agg.

https://docs.djangoproject.com/en/1.8/ref/models/querysets/#extra http://www.postgresql.org/docs/9.4/static/sql-expressions.html#SYNTAX-AGGREGATES

Assuming the table you posted is a ManyToManyRelation with item actually being a reference to your model MyModel, with the model table being mymodel and the mapping table being mymodel_value:

MyModel.objects.extra(select={
    'values':
        """
        SELECT string_agg(value, ', ' ORDER BY value)
          FROM mymodel_value
         WHERE mymodel.id=mymodel_value.item
        """
}).values('values')

The resulting dictionary will have an entry values with a string as value that is the concatenated (aggregated) list of values for each item.

Try this in the Django shell (./manage.py shell). You might have to add more tables to the subselect in case those have not been added by the ORM, yet. (The main model table should definitely be there, already.) It depends on how complex the model relations are.

Turn on DB logging to inspect the query that the ORM generates.