GROUP_CONCAT equivalent in Django

Question

Say I have the following table called fruits:

id | type   | name
-----------------
 0 | apple  | fuji
 1 | apple  | mac
 2 | orange | navel

My goal is to ultimately come up with a count of the different types and a comma-delimited list of the names:

apple, 2, "fuji,mac"
orange, 1, "navel"

This can be easily done with GROUP_CONCAT in MySQL but I'm having trouble with the Django equivalent. This is what I have so far but I am missing the GROUP_CONCAT stuff:

query_set = Fruits.objects.values('type').annotate(count=Count('type')).order_by('-count')

I would like to avoid using raw SQL queries if possible.

Any help would be greatly appreciated!

Thanks! =)

Answer 1

The Django ORM does not support this; if you don't want to use raw SQL then you'll need to group and join.

Answer 2

You can create your own Aggregate Function (doc)

from django.db.models import Aggregate

class Concat(Aggregate):
    function = 'GROUP_CONCAT'
    template = '%(function)s(%(distinct)s%(expressions)s)'

    def __init__(self, expression, distinct=False, **extra):
        super(Concat, self).__init__(
            expression,
            distinct='DISTINCT ' if distinct else '',
            output_field=CharField(),
            **extra)

and use it simply as:

query_set = Fruits.objects.values('type').annotate(count=Count('type'),
                       name = Concat('name')).order_by('-count')

I am using django 1.8 and mysql 4.0.3

Answer 3

NOTICE that Django (>=1.8) provides Database functions support. https://docs.djangoproject.com/en/dev/ref/models/database-functions/#concat

Here is an enhanced version of Shashank Singla

from django.db.models import Aggregate, CharField


class GroupConcat(Aggregate):
    function = 'GROUP_CONCAT'
    template = '%(function)s(%(distinct)s%(expressions)s%(ordering)s%(separator)s)'

    def __init__(self, expression, distinct=False, ordering=None, separator=',', **extra):
        super(GroupConcat, self).__init__(
            expression,
            distinct='DISTINCT ' if distinct else '',
            ordering=' ORDER BY %s' % ordering if ordering is not None else '',
            separator=' SEPARATOR "%s"' % separator,
            output_field=CharField(),
            **extra
        )

Usage:

LogModel.objects.values('level', 'info').annotate(
    count=Count(1), time=GroupConcat('time', ordering='time DESC', separator=' | ')
).order_by('-time', '-count')

Answer 4

As of Django 1.8 you can use Func() expressions.

query_set = Fruits.objects.values('type').annotate(count=Count('type'), name = Func(F('name'), 'GROUP_BY')).order_by('-count')

Answer 5

If you don't mind doing this in your template the Django template tag regroup accomplishes this

Answer 6

Use GroupConcat from the Django-MySQL package ( https://django-mysql.readthedocs.org/en/latest/aggregates.html#django_mysql.models.GroupConcat ) which I maintain. With it you can do it simply like:

>>> from django_mysql.models import GroupConcat
>>> Fruits.objects.annotate(
...     count=Count('type'),
...     types_list=GroupConcat('type'),
... ).order_by('-count').values('type', 'count', 'types_list')
[{'type': 'apple', 'count': 2, 'types_list': 'fuji,mac'},
 {'type': 'orange', 'count': 1, 'types_list': 'navel'}]

Answer 7

If you are using PostgreSQL, you can use ArrayAgg to aggregate all of the values into an array.

https://www.postgresql.org/docs/9.5/static/functions-aggregate.html

Answer 8

Not supported by Django ORM, but you can build your own aggregator.

It's actually pretty straightforward, here is a link to a how-to that does just that, with GROUP_CONCAT for SQLite: http://harkablog.com/inside-the-django-orm-aggregates.html

Note however, that it might be necessary to handle different SQL dialects separately. For example, the SQLite docs say about group_concat:

The order of the concatenated elements is arbitrary

While MySQL allows you to specify the order.

I guess that may be a reason why GROUP_CONCAT it's not implemented in Django at the moment.

Answer 9

To complete the answer of @WeizhongTu, Notice that you can not use the keyword SEPARATOR with SQLITE. In cases where you are using MySQL and SQLite for your tests, you can write :

class GroupConcat(Aggregate):
    function = 'GROUP_CONCAT'
    separator = ','

    def __init__(self, expression, distinct=False, ordering=None, **extra):
        super(GroupConcat, self).__init__(expression,
                                          distinct='DISTINCT ' if distinct else '',
                                          ordering=' ORDER BY %s' % ordering if ordering is not None else '',
                                          output_field=CharField(),
                                          **extra)

    def as_mysql(self, compiler, connection, separator=separator):
        return super().as_sql(compiler,
                              connection,
                              template='%(function)s(%(distinct)s%(expressions)s%(ordering)s%(separator)s)',
                              separator=' SEPARATOR \'%s\'' % separator)

    def as_sql(self, compiler, connection, **extra):
        return super().as_sql(compiler,
                              connection,
                              template='%(function)s(%(distinct)s%(expressions)s%(ordering)s)',
                              **extra)