Django Rest Framework metrics per user requests

问题

I'm currently working on an Api build with Django Rest Framework, uwsgi nginx and memcached.

I would like to know what is the best way to get users statistics like number of requests per user? Taking in consideration that the infrastructure is probably going to scale to multiple servers.

And is there is a way to determine if the response was retrieve from cache or from the application?

What I'm thinking is processing the Nginx logs to separate the request by user and apply all the calculations.

回答1:

First: You might find drf-tracking to be a useful project, but it stores the response to every request in the database, which we found kind of crazy.

The solution for this that we developed instead is a a mixin that borrows heavily from drf-tracking, but that only logs statistics. This solution uses our cache server (Redis), so it's very fast.

It's very simple to drop in if you're already using Redis:

class LoggingMixin(object):
    """Log requests to Redis

    This draws inspiration from the code that can be found at: https://github.com/aschn/drf-tracking/blob/master/rest_framework_tracking/mixins.py

    The big distinctions, however, are that this code uses Redis for greater
    speed, and that it logs significantly less information.

    We want to know:
     - How many queries in last X days, total?
     - How many queries ever, total?
     - How many queries total made by user X?
     - How many queries per day made by user X?
    """

    def initial(self, request, *args, **kwargs):
        super(LoggingMixin, self).initial(request, *args, **kwargs)

        d = date.today().isoformat()
        user = request.user
        endpoint = request.resolver_match.url_name

        r = redis.StrictRedis(
            host=settings.REDIS_HOST,
            port=settings.REDIS_PORT,
            db=settings.REDIS_DATABASES['STATS'],
        )
        pipe = r.pipeline()

        # Global and daily tallies for all URLs.
        pipe.incr('api:v3.count')
        pipe.incr('api:v3.d:%s.count' % d)

        # Use a sorted set to store the user stats, with the score representing
        # the number of queries the user made total or on a given day.
        pipe.zincrby('api:v3.user.counts', user.pk)
        pipe.zincrby('api:v3.user.d:%s.counts' % d, user.pk)

        # Use a sorted set to store all the endpoints with score representing
        # the number of queries the endpoint received total or on a given day.
        pipe.zincrby('api:v3.endpoint.counts', endpoint)
        pipe.zincrby('api:v3.endpoint.d:%s.counts' % d, endpoint)

        pipe.execute()

Put that somewhere in your project, and then add the mixin to your various views, like so:

class ThingViewSet(LoggingMixin, viewsets.ModelViewSet):
    # More stuff here.

Some notes about the class:

It uses Redis pipelines to make all of the Redis queries hit the server with one request instead of six.
It uses Sorted Sets to keep track of which endpoints in your API are being used the most and which users are using the API the most.
It creates a few new keys in your cache per day -- there might be better ways to do this, but I can't find any.

This should be a fairly flexible starting point for logging the API.

回答2:

What I do usually is that I have a centralized cache server (Redis), I log all requests to there with all the custom calculations or fields I need. Then you can build your own dashboard.

Go with Logstash from the Elasticsearch company. Very well done saves you time and scales very good. I'd say give it a shot http://michael.bouvy.net/blog/en/2013/11/19/collect-visualize-your-logs-logstash-elasticsearch-redis-kibana/

来源：https://stackoverflow.com/questions/23932825/django-rest-framework-metrics-per-user-requests

标签

django

nginx

django-rest-framework