DRF throttles 限流源码阅读

限流

流程分析

入口
实现

1.0 限流在 view 中的入口

dispatch
initial
check_throttles: 限流器检查

views.py

from django.views.generic import View

class APIView(View):
    throttle_classes = api_settings.DEFAULT_THROTTLE_CLASSES # setting.py

    def get_throttles(self):
        """
        Instantiates and returns the list of throttles that this view uses.
        """
        return [throttle() for throttle in self.throttle_classes]

    def throttled(self, request, wait):
        """
        If request is throttled, determine what kind of exception to raise.
        """
        raise exceptions.Throttled(wait)

    def dispatch(self, request, *args, **kwargs):
        """
        `.dispatch()` is pretty much the same as Django's regular dispatch,
        but with extra hooks for startup, finalize, and exception handling.
        """
        request = self.initialize_request(request, *args, **kwargs)
        self.request = request
        try:
            self.initial(request, *args, **kwargs)
            handler = getattr(self, request.method.lower(),
                                  self.http_method_not_allowed)
            response = handler(request, *args, **kwargs)
        except Exception as exc:
            response = self.handle_exception(exc)
        self.response = self.finalize_response(request, response, *args, **kwargs)
        return self.response

    def initial(self, request, *args, **kwargs):
        """
        Runs anything that needs to occur prior to calling the method handler.
        """
        # ...
        # Ensure that the incoming request is permitted
        self.perform_authentication(request)
        self.check_permissions(request)
        self.check_throttles(request)

    def check_throttles(self, request):
        """
        Check if request should be throttled.
        Raises an appropriate exception if the request is throttled.
        """
        throttle_durations = []
        # 遍历限流类实例列表(DEFAULT_THROTTLE_CLASSES)
        for throttle in self.get_throttles():
            # 当前限流类是否允许请求, 不允许加入到throttle_durations
            if not throttle.allow_request(request, self):
                throttle_durations.append(throttle.wait())
        if throttle_durations:
            # raise 429 限流异常
            self.throttled(request, duration)

2.0 限流具体实现

2.1 相关文件

settings.py
throttling.py

settings.py

DEFAULTS = {
    # 默认不限流
    'DEFAULT_THROTTLE_CLASSES': [],
    # 限流rate
    'DEFAULT_THROTTLE_RATES': {
        'user': None,
        'anon': None,
    },
}

2.2 限流类继承关系

BaseThrottle: 定义了check_throttles中throttle.allow_request接口
- allow_request: 抽象方法，子类实现
- get_ident: 默认去客户端ip
- wait: 等多
SimpleRateThrottle(BaseThrottle): 实现了(3/10s)基于redis中list的限流策略
- scope = None
- get_cache_key: 抽象方法，子类实现
- get_rate:
  - getattr(self, 'scope', None)
  - self.THROTTLE_RATES[self.scope]
AnonRateThrottle(SimpleRateThrottle)
- scope = 'anon'
- get_cache_key(self, request, view): 实现了为完成认证的用户，基于ip限制请求次数
- 优先级:
UserRateThrottle(SimpleRateThrottle)
- scope = 'user'
- get_cache_key:
  - ident = request.user.pk
ScopedRateThrottle(SimpleRateThrottle)
- scope_attr = 'throttle_scope'
- get_cache_key:
  - scope: getattr(view, self.scope_attr, None)
  - ident: user.pk

意义:

BaseThrottle实现了，view与限流的入口.
SimpleRateThrottle: 基于cache(redis list)实现了基本限流策略
- get_rate:
  - 优先自己的scope
  - 未实现时，使用api_settings.DEFAULT_THROTTLE_RATES
AnonRateThrottle: 实现了未认证的用户，基于ip限制请求次数
UserRateThrottle: 实现了认证用户，基于用户id限制请求次数
ScopedRateThrottle: 实现了针对部分APIView类中定义了throttle_scope的视图进行限流.

2.2.1 BaseThrottle

allow_request: 抽象方法，子类实现
get_ident: 默认去客户端ip
wait: 等多久

class BaseThrottle:
    """
    Rate throttling of requests.
    """

    def allow_request(self, request, view):
        raise NotImplementedError('.allow_request() must be overridden')

    def get_ident(self, request):
        return client_ip

    def wait(self):
        """
        距离下一次请求，还有多久
        """
        return None

2.2 SimpleRateThrottle

基于django cache实现的限流, (number_of_requests:请求次数/period时间间隔)

get_cache_key: 抽象方法

from django.core.cache import cache as default_cache

class SimpleRateThrottle(BaseThrottle):
    """
    A simple cache implementation, that only requires `.get_cache_key()`
    to be overridden.

    The rate (requests / seconds) is set by a `rate` attribute on the View
    class.  The attribute is a string of the form 'number_of_requests/period'.

    Period should be one of: ('s', 'sec', 'm', 'min', 'h', 'hour', 'd', 'day')

    Previous request information used for throttling is stored in the cache.
    """
    cache = default_cache
    timer = time.time
    cache_format = 'throttle_%(scope)s_%(ident)s'
    scope = None
    THROTTLE_RATES = api_settings.DEFAULT_THROTTLE_RATES

    def __init__(self):
        if not getattr(self, 'rate', None):
            self.rate = self.get_rate()
        self.num_requests, self.duration = self.parse_rate(self.rate)

    def get_cache_key(self, request, view):
        """
        Should return a unique cache-key which can be used for throttling.
        Must be overridden.

        May return `None` if the request should not be throttled.
        """
        raise NotImplementedError('.get_cache_key() must be overridden')

    def get_rate(self):
        """
        Determine the string representation of the allowed request rate.
        """
        if not getattr(self, 'scope', None):
            msg = ("You must set either `.scope` or `.rate` for '%s' throttle" %
                   self.__class__.__name__)
            raise ImproperlyConfigured(msg)

        try:
            return self.THROTTLE_RATES[self.scope]
        except KeyError:
            msg = "No default throttle rate set for '%s' scope" % self.scope
            raise ImproperlyConfigured(msg)

    def parse_rate(self, rate):
        """
        Given the request rate string, return a two tuple of:
        <allowed number of requests>, <period of time in seconds>
        """
        if rate is None:
            return (None, None)
        num, period = rate.split('/')
        num_requests = int(num)
        duration = {'s': 1, 'm': 60, 'h': 3600, 'd': 86400}[period[0]]
        return (num_requests, duration)

    def allow_request(self, request, view):
        """
        Implement the check to see if the request should be throttled.

        On success calls `throttle_success`.
        On failure calls `throttle_failure`.
        """
        if self.rate is None:
            return True

        self.key = self.get_cache_key(request, view)
        if self.key is None:
            return True

        self.history = self.cache.get(self.key, [])
        self.now = self.timer()

        # Drop any requests from the history which have now passed the
        # throttle duration
        while self.history and self.history[-1] <= self.now - self.duration:
            self.history.pop()
        if len(self.history) >= self.num_requests:
            return self.throttle_failure()
        return self.throttle_success()

    def throttle_success(self):
        """
        Inserts the current request's timestamp along with the key
        into the cache.

        "2/10s" 10s内请求2次
        num_requests: 2
        duration: 10s

        第1次: timestamp
        key1: []
        # 时间范围[timestamp-10, timestamp], 10s范围内的请求数量，是否大于num_requests
        --insert 0, self.now
        key1: [timestamp+1]


        第2次: timestamp+2
        key1: [timestamp+1]
        # self.now = timestamp+1
        # self.now - self.duration = timestamp + 1 - 10.  
        # 时间范围[timestamp-9, timestamp+1], 10s范围内的请求数量，是否大于num_requests
        while self.history and self.history[-1] <= self.now - self.duration:
            self.history.pop()  # 移除10s之外的请求时间
        if len(self.history) >= self.num_requests:
            return self.throttle_failure()
        # 请求成功，插入当前时间戳
        key1: [timestamp+1, timestamp]

        第3次: timestamp+2
        key1: [timestamp+1, timestamp]
        # self.now = timestamp+2
        # self.now - self.duration = timestamp + 2 - 10.  
        # 时间范围[timestamp-8, timestamp+2], 10s范围内的请求数量，是否大于num_requests
        while self.history and self.history[-1] <= self.now - self.duration:
            self.history.pop()  # 移除10s之外的请求时间
        if len(self.history) >= self.num_requests:
            return self.throttle_failure() # 以前请求成功了两次，第三次请求失败
        
        """
        self.history.insert(0, self.now)
        self.cache.set(self.key, self.history, self.duration)
        return True

    def throttle_failure(self):
        """
        Called when a request to the API has failed due to throttling.
        """
        return False

    def wait(self):
        """
        Returns the recommended next request time in seconds.
        """
        if self.history:
            remaining_duration = self.duration - (self.now - self.history[-1])
        else:
            remaining_duration = self.duration

        available_requests = self.num_requests - len(self.history) + 1
        if available_requests <= 0:
            return None

        return remaining_duration / float(available_requests)

3.0 应用场景

全局限流
APIView限流

3.1 全局限流

参考刘江drf教程
基于UserRateThrottle, 实现全局用户限流
burst: 一分钟内最多请求60次
sustained: 一天内最多请求1000次

# 第一个限流子类
class BurstRateThrottle(UserRateThrottle):
    scope = 'burst' # 主要用于控制爆发期的访问频率

# 第二个限流子类 
class SustainedRateThrottle(UserRateThrottle):
    scope = 'sustained' # 主要用于控制持续时期的访问频率

settings.py

REST_FRAMEWORK = {
    # 配置我们自定义的限流类
    'DEFAULT_THROTTLE_CLASSES': (
        'example.throttles.BurstRateThrottle',
        'example.throttles.SustainedRateThrottle'
    ),
    'DEFAULT_THROTTLE_RATES': {
        'burst': '60/min', # 每分钟最多60次，控制爆发期
        'sustained': '1000/day' # 每天最多100次，控制持续访问 
    }
}

3.2 特定APIview限流

ScopedRateThrottle
针对APIView定义了throttle_scope的类实现限流
ContactListView: 每个用户, 联系人列表, 限制每天1000次
ContactDetailView: 每个用户, 联系人详情, 限制每天1000次
UploadView: 每个用户，上传文件，限制每天20次

settings.py

REST_FRAMEWORK = {
        # 注册特定API限流类, view必须定义throttle_scope属性
       'DEFAULT_THROTTLE_CLASSES': (
           'rest_framework.throttling.ScopedRateThrottle', 
        ),
        'DEFAULT_THROTTLE_RATES': {
            'contacts': '1000/day', # 字典的键，对应视图中的scope属性 
            'uploads': '20/day'
        }
}

views.py

from rest_framework.views import APIView

class ContactListView(APIView):
    throttle_scope = 'contacts' 
    # 定义scope属性，并提供一个字符串，用于去settings中查找设置的值 ...

class ContactDetailView(APIView): 
    throttle_scope = 'contacts' 

class UploadView(APIView): 
    throttle_scope = 'uploads' # 同上

来源：oschina

链接：https://my.oschina.net/tplinuxhyh/blog/4527342

标签

def

ApiView