How To Rate-Limit An API [closed]

Answer 1

For an ideal amount of performance ,you can run a light-weight web framework with functions for managing logs on an in-memory database for monitoring and logging the traffic data , be it based on IP or User or Service called by user. The more important choice is the data storage you want to employ.

Best and most used free options are :

redis.io advanced key-value store

ehcache standards-based cache, actively developed, maintained and supported as a professional open source project by Terracotta

hazelcast an open source In-Memory Data Grid for faster execution and seamless elastic scalability

VoltDB an in-memory operational database

Answer 2

Try nginx. Rate limiting can be done easily by writing simple changes in the configuration file. Moreover, nginx is fast.

Answer 3

I definitely would not recommend doing this with MySQL - the problem isn't so much reads or the inefficiency in the algorithm you're highlighting there - but writes. As volumes go up you'll start getting into multi-second writes. We use REDIS as storage as another poster already mentioned - it has atomic increment/decrement functions which are exactly what you need + it's extremely fast (in memory) - you just have to manage sharding at ultra high volumes (but that ultra-high is many orders of magnitude above MySQL). Another option if you're not familiar with REDIS is doing in Memcached - but it's not quite as nice on the operations level.

A further option is still is to use something like 3scale (http://www.3scale.net) which effectively does all this for you + other stuff (analytics, key management, developer docs etc.). There are code plugins for a whole bunch of languages (https://support.3scale.net/libraries) and these connect to the infrastructure. You can also use the Varnish Libmod (https://github.com/3scale/libvmod-3scale/) and plug that into a Varnish cache in front of the API.

Answer 4

You can try to use Redis, there are few pattern for rate limiting

Answer 5

I'm currently investigating this issue as well. My current plan (Note this is with a LAMP stack!) is to implement this using APC's cacheing functions. When a request is received, I check to see if that IP is stored in APC's cache. If it is, then check to see if it is greater than 'X' where 'X' is the max requests per unit time. If it isn't, then create the cache entry for that IP.

This system means that no database access is required to check the rate limiting, and it doesn't rely on anything like a MongoDB or Redis server. It does assume you are using PHP with APC; if you aren't, then memcached might work instead.