What are the connection limits for Google Cloud SQL from App Engine, and how to best reuse DB connections?

Question

I have a Google App Engine app that uses a Google Cloud SQL instance for storing data. I need my instance to be able to serve hundreds of clients at a time, via restful cal

Answer 1

I read from the documentation and noticed there's a 12 connection / instance limit:

Look for "Each App Engine instance cannot have more than 12 concurrent connections to a Google Cloud SQL instance." in https://developers.google.com/appengine/docs/python/cloud-sql/

Answer 2

Short answer: Your queries are probably too slow and the mysql server doesn't have enough threads to process all of the requests you are trying to send it.

Long Answer:

As background, Cloud SQL has two limits that are relevant here:

• Connections: These correspond to the 'conn' object in your code. There is a corresponding datastructure on the server. Once you have too many of these objects (currently configured to 1000), the least recently used will automatically be closed. When a connection gets closed underneath you, you'll get an unknown connection error (ApplicationError: 1007) the next time you try to use that connection.
• Concurrent Requests: These are queries that are executing on the server. Each executing query ties up a thread in the server, so there is a limit of 100. When there are too many concurrent requests, subsequent requests will be rejected with the error you are getting (ApplicationError: 1033)

It doesn't sound like the connection limit is affecting you, but I wanted to mention it just in case.

When it comes to Concurrent Requests, increasing the limit might help, but it usually makes the problem worse. There are two cases we've seen in the past:

• Deadlock: A long running query is locking a critical row of the database. All subsequent queries block on that lock. The app times out on those queries, but they keep running on the server, tying up those threads until the deadlock timeout triggers.
• Slow Queries: Each query is really, really slow. This usually happens when the query requires a temporary file sort. The application times out and retries the query while the first try of the query is still running and counting against the concurrent request limit. If you can find your average query time, you can get an estimate of how many QPS your mysql instance can support (e.g. 5 ms per query means 200 QPS for each thread. Since there are 100 threads, you could do 20,000 QPS. 50 ms per query means 2000 QPS.)

You should use EXPLAIN and SHOW ENGINE INNODB STATUS to see which of the two problems is going on.

Of course, it is also possible that you are just driving a ton of traffic at your instance and there just aren't enough threads. In that case, you'll probably be maxing out the cpu for the instance anyway, so adding more threads won't help.