When does the App Engine scheduler use a new thread vs. a new instance?

Question

If I set threadsafe: true in my app.yaml file, what are the rules that govern when a new instance will be created to serve a request, versus when a

Answer 1

The following set of rules are currently used to determine if a given instance can accept a new request:

if processing more than N concurrent requests (today N=10): false
elif exceeding the soft memory limit: false
elif exceeding the instance class CPU limit: false
elif warming up: false
else true

The following of total CPU/core limits currently apply to each instance classes:

CLASS 1: 600MHz 1 core
CLASS 2: 1.2GHz 1 core
CLASS 4: 2.4GHz 1 core
CLASS 8: 4.8GHz 2 core

So only a B8 instance can process up to 2 fully CPU bound requests in parallel.

Setting threadsafe: true (Python) or <threadsafe>true</threadsafe> (Java) for instances classes < 8 would not allow more than one CPU bound requests to be processed in parallel on a single instance.

If you are not fully CPU bound or doing I/O, the Python and Java runtime will spawn new threads for handling new request up to 10 concurrent requests with threadsafe: true

Also note that even though the Go runtime is single threaded, it does support concurrent requests: It will spawn 1 goroutine per requests and yield control between goroutines while they are performing I/O.

Answer 2

If I set threadsafe: true in my app.yaml file, what are the rules that govern when a new instance will be created to serve a request, versus when a new thread will be created on an existing instance?

Like people are saying here, if a previous instance is already using 10 threads, a new instance with a new thread would be initiated. A new thread will be created if all other threads are busy, they must be either waiting for some response or with computing results.

If I have an app which performs something computationally intensive on each request, does multi-threading buy me anything? In other words, is an instance a multi-core instance or a single core?

Now this question is very controversial. Everyone knows the answer but still they are skeptical. Multi-threading can never buy you any good if your task is based on just computations unless you're using a multi-core processor, don't ask me why a multi-core processor will help better, you know the answer. Now google app engine is not sophisticated enough to decide that when new threads should be dispatched to the other processor/core(if it exists), only new instances are dispatched to the other core/processor. Want your thread to run in the other core/processor? Well, throw some skills there and booya! Remember, it's upto you to decide if threads should run in other cores/processors, the engine can not take the responsibility for such because this could lead to so many confusions, the engine is not God. In short, by default the instance is single core, the engine can't decide for you when it should go multi-core.

Or, are new threads only spun up when existing threads are waiting on IO?

The first part of my answer clears this out. Yes, they only spun up when existing threads are busy, this is how threadsafe works, to prevent deadlocks.

Now I can tell you this all, from my personal experience, I worked on the app engine for many months and programmed/debugged/tested apps that were highly dependent on the threadsafe architecture. If you want I can add references(I don't have references, just personal experience, but I'm ready to search and put things on the table for you), but I don't think they are needed in this case, threadsafe works in obvious ways which I have validated myself.

Answer 3

Read the next messages from link which was suggested by Kyle Finley

Jeff Schnitzer: Is there still a hard limit of 10 threads?

Yes, but probably not for the reason you expect. The primary issue we run into is memory management. If we raised the default to 100, many apps would then see out-of-memory deaths (more than they do now), and these deaths show up differently for python/java/go. The right path forward is more intelligent algorithms wrt memory, providing configurability, and so on. This is an example of the kinds of projects we work on for the scheduler, but as with any team we have to prioritize our projects. I'd recommend filing this (or any other desired scheduler enhancements) on the public issue tracker so they can get feedback/data/votes.