error: can't start new thread

Question

I have a site that runs with follow configuration:

Django + mod-wsgi + apache

In one of user\'s request, I send another HTTP request to another service, and

Answer 1

You are starting more threads than can be handled by your system. There is a limit to the number of threads that can be active for one process.

Your application is starting threads faster than the threads are running to completion. If you need to start many threads you need to do it in a more controlled manner I would suggest using a thread pool.

Answer 2

I think the best way in your case is to set socket timeout instead of spawning thread:

h = httplib.HTTPSConnection(self.config['server'], 
                            timeout=self.config['timeout'])

Also you can set global default timeout with socket.setdefaulttimeout() function.

Update: See answers to Is there any way to kill a Thread in Python? question (there are several quite informative) to understand why. Thread.__stop() doesn't terminate thread, but rather set internal flag so that it's considered already stopped.

Answer 3

I was running on a similar situation, but my process needed a lot of threads running to take care of a lot of connections.

I counted the number of threads with the command:

ps -fLu user | wc -l

It displayed 4098.

I switched to the user and looked to system limits:

sudo -u myuser -s /bin/bash

ulimit -u

Got 4096 as response.

So, I edited /etc/security/limits.d/30-myuser.conf and added the lines:

myuser hard nproc 16384

myuser soft nproc 16384

Restarted the service and now it's running with 7017 threads.

Ps. I have a 32 cores server and I'm handling 18k simultaneous connections with this configuration.

Answer 4

in my case, I added this code before starting a new thread. It gives the app a max limit of running threads will wait

while threading.active_count()>150 :
    time.sleep(5)
getting.start()

Note: It is not a good solution but I had to find a workaround to solve that issue and this worked for me.

Answer 5

The "can't start new thread" error almost certainly due to the fact that you have already have too many threads running within your python process, and due to a resource limit of some kind the request to create a new thread is refused.

You should probably look at the number of threads you're creating; the maximum number you will be able to create will be determined by your environment, but it should be in the order of hundreds at least.

It would probably be a good idea to re-think your architecture here; seeing as this is running asynchronously anyhow, perhaps you could use a pool of threads to fetch resources from another site instead of always starting up a thread for every request.

Another improvement to consider is your use of Thread.join and Thread.stop; this would probably be better accomplished by providing a timeout value to the constructor of HTTPSConnection.

Answer 6

I completely rewrite code from httplib to pycurl.

c = pycurl.Curl()
c.setopt(pycurl.FOLLOWLOCATION, 1)
c.setopt(pycurl.MAXREDIRS, 5)
c.setopt(pycurl.CONNECTTIMEOUT, CONNECTION_TIMEOUT)
c.setopt(pycurl.TIMEOUT, COOPERATION_TIMEOUT)
c.setopt(pycurl.NOSIGNAL, 1)
c.setopt(pycurl.POST, 1)
c.setopt(pycurl.SSL_VERIFYHOST, 0)
c.setopt(pycurl.SSL_VERIFYPEER, 0)
c.setopt(pycurl.URL, "https://"+server+path)
c.setopt(pycurl.POSTFIELDS,sended_data)

b = StringIO.StringIO()
c.setopt(pycurl.WRITEFUNCTION, b.write)

c.perform()

something like that.

And I testing it now. Thanks all of you for help.