Threads in twisted… how to use them properly?

Question

I need to write a simple app that runs two threads: - thread 1: runs at timed periods, let's say every 1 minute - thread 2: just a 'normal' while True loop that does 'stuff'

if not the requirement to run at timed interval I would have not looked at twisted at all, but simple sleep(60) is not good enough and construction like:

l = task.LoopingCall(timed_thread)
l.start(60.0)
reactor.run()

Looked really simple to achieve what I wanted there.

Now, how do I 'properly' add another thread?

I see two options here:

• Use threading library and run two 'python threads' one executing my while loop, and another running reactor.run(). But Google seems to object this approach and suggests using twisted threading
• Use twisted threading. That's what I've tried, but somehow this looks bit clumsy to me.

Here's what I came up with:

def timed_thread():
    print 'i will be called every 1 minute'
    return

def normal_thread():
    print 'this is a normal thread'
    time.sleep(30)
    return

l = task.LoopingCall(timed_thread)
l.start(60.0)
reactor.callInThread(normal_thread)
reactor.run()

That seems to work, but! I can't stop the app. If I press ^C it wouldn't do anything (without 'callInThread' it just stops as you'd expect it to). ^Z bombs out to shell, and if I then do 'kill %1' it seems to kill the process (shell reports that), but the 'normal' thread keeps on running. kill PID wouldn't get rid of it, and the only cure is kill -9. Really strange.

So. What am I doing wrong? Is it a correct approach to implement two threads in twisted? Should I not bother with twisted? What other 'standard' alternatives are to implement timed calls? ('Standard' I mean I can easy_install or yum install them, I don't want to start downloading and using some random scripts from random web pages).

Answer 1

Assuming that your main is relatively non-blocking:

import random
from twisted.internet import task

class MyProcess:
  def __init__(self):
    self.stats = []
    self.lp = None
  def myloopingCall(self):
    print "I have %s stats" % len(self.stats)
  def myMainFunction(self,reactor):
    self.stats.append(random.random())
    reactor.callLater(0,self.myMainFunction,reactor)
  def start(self,reactor):
    self.lp = task.LoopingCall(self.myloopingCall)
    self.lp.start(2)
    reactor.callLater(0,self.myMainFunction,reactor)
  def stop(self):
    if self.lp is not None:
      self.lp.stop()
    print "I'm done"

if __name__ == '__main__':
  myproc = MyProcess()
  from twisted.internet import reactor
  reactor.callWhenRunning(myproc.start,reactor)
  reactor.addSystemEventTrigger('during','shutdown',myproc.stop)
  reactor.callLater(10,reactor.stop)
  reactor.run()

$ python bleh.py
I have 0 stats
I have 33375 stats
I have 66786 stats
I have 100254 stats
I have 133625 stats
I'm done

Answer 2

You didn't explain why you actually need threads here. If you had, I might have been able to explain why you don't need them. ;)

That aside, I can confirm that your basic understanding of things is correct. One possible misunderstanding I can clear up, though, is the notion that "python threads" and "Twisted threads" are at all different from each other. They're not. Python provides a threading library. All of Twisted's thread APIs are implemented in terms of Python's threading library. Only the API is different.

As far as shutdown goes, you have two options.

• Start your run-forever thread using Python's threading APIs directly and make the thread a daemon. Your process can exit even while daemon threads are still running. A possible problem with this solution is that some versions of Python have issues with daemon threads that will lead to a crash at shutdown time.
• Create your thread using either Twisted's APIs or the stdlib threading APIs but also add a Twisted shutdown hook using reactor.addSystemEventTrigger('before', 'shutdown', f). In that hook, communicate with the work thread and tell it to shut down. For example, you could share a threading.Event between the Twisted thread and your work thread and have the hook set it. The work thread can periodically check to see if it has been set and exit when it notices that it has been. Aside from not crashing, this gives another advantage over daemon threads - it will let you run some cleanup or finalization code in your work thread before the process exits.