Tweepy. Make stream run forever

Question

I am relatively new to tweepy python library. I want to be sure that my stream python script always runs on a remote server. So it would be great if someone will share the best practices on how to make it happen.

Right now I am doing it this way:

if __name__ == '__main__':

while True:

    try:

        # create instance of the tweepy tweet stream listener
        listener = TweetStreamListener()

        # set twitter keys/tokens
        auth = OAuthHandler(CONSUMER_KEY, CONSUMER_SECRET)
        auth.set_access_token(ACCESS_TOKEN, ACCESS_TOKEN_SECRET)

        # create instance of the tweepy stream
        stream = Stream(auth, listener)

        stream.userstream()
    except Exception as e:
        print "Error. Restarting Stream.... Error: "
        print e.__doc__
        print e.message
        time.sleep(5)

And I return False on each of the methods: on_error(), on_disconnect(), on_timeout(). So, by returning False the stream stops and then reconnects in the infinite loop.

Answer 1

Here's how I do mine and it's been running for almost a year, on two computers to handle the errors that stop the stream here and there.

#They don't need to be in the loop.
auth = OAuthHandler(CONSUMER_KEY, CONSUMER_SECRET)
auth.set_access_token(ACCESS_TOKEN, ACCESS_TOKEN_SECRET)

while True:
    listener = TweetStreamListener()
    stream = Stream(auth, listener, timeout=60)

    try:
        stream.userstream()

    except Exception, e:
        print "Error. Restarting Stream.... Error: "
        print e.__doc__
        print e.message

To make sure that it runs forever, you should redefine the on_error method to handle the time between reconnection attempts. Your 5 seconds sleeping will hinder your chances to a successful reconnect because Twitter will see that you tried to do it too frequently. But that's another question.

Answer 2

Just my two cents.

I received lots of Error 420, which was weird because I didn't ask for too much keywords to the stream API.

So I figured out that the on_data() method of the stream listener class must always return True.

Mine returned False sometimes, so tweepy cut the connection, and recreate it directly as it was in a loop, twitter didn't like it much...