Streaming API vs Rest API?

Question

The canonical example here is Twitter\'s API. I understand conceptually how the REST API works, essentially its just a query to their server for your particular request in

Answer 1

Twitter's stream API is that it's essentially a long-running request that's left open, data is pushed into it as and when it becomes available.

The repercussion of that is that the server will have to be able to deal with lots of concurrent open HTTP connections (one per client). A lot of existing servers don't manage that well, for example Java servlet engines assign one Thread per request which can (a) get quite expensive and (b) quickly hits the normal max-threads setting and prevents subsequent connections.

As you guessed the Node.js model fits the idea of a streaming connection much better than say a servlet model does. Both requests and responses are exposed as streams in Node.js, but don't occupy an entire thread or process, which means that you could continue pushing data into the stream for as long as it remained open without tying up excessive resources (although this is subjective). In theory you could have a lot of concurrent open responses connected to a single process and only write to each one when necessary.

If you haven't looked at it already the HTTP docs for Node.js might be useful.

I'd also take a look at technoweenie's Twitter client to see what the consumer end of that API looks like with Node.js, the stream() function in particular.