Node.js - Monitoring a database for changes

Question

I am using a node.js server to create a \'close to real-time\' socket between my web app and a database. Currently I am using MySQL which I am polling every second in node

Answer 1

I wouldn't personally use a polling mechanism for this. I think this is a pretty good use case for a pub/sub mq as a component on top of the database that allows consumers to subscribe to specific channels for change events on entities that they care about.

Ex:

1. Someone requests that a model be changed
2. Model broadcasts change event
3. Queue up change to be persisted in the database
4. Fire off a change set on a specific channel in a message queue for distribution to all interested parties

You can use a very simple in process pub/sub mechanism for this type of thing using nodes EventEmitter, and as you need to scale, have durability requirements, or need a cross language MQ you can go to a technology like rabbitmq, zeromq, etc. I've started to implement something very lightweight to do just this in one of my applications: https://github.com/jmoyers/mettle/blob/master/src/pubsub.coffee

It boils down to something like:

pubsub.sub('users.*', function(updates){
    // Interested party handles updates for user objects
});

That way you aren't putting stupid polling pressure on your database. In fact, change distribution is completely independent of writing to your database

Josh

Answer 2

I agree with the answer given by @Josh however if for whatever reason you are forced to monitor the state of a MySQL database then it is best to perform any polling server-side to significantly reduce the load on the server. One such technique I have used is to create a stored procedure that monitors some query result N times with a sleep.

DELIMITER //

CREATE PROCEDURE waitForResults(OUT c INT)
BEGIN
  DECLARE n INT DEFAULT 20;
  DECLARE x INT;

  WHILE n > 0 DO
    SELECT SLEEP(0.5) INTO x;
    SELECT COUNT(*) FROM `jobs` INTO c;
    IF (c > 0) THEN SET n = 0;
    ELSE SET n = n - 1;
    END IF;
  END WHILE;

END //

DELIMITER ;

You can then query the database when this stored procedure returns. Knowing something has changed and then call the procedure again to wait on results.