Express app.post getting called multiple times

Question

I have been building a youtube video conversion app which streams youtube videos using youtube-dl and saves them, everything was working fine until I attempted to stream a video that was over an hour long. When the task was anywhere between 50% - 100% complete or 40-80seconds in, would be when the entire block of code would get re-run resulting in multiple simultaneous streams occurring. The response can therefor never get sent as it waits for the pipe to finish. Adding next(); outside the stream function allowed the conversion to complete with out any interruption or reruns of the code block, however it resulted in the following error when attempting to send the response:

throw new Error('Can\'t set headers after they are sent.');

This is the Node.js code block in question:

app.post('/convert/', function (req, res, next){
  var url = 'http://www.youtube.com/'+req.body.url;
  var quality = req.body.quality;
  var socketId = req.body.socketId;


stream = youtubedl(url, 
  ['-f ' + quality],

  // // Additional options can be given for calling `child_process.execFile()`. 
  { cwd: __dirname });

stream.on('info', function(info) {

  console.log('Download started');
  console.log('filename: ' + info._filename);
  console.log('size: ' + info.size);
  console.log('format: ' + info.format);

  var fileName = info._filename;
  var videoId = info.id;
  var videoTitle = info.title;
  videoTitle = videoTitle.replace(/[^a-zA-Z0-9\s]/g, '');
  console.log(videoTitle);

  var mp4 = 'downloads/'+videoTitle+'-'+info.format_id+'.mp4';
  fs.writeFile(mp4, "file", function(err) {
      if(err) {
          return console.log(err);
      }

    console.log("The file was saved!");
    var stat = fs.statSync(mp4);
    var str = progress({
      length: info.size,
      time: 100
    });

    str.on('progress', function(progress) {
      io.to(global.socketid).emit('progressVideo',{progress: progress.percentage});
      console.log(info.size);
      console.log(progress.transferred);
      console.log(progress.percentage);
      console.log(progress.remaining);
    });

    var pipe = stream.pipe(str).pipe(fs.createWriteStream(mp4));

    pipe.on('finish', function () {
      console.log("stream finished.");
      res.json(videoTitle+'-'+info.format_id+'.mp4');
    });
  });
 });
 // next();
});

Called by some Angular code.

// Sends youtube link to backend
$scope.getVideo = function(youtubeLink, resolution){
    var cleanedLink = youtubeLink.substring(24);
    var url = {url: cleanedLink, quality: resolution};
    $http.post('/convert/', url).success(function (response){
      // Do some stuff
    });
}

Confused as to why it was getting run more then once, I slowly removed more and more code until I was left with this simple test.

app.post('/convert/', function (req, res, next){
  console.log('hello!');
});

Which was called by an ng-click event and after waiting a minute or so the console also printed out two and then three hello! statements. I am completely lost as to why this happens. If anyone could shed some light on this for me it would be greatly appreciated.

Answer 1

So after just getting down to a very basic post route and logging out some text but not returning a response, I came to the conclusion the issue resides with node. I decided to record the length of time between the console.log statements which turned out to be every 2 minutes. With this I was able to find out that node has a default timeout of 2 minutes if a response is not sent back to the client.

I was able to set the response to never timeout with the following code:

res.connection.setTimeout(0);

I hope this helps anyone else that needs to hold connections for large periods of times for file conversions/transfers etc...