Azure WebJob Command Timeout

Question

We are having issue with Azure Web Jobs. We created a C# console application, zipped it, and created the new Web Job. It's a c# console app that will constantly hit one of our web services to process items in queue.

Whenever we run the Web Job, we are getting the following error:

'cmd /c xxxxxxxx....' aborted due to no output and CPU activity for 121 seconds. You may increase SCM_COMMAND_IDLE_TIMEOUT setting to solve the issue

When we increased the SCM_COMMAND_IDLE_TIMEOUT to 600 (10 minutes). The job DOES run for 10 minutes - and then we get the same error with the same 121 seconds error.

What are we doing wrong?

Here is the console app code:

static void Main(string[] args)
    {

        bool ThereAreItemsInQueue = true;
        int Counter = 1;
        DateTime StartTime = DateTime.Now;

        while(ThereAreItemsInQueue)
        {
            Task.Run(() => {
                try
                {
                    //DEQUEUE
                    byte[] response = HttpHelper.HttpPOST(@"xxxxxxxxxxxxx", new byte[0]);
                    string strResponse = System.Text.Encoding.Default.GetString(response);
                    System.Diagnostics.Trace.TraceError("Attempt #" + Counter + "DEQUEUE FINISHED. Response:" + strResponse);

                    //CHECK IF THE QUEUE IS EMPTY
                    if (strResponse.Contains("Were Done"))
                        ThereAreItemsInQueue = false;

                }
                catch(Exception ex)
                {
                    System.Diagnostics.Trace.TraceError("Error Has Occured on attempt #" + Counter + "." + ex.Message + "\r" + ex.StackTrace);
                }

            });

            System.Threading.Thread.Sleep(5000);

            //SEE IF THIS HAS BEEN RUNNING FOR MORE THAN 24 HOURS
            if (DateTime.Now.Subtract(StartTime).TotalHours >= 24)
                ThereAreItemsInQueue = false;

            Counter++;
        }

    }

Are we approaching this problem the wrong way?

Note: each HttpHelper.HttpPOST request takes about 2 seconds - so that's not the issue.

Note2: We are using Task.Run to create "set-it-and-forget-it" type of requests.

Note3: The website setting of "Always On" - is turned on.

Answer 1

For triggered WebJobs the way to increase idle timeout is using the app setting: WEBJOBS_IDLE_TIMEOUT. Set it to your desired timeout in seconds.

The error is confusing and only refers to idle timeout during deployment.

https://github.com/projectkudu/kudu/wiki/Web-jobs#configuration-settings

Answer 2

This seems to have solved my problem:

if (Counter % 25 == 0)
   Console.WriteLine("Heartbeat");

I guess you have to keep writing out to console to keep the JOB running.