Strategy for feeding a watchdog in a multitask environment

Question

Having moved some embedded code to FreeRTOS, I'm left with an interesting dilemma about the watchdog. The watchdog timer is a must for our application. Using FreeRTOS has been a huge boon for us too. When the application was more single-tasked, it fed the watchdog at timely points in its logic flow so that we could make sure the task was making logical progress in a timely fashion.

With multiple tasks though, that's not easy. One task could be bound up for some reason, not making progress, but another is doing just fine and making enough progress to keep the watchdog fed happily.

One thought was to launch a separate task solely to feed the watchdog, and then use some counters that the other tasks increment regularly, when the watchdog task ticks, it would make sure that all the counters looked like progress was being made on all the other tasks, and if so, go ahead and feed the watchdog.

I'm curious what others have done in situations like this?

Answer 1

A watchdog task that monitors the status of all the other tasks is a good solution. But instead of a counter, consider using a status flag for each task. The status flag should have three possible values: UNKNOWN, ALIVE, and ASLEEP. When a periodic task runs, it sets the flag to ALIVE. Tasks that block on an asynchronous event should set their flag to ASLEEP before they block and ALIVE when the run. When the watchdog monitor task runs it should kick the watchdog if every task is either ALIVE or ASLEEP. Then the watchdog monitor task should set all of the ALIVE flags to UNKNOWN. (ASLEEP flags should remain ASLEEP.) The tasks with the UNKNOWN flag must run and set their flags to ALIVE or ASLEEP again before the monitor task will kick the watchdog again.

See the "Multitasking" section of this article for more details: http://www.embedded.com/design/debug-and-optimization/4402288/Watchdog-Timers

Answer 2

This is indeed a big pain with watchdog timers.

My boards have an LED on a GPIO line, so I flash that in a while/sleep loop, (750ms on, 250ms off), in a next-to-lowest priority thread, (lowest is idle thread which just goes onto low power mode in a loop). I have put a wdog feed in the LED-flash thread.

This helps with complete crashes and higher-priority threads that CPU loop, but doesn't help if the system deadlocks. Luckily, my message-passing designs do not deadlock, (well, not often, anyway:).

Answer 3

Do not forget to handle possible situation where tasks are deleted, or dormant for longer periods of time. If those tasks were previously checked in with a watchdog task, they also need to have a 'check out' mechanism.

In other words, the list of tasks for which a watchdog task is responsible should be dynamic, and it should be organized so that some wild code cannot easily delete the task from the list.

I know, easier said then done...

Answer 4

I've design the solution using the FreeRTOS timers:

1. SystemSupervisor SW Timer which feed the HW WD. FreeRTOS Failure causes reset.
2. Each task creates "its own" SW timer with SystemReset function.
3. Each task responsible to "manually" reload its timer before it expired.
4. SystemReset function saves data before commiting a suiside

Here is some pseudo-code listing:

//---------------------------------
//
// System WD
//
void WD_init(void)
{
HW_WD_Init();
    //  Read Saved Failure data, Send to Monitor
    //  Create Monitor timer
    xTimerCreate(   "System WD",        //  Name
                    HW_WD_INTERVAL/2,   //  Reload value
                    TRUE,               //  Auto Reload
                    0,                  //  Timed ID (Data per timer)
                    SYS_WD_Feed);
}
void SYS_WD_Feed(void)
{
    HW_WD_Feed();
}

//-------------------------
//   Tasks WD
//
WD_Handler WD_Create()
{
    return xTimerCreate(   "",                 //  Name
                           100,                //  Dummy Reload value
                           FALSE,               //  Auto Reload
                           pxCurrentTCB,       //  Timed ID (Data per timer)
                           Task_WD_Reset);
}

Task_WD_Reset(pxTimer)
{
    TaskHandler_t th = pvTimerGetTimerID(pxTimer)
    // Save Task Name and Status
    //  Reset
}

Task_WD_Feed(WD_Handler, ms)
{
    xTimerChangePeriod(WD_Handler, ms / portTICK_PERIOD_MS, 100);
}