Test Automation with Embedded Hardware

Question

Has anyone had success automating testing directly on embedded hardware?

Specifically, I am thinking of automating a battery of unit tests for hardw

Answer 1

We have had good results with automated testing on our embedded systems. We have test written in high level (easy to program and debug) languages that run on dedicated test machines. These test generally do sanity checking or generate random inputs into the devices, then check for correct behavior. There is a lot of work to generate and maintain these tests. We designed a framework and then let interns work on the tests themselves.

It's not a perfect solution, and the tests are certainly prone to errors, but the most important part is to improve on your existing coverage holes. Find the biggest hole and design something to cover it in an automated fashion, even if it isn't perfect or won't cover the entire feature. Later when all of your stuff is covered somewhat, you can come back and address the worst coverage or the most critical features.

Some things to consider:

• What is the penalty of a firmware bug? How easier is it to update firmware in the field.
• What kind of coverage do my test provide? Is it a simple sanity check? Is it configurable enough that it can test many different scenarios?
• Once a test has failed, how will you reproduce that value in order to debug it? Did you log all the device and test settings so you can eliminate as many variables as possible? Device configuration, firmware version, test software version, all external inputs, all observed behavior?
• What are you testing against? Is the spec clear enough on what the expected behavior of the device you are testing or are you validating against what you think the code should do?