In-target test automation set to boost embedded software quality

Even so, many embedded products are released without the necessary and/or desirable level of testing. Consequently, software quality is receiving increasing focus in embedded application development. Most existing software testing tools can only execute the tests on a pc, but the actual system will have numerous differences, including hardware interfaces, timing issues and memory constraints. Therefore, it is important to run as many software tests as possible on the actual embedded hardware to minimise interplatform differences. The challenge is to find automated in-target software testing tools that can be integrated into developers' embedded development environments. These tools should be able to analyse the application source code, autogenerate a test suite and run it automatically on the target board. However, a test report showing 100% success may not be sufficient as the test procedure itself may not have been applied to more than a small fraction of the code. Understanding the quality of test procedures thus becomes critical in judging whether a product is ready for release. Unit tests A key element of the software testing process is 'unit testing'. Unit tests call up C programming functions with different combinations of input parameter values to drive the code through different execution paths. Other than for trivial functions, writing unit tests by hand is laborious, costly and tedious, leaving the possibility that many possible execution paths remain untested. A detailed analysis of how input parameters drive the code is therefore required, so as many of the important execution paths throughout a function as possible are exercised. Another challenge is maintaining the synchronisation of unit tests with code under development. This is often exacerbated when delivery schedules become compressed. If source code and unit tests become out of sync, the tests become less useful – at the time they are needed most. Unit test tools available to pc developers are less useful to embedded developers as they rarely manage compilation, downloading and execution of the test suites on embedded boards. Tools that create unit tests automatically, build them into the embedded application and allow them to run on the target via a convenient pathway – such as a Jtag debugger – will result in optimal usage of time and maximum productivity. An effective approach is to use an embedded test automation system that is integrated into the C/C++ development integrated development environment (IDE), so that synchronisation is easy to maintain and ancillary tools, such as Jtag debug probes, can be used to full effect. This new breed of professional embedded test automation tools can analyse the source code and generate and execute suitable test suites automatically on the target board. An example is Atollic's TrueVERIFIER, which autogenerates a test suite implemented in C and makes many function calls with different combinations of input parameters, thus driving a large number of different execution paths. Figure 1 shows how a trivial C function can be tested, by calling it many times with different input parameters which have been selected and generated automatically by the testing tool. Once the test suite has been generated, it must be compiled, linked and executed on the target system. Many low cost unit test tools exist that only run unit tests on a Windows based pc. The inability of these tools to integrate with the embedded tool development environment results means test cannot be executed on the target board. However, TrueVERIFIER integrates into a professional embedded IDE, enabling synchronisation of test with code development, and takes advantage of integration with tools such as Jtag debug probes. Measuring test quality Once code has been developed and tested, the focus shifts to understanding what actually happened during testing. Code coverage measurement, performed using dynamic execution flow analysis, is commonly used to study which parts of the code have been tested. Some code coverage analysis types are: • Statement or block coverage. This measures how many C statements or code blocks have been executed during a test session, but does not measure how branches affect which C statements or code blocks are executed. • Function coverage. This measures which or how many of the C functions have been called during a test session, but not which, or how many, function calls in a code section are executed or the quality of the testing of the function itself. • Function call coverage. Measures which or how many of the function calls in a code section have been called during a test session. • Branch coverage. Measures whether all code blocks and alternate branch paths have been executed in a code section – such the if and the else part in an if-else statement. Branch coverage typically requires a code section to be executed a sufficient number of times, so that all alternative branch directions will be tested. As all branch paths must be executed, all corresponding code blocks are executed as well. • Modified Condition/Decision Coverage (MC/DC). An advanced type of code coverage analysis which is applied to applications of which the highest reliability is expected. It extends branch coverage with the additional requirement that all sub expressions in complex decisions – such as in a complex if-statement – must drive the branch decision independently of the other sub-expressions. The few test quality measurement tools that have been available have been limited by such problems as: testing using weak types of code coverage analysis only; testing in pc environments and not on the embedded target; expense; difficult to use; and lacking integration. An example of an integrated tool that supports a range of code coverage analysis types is TrueANALYZER. Connecting to the target via a Jtag probe, the embedded application runs directly on the target board with the code coverage information recorded for later inspection and quantitative analysis. Figure 2 shows a trivial code section that contains three blocks: a red code block, which is always executed; a green block, sometimes executed depending on the branch decision made in the if-statement; and a blue block, again, always executed. The branch decision in the if-statement drives the selection of one of the two possible execution paths: either directly from the red to the blue code block; or passing from red to blue via the green code block. Figure 3 shows how TrueANALYZER tool can detect the different types of code coverage, including statement/block coverage, function coverage, function call coverage, branch coverage and MC/DC. TrueVERIFIER and TrueANALYZER are integrated in Atollic TrueSTUDIO C/C++ IDE. These embedded test tools will help software developers to be confident of delivering high quality software and make a significant contribution to new products that work first time. Magnus Unemyr, vp sales and marketing, and Mark Moran, US general manager, are with Atollic.