Software solutions are increasingly central to safety-critical and safety-related systems, meaning software malfunctions now present a real threat of injury, loss of life, essential service interruption, or environmental damage.
Consequently, there are widely recognised standards that exist, against which software developers can certify software safety. For example, in the automotive sector, these include ISO 26262 (Road vehicles – Functional safety).
The application developer must demonstrate that application software, and the methods, processes, and tool chains used to develop it, comply with the relevant functional safety standards. However, significant parts of the tool chain lie outside the developer’s control. Compiler validation has become key for safety-critical system developers. Virtually no compiler is bug-free, making it vital to know where a compiler malfunctions so errors can be avoided.
Much of the source code that typically ends up as a compiled binary in an application never goes through the compiler under exactly the same use-case, set of compiler options, and target hardware environment being used by the developer. Part of the code that typically ends up in an application comprises pre-compiled library functions, such as those in the C Standard Library (libc) often supplied in binary format as part of a software development kit (SDK).
It is widely believed that because a library is supplied in binary format, it is insensitive to any particular use-case, but this is not the case. The library’s header files are compiled with the application and contain many definitions including function-like macros and, in C++, templates. They make library components use-case sensitive. Even if the library was pre-qualified by the SDK supplier using the same compiler delivered with the SDK, the matching use-case, compiler options, and target hardware environment requirements are unlikely to have been met, making it difficult to demonstrate functional safety standard compliance.
Full traceability
A requirements-based test suite with full traceability from individual test results back to requirements derived from the ISO C language specification – like the SuperGuard C Library Safety Qualification Suite – can help overcome this limitation.
SuperGuard can support qualification of C Standard Library implementations for safety-critical applications both for unmodified third-party library implementations and self-developed or self-maintained implementations.
Software library qualification is critical because code from the library is linked into the application and installed onto the target device. If a library component is defective, the entire application’s functional safety is jeopardized.
Functional safety standards generally share common objectives: to verify that the library implementation complies to its specification. ISO 26262 provides two library qualification routes, detailed in Part 8 and Part 6. SuperGuard can be used in both.
Qualification of software components
In terms of being characterized as a commercial off-the-shelf product, libraries are covered in Part 8, Clause 12: ‘Qualification of Software Components’. This addresses the need to qualify existing software components such as libraries “to provide evidence for their suitability for re-use”. It specifically mentions “software libraries from third-party suppliers” but also applies to open-source and re-used internal software.
Clause 12 states a qualification pre-requisite is a statement of the software component requirements. It also suggests that evidence that a software component complies with these requirements should “primarily be based on requirements-based tests”, that this can be achieved through the “application of a dedicated qualification test suite”, that it should “cover both normal operating conditions and behaviour in the case of failure”, and that it should “display no known errors that may lead to a violation of safety requirements”.
Fortunately, for both C and C++, the library specification is publicly accessible, providing the starting point for rigorous requirements-based testing. However, the language specification is not written as a list of requirements. All SuperGuard’s library tests are firmly based on requirements derived from the ISO C language definition.
To meet the requirement that the test suite should cover both normal and failure conditions, it should exercise each function both within and outside its boundary conditions and verify the function’s error handling. The requirements derived from the ISO C specification to create SuperGuard include verification of the required failure behaviour defined in the specification.
Reliable evidence of there being no known errors that could lead to a violation of safety requirements not only requires effective requirements-based testing. It also relies on structural code coverage for Automotive Safety Integrity Level D (ASIL D). To ensure evidence of completeness, SuperGuard provides high structural code coverage and high Modified Condition/Decision Coverage (MC/DC).
The SuperGuard C Library Safety Qualification Suite also includes analysis and testing of equivalence classes and boundary values, and error guessing based on the best available knowledge and experience of a library function’s behaviour.
Product development at the software level
Qualification of the C Standard Library as detailed in Part 6 is more involved than the approach in Part 8 as Part 6 addresses all software development phases.
Part 6, Table 7: ‘Methods for software unit verification’ includes requirements-based testing as a recommendation for all ASIL levels. SuperGuard comprehensively supports requirements-based testing, although in practice developers should use more than one of the methods listed to verify a C Standard Library implementation.
SuperGuard also employs all the methods listed in Part 6, Table 8: ‘Methods for deriving test cases for software unit testing’. Method 1a, ‘Analysis of requirements’ breaks down the C Standard Library specification into testable requirements. The C Standard Library description is a mix of (sometimes implicit) definitions, restrictions on the use of functions, and definitions of function behaviour.
In SuperGuard, these have been translated into testable requirements forming the basis of the test suite, including requirements for handling anomalous cases defined in the language standard.
Above: An example of a SuperGuard report
Methods 1b, ‘Generation and analysis of equivalence classes’, and 1c, ‘Analysis of boundary values’, relate to partitioning the input value domains of the functions and are addressed in SuperGuard's test specifications, providing the link between requirements and tests.
Method 1d, ‘Error guessing based on knowledge or experience’ is based on Solid Sands' knowledge of difficult implementation areas for the C library, and on the regression tests collected since the initial development of Solid Sands’ SuperTest compiler test and verification suite more than 30 years ago.
In this approach, SuperGuard plays a substantial role on the right-hand side of the V-Model for software development. In relation to Part 6, Clause 9: ‘Software unit verification’, it addresses:
- Compliance of the standard C library implementation to its requirements
- Verification of the hardware-software interface by running the tests on the target hardware
- Confidence in the absence of unintended functionalities by failure case verification and by monitoring code coverage
- Verification of resource requirements by running the tests on the target hardware
By generating a comprehensive qualification report tailored to ISO 26262 certification organisations, the SuperGuard C Library Safety Qualification Suite helps to alleviate much of the burden of demonstrating the integrity of library components used in safety-critical applications.
Author details: Marcel Beemster, CTO, Solid Sands