The increasing complexity of embedded systems is challenging traditional development and testing methods. This trend is exemplified by Volvo’s new electric flagship model EX90, which features advanced technologies like LiDAR sensors, NVIDIA’s computing system and Qualcomm’s ‘Snapdragon Cockpit Platform’, all integrated with lots of inhouse-developed software. Originally announced in 2022, production delays pushed deliveries to early September 2024, with the first cars shipping without working LiDAR sensors. These delays, at least partly due to the increased system complexity, have cost billions and highlight the challenges in software development and verification for modern embedded systems.
To address these challenges, the concept of Continuous Observability has emerged. Observability in software systems refers to the ability to measure a system’s internal states by examining its outputs, such as logs, system traces and memory dumps. Traditionally, observability has been reactive, with developers using debugging tools only when problems are detected. However, this approach is insufficient for addressing sporadic issues or problems that occur outside the development lab.
Continuous Observability takes a proactive approach, enabling data collection by default and automating reporting. This ensures that diagnostic data is always available for analysis when issues arise. The approach spans the entire product lifecycle, from early software development to deployment in the field. By combining automated monitoring with deep observability, hidden bugs, performance issues and system anomalies can be detected earlier, speeding up issue resolution and improving software quality.
The growing complexity of embedded systems is a key driver for adopting Continuous Observability. Modern embedded systems often integrate various software functions through multithreading and asynchronous events, leading to unpredictable variations in the software execution. This complexity makes verification and debugging more challenging, and testing every possible scenario becomes unrealistic. Even after deployment, embedded software typically contains about three bugs per 1000 lines of code.
Edge connectivity further increases system complexity while also exposing systems to cyber threats. However, the increasing connectivity also offers a potential remedy through over-the-air (OTA) updates and remote observability. The combination of observability and OTA updates is transforming how developers address the challenges of software complexity.
Case in Point – SDVs
The rise of Software-Defined Vehicles (SDVs) exemplifies the transformation in embedded systems development. SDVs reimagine automotive architecture by placing software at the core of vehicle functionality. Unlike traditional vehicles with self-contained electronic modules, SDVs use centralized high-performance computing, partitioning functions into software layers. This approach enables OTA software updates, dynamic feature deployment and constant connectivity with external systems.
While SDVs offer undeniable benefits such as rapid innovation and reduced hardware complexity, they also introduce new challenges. Frequent updates, diverse configurations and continual software evolution create an environment where it’s nearly impossible to fully test every potential scenario that a vehicle might encounter over its lifecycle. This clash between traditional, hardware-focused approaches and the agile, software-defined future of SDVs places immense pressure on developers to balance safety, performance and innovation.
In this context, Continuous Observability becomes essential. It provides constant insights into system behaviour, helping developers ensure that new software updates, feature rollouts, and system configurations don’t compromise core functionalities like safety or performance. The SDV trend extends beyond the automotive industry, influencing sectors like aerospace, industrial automation, and medtech, making Continuous Observability a necessity across various industries.
Observability-driven development
Observability-Driven Development (ODD) is emerging as a key strategy to manage the growing complexity of embedded systems, and it aligns seamlessly with Percepio’s Continuous Observability concept. ODD integrates observability into every stage of the development cycle, providing real-time visibility into system behaviour from early software development through to post-deployment, using software tracing as a key utility. It can be extended with advanced monitoring of software performance metrics, making systems “self-aware” with respect to abnormal behaviour and potential failure risks in the runtime software.
By continuously monitoring system performance, ODD helps teams stay agile, even as software evolves post-launch. In sectors with high costs of failure ODD offers an invaluable layer of security, preventing costly recalls, downtime or performance degradation. As system complexity grows, ODD ensures that developers have the insights they need to maintain control over their products, regardless of how often software changes occur.
Continuous observability
The consequences of failing to manage the complexity of modern embedded systems can be profound, especially in industries where system reliability is non-negotiable. Among several high-profile examples is a leading automotive manufacturer having to recall over a million vehicles delivered between 2020 and 2022 due to a software and sensors combination system error in the vehicle’s Occupant Classification System (OCS), potentially affecting airbag triggering. The issue was not caught during testing, resulting in significant financial costs and reputational damage. In 2024 the discovery of the Linux/OpenSSL XZ backdoor, a meticulously planned supply chain attack, highlighted a new and enlarged attack surface for cyber threats – compromised open-source software components. This injected malicious code via the build system without changing the source code and was detected by chance through runtime measurements showing discrepancies in software execution times.
These examples demonstrate that traditional testing and debugging methods are no longer sufficient to manage the complexity of today’s interconnected, software-driven devices. Software issues in critical systems have tangible, often high-cost consequences. The inability to foresee every possible combination of conditions in which a system might operate makes it virtually impossible to test every edge case before launch.
This is where Continuous Observability and the adoption of ODD come into play. By implementing ODD, developers can identify and mitigate issues much earlier, before they manifest into large-scale recalls or delays. With Continuous Observability, they gain ongoing insights into how their systems behave in real-world environments, helping them detect anomalies, optimise performance and ensure the safety and reliability of their products - even post-deployment.
In industries such as automotive, medtech and aerospace, where the cost of failure is exceptionally high, the benefits of ODD and Continuous Observability are crucial. Whether it’s preventing a life-threatening medical device malfunction or avoiding a large-scale automotive recall, the ability to continuously monitor, analyse and improve software performance across a product’s lifecycle is becoming an indispensable part of modern development strategies.
The path forward
ODD requires a “shift left” approach, where observability is designed-in early to support all following stages of product development and maintenance. This is not only a concern for software QA and test management but a key factor in the performance and competitiveness of product development organizations.
As the complexity of embedded systems continues to evolve and we move further into an era of software-defined everything, the question is no longer if developers should adopt Continuous Observability and ODD practices, but when and how.
With Tracealyzer for development, Detect for system testing, and DevAlert for deployment, Percepio offers a comprehensive Continuous Observability solution that spans the entire product lifecycle. These tools provide real-time visibility into complex embedded software behaviour, enabling developers to accelerate debugging, improve product quality and reduce deployment risks.
By scaling from small IoT nodes to powerful multicore SoCs, this solution exemplifies how continuous observability can be implemented across various embedded systems, addressing the growing complexity challenges.
Author details: Andreas Lifvendahl, CEO, Percepio
