Consequently, the test strategies of the past are no longer adequate given the high degree of detail to which functions need to be examined and the huge number of different potential scenarios that must be played out.
Because of this, far greater emphasis is now being placed on acquiring test data through simulation and conducting parallelised automated test processes to complement real world testing.
“The test environment is changing and if you look back over the past few years we’ve realised, as an industry, that the move toward autonomous vehicles is far more complex and complicated,” said Ashish Naik, Director of Business Development, ADAS/AD, at National Instruments.
“We’ve also seen the industry shift its focus, to some extent, to electrification. That will provide quicker ‘wins’ for the industry and consumers.
“But despite that a lot of work is being undertaken in terms of autonomy but we need to better understand how to test this technology and to allow consumers to get use to using it.”
According to Naik, that will involve securing sufficient consumer trust and so automobile brands will have to be able to convince the car buying public that their models are completely safe.
“Also, by giving vehicles autonomy, the responsibility for any accidents is placed on to the manufacturer, rather than the driver. If any incident happens where lives are lost, the damage done to the manufacturer’s reputation could be irrevocable – that’s why comprehensive testing is paramount,” added Nicholas Keel, Chief Strategic Business Development Manager - ADAS and Autonomous Vehicle Test, National Instruments.
Another issue is that until now, innovation in the automobile business has always been driven by the vehicle manufacturers and their tier-1 technology partners.
“Vehicle autonomy is set to shake things up,” suggested Keel.
Naik agreed. “You’re seeing a much broader development ecosystem that now includes technology companies and start-ups when it comes to the development of new technologies, and that has huge implications for test and simulation.
“From the testing perspective we are having to address disruptive technologies with new tools. A lot of test companies claim to be able to address all aspects of the testing environment, but engineers need to be able to separate the hype from the reality.
“NI is developing partnerships with companies, like Seagate, who are experts in data recording. Likewise, we are working with Nvidia which has a strong background in simulation. All of which requires a significant mind shift in the way we and the automotive industry work together. In addressing the twin challenges of autonomy and electrification there is a need for partnership and a more practical approach.”
Whether working with recognised passenger vehicle manufacturers, start-up manufacturers or technology multinationals looking to expand their influence, there will be clear distinctions between how these parties’ approach autonomous driving.
The established vehicle manufacturers will have legacy models still in production, so they will be transitioning to driverless cars in incremental phases. In contrast, many of the new entrants are going to be intent on disrupting the market and making an impact by getting ahead of the competition.
Above: A conventional V diagram - outlining the key stages in vehicle system validation -with large amount of rework at the coding, prototype and HIL validation points
“Huge quantities of testing need to be done, so that it can be ascertained that the AI algorithms being used are robust enough - with as many variables being factored in and scenarios examined as possible,” said Keel. “It is not just simply the number of hours (or miles) they have spent on the road, but also exposure to a much larger number of different situations.”
Redefining automobile test procedures
One of the biggest problems for automotive engineers is that they are not able to completely envisage what all the different scenarios might be. Testing must go beyond the scenarios that they are already familiar with, exploring ones that, although incredibly rare, could still feasibly happen.
The dynamics that are now in place mean that for each new vehicle model being developed, the test coverage (in terms of miles driven) is at least two orders of magnitude greater than it was only 6 years ago. “Furthermore, such exponential increases will continue to be witnessed as the automotive industry moves further into the autonomous driving era - and that means major scalability issues,” said Keel.
Consequently, automobile test strategies must be able to fulfil a number of objectives such as: performing a dramatically greater number of different test cases; assuring maximum reliability via the addition of more fault cases to the test scenarios and conditions under examination and a significant lowering of the costs that are associated with testing activities (to compensate for the rising volumes involved), while keeping the time needed for this testing as short as possible so that time-to market expectations can still be met.
Testing on the road was always the foundation of automobile model development but the complexity of modern system design, with so many different functions being integrated, now makes the process too expensive and time-consuming.
Though road tests will always have a vital part to play and are in fact growing in quantity compared to previous decades, the extensive testing being done within the virtual domain is growing far faster - and dominating overall test coverage.
“It will be via this that the sheer scale of testing required may be attended to and the majority of issues uncovered early on,” suggested Keel.
This alteration to how testing is approached calls for a skewing of the traditional V diagram arrangement that engineers normally refer to during product validation. Through such skewing, a more acute focus is going to be placed on the use of virtual prototypes, rather than physical ones, with massive amounts of software-in-the-loop (SIL) testing needing to be carried out on multiple test rigs running in parallel.
“The industry needs a set of common tools and platforms as more data is used and computational capacity increases,” said Naik. “It will take time to change minds as we look to share data and help companies improve the quality of testing. It’s a more joined up approach that will help companies learn, educate and connect with one another and in the process share what does and doesn’t work. At present no one knows what constitutes ‘best practice’.”
According to Naik it’s an “exciting time” and, “different skills sets are required when it comes to test, simulation and data. I think more companies are planning for long term success and rather than cutting corners are looking to build greater transparency and trust.”
As more advanced systems are added to vehicles, so data volumes are going to grow even further.
Once a large body of simulation-based testing data has been amassed, from sensors and the like, it will then be complemented by the acquisition of a smaller proportion of hardware-in-the-loop (HIL) test data. But that means there will then be an even smaller proportion of actual road-based testing done. To give some idea of what the test breakdown is likely to be for autonomous vehicle models now starting to be developed, estimates suggest that around 1 million hours of road testing will be embarked upon once at least 1 billion hours of virtual test work has already been completed.
“Any virtual testing conducted will only be as good as the quality of the simulation data it is based on. This is why it is still important to have HIL and road testing, so that real world dynamics can be brought in,” explained Keel.
Through early road-testing work, large repositories of data may be created and used to complement simulation data that will be of higher fidelity than is possible within the purely virtual domain. Many miles of road testing will get clocked up, and then digitised. From these digitised miles, reusable real-world scenarios for actioning in SIL and HIL can subsequently be constructed.
It is clear that the size of the test problem that autonomous driving presents is unlike anything the automotive industry has had to deal with before so the way that testing is performed needs to be entirely reassessed.
Currently, vehicle validation workflows are comprised of many different test elements, which are each isolated from one another. This significantly limits their efficiency, as well as making them difficult to scale. By connecting each of these test elements and their data together, it will be possible to implement a better optimised workflow that increases test coverage and shortens development schedules.
“NI has clear areas of strength – data recording, hardware in the loop, data validation – but for effective simulation we will need a more open approach and we need to ensure that any investments or acquisitions that are made can be connected to others so that tools are better aligned,” said Naik.
Above: An example of the V diagrams that are now going to be followed - with SIL validation becoming a vital step and reducing the rework activity
According to Naik an open and adaptable system architecture will be vital in supporting the automotive sector’s transition into full vehicle autonomy.
It will need to accommodate an extensive array of different interfacing options and feature a convenient modular construction methodology and through its use, manufacturers will have access to a streamlined, connected workflow that will deliver comprehensive testing of their latest vehicle models - rather than the disjointed arrangements that they had before (made up of a multitude of disparate pieces). They will also have the ability to adapt their testing operations as the standards that must be conformed to change over time.
“NI is also serving in a consultative role. On one side, our teams are talking to the established car manufacturers, mobility providers and the tier 1 systems integrators about their plans for autonomous driving implementation and advising them accordingly. Simultaneously we are conversing with the tier 2 suppliers to keep fully aware of their product roadmaps, in terms of the supporting imaging devices, the graphics processing units and the networking technology that will be introduced in the years ahead,” explained Keel. “This will mean that the nature of testing will be able to evolve as the automotive industry moves closer to level 5 vehicle autonomy, with all the stakeholders involved being totally prepared for it.”
NI is working with the likes of Seagate on next generation data storage and Konrad for state-of-the-art compression algorithms that will shrink the amount of data needing to be stored.
Today it is no longer possible to carry out testing in just real-world conditions and while still important, it must be supplemented by other activities that can cope with the staggering volume of tests involved, while adhering to both challenging time and budgetary constraints.
To make testing more cost-effective and to keep this work within acceptable timeframes, greater reliance on automation is now being mandated. High-volume parallelised testing will be essential - as it can be done much quicker and at a lower total cost than conventional physical testing.
It also provides the option to scale up test capabilities to meet heightened demands. Via automated testing, engineers can gain valuable insight into every aspect of the vehicle’s operation, with component parts and sub-systems being more accurately characterised and any potential issues being much easier to identify.
