Smart metering needs smarter test

New equipment, technologies and software are being developed for almost every part of the infrastructure. Such large scale change presents major challenges to the companies involved and the financial and reputational impact of failure could be huge. Not only does the transition need to be managed, but extensive testing is also required to ensure reliability and continuity in supply and billing. Much of the testing needed is 'hands on', with dedicated test labs that mirror the domestic metering environment and large scale testing performed with simulators and test harnesses. Developing a test strategy In the multivendor, multitier, high volume world of smart metering infrastructure, a robust test strategy is essential. The test strategy should give a clear view on the approach to testing, including: • What the tests will cover at a high level • Mapping of the types of testing used to phases • Detailed entry and exit criteria between test phases • How risks and issues will be managed and the defect lifecycle to be used • Clearly defined testing roles and responsibilities The test strategy needs to be tailored to the needs of the project and signed off by stakeholders. If there are third party vendors testing ahead of a combined system test phase, it is important their test plans are understood and suitable checks (entry criteria) are in place to ensure acceptable quality ahead of a joint phase. Since much of the hardware and software in smart metering networks is new, hardware problems are often more difficult and can take longer to resolve than software defects. Test planning needs to account for the difference between software and hardware development lifecycles, which is why testing should be carried out early in the lifecycle. Requirements validation and document reviews can uncover design defects long before units go into production and the incremental cost of a requirements or document review is negligible when considered against the cost implications of fixing design defects after hardware has gone into production. When a vendor is late with a delivery or testing time is squeezed, test activity must be prioritised and the basis for prioritisation should be defined in the test strategy. Typically, a risk based approach is adopted: high risk areas are tested first, while lower risk tests might be descoped to save time. Whenever such descoping is undertaken, it must be carefully managed with business and technical stakeholders fully aware of the risks of the approach. Managing system testing While each component of the smart metering infrastructure must be tested, one of the most complex aspects of the quality assurance process is bringing the elements together for system testing. As the number of vendors and integration touch points increases, so does system complexity and the need for robust test management and coordination of suppliers, internal teams and customers. Regular communication is vital and, as testing progresses, daily defect calls with all suppliers help to ensure that all parties understand the issues and can disseminate information to diagnose and resolve issues quickly. This sharing also helps to reduce defects being passed from one supplier to another. When working in multivendor environments, SQS typically maintains requirements, test cases and test results in a shared test management software repository; ensuring a 'single source of truth' for the status of testing, defects and reporting. All reporting and defect management is driven from the repository, so all stakeholders have the same view of the project and testing status. Testing, tools and environments With 30million smart meters expected to be installed by 2019, the scale of smart metering networks is staggering. Alongside traditional test management systems for unit, system, integration and acceptance testing, specialist test environments and tools are required to provide sufficient test coverage of such large systems. Performance testing requires test harnesses and simulators that can handle very large data volumes. Smart Meters typically take half hourly meter readings – more than 4000 times the number taken from a traditional meter. With more than 30m households on the network, large scale test tools are required to show that back end billing and analysis systems can collect, store and process the huge volume of data. Scheduling usage of and managing testing environments is critical. Test environment builds must be repeatable and configuration definitions fully documented. For example, each device on the consumer's premises has firmware supported by a specific version of software on the head end. Device firmware and head end software need to align to ensure that testing is performed on vendor supported configurations; this is particularly important when carrying out end to end testing of the entire solution or integration testing of two or more components. It is important to ensure that all components are configured correctly ahead of testing starting and any new build should be 'smoke tested' with a subset of high priority tests to establish whether there are any fundamental issues ahead of full integration testing. Attention to detail is paramount; inaccurate or incomplete information can lead to misconfigurations and delay. Purpose built smart metering test labs can replicate the consumer environment, allowing realistic and representative tests to be carried out safely and efficiently. In addition to their use in certification tests, using real meters in the lab increases confidence in the test results, albeit at the cost of introducing a schedule dependency upon hardware and software availability to support the meters. Testing in the lab does have its limits and so must be backed up with field trials. Testing in the field increases confidence that systems will work in real world environments, introducing variables such as: meter position; low signal coverage; age of the property; household wiring; improperly sized meter enclosures; and wireless network connectivity problems. Meeting the system testing challenge Delivering the new smart meter infrastructure requires a coordinated approach from customers and suppliers of hardware, software and infrastructure. When testing such a complex system, long lead times for environments and equipment can easily push projects off track or lead to some aspects of testing being descoped. To keep quality high and minimise the risk of costly failures, test planning should be considered as early as possible. Having a test manager or tester involved in the review process alongside other stakeholders when design documents are being drafted helps issues to be identified earlier. Invariably, defects and issues found in design documents are far easier and cheaper to resolve in development and test rather than production, when millions of consumers are depending on the new smart metering infrastructure for their energy supply. Jack Coxeter is smart metering test manager, and William Leacock is smart metering test lead, with SQS UK.