But today you do not need it to be in the same room or even the same country for that to happen.
In an environment where people working on the same project could not just be in different offices but working halfway across the country at home or in a completely different country, getting hardware into the hands of developers is easier said than done. The recent post-Covid supply-chain crunch exacerbated the problem as even mass-produced embedded boards popular for prototyping such as the Raspberry Pi became tricky to source at time.
This is where the board or device farm comes in: real hardware but sitting in a lab somewhere else. Device farms quickly became a popular choice for developers working on apps for smart mobile devices that you could log into using Amazon Web Services and similar offerings. One of the big attractions of this model was that it provided access to several different devices so the developers could see how changes in screen size and processor would affect software behaviour. This is now moving into the virtual domain with services from vendors such as Corellium. Now that Arm servers are common in the cloud, the multicore Cortex-A processors on those blades can often run the app code as fast as that on a dedicated processor for the target handset.
Virtualised processors in servers can substitute for real embedded hardware to an extent, particularly for running unit tests, though there is a significant overhead when it comes to building accurate models of I/O and peripherals, which points to the device farm having a longer shelf life in development cycles.
Typically, the device farm supports two types of use-case: interactive debug; and automated testing as part of a continuous-integration flow. Because the boards can be controlled remotely, it makes sense to have them turned over to automated testing when engineers do not need to work interactively. Software like Jenkins can schedule many tests and compile reports for post-analysis. Tools such as Github’s bisect can work out which altered modules caused a failure before the next cycle of changes.
To support this kind of automated regression testing, companies such as Cambridge-based consultancy Collabora have set up their own racks of embedded devices. According to consultant software engineer Laura Nao, in her description of the farm at the FOSDEM open-source conference in February, many of the 200-plus devices in the racks are Google Chromebooks used for testing application code, but a significant minority are embedded boards. Because they are used primarily for continuous-integration testing, the company uses the Lava toolkit developed by Arm-backed open-source group Linaro.
However, as it was developed primarily for batched tests, Lava is not a good fit for interactive. To support interactive debugging on Linux-based boards, various teams have developed their own collections of drivers and console routines. But there are off-the-shelf options, such as Labgrid developed by German company Pengutronix. This was written in Python initially for its own 19in racks of embedded boards and devices, but which has become a popular readymade option for interactive debugging and control of board farms.
Practical issues
Though the software infrastructure is improving, there are numerous practical issues around running device farms. A birds-of-a-feather session organised by Pengutronix at last autumn’s Embedded Linux Conference Europe, held in Dublin, underlined how some quite low-level issues, such as power delivery and reliability issues with USB and memory cards, can cause problems. There is also the question of how to handle boards that are not intended to run Linux as many of the software tools are designed to take advantage of that operating system.
The question of whether you can turn it off and on again has turned out to be one of the fundamental issues. And farm operators are working to come up with several ways to solve that problem. A secondary issue comes down to the diversity of power supply options for embedded boards. Though it would often be cleaner to have a network-controlled power-distribution unit providing the same 5V DC to every board in a rack, that is probably not a realistic option though the gradual rise in power over Ethernet and USB-C may eventually solve that problem. In the meantime, engineers building custom setups have in some cases turned to home-automation devices such as Sonoff’s Wifi-controlled power sockets to control the supply to the various wall-wart supplies the boards need.
USB also often needs the ability to be turned off and on physically when things go wrong, again using some kind of relay to control it. Another less obvious requirement surrounds the non-volatile storage need by each board. Very often the boot image comes from a USB thumb drive or an SD card, which somehow needs to be switched out and updated without physical contact.
One answer is to rely on firmware loaded onto the board to manage over-the-air updates, which themselves are becoming more common, though this takes longer to set up. The other option is to use a multiplexing interface that resembles the debug cable used on emulators. The cable plugs into the SD card slot and presents a built-in SC card to the target but also adds the ability to program the flash from over the network.
Some suppliers have rolled up these functions into products aimed at managing device farms. Timesys has developed two hardware units to manage device farms. The first is the Zombie, which coordinates four devices and provides regular USB and Ethernet connections to the boards being tested. For the situations where power control and SD card control is needed, a second unit, the IO-CX plugs into the Zombie. This provides an SD-multiplexing cable, and power-switchable ethernet and USB ports as well as programmable general-purpose I/O pins to provide external stimulus to the embedded device.
Segger’s model for building a similar setup is to use a network switch with support for power-over-Ethernet connected to the company’s JTAG debug probe hardware. Last month the company launched an optional cable and header that gives the J-Link probe control over the power supplied to the target board through a USB 2.0 port.
In its farm, Collabora performs active health monitoring, in combination with the reporting features in Lava to spot potential problems. For example, if a test fails, it is rarely immediately clear whether it is an intermittent issue caused by a software conflict or caused by a looming hardware problem, such as an onboard backup battery failing or a cable having worked loose because it was disturbed when a nearby device was replaced.
Managing complexity
The complexity of managing device farms has seen some service providers move into the embedded space and some of the device makers build their own farms to act as remote evaluation facilities. STMicroelectronics, for example, has built a device farm to offer remote evaluation and development access to boards based on its Arm-based microcontrollers.
Using All-Hardware’s managed farm, you can have your own boards hosted and managed remotely or share access to popular third-party development boards, such as those from ST. The company designed its own managed racks that include a webcam over any board with onboard displays or indicators so that engineers working remotely can see through a web interface how it responds.
Though a comparatively recent addition to the armoury of development tools for embedded projects, device farms are gradually growing an infrastructure of DIY and commercial offerings.
Though improvements in simulation technology may restrict how many of the boards a team needs to have hosted, remote hardware is likely become a mainstay of development.
