Testing times for engineers

Historically, 'testbench generator' and 'testbench automation' tools have fallen into one of three categories: scripts which generate a skeleton HDL file, which has to be filled in by the user; tools which allow the user to create waveforms on device inputs; and programming languages, such as e and SystemVerilog. These programming languages provide constructs which can be used to create specific test input sequences and to define and record functional coverage metrics. But none of these tools provides any significant assistance in creating self checking, reactive testbenches. Maia is a new class of tool which automates these time consuming and complex tasks. Given a description of the required inputs to a device and the expected outputs, a Maia compiler outputs a complete self checking testbench for that device. Here are some curious facts: less than 1% of engineers actually prefer writing testbenches to writing RTL code; less than 10% of engineers actually like writing testbenches, because it can take three days to write a self checking testbench; almost 50% of engineers never write a self checking testbench; and less than 20% of engineers have the inclination – or the ability – to learn and use high level 'verification languages'. OK, at this point I'll come clean. I found the first three numbers on Google in about five minutes (they're quotes from well respected industry figures). I made up the fourth number, although I suspect that not too many people would disagree with me. The fifth number, however, is important, because I bought my house with it. Back in 2000, I took on a job with a well known IP vendor, writing Specman/e and Verilog. This vendor, which had been using e since 1998, had a policy that engineers should verify their own work with e, before handing the assembled device over to the verification team. Over time, however, it became obvious that many of the (very smart) engineers in this company either wouldn't, or couldn't, use e effectively and that external help was required. This was despite the fact that e is simple, well designed and (for a programmer, at least) easy to learn. With hindsight, this shouldn't have come as a surprise. If your brain is wired to excel at designing translation look aside buffers or Viterbi decoders, then it's unlikely to also excel at coverage closure, aspect orientation, class inheritance or the myriad other skills required for successful verification. Design and verification are different skills. Engineers are not programmers; that's why we have verification departments. There is clearly a problem here that needs addressing, since most engineers don't have access to verification specialists and must, therefore, verify their own work. There are cases where a simple fpga can be built with little or no verification, then debugged and corrected in system. In most cases, however, you'll have to bite the bullet and write a testbench. Given this is the case, let's look at what exactly is required to write a self-checking testbench manually and what parts of this can be eliminated by a smart automation tool: * You will need to handle datasets for the device under test (DUT) inputs and the expected outputs. In both cases, the required data might be hard coded directly into your testbench, generated algorithmically or read from a file * You will need to drive the inputs and write checkers for the outputs. You may also need to confirm that all inputs and outputs meet their timing specifications * If your device is pipelined, your checkers will also need to be pipelined. You'll need to load and flush the checker pipeline and only check outputs after the required pipeline delay * If your device has an unknown or variable latency, you'll need to add additional threads to allow your checkers to wait for, and respond to, the outputs at arbitrary times * You may need to add code to start checking specific outputs only when a given trigger condition is detected * You may need to drive or check internal nodes in the design, using force/release semantics * You will need to manage a potentially large number of threads, ensuring that they communicate correctly * You will need to create clock and reset generators, advance time appropriately, handle bidirectional signals at the DUT I/Os, maintain pass/fail statistics and report errors. It may now be obvious why most engineers don't write self checking testbenches. Testbenches are, by and large, complex pieces of error prone software engineering and writing a good one will generally take longer than creating the design itself. However, some reflection shows that none of these steps actually adds any value to the design process. This is grunt work – work for the sake of work. It has to be repeated for everything you ever design, with minor variations for each project. In fact, the only steps that add any value, and which require any 'intelligence' are the only steps not listed above: the definition of the input sequences which will exercise your design and the outputs which are expected from it. Maia was designed with the explicit purpose of automating every step in this list. It was designed to be used by engineers, but not nby programmers. To use it successfully, you only need to be able to define the required input sequences to the device and the expected outputs. A Maia program takes on one of two basic forms. At its simplest level, a program is just a declaration of the DUT, together with a list of test vectors. While this form of test may be suitable for testing simple modules, it has a number of limitations. The test vectors are executed in a linear sequence, with no control flow constructs, and are limited to driving and testing explicitly coded constant values. In its second form, a Maia program has the look and feel of C code, albeit with a number of extensions and simplifications (although Maia will compile many simple C programs without modification). The Maia compiler (mtv) produces an HDL testbench, so a simulator is still required to run the testbench. This process is automated by the compiler driver, rtv. If the testbench is in file test.tv and the Verilog MAC code is in file test.v, then the entire test can be run in a single step as follows: > rtv test.tv test.v 257 vectors executed; 257 passes, 0 fails If the Verilog MAC code had contained errors, the testbench output would have reported the times at which the errors occurred and the corresponding line numbers in test.tv, together with the actual and expected values of any incorrect outputs. Maia, which was released at the end of 2009, is currently available as a free download from www.maia-eda.net and this release will continue to be freely available. However, the Q1 2010 release will be available on an annual subscription basis and this is expected to be less than £350.