AMD has worked closely with ESnet since 2018 to integrate powerful adaptive computing for the smart and programmable network nodes of ESnet6. ESnet6’s extreme scale packet monitoring system uses AMD Alveo U280 FPGA-based network-attached accelerator cards at the core network switching nodes to enable high-touch packet processing and help improve the accuracy of network monitoring and management. The programmable hardware allows for new capabilities to be added for continuous innovation in the network.
In order to customise AMD Alveo U280 2x100Gb/s accelerators as network interface cards (NIC) for ESnet6, the OpenNIC overlay – developed by AMD – was used to provide standard network-interface and host-attachment hardware, allowing experimental networking functions to be implemented on the Alveo card.
OpenNIC has since been open sourced after being successfully used and developed by ESnet, as well as various other academic research groups. Also important for rapid innovation by non-hardware experts was the use of the AMD VitisNetP4 development tools for compiling the P4 packet processing language to FPGA hardware.
AMD Alveo U280 cards, using OpenNIC and VitisNetP4, are being deployed on every node of the ESnet6 network. The high-touch approach based on FPGA-accelerated processing allows every packet in the ESnet6 network to be monitored at extremely high transfer rates to enable deep insights into the behaviour of the network, as well as helping to rapidly detect and correct any problems and hot spots in the network.
The Alveo U280 card with OpenNIC platform also supplies the adaptability to allow the continuous roll-out of new capabilities to the end user community as needs evolve over the lifetime of ESnet6.
The benefits of adaptive computing at demanding network line rates are also applied to the edges of the ESnet6 network. Globally distributed scientific instruments are the primary sources of the ‘big data” science flows that ESnet was designed to support. The Alveo card with OpenNIC can be used for customised data filtering and shaping before transmission over ESnet.
For supercomputing facilities, which are among the destinations for these large data sets, the Alveo card with OpenNIC can be used for load balancing across compute and storage servers.
“AMD is honoured to provide ESnet with leading-edge technology and expertise needed to help bring the new network into service,” said Ivo Bolsens, senior vice president and CTO of Adaptive and Embedded Computing Group, at AMD. “AMD FPGA-based adaptive networking capabilities make up a core building block of the ESnet6 extreme-scale packet processing system and provide the underpinning for the DOE’s future vision of distributed e-science.”
Headquartered at Lawrence Berkeley National Laboratory (Berkeley Lab), ESnet serves as the “data circulatory system” for the DOE, connecting all of its national laboratories, and tens of thousands of DOE-funded researchers. This includes DOE’s premier scientific instruments and supercomputer centres – including the world’s first exascale system, the Frontier supercomputer.
“ESnet6 is built to support the DOE’s multi-billion-dollar investment in scientific research that leads to breakthroughs that impact our everyday lives. It represents a transformational change in the way networks are built for scientific research,” explained Inder Monga, executive director of ESnet.
With over 46 Terabits per second of bandwidth, ESnet6 features a significant increase in bandwidth over prior generations to support unique needs of scientific research. With this boost in capacity, along with new service capabilities, scientists can more quickly process, analyse, visualize, share, and store the mountains of research data produced by experiments, modelling and simulations. Data rapidly moves between the interconnected sites and collaborators, accelerating time to discovery.
