Facebook's new data centre is home to numerous innovations

At the top end of the scale, data centres are huge operations; large buildings crammed with unbelievable numbers of servers and vast amounts of data storage. And they consume huge amounts of power; not only to drive the servers, but also to remove the heat they generate. Nobody knows for sure how many data centres there are around the world. But an educated guess by those 'in the know' suggests that data centres consume between 2 and 3% of all electricity generated in the US. It's no surprise that energy efficiency is rising very quickly to the top of the agenda. Social networking pioneer Facebook is leading efforts to address data centre energy consumption with the launch of the Open Compute project - www.opencompute.org . While the goal of the project is to develop an open forum to drive data centre efficiency, the initiative has grown out of Facebook's desire to be as green as it can. Until recently, Facebook's website has been hosted in a number of leased facilities. But it has just opened its own data centre in Oregon, which it has designed from a 'clean sheet of paper'. Open Compute is the result of two year's development work by a team at Facebook's Palo Alto headquarters. Hardware engineering manager on the team is Amir Michael. He said his main focus has been to drive the Open Compute effort. "My job has been to reduce the overall infrastructure cost. We looked at a variety of ideas, but all revolved around the data centre – and servers in particular." In fact, the project had a wide remit: everything from the power grid to gates on the processors was up for examination. According to Michael, the best optimisations could only be achieved by modifying the way the data centre and the servers work together. "The 'big win' comes when you modify both," he said. "We had two teams working on the project. One was designing a custom data centre, the other was designing custom servers." And he said there were a number of overlaps between the two teams. Along with innovative server design, Michael believes there have also been 'big wins' in the way the servers are powered and in how they are cooled. "The environment within which the servers operate has a big impact," he asserted. The two year project split nicely into a year of research and a year of execution. "In the first year," Michael noted, "we began to understand what the design would look like. Getting the whole project done in two years is quick for this industry and we went from breaking ground at the Oregon site to powering up the servers in about a year." Facebook is currently hosted at four sites: along with its Oregon data centre, leased facilities are used in California, Virginia and North Carolina. A project is now underway to construct another Facebook data centre in North Carolina. An indication of the scale of these operations is the fact that Facebook can host all traffic from its West Coast and Asian users at the Californian facility. While the design of the servers is innovative, the processors that feature are not. For the moment, Facebook has specified what Michael calls 'same source processors' for the servers. "We started out with devices from Intel and AMD and each has its own strengths and weaknesses." Intel processors are used for front end tasks, while AMD processors are associated with memory operations. "Our operations require a lot of memory," Michael noted, "and the AMD parts have an extra memory channel." Michael accepted that Intel and AMD were 'safe choices'. "We were familiar with the parts and we can bet on them to have competitive products when we want to evaluate new devices." Having said that, he noted that new products were reviewed constantly. Along with ARM based devices and those from Tilera (see NE, 13 July 2011), Facebook is also evaluating Atom based devices and processors aimed at desktop applications. "Whatever we can get our hands on to see if it makes sense," Michael observed. "We see how they perform and plug the results into a spreadsheet to provide a definitive view." The team developed a tray based design for the servers at Oregon. Michael said the design was driven not only by the quest for greater power efficiency, but also by the need for easier maintainability. "Many data centre technicians spend their days swapping things in and out. We wanted to make their job as easy as possible." A time and motion study on the new design showed a factor of ten improvement. "Every hour a server is out of commission costs money," he noted, "and our aim was that less than 1% of servers would be offline at any time." Not only are the trays easier to remove, they have fewer parts and weigh 3kg less than a 6U server. Trays are then grouped into what are called triplets, racking systems designed specially for the job. "We looked at 19in racks," Michael noted, "but decided to simplify them while keeping the form factor." Previously, there were 40 servers per rack. Now, the triplet scheme boosts this to 45. "A typical network card supports up to 48 servers," he added, "so we don't use all the capacity and this adds cost. Although we're now putting 45 servers, with extra capacity should we decide to expand." In building its own data centres, Facebook has looked at the entire cost of ownership; and power is, without doubt, one of the largest costs. Michael said: "How much work can we get out of a data centre? How much does that cost? And how many dollars does it take to run the data centre?". One of the first design considerations was the power consumed by the data centre. A top line indication of how efficiently this is used is given by the Power Usage Effectiveness ratio, or PUE, which relates total power supplied to a facility and the power consumed by the servers. The US Environmental Protection Agency has a target figure of 1.5; which implies 33% of the power supplied is consumed by air conditioning and similar services. Michael said: "Our target was 1.15 and, as we got further down the design road, we revised that to 1.05. We have ended up with a figure of 1.07, where some large data centres typically have a figure of 2." The Oregon facility has a series of buildings, each consuming 'tens of MW'. Plans show the company is looking to expand this with the construction of a power transmission line that could supply 120MW. The computing efficiency is complemented with an energy efficient evaporative cooling system, led lighting and a tracking photovoltaic solar array generating 100kW. Power supply efficiency has also been targeted. "Most devices are more than 94% efficient," Michael claimed, "and some next generation parts will be more than 95% efficient." Facebook has also looked to eliminate uninterruptible power systems (UPS) where possible. "There's a number of reasons," Michael pointed out, "including conversion losses and the sheer cost of UPS. If you look at a data centre UPS, it costs around $2/W. Our target is 25c/W." The Oregon facility is also powered by a 480V supply, rather than 208V, to reduce losses. "Rather than convert ac to dc, store the power in a battery, then convert it back to ac," he said, "we're taking batteries and putting them closer to servers." The triplets include one lead acid battery for every six servers. Open to all In what might seem to be an unusual move, Facebook has made all this information available via Open Compute. John Kenevey, program manager with Facebook, explained. "The project resonates with Facebook's basic idea of sharing. But we also felt the technology itself is not a differentiator for us; we compete at the product level. The idea of Open Compute is to open up data centre innovation. Most innovation over the last couple of decades has come in the consumer space, so one thing we're focused on is developing a community and a contributions model. And we already have contributors lining up to share their IP with us." Alluding to the success of the Linux software distributor, Kenevey claimed that Open Compute wants to be the 'Red Hat' of the data centre industry. The two year Oregon data centre project appears to have been a success. The facility consumes 38% less energy to do the same work as existing sites, while costing 24% less. According to the company, it's 'vanity free'. Server boasts 384 Atom processors Low power server pioneer SeaMicro has launched what it says is the most energy efficient 64bit x86 server. The SM10000-64HD combines SeaMicro's server architecture with 384 Intel Atom N570 dual core processors, boosting compute density by 150% and its compute/W metric by 20%. All processors are housed in a 10 rack unit. Optimised for 64bit internet data centres, SeaMicro's SM10000-64HD system comprises: • 384 x86 dual core 1.66GHz Intel Atom processors, • 1.536Tbyte of DDR3 dram, • up to 64 SATA solid state or hard disk drives, • 8 to 64 1Gbit Ethernet uplinks. Keeping your cool Increasing energy costs mean there is no way data centres can continue to operate with air cooling models that have been in place for more than a decade, says Jason Preston, director of 2bm. "It is time to reconsider the way data centre cooling is delivered and explore opportunities for reducing the primary source of energy consumption." Preston says a compelling technology, is fresh air evaporative cooling. The approach combines external air with mechanically generated hot air from the data centre to create the right temperature – typically 21°C. Once the external temperature exceeds 19°C, the system switches to evaporative cooling, passing the fresh air over water saturated pads and mixing it with hot data centre air. The result allows companies to use fresh air, even in ambient temperatures as high as 38°. "The payback on this technology is also compelling: most organisations will save up to 90% of their cooling costs and get a return on their investment within two years," said Preston. The approach also qualifies for the Enhanced Capital Allowance, which allows the entire investment to be claimed against tax in the first year.