Wow. I had an idea I had been punting around about putting together a place for Data Center designers and engineers to openly discuss new ideas and concepts, perhaps even growing to an organization that could have a test lab to put ideas into action. I’m glad to see that Facebook has started it out with the Open Compute Project and put their designs into the public eye.
Looking at the Open Compute Project’s website, I see a lot of innovative ideas. I like the “triplet rack” concept. It is a great idea when you have clusters of cabinets that create your rows. I’d have used a “doublet rack” in places had I thought to ask our manufacturer for something like it. The concept of splitting power between “clean utility” and UPS power is not new, but the way that Facebook is implementing it is interesting. My conversation with myself today will focus on power.
First, no server major commodity server manufacturer (HP, IBM, Dell) offers a 277v power supply. This means either the manufacturer would have to create one for you, or you would have to source from a 3rd party. I see problems with this, especially if you leverage manufacturer support for your maintenance. The biggest being you would have to stock a fair number of replacement power supplies since they are a non-standard part. I don’t think that would fly in my organization, I doubt it would fly in other organizations.
Second, while they have two power paths, a 277v AC path and a 48v DC path, they terminate into a single power supply. In a scale out computing scenario this is fine as the loss of one individual node is usually trivial, but if you are hosting mission critical databases you are going to want to make sure that you don’t lose a node because of a failed power supply.
Third, it looks like Facebook has a fairly homogeneous server stack. This has not been typical in my experience, and I expect that most others in my position would feel the same way. A homogeneous server stack allows you to perform a number of optimizations that you wouldn’t be able to otherwise. The typical cabinets in my server room have a variety of 1U, 2U, 4U and blade enclosures to service the disparate needs. Since our data center group does not specify servers we have very little control over what comes in as long as there is capacity (space, power, cooling) to support the equipment. This often means that we have to support servers that are purchased as a proof of concept or because of an unbelievable discount from a vendor trying to find footprint in our data center. This means that any group of three cabinets I have in my data center my not match any other group. Our initial design criteria for our current data center was 10kW per server cabinet for the first phase, growing to 15kW with the second phase and 20kW in the third and final phase. Right now we are in the 6-8kW range for most of our cabinets, but increasing use of blade servers will move us closer to 15kW very soon. Facebook is looking at 56kW for two of their three cabinet pods, that puts each cabinet at around 9kW. For enterprise data centers with a mix of server types and a industry push towards virtualization and higher density platforms, how well does this play?
I’m sure I’ll have more to say as I continue to digest…
