Our first two colocation builds were relatively low power affairs. Both our Las Vegas, Nevada facility and Fremont, California facility offer 15A 120V of power per cabinet. Our latest facility in Sunnyvale, California has 208V 30A per rack. In our original lab and in the subsequent facilities, testing has traditionally occurred on 120V circuits which are standard in the USA. With 208V we have noticed some significant changes.
120V or 208V which is better?
120V (if you live in the USA) is extremely easy to find. 15A 120V wiring has been standard for residential and many office buildings for some time now. In the Silicon Valley, California, many small offices are abuzz with the sound of servers and workstations because 120V is available.
In terms of power supply efficiency, 208V is generally around 2% more efficient with figures commonly quoted in the 1-3% range. That degree of impact is noticeable but is not necessarily a game changer, especially in low density racks. The difference between 200w and 204w for a server is often negligible on a single server level. At a rack level, with 5000w usable (as an example) 2% better efficiency might mean about 100w difference, or enough to power a handful of very low power nodes.
Operating at a higher voltage requires less current to drive the same amount of power. For example, a 400w dual processor server might require around 3.33A on a 120V circuit but would only require about 1.92A on a 208V circuit. The simple formula you can use for most basic “plug a box into a datacenter PDU” scenarios is: Volts x Amps = Watts. Here is an example of how a 400w server varies based on input voltage.
Given the varying power situations servers are installed in, we now have the ability to test on both 208V and 120V racks for our testing. Moving up to 220V-240V generally yields another 0.25% or so gain in efficiency. There are more exotic power setups but if you are using standard colocation, there is a good chance you will use one of these options.
How does this impact how much power I can use for my servers and networking gear?
The difference between 120V and 208V practically is that 208V circuits tend to deliver more power and are (generally) designed to be more robust. One must take into account that only about 80% of the total power will be considered usable for a variety of reasons. Take two common examples:
- The standard 15A 120V circuit has capacity for 1.8kW * 80% = 1.44kW
- The standard 30A 208V circuit has capacity for 6.24kW * 80% = (about) 5kW
That is a huge difference. Also do remember that the 208V is more power efficient to the tune of 2%. 120V is considered a lower density option but is readily available. An example of this is the 8x GPU server running in Bank 1 of our Schneider Electric APC AP8441 PDU
When you have systems this large, they cannot be run on a standard 15A 120V circuit.
120V or 208V which can my server/ storage/ switch use?
Generally speaking, most modern servers are rated for 100-240V or 250V operation, much like the majority of today’s consumer electronics. The simple fact is that building a global power supply is oftentimes more cost efficient than building a narrow voltage input range power supply. There is great benefit in being able to simply match a power cord and get running. Do check the power supply (or manual) but if the included cable has a IEC320 C13 end, then you are likely OK on power.
While servers may have power supplies that can span voltage ranges, the power distribution unit and battery backup side of things can be a bit different. Those components due to plugs and inefficiencies are often rated at much narrower voltage ranges (e.g. 208V only, 208-240V and etc.) When we started building out the new Sunnyvale, California facility, we used Schneider Electric APC AP8441 PDUs which are rated for 200V/208V/230V 30A input.
Conclusion
This was not meant to be the most comprehensive guide on the subject by any means, but it should help with an extremely high-level overview of why one would use a higher-power rack and some of the implications. We do now have the capability to test machines in both low power 120V 15A environments as well as higher power 208V 30A environments. There are many more higher power and higher efficiency options out there but using general off the shelf hardware and colocation these are common options. One note is that lower-end gear with external power supplies may work fine on 208V but may require an adapter to work in the higher power PDUs. For example, the North American shipping Mikrotik CRS226-24G-2S+RM has an external power supply which is not easy to power using a standard IEC320 C14 outlet. More on this in the near future.
Some factors for the data center operator are that 208V circuits take up two panel spaces in the electrical panel instead of one. If you don’t have much panel space, that can be an issue.
Also, 208V power strips (vs. 120V gear), and especially 30A power strips (vs. 20A gear) are generally much more expensive, between 40-60% more. Which adds up if you provide power strips.
And 30A power requires running 10AWG wires, which take up more space in conduit, leaving less fill capability (& cost more (when you run miles of it).
Where the hell did you found 208V. In Europe the 230V is a standard now. (220V is a rare find) However you can get between 230-240V depending on how far and how many users are from the last transformer to your house.
Yup, none of the crappy low voltages here. 240 was the standard here in Oz but it has been dropped a little to 230VAC. Doesn’t mean much when we regularly see 250+ with all the the grid-inject solar that is around that jacks voltages higher. UPS is the magic savior.
I did notice, none of the gear in the article was on a UPS, one might assume that the site is on UPS/genset protection. Any racks I have or run have a UPS so all gear is fed off them, not direct mains. There goes the power-factor and power usage maths in the article.
While all of those things are true, they also don’t really matter. When designing a datacenter you work with the maximum power that the room has to be able to handle – power distribution, cooling, etc. are all based around that and it ends up being the single number often used to compare the size of datacenters. When you’re looking at say a 1 megawatt datacenter, it is far more economical to do your power distribution at high voltages. Sure a 30A 208V breaker takes two spaces instead of just one – but to get 5Kw of power to a rack you either need that 2-slot 208V 30A breaker, or 4 110V 15A breakers – suddenly it is the 208V option that is using half as much panel space. And it follows all down the chain – you pay more for a single power strip and need far fewer of them, you run wire that is slightly thicker diameter, but only need 1/4 as many runs.
And in response to the other post about “where does 208V come from” – in my experience the more unusual voltages are usually related to 3-phase power, which is another option you will see in datacenters. For example in our datacenter at work we run a pair of 30A 3-phase circuits to every rack, and our power-strips convert it into regular single-phase power. So going into the power bar there are 4 wires – 3 live phases and one ground. Inside the power strip if a plug connects from the first live wire to ground, you get a plug with 110V on phase 1. Connecting the second live wire to ground is a 110V plug on phase 2. Or you can connect across phases – a plug connected across the live wires for phase 1 and 2 gives you a 208V plug on phase 1/2. Our power bars are all split into four sections – the majority of the plugs are 208V and are divided evenly into groups for phase 1/2, phase 2/3, and phase 1/3, and there is also a small section in the middle with a few 110V plugs on phase-1 for the occasional odd piece of equipment that needs it. In a way it is also a further extension of what I was saying above – running a single 30A 3-phase circuit to a rack gives us 15Kw of power, needing only a single breaker and a single 4-conductor wire run to the rack.
208V is a standard in US datacenters but is not common outside of that.
Psst.. Please correct the “Stroage” in the title. I’ve tried not to say anything but my OCD’s been killing me every time I browse the front page!
208V is what you get from 240V power in a 3-phase “Wye” system. That lets you get 120V between any phase leg and neutral and 208V between any two phase legs. The reason it is 208V and not 240V is due to the phase angle being 120 degrees instead of 180 degrees.
This is also used outside of datacenters, wherever there is a large electrical load – for example, an apartment building will usually be served with 3-phase power, with the feeds to each apartment being assigned to 2 legs of the feed to provide 120V/208V. The idea is to “balance the legs” so that each leg is supplying approximately the same amount of current, with neutral current being as low as possible. Even single-family residences can have 208V – on my street there are a number of large apartment buildings, so the power on the poles is 3-phase. Single-family houses get 2 legs, as described above.
Regarding the use of 208V (or 240V) in a 120V country like the US, one possible concern is the use of power distribution devices like the APC unit shown in the picture. It can be tempting to use a 208V/240V-only power distribution unit. These will normally have IEC connectors for the input and output instead of the US 5-15R straight blade receptacles. The IEC receptacles will usually be IEC C13, which is also what is used on most servers (in the matching C14 plug). There is no distinction on those IEC connectors for 120V vs. 208V/240V. So you have an outlet strip and servers happily running at 208V/240V. Sometime later you add another device which is either 120V only or manually selectable 120V / 208V/240V. Not remembering that your outlet strip is providing 208V/240V, you plug in the device. Poof! Hopefully it is only the new device that blows its fuse or goes up in smoke. But you have the possibility of tripping the circuit breaker / fuse in either the outlet strip or the datacenter’s power panel feeding it. In either case, all of your servers are down until you can reset. If you’re truly unfortunate and this happens at an off hour in a datacenter without 24H on-site coverage (some datacenters have only security personnel during off hours; some have an on-duty tech at all times but require an electrician instead of a tech to reset breakers) you may be down for an extended length of time while they find someone who has both the key to the electrical panel and permission to use it.
> “120V or 208V …”.
It depends who is paying for what. If you’re an end user you will likely opt for what is affordable; based on an uninformed opinion.
You will buy 120V, because the plug fits in the wall; no need to hire an Electrician just to plug in a Lamp.
Where the Consumer (end user) needs more than 15A (like for an Oven or Dryer) they are stuck with the plug in the Laundry Room or Kitchen.
IF you consider things from a viewpoint of efficiency (you are the one paying for the electricity, or there is a penalty from the Power Company for wastefulness) then HIGHER voltage is more efficient.
See: https://www.google.ca/url?q=http://www.apcdistributors.com/white-papers/Power/WP-27%2520Efficiency%2520and%2520Other%2520Benefits%2520of%2520208%2520Volt%2520Over%2520120%2520Volt%2520Input%2520for%2520IT%2520Equipment.pdf&sa=U&ved=0CA4QFjABahUKEwjDotaC7ZDIAhWJl4gKHRGCCsg&sig2=ldNoPH953ccN6VwhHEO5GA&usg=AFQjCNGo0XKDlUn-6YbL5DcX45_XEoChow
(Can’t get direct URL), but here’s a quote:
“Server power supplies, when considered as a group, have a significant impact on energy efficiency of a data center. This has prompted some data center managers to specify high efficiency power supplies. The easiest way to improve the efficiency of existing server power supplies is simply to increase their input voltage. Figure 1 illustrates a 1-3 percentage point improvement in power supply efficiency with increased voltage. This represents a savings of $4 to $31 per year per server in electrical cost by doing nothing more than moving equipment to a higher voltage source.”
Datacenters want to offer the highest voltage that can be used.
Terry Kennedy, or anyone who is qualified to answer this, I have a number of 220v PSU’s that I need to power in a small bitcoin mining operation. I need a PDU for these and I see a nice heavy cable on a 60AMP 208 V pdu. Could this pdu work for me or does it have to be 208? having a hard time finding a 220v PDU for my PSU’s Thank you for anyone who can answer. My assumption is that they may not provide enough power for the 220v psu’s or be somehow off phase?
Thanks for a great article, also enjoyed reading the commentary. Just thought I’d mention that 208V is also found in defense applications, at least in our experience. We have several military clients who have requested 208V power supplies for various equipment such as radar, onboard power for armored vehicles, command and control platforms, and other electronic warfare equipment. Here are some examples of the 208V power units for these applications, for anyone interesting in comparing differences to those used in data centers: http://aegispower.com/index.php/products/three-phase
For office space 208v is the correct option for a MDF or IDF, due to the continued growth of PoE. Example: Cisco 4510 with four power supplies at 120v can only power about 3 blades of PoE devices (WAPs and Phones). Using 208v with the same equipment you can fully power all 8 blades (384 ports). Don’t get caught with your network equipment out of power for tomorrow’s demand.
If I might add, there is also the issue of cooling….higher voltage means less current to drive the same load. Less current means less heat. Less heat means less cooling required. Perhaps this isn’t an issue in less densely populated racks, but in a data center full of racks, filled with routers, switches, and servers, that can make a difference.
Furthermore, almost if not every device Cisco has made in the last 8+ years has been outfitted with a power-supply capable of auto-sensing line power. Same is true for Dell and HP servers.