It wasn’t that long ago that the HP N54 Microserver was serving as a dedicated ESXi host, my thoughts here. Though within a few months, I found myself running into it’s limitations – a distinct lack of cores and physical memory. If the wind blew in the wrong direction, the diminutive AMD N54 dual-core would be pegged at maximum capacity, commonly holding tasks hostage while it valiantly struggled. In it’s defence, it was never intended for such heavy lifting, especially not in a hypervisor environment. A low-power NAS-only setup it handles handsomely, but honestly, not much else. In addition, I found myself increasingly working on my Macbook whilst out and about, so I decided to re-purpose my desktop machine to replace the N54.
For a while the new machine performed well – an Intel i5-3570 (‘Ivy Bridge’ 3.4ghz with 6MB of L3), 16GB of RAM and a mid-range desktop board (Gigabyte-H77). However, two issues became evident. On Intel’s consumer desktop boards, 4 DIMM slots, or 32GB of RAM, is the absolute maximum. All slots were already taken (4 x 4GB). That’s alot for a single OS, but when running a dozen at the same time, physical memory quickly becomes scarce. ESXi does have memory compression, de-duplication and SSD swapping available, but there’s only so far you can take over commitment.
Secondly, without Hyper-Threading, the i5’s quad-cores had to carefully provisioned. One of the glaring limitations of ESXi’s CPU scheduler is the requirement to have the number of vCPU cores free at any one time, that are shared between VMs, or the VM is held up in a ‘ready’ state (manifested as %RDY). In short, cramming every VM with the maximum number of available cores (a common rookie mistake) is a guaranteed recipe for abysmal performance – this applies to Workstation/Fusion as well. Hyper-V does not have this limitation, but instead has a whole host of other quirks of it’s own, but we won’t get into that. Over-allocate and you run into situations where VMs are waiting on X number of multiple physical CPU cores to be available, otherwise known as co-stop (or %CSTP).
In any case, more RAM required, more cores required – individual core speed wasn’t a big concern, but parallelism was. I kept an eye open until a second-hand 2011 Dell Precision T5500 appeared at an undeniably bargain price, cheaper than even a current-model and technically far-inferior G8 Microserver. Produced in either a 1U rack format, or tower/workstation form, it comes in a variety of configurations.
The one I picked up was a dual-socket Intel Xeon 5630 (‘Westmere’ 2.8ghz x 4 cores on FCLGA1366), to make 16 logical cores. Along with this, it has 9 DIMM slots, which accept up to 72GB of ECC registered/unbuffered DIMMs (UPDATE: Thanks to Zdravko, the T5500 unofficially supports 144GB-288GB of RAM). Though manufactured on a 32nm process (vs the i5’s 22nm), and having lower maximum clock speeds, the Xeons are ideal for an ESXi server, where the quantity of cores, rather than the individual speed, is more important. 12MB on-die L3 cache also doesn’t hurt.
The two-bay hard drive tray on the left folds down to allow access to the first row of DIMM slots. The CPU riser on the right unclips and slides out. Front panel also disassembles easily.
Used ECC RAM is often cheaper than consumer-grade RAM, because it’s generally slower clock-for-clock, due to it’s parity checking, and often companies are constantly offloading decommissioned / out-of-contract servers and components, which leads to a healthy supply of second-hand DIMMs and service parts. I picked up 48GB of Micron DDR3 (6 x 8GB 1066mhz) from an ex-HP server from eBay to start off. It all slots into a custom Intel X5520-based board, with somewhat peculiar PSU wiring from the 875W PSU, especially to the 2nd CPU’s riser board. A Broadcom BCM5754 GBit NIC and half a dozen USB ports make it a well-rounded option. There are also a bunch of PCI-E slots, if you later wanted to pass-through a GPU.
CPU riser card houses the second socket, three additional DIMMS and associated cooling fans.
So 4 times the logical cores, 3 times the physical RAM, surely it’s all smooth sailing right? Not yet, there were some important lessons I learnt while setting it up, which thankfully, was the purpose of the exercise:
* The T5500 case is more or less packed to the brim. It’s sized approximately the same as a standard MIDI Tower case, but houses a server class board/cooling/PSU and dual sockets, one on a riser card, along with 9 DIMM slots. As a result, it has barely enough room to fit in a DVD drive and 2 3.5″ bays. There is one additional flex bay, which can be used for either (though an optional flex bracket is required). Luckily, if you want storage arrays, there’s an e-SATA port, or if you want better performance – plenty of external SAS options (the PERC 6 being the fan favourite).
* Hyper-Threading and VT-D is disabled by default in the BIOS. There are also four diagnostic lights on the front of the machine, the service manual outlines the codes.
* There is a chassis intrusion switch, which can optionally trigger a silent alarm in the BIOS. I have no use for such a feature, but as I learnt after many hours of head-scratching, that triggering this alarm while the machine is ON, results in CPU fan 2 spinning up to full speed, emulating a vacuum cleaner until the option is cleared in BIOS. Maybe it’s a quirk, or some kind of twisted joke by the engineers. Resetting this option has a 50/50 chance of solving the problem, even BIOS resets, battery pulls and BIOS updates did not guarantee a solution. Pulling the switch entirely also did not help, but disassembling the switch to keep the two sides permanently disconnected (ie. when the case is open, the circuit is closed), then taping it up and returning it to place did work. Additionally, a simple jumper across two of the pins on the switch header will also work. Though I had issues even locating a jumper at all … how far we’ve come.
* ESXi is NUMA-aware, so switch that on in the BIOS. Think of it as localised memory access for better performance. Additionally, though it’s not strictly required, Dell also distribute a Dell-specific ESXi 5.5U1 build, which can be downloaded here. As far as I can tell, IPMI is not supported. Even if you don’t use this build, ESXi 5.5 supports the built-in NIC, no need to roll your own.
* The T5500 weighs ALOT. Around 20KG to be exact, without any drives. My desktop i5 machine, fully loaded with 9 x 3.5″ spinning hard disks and power supply, only weighs a touch over 15KG.
* Cooling wise, two large fans draw air in the front of the case, through a removable plastic duct, which blows over CPU1 and the first array of DIMM slots. CPU2 has it’s own Sunon fan which pushes exhaust air out the rear of the case. The comparison between server and consumer grade hardware is night and day. Everything is designed to be swappable, solid and easy to open.
How does it perform? Better than I had expected. With a half-dozen Linux VMs (incl. 1 x MySQL and 1 x PostgreSQL DBs), a Plex Media Server doing transcode (Plex uses an incredibly inefficient Java transcoder), three Windows servers VMs (AD, workstation, Exchange), Nagios and analytics services (Quest Foglight / vCenter Operations Manager), all suitable provisioned with plenty of CPU/RAM, it didn’t even bat an eyelid. Additionally, I ran two nested Hyper-V servers (on full Server 2012 host VMs, each with 6 cores/8GB allocated) and there was no noticeable latency on neighbouring VMs. Co-stop was non-existent, peak CPU load was well under 50%, and ready time in the low single-digits. As far as servers go, the T5500 is still in the kiddie pool, but it’s plenty for a home server.
If you want to enable EVC on your cluster with the T5500, make sure you select Westmere compatibility.
The T5500 was just the first part of the puzzle, because the NAS was also due for an upgrade. If there’s one thing that makes home networks sweeter, it’s faster storage. I decided to do everything at the same time and upgrade the NAS, but juggling was required. It all comes together in Part 2.
Pretty, the Dell Precision is not.