[Guide] Budget Home Server Upgrade (Pt. 1): Dell Precision T5500

It wasn’t that long ago that the HP N54 Microserver was serving as a dedicated ESXi host, my thoughts here. Though within a few months, I found myself running into it’s limitations – a distinct lack of cores and physical memory. If the wind blew in the wrong direction, the diminutive AMD N54 dual-core would be pegged at maximum capacity, commonly holding tasks hostage while it valiantly struggled. In it’s defence, it was never intended for such heavy lifting, especially not in a hypervisor environment. A low-power NAS-only setup it handles handsomely, but honestly, not much else. In addition, I found myself increasingly working on my Macbook whilst out and about, so I decided to re-purpose my desktop machine to replace the N54.

For a while the new machine performed well – an Intel i5-3570 (‘Ivy Bridge’ 3.4ghz with 6MB of L3), 16GB of RAM and a mid-range desktop board (Gigabyte-H77). However, two issues became evident. On Intel’s consumer desktop boards, 4 DIMM slots, or 32GB of RAM, is the absolute maximum. All slots were already taken (4 x 4GB). That’s alot for a single OS, but when running a dozen at the same time, physical memory quickly becomes scarce. ESXi does have memory compression, de-duplication and SSD swapping available, but there’s only so far you can take over commitment.

Secondly, without Hyper-Threading, the i5’s quad-cores had to carefully provisioned. One of the glaring limitations of ESXi’s CPU scheduler is the requirement to have the number of vCPU cores free at any one time, that are shared between VMs, or the VM is held up in a ‘ready’ state (manifested as %RDY). In short, cramming every VM with the maximum number of available cores (a common rookie mistake) is a guaranteed recipe for abysmal performance – this applies to Workstation/Fusion as well. Hyper-V does not have this limitation, but instead has a whole host of other quirks of it’s own, but we won’t get into that. Over-allocate and you run into situations where VMs are waiting on X number of multiple physical CPU cores to be available, otherwise known as co-stop (or %CSTP).

In any case, more RAM required, more cores required – individual core speed wasn’t a big concern, but parallelism was. I kept an eye open until a second-hand 2011 Dell Precision T5500 appeared at an undeniably bargain price, cheaper than even a current-model and technically far-inferior G8 Microserver. Produced in either a 1U rack format, or tower/workstation form, it comes in a variety of configurations.

The one I picked up was a dual-socket Intel Xeon 5630 (‘Westmere’ 2.8ghz x 4 cores on FCLGA1366), to make 16 logical cores. Along with this, it has 9 DIMM slots, which accept up to 72GB of ECC registered/unbuffered DIMMs (UPDATE: Thanks to Zdravko, the T5500 unofficially supports 144GB-288GB of RAM). Though manufactured on a 32nm process (vs the i5’s 22nm), and having lower maximum clock speeds, the Xeons are ideal for an ESXi server, where the quantity of cores, rather than the individual speed, is more important. 12MB on-die L3 cache also doesn’t hurt.

DSC_0161

The two-bay hard drive tray on the left folds down to allow access to the first row of DIMM slots. The CPU riser on the right unclips and slides out. Front panel also disassembles easily.

Used ECC RAM is often cheaper than consumer-grade RAM, because it’s generally slower clock-for-clock, due to it’s parity checking, and often companies are constantly offloading decommissioned / out-of-contract servers and components, which leads to a healthy supply of second-hand DIMMs and service parts. I picked up 48GB of Micron DDR3 (6 x 8GB 1066mhz) from an ex-HP server from eBay to start off. It all slots into a custom Intel X5520-based board, with somewhat peculiar PSU wiring from the 875W PSU, especially to the 2nd CPU’s riser board. A Broadcom BCM5754 GBit NIC and half a dozen USB ports make it a well-rounded option. There are also a bunch of PCI-E slots, if you later wanted to pass-through a GPU.

DSC_0173

CPU riser card houses the second socket, three additional DIMMS and associated cooling fans.

So 4 times the logical cores, 3 times the physical RAM, surely it’s all smooth sailing right? Not yet, there were some important lessons I learnt while setting it up, which thankfully, was the purpose of the exercise:

* The T5500 case is more or less packed to the brim. It’s sized approximately the same as a standard MIDI Tower case, but houses a server class board/cooling/PSU and dual sockets, one on a riser card, along with 9 DIMM slots. As a result, it has barely enough room to fit in a DVD drive and 2 3.5″ bays. There is one additional flex bay, which can be used for either (though an optional flex bracket is required). Luckily, if you want storage arrays, there’s an e-SATA port, or if you want better performance – plenty of external SAS options (the PERC 6 being the fan favourite).

* Hyper-Threading and VT-D is disabled by default in the BIOS. There are also four diagnostic lights on the front of the machine, the service manual outlines the codes.

DSC_0190

* There is a chassis intrusion switch, which can optionally trigger a silent alarm in the BIOS. I have no use for such a feature, but as I learnt after many hours of head-scratching, that triggering this alarm while the machine is ON, results in CPU fan 2 spinning up to full speed, emulating a vacuum cleaner until the option is cleared in BIOS. Maybe it’s a quirk, or some kind of twisted joke by the engineers. Resetting this option has a 50/50 chance of solving the problem, even BIOS resets, battery pulls and BIOS updates did not guarantee a solution. Pulling the switch entirely also did not help, but disassembling the switch to keep the two sides permanently disconnected (ie. when the case is open, the circuit is closed), then taping it up and returning it to place did work. Additionally, a simple jumper across two of the pins on the switch header will also work. Though I had issues even locating a jumper at all … how far we’ve come.

* ESXi is NUMA-aware, so switch that on in the BIOS. Think of it as localised memory access for better performance. Additionally, though it’s not strictly required, Dell also distribute a Dell-specific ESXi 5.5U1 build, which can be downloaded here. As far as I can tell, IPMI is not supported. Even if you don’t use this build, ESXi 5.5 supports the built-in NIC, no need to roll your own.

* The T5500 weighs ALOT. Around 20KG to be exact, without any drives. My desktop i5 machine, fully loaded with 9 x 3.5″ spinning hard disks and power supply, only weighs a touch over 15KG.

* Cooling wise, two large fans draw air in the front of the case, through a removable plastic duct, which blows over CPU1 and the first array of DIMM slots. CPU2 has it’s own Sunon fan which pushes exhaust air out the rear of the case. The comparison between server and consumer grade hardware is night and day. Everything is designed to be swappable, solid and easy to open.

How does it perform? Better than I had expected. With a half-dozen Linux VMs (incl. 1 x MySQL and 1 x PostgreSQL DBs), a Plex Media Server doing transcode (Plex uses an incredibly inefficient Java transcoder), three Windows servers VMs (AD, workstation, Exchange), Nagios and analytics services (Quest Foglight / vCenter Operations Manager), all suitable provisioned with plenty of CPU/RAM, it didn’t even bat an eyelid. Additionally, I ran two nested Hyper-V servers (on full Server 2012 host VMs, each with 6 cores/8GB allocated) and there was no noticeable latency on neighbouring VMs. Co-stop was non-existent, peak CPU load was well under 50%, and ready time in the low single-digits. As far as servers go, the T5500 is still in the kiddie pool, but it’s plenty for a home server.

vcenter

If you want to enable EVC on your cluster with the T5500, make sure you select Westmere compatibility.

The T5500 was just the first part of the puzzle, because the NAS was also due for an upgrade. If there’s one thing that makes home networks sweeter, it’s faster storage. I decided to do everything at the same time and upgrade the NAS, but juggling was required. It all comes together in Part 2.

a1ii-800Pretty, the Dell Precision is not.

Advertisements

12 thoughts on “[Guide] Budget Home Server Upgrade (Pt. 1): Dell Precision T5500

  1. I have two T5500’s. One of which is always on and always handling more than I can ever imagining any machine handling. With that said, my secondary unit that I’ve been working on has been problematic. It started with random shut-downs. Sys Diags showed DIMM problems. Replaced motherboard—totally non-functional. Replaced “new” motherboard with another “new version 2” motherboard, also dded a 2nd PSU daughterboard/riser

    All seems to work, except the occasional orange light that won’t allow me into BIOS or POST. Removing the 2nd CPU riser and reinserting it seems to resolve the problem.

    I’m wondering…Is it the PSU that was the problem all along? Could it possibly be the second CPU? Just wondering if anyone reading this has had similar “finicky” problems.

  2. dell t 5500 ram24 gb processor 3.7 3.6 i have a problem in my i7 upgade because i have a problem in dara transferring to another device ( memory card or usb) for sharing songs movies so the problem is speed is very slow for the anthor things so please kindly give me the soulation for this problem tnanks.
    name zohiab khan cell nbr oe email 03038682635 khyber.edu.pk@gmail.com

  3. “…technically far-inferior G8 Microserver…”
    While absolutely true, I am even scared to think how much juice this beast drinks. I probably could power my G8 for weeks instead of days for same money…

    1. Not that it matters now, and it’s a silly comparison, but let’s do the numbers. According to my watt meter, the T5500 consumes around 140w of instantaneous power. Admittedly, this is roughly 4 times the power consumption of the G8 (unfortunately, I never tested my G7 when I had it). In my area, this equates to an increase in power bill of about $12 a month.

      It’s important to note that I purchased the T5500 second hand (the only state in which I’d recommend buying one), and it was actually substantially cheaper than a brand new G8. In fact, it was only after I purchased 48GB of 2nd-hand ECC server RAM that there was price parity between the two. The G8 is horribly overpriced. Notice that I titled the article ‘budget’, because a good deal came up.

      Which left me with this entertaining choice, so for the same price:

      * 2 cores vs 16 cores
      * 32GB max RAM vs 288GB max RAM
      * 2GB of included RAM vs 48GB of included ECC RAM
      * 2400 vs 17300 on CPU Passmark
      * 2MB on-die cache vs 24MB on-die cache
      * Dual vs triple channel memory bandwidth
      * +AES hardware acceleration
      * +VT-D hardware acceleration
      * +Bunch of extra PCI-E / addon slots

      So, unfortunately for the G8 and Microserver owners, the only area where the G8 does have the advantage is in power consumption. For anything more than a medium workload, the G8 is entirely unsuitable, and I’d even say, a poor bang-for-buck, performance wise. I’ve noticed multiple VM, RAID6 and transcoding max out the Xeon at times (individually), I’d shudder to think what it would do to a Celeron. Horses for courses, I’d definitely recommend a 2nd-hand T5500 for those who don’t like waiting.

  4. What’s great about the Dell Precision T5500 is the (undocumented) fact that you can put more than the officially supported 72 GB of RAM into this system. I just bought three refurbished T5500 systems for my own homelab and successfully populated all 9 DIMM sockets in one of the systems with 16 GB RAM modules – the system now totals to 144 GB of RAM! It seems that you could even put a total of 288 GB of RAM into one single T5500 dual-cpu system if using 32 GB DIMM modules – excellent value for the money!

      1. Have you guys had any issues with the riser connections? I recently had to buy a new replacement motherboard because when I pulled off the cpu riser, the connection covers came off the main mobo.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s