Home Server Setup

In late 2014, I upgraded my home server from an underpowered HP Microserver to an industrial-strength Dell server. They are surprisingly cheap second-hand, less than a few hundred, and this one only required some ECC RAM to get going. Even though the processor generation is a few years old, it still has plentiful ability to run everything I want, and the components (board / power supply / chassis) are built to a much higher standard than consumer parts.

My aim is to have the server handle all the storage/media/services in my home, as well as function as a testing lab for certifications and new technologies. I will continue to update this page as time goes on, with explanations peppered throughout, in case anybody is thinking of doing something similar.

The current setup:

  • Dell T5500 Dual Socket Workstation
  • 2x Intel Xeon E5630 2.5ghz (Total 24MB cache, 8 Core, 16 threads, Westmere 32nm)
  • 48GB ECC DDR3 RAM (expandable to 288GB), 6x8GB sticks.
    • Total RAM usage sits around 20-30GB at the moment, depending on what extra VMs are running. Any free RAM is used for write-caching. Sysctl.conf can be adjusted to allow for more aggressive caching, as the system is UPS-backed.
    • 3 more 8GB sticks can be added for a total of 72GB with this setup, before the more costly option of changing sticks to 16GB/32GB sticks.
  • 6x 3TB WD Green/Red drives in RAID6 (mdadm), 2x internal, 4x in Nexstar HX4R
  • 18TB Total, 12TB usable space, encfs encrypted
    • The RAID6 setup allows for two simultaneous drive failures before data is at risk. The four drives in the external enclosure are connected via eSATA.
    • The bottleneck is with the eSata connection, especially on heavy reads. I plan to eventually upgrade the external enclosure to a larger one (maybe 8 or 10 bay) with a USB 3 connection.
  • Sandisk 32GB USB Stick (Host OS)
  • Ubuntu 14.04 LTS (Kernel 3.13) running Samba, Syncthing, KVM, and DHCP.
    • I found best results running file-syncing services on the host directly, as inotify file watchers for syncing are more efficient than periodic scans from a guest OS.
    • The isc-dhcp-server on the host is for the secondary, internal-only network. KVM has limited guest/host communication, so file sharing can either be done via 9p (an inbuilt VirtIO device), or the more traditional SMB sharing on a dedicated network. I ran into a few POSIX issues with 9P. According to this post, 9P has been deprecated anyway.
    • Nearly all the VMs are running Ubuntu 14.04 LTS and have 1-2 CPU cores (<5% usage) and 1-2GB allocated each, with the exception of the Media VM, which runs Plex’s very heavy transcoder.
    • All VMs also run the NRPE, to feed server statistics back to Nagios for logging, and/or Rsyslog/Filebeat for logging. VM backups are rsynced by cron. Backups are mirrored to multiple offsite locations and rotated on a monthly basis. VM activity is monitoring by virt-top or virt-manager and Nagios.
  • KVM machines:
    • Important VMs:
      • ‘Net Master’ – DHCP/DNS server – DNS master server (bind9 w/network wide adblocking), DHCP Server (isc-dhcp-server), Web Server (nginx for bootstrapping), Squid-deb-proxy for apt package caching (previously apt-cacher-ng [constant hash sum mismatches, no longer maintained] and apt-mirror [excessive disk space requirements]).
      • ‘Net Slave’ – DNS server, replicates from Net Master.
      • ‘VPN/Proxy’ – NGinx reverse proxy, mainly simplifies port-forwarding from the router, and adds robust SSL to all internal services which are available externally. Remote SSH access. [Previously – OpenVPN access server, but retired due to lack of use].
      • ‘Identity/Authentication’Later: setting up FreeRADIUS server and Kerberos/network authentication.
      • ‘Web Server’ – RSS aggregator (Tiny Tiny RSS on Nginx/PostgreSQL), Usenet search aggregator (NZBHydra which forwards downloads to the download VM). Kept on this server for responsiveness, as router QOS prioritizes the VMs in this group.
      • ‘Console’ – General purpose VM, used for doing file operations and other things when I’m working remotely. Also runs Flexget for automatic podcast, Youtube and other miscellaneous downloads / conversion.
    • Secondary VMs:
      • Configuration Management’ – Puppet master (manifests and source files are kept on the RAID).
      • ‘Downloader’ – Torrents (transmission-server), Usenet (SABNZBdplus), Couchpotato (movie management), Sickrage (TV show management, previously Sickbeard). Later: Migrate to Sonarr once mature.
      • ‘Monitoring’Nagios 3.x server (for service monitoring/alerts, short/long term availability and performance graphs and reports), pnp4nagios for nicer graphs. Logs and records are retained for around 5 years. Any service downtime or high VM load sends immediate alerts.
      • ‘Logging’ – Centralized network logging (rsyslog server, which receives logs from all VMs, parses them and drops them onto the RAID array, log rotation).
      • ‘Mail Forwarding’ Postfix w/smarthost for forwarding local mail to remote mail server for sending. Neater than configuring many instances of forwarding on all VMs (and the associated security issues). Clients use SSMTP to forward this VM.
      • ‘Media’Plex Media Server (which feeds to the Apple TV, mobile devices and web frontends), PlexConnect (for spoofing the older Apple TV into working with Plex), PlexPy (addon for managing Plex libraries in-depth).
      • ‘Syncing’Wallabag (a read-it-later service, similar to Pocket), Nextcloud (the actively developed fork of Owncloud, for remote document, photo and file access). Note that Quickpic, the Android gallery app, supports Owncloud/Nextcloud services. Later: Self-hosted calendar/contact lists.
      • ‘ELK’Elasticsearch, Logstash, Kibana server for log collection, parsing and fancy graphing. ELK stack is one of the industry favorites, but resource requirements are higher than Nagios/Rsyslog.
    • Testing VMs:
      • ‘Ubuntu Testing’ – For testing new changes and packages before rolling out to the LAN.
      • ‘Red Hat 7’ – For testing CentOS/Redhat changes. Red Hat provide a free license for home use.
      • ‘Windows’ – For testing purposes, runs Windows Server 2008 as Domain Controller on separate subnet. Heavier than nearly all the other VMs combined.