I wanted to build a reasonably powerful ESXi Whitebox platform on which to run a lab with my work related virtual machines (I am a Systems Engineer employee of Cisco). I want to run Cisco Prime Infrastructure, Cisco Identity Services Engine, Cisco APIC and Cisco Modelling Labs all on this one box simultaneously as well as a Windows AD controller and a Web server. As well as all this I want to run this box as a home network attached storage (NAS) box.
I had several key requirements. First it had to be small - I don't want a large tower case dominating my small study - I need this to be a small form factor machine. Secondly it had to have low power consumption since the box would be left for much of the day whilst doing lab work and for the media server function. I expect it to come in at about 50W utilisation when idle. So whilst thats not a lot I certainly don't want it on 24x7 consuming that much power. All VMs other than the home media centre VM would be powered down to save on power when I am am not using the work lab VM's. I want it consuming just a trickle of power when idle; let's face it, power bills are not going to get any lower.... I also want it to look "nice" in a geek sort of way! Tech should look the part in my book - just look how successful Apple have been! If it's good enough for them then it's good enough for me.....
A principle requirement was that the machine would need 32G of RAM. ESXi works best when the machines are loaded into RAM as much as possible to save using the disks. IO read-write speed (IOPS in geek speak) is not a strong point on none-RAID attached commodity HDD 7200rpm disks. True, a SSD could be used as storage (more on this later) however, their capacity/cost ratio is not good in comparison with a traditional HDD.
I narrowed my list of components down to:
CPU = Intel 4570S
Motherboard = Gigabyte B85M-D3H
Memory = 2 x kits of G.Skill 16Gig RipjawsX 1600 DDR3 CL10
Hard Disks = > 2 x WD Caviar 1TB Blue
SSD = Crucial 256G MX100
LAN Adapter = Intel EXPI9301CTBLK
Case = Silverstone SU02
Be careful if you are building a Whitebox; you need to ensure the components are supported on the VMware HCL. If they are not the installation will not be allowed to continue. By all means copy my list of parts if you need to :-) The HCL is quite liberal when it comes to CPUs, memory and disks. However, it is very strict on LAN adapters and RAID cards.
RATIONALE
CPU
I am familiar with Intel chipsets and motherboards so decided to stick to this route. I am sure AMD would be good as well (perhaps more so for all I know) but it's one less variable for me by sticking with an Intel CPU/MB combination. I was not prepared to pay the £100+ premium for an i7 CPU as much as the hyper-threading would be nice to have. Typically the CPU is much less important that disk speed and RAM availability so I decided to spend that extra £100 in these two departments and settled for the i5 CPU series and specifically the 4570S. Note that trailing "S"..... That means the low power TDP version of this CPU which consumes 20W less than the none S version. This is important to reduce the power utilisation of the completed box. The 4570S also fully supports virtualisation.
I also ditched the Intel stock CPU fan and used a AKASA CPU fan instead for an extra £12.
MOTHERBOARD
I did not need the all-singing, all-dancing motherboards that can be bought nowadays. A simple Intel B85M chipset is more than enough and these can be had at £50 or so and I therefore opted for the Gigabyte B85M-D3H. Take note of the last three characters - there are some versions with only two memory slots if you get that combination wrong. This model has four memory slots to give a maximum of 32G. This motherboard in conjunction with the CPU also supplies on board graphics with no need for a graphics adapter. Ok, maybe running the latest games is not going to be too successful with onboard graphics but that is not the purpose of this machine. Nevertheless, the graphics performance at 1900x1200 is certainly very good. Another reason I decided on this board is because it is Micro-ATX in form factor to fit into a smaller case. I have also had good experiences with Gigabyte motherboards in the past, finding them very reliable.
MEMORY
RAM is the most expensive item in the entire system at £125 per 16G in Autumn 2014 prices. However, for my purposes so that I can run multiple VMs it has to be 32G. If you need to save some money then 16G will still be OK depending on what you want to do. and you can do a LOT with 16G. However, memory is king when it comes to performance - more memory equals less disk access, especially with multiple VM's loaded. I did not want to spend a fortune on high performance memory - having memory with the lowest possible latencies is not important so I settled for CL10 memory. The G.Skill 16G kits come as 2 x 8G sticks so you need two such kits for 32G. This will fully populate the four memory slots on the B85-D3H. I checked carefully on the Gigabyte memory compatibility list before selecting this combination and it can be a bit confusing. However, I can confirm that the combination I selected works very well.
HARD DISKS
Lets face it, you can never have enough storage. I did not want to buy a RAID card (due to cost and the extra power it would consume) so I chose to use standard HDDs. By teh way, ESXi doe snot support motherboard RAID either so forget that idea! :) I selected the Western Digital Caviar Blue series in the 1TB capacity. These are only £40 each. The higher performing WD Black series are much more expensive for the same storage capacity - more than double - and I was not prepared to pay that premium. So I just bought two of the "blues" and connected them up to the SATA 6G ports of the motherboard. These constitute two Datastores totalling 2Gbytes on which to store the VMs.
I also use a Crucial MX100 256G SSD to enable ESXi to write its swap to the SSD if it should run out of 32G of memory. Running out of memory is not likely because I will not massively overcommit the memory as it would be if running a shed load of VDI instances for instance. Also, for a home server 32G is a fair amount. However, I will select 50G of that SSD drive for the ESXi swap and will also allocate the smaller virtual machines storage on the SSD. I'd also love to be able to do host swap as well where each VM can be allocated some of that SSD in the event the individual VM needs to swap; however, that is not supported on evaluation ESXi hypervisors - it needs Enterprise plus licensing. I have also installed ESXi to the SSD to speed up loading. ESXi runs entirely from memory once loaded so this just speeds things up when twiddling thumbs after the VSphere client is loaded :) Certain VM's (for instance my small footprint Windows 2008 AD/DNS/DHCP server and the small Web Server (Win2008 again)) are installed onto the SSD. Larger VMs are loaded onto the HDDs.
LAN ADAPTER
Motherboard built-on LAN adapters rarely work with ESXi without special hacks. And even when you can hack them to work you can't support 802.1q tagging, a feature that I want to use so that some of my VM's can use the LAB network and the media server can use my home network (via my Cisco 3650C switch of course ;-) ). I therefore bought an Intel EXPI930 LAN adapter to facilitate this requirement. This is small and low power consuming and can be had for £20 or so. Importantly, it is also on the VMware Whitebox HCL.
 |
| LAN adapter bottom left of Picture |
CASE AND PSU
Because I want the server to be small I bought the SU02 from Silverstone. This is a lovely case to use. It is small but not overly so. You need to be sure that you use a Micro-ATX form factor motherboard since a regular ATX motherboard will not fit. Such a small case mandated that the PSU was a semi-modular sort so that I did not have to try and hide lots of unnecessary cables since there would not be the room to do so in this small case. There are reviews of this case on the web and I can confirm it is a nice case to work on - as long as the components fit!
 |
| Very nice PSU. |
 |
| Top view. The fan at top left is not included as standard. |
 |
| Server with case lid back on |
I have a confession. I'm a bit disappointed by the idle power of the server. It runs idle at about 50W which is more than I had hoped and too much for it to be left on 24x7 as a media server. I now wish I had a Gold PSU instead of a bronze. Not to worry, when I build another rig I will repurpose this PSU into that.
The machine runs whisper quiet and runs like a rocket! I am very impressed! :)
