Home Lab Gen IV – Part II: Design Considerations
I have decided to move my Home Lab away from Gen III into Gen IV. In doing this I am going to follow my best practices laid out in my ‘Home Lab Generations’ and ‘VMware Home Labs: a Definitive guide’. As you read through the “Home Lab Generations page” you should notice a theme around planning each generation and documenting its outcomes and unplanned items. In this blog post, I am going to start laying out Design Considerations which include the ‘Initial use case/goals and needed Resources as they relate to GEN IV.
Design Considerations:
Initial Use case / goals:
- Support vSphere 6.5 and vSAN 6.6 All Flash Environment
- Move networking vSAN and FT to high-speed InfiniBand
- Support headless ESXi Environment
Resources needed:
- To meet the initial use case/goals I’m will be investing quite a bit into this total refresh.
- Some initial GEN IV resource choices (Still in the works and not all proven)
- Reuse the 3 x Antec Cases / Power Supplies (No cost)
- BitFenix Case to support Windows 10 PC
- Mobo: Gigabyte MX31-BSO ($140 x 3 Ebay)
- RAM: 32GB DDR4 ($144 x 3 Jet.com)
- CPU: E3-1230v5 Xeon ($245 x 3 Jet.com)
- IBM M5210 SAS RAID (Found on Ebay $75 x 3)
- Mini SAS SFF-8643 to (4) 29pin SFF-8482 (Amazon 3 x $18)
- 12 x 200GB Enterprise Class SDD (Found on an Ebay lot deal)
- InfiniBand (All on Ebay)
- Mellanox IS5022 40Gb/s 8 Port Switch ($250) << Do not recommend, no Subnet Manager
- Mellanox ConnectX HCA rebranded as HP INFINIBAND 4X DDR PCI-E HCA CARD 452372-001 ($35 x 3)
- 6 x Infiniband Cables (Mellanox MCC4N26C-003 Cable ($60)
Notes around some of the choices above:
Mellanox IS5022 – I liked the footprint of this device over it not having built-in IB subnet manager. An IB subnet manager is required to manage the “network paths” on an IB network. Without an active IB subnet manager available your IB HCAs will not connect. Since the IS5022 I chose does not have an IB Subnet manager I will need a place to supply this service. I can choose an ESXi Host or a Windows Host. Since my ESXi hosts will be going up and down I plan to use my Windows PC as my subnet manager as it is always on and available.
Mellanox ConnectX HCA rebranded as HP INFINIBAND 4X DDR PCI-E HCA CARD 452372-001 – I initially choose these HCAs based on some other blog posts. They are at an attractive price point but they are much older and no longer have driver support. I was able to get them operational with ESXi 6.0 and will be soon working with them on ESXi 6.5 My advice is Mellanox has great products and support, however, I would recommend if you can afford it to go with a newer card that supports ESXi and save yourself the trouble of modifying ESXi software vibs. I’ll post more on this topic as I start deploying them with ESXi 6.5
Windows PC – I repurposed my Gigabyte Z68XP-UD3, i7 CPU, 90GB SSD, 16GB of DDR3 and then bought a cheap BitFenix case to build out my Windows PC. This PC will serve as my Plex Home Media Server and IB Subnet Manager. I also plan to run Workstation 10 and set up various service VM’s (AD, vCenter Server / VUM, DNS, etc). So far it’s working pretty well but this Mobo has been known to give me issues.
ESXi Hosts – I have 3 Antec Sonata cases, one that I have had since 2003 that I will reuse in this environment. I choose the following parts to make up my new ESXi Hosts: Mobo: Gigabyte MX31-BSO, 32GB DDR4, E3-1230v5 Xeon, and 4 x 200GB SSDs per host. This mobo is a bit limited on the ports but so far it seems to be working out well. For the boot disk, I plan to use the onboard USB port and a 64GB USB Stick. However, the question should be – What am I going to so with those 6 x 1TB SATA disks from GEN III — I put them into my old IX4
Here are a few PICs of the current build:
Next Post I’ll be going over the Gigabyte Motherboard.
If you like my ‘no-nonsense’ blog articles that get straight to the point… then post a comment or let me know… Else, I’ll start writing boring blog content.
vmexplorer 