vmware

Home Lab Gen V Build: Part 5 – Motherboard Install

Posted on Updated on

In this video I’ll be reviewing what it took to install the motherboard, what to look for when you buy a used CPU, and some tips on heat sinks/coolers.

More information on the overall components can be found here: https://vmexplorer.com/2020/01/27/home-lab-gen-v-the-quest-for-more-cores-part-i-design-considerations/

If you like my ‘no-nonsense’ videos and blogs that get straight to the point… then post a comment or let me know… Else, I’ll start posting really boring content!

Home Lab Gen V Build: Part 4 – First Install Steps

Posted on Updated on

In this video I’ll be covering the first installation steps I take to prepare the case and power supply for the build.

If you like my ‘no-nonsense’ videos and blogs that get straight to the point… then post a comment or let me know… Else, I’ll start posting really boring content!

Home Lab Gen V Build: Part 3 – Motherboard Overview

Posted on Updated on

In this video I’ll cover an overview of the X79 motherboard I choose from my Gen V Home lab.

If you like my ‘no-nonsense’ videos and blogs that get straight to the point… then post a comment or let me know… Else, I’ll start posting really boring content!

Home Lab GEN V Build: Part 2 – Case Selection

Posted on Updated on

In this video I’ll cover the Rosewill RISE Glow case. Its the case I’m using in my Generation V Home Lab build.

If you like my ‘no-nonsense’ videos and blogs that get straight to the point… then post a comment or let me know… Else, I’ll start posting really boring content!

Home Lab GEN V Build: Part 1 – Home Lab Gen V Build Components Overview

Posted on Updated on

In this vlog I’ll be covering all the components that go into my Gen V build.

If you like my ‘no-nonsense’ videos and blogs that get straight to the point… then post a comment or let me know… Else, I’ll start posting really boring content!

 

 

Home Lab GEN V: The Quest for More Cores! Design Considerations

Posted on Updated on

I have decided to update my Home Lab into Generation V. In doing this I am going to follow my best practices laid out in my ‘Home Lab Generations’ and ‘VMware Home Labs: a Definitive guide’. As you read through the “Home Lab Generations page” you should notice a theme around planning each generation and documenting its outcomes and unplanned items. In this blog post, I am going to start laying out Design Considerations which include the ‘Initial use case/goals and needed Resources as they relate to GEN V.

First off, lets answer why am I updating my home lab.  Over the past 4+ Home Lab generations I had deemed that CPU’s with 4 Physical Cores with up to 32GB RAM would meet the demands of my use cases and, in most cases it did.  However, most recently I starting having resource constraints when I wanted to use multiple VMware products.  This caused me to do a bit of shuffling to be able to run the software I wanted.  Now this is not the fault of VMware, its just that there are so many products that have resource demands and my current home lab was undersized.  Additionally, the fan noise from the InfiniBand switch and others was just to loud.

First – Here are my initial use case and goals:

  • Be able to run vSphere 6.x and vSAN Environment
  • Reuse as much as possible from Gen IV Home lab, this will keep costs down
  • Choose products that bring value to the goals, are cost effective, and if they are on the VMware HCL that a plus but not necessary for a home lab
  • Move networking (vSAN / FT) from 40Gb InfiniBand to 10Gbe Switch
  • Have enough CPU cores and RAM to be able to support multiple VMware products (ESXi, VCSA, vSAN, vRO, vRA, NSX, LogInsight)
  • Be able to fit the the environment into 3 ESXi Hosts
  • The environment should run well, but doesn’t have to be a production level environment

Second – Evaluate Software, Hardware, and VM requirements:

Before I run off and start buying items. I need to look at the software requirements on the hardware.  Using the table from my ‘HOME LABS: A DEFINITIVE GUIDE’, I can start to figure out how much CPU, RAM, and Disk space I’ll need.  Through experience working with these products I already know my Dual Port 10Gbe per host network is adequate to support these products.  Its the other items I’m concerned with in this build.

Using this information I can quickly see I need the following across all my hosts:

Lastly I figure I run between 20-30 VM’s for testing, these could be Windows, Linux, etc. which can be over subscribed

  • CPU: 30 x 2 vCPU = ~60 vCPU
  • RAM: 30 X 8GB = ~240 GB
  • Disk: 30 x 30GB = ~900GB

Total resource needs for the Cluster:

  • CPUs: 92 cores
  • RAM:  335GB
  • Disk: 2.2TB

Third – Home Lab Design Considerations

As you can see form the totals above my existing Gen IV Home Lab would not be able to keep up.  Lets do keep in mind the totals for the CPU/RAM are a 1:1 ratio and doesn’t take in consolidation.  For a home lab I should be able to reduce these numbers quite a bit. What I do next is review my Home lab Design Considerations.  This plus the information from step two will help me to decide which hardware to select.

Home Lab Design Considerations:

Design Considerations Description
Initial Cost How much does the Home lab solution cost to build out?

This is always top of mind for me and I do a lot of cost comparisons, research, and evaluation.  For this build I found that reusing what I have plus purchasing a few more items kept my cost lower with more value then buying new or even used hardware.

Noise When the home lab is running how much noise will it produce, and are the noise levels appropriate for your use case?

In my design I’m looking to reduce main fan noise.  My lab is in my home office so it needs to be whisper quiet.

Heat / Power Consumption Does the home lab produce to much heat for the intended location?

Heat/Power is always a balancing act.  I want something that will not heat up my room, has enough cores to to do the job but doesn’t consume so much power I don’t want to turn it on.

Monthly Operational Cost Based on Power (watts) and the average cost for electricity for the USA, cost is an estimate if running for 24x7x30 days?

My home lab power needs are as follows:

  • 3 x Gen V Servers ~135 Watts | 1 x Workstation PC ~75 Watts | Network / Misc = 45 Watts = 525 Watts
  • By going to my local power provider I find the most per kw is ~.24 cents
  • Using an online calculator the most I can expect to pay if I run the system for 8 hours a day is – $35 Month

APS Rate GuideElectricity Bill Calculator

Foot Print Space and Flexibility How much space does the solution take up. Based on the type of product you choose, how flexible is the solution when hardware or other changes are needed to expand?

What I’m really looking for are 3 x Tall tower PC Cases with maximum flexibility. This means where the power supplies are located, the amount of drives it can hold, fits many different motherboards, and has vertical / horizontal slots for Host Cards.

Bleeding Edge

VMware products

Software products are constantly adding new requirements for home labs (example: 10Gbe Networks, or more HDD/SDD) How does the solution align to bleeding edge products without major over haul?

If I build this system beefy enough I should be able to be in the position to run just about any software that comes my way.

Hands on Software Measures viability from the ESXi layer through the entire stack of products

My new system should be designed to accommodate the software stack mentioned in step 2

Hands on Hardware Considers the effectiveness of the hardware solution to real world technologies

Choosing a system that allows the most flexibility is key here to be future looking

ESXi / vSAN HCL Support How does the hardware align to the Hardware compatibility guides

Not top of mind for a Home lab as I’m not looking for VMware to support it.  However, the closer I can get to the HCL the better off I will be.

HCI

Hyper Converged Infrastructure

How well does the solution adapt to HCI (vSAN)

 

I should ensure that JBOD Disk Controller and NIC both have a PCIe 8x slot or better and I can fit many drives into my case.

Refresh Cost Financially, what would it take to refresh, replace, or update the hardware solution

Consider how adaptable is the solution to changing hardware and software demands.

I want to choose products that are cost effective but I can reuse down the road. This should put my lab in a potion to keep costs down.

Step Four – Choosing Hardware

Based on my estimations above I’m going to need a very flexible case, Dual CPU Mobo, lots of RAM, and good network connectivity.

Here is what each Host will have:

  • Rosewill RISE Case
  • JINGSHA EATX X79 Dual CPU motherboard
  • 128GB DDR3 ECC RAM
  • 4 x 200 SAS SSD
  • 4 x 600 SAS HDD
  • 1 x IBM 5210 JBOD
  • 1 x HP 593742-001 NC523SFP DUAL PORT SFP+ 10Gb
  • Connect into a MikoTik 10gbe CN309

Here are the resources I’ll need to build out my 3 hosts:

  • To meet the initial use case/goals I’m will be investing quite a bit into this total refresh.
  • Here are some of the initial GEN V resource choices (Still in the works and not all proven out)
    • Purchase Items:
      • Mobo: JINGSHA EATX X79 Dual CPU motherboard LGA 2011 Supports Xeon v2 processor ($86 Alibaba)
      • Mobo Stands: 4mm Nylon Plastic Pillar (Amazon $8)
      • RAM: 128GB DDR3 ECC (Ebay $110)
      • CPU: Xeon E5-2640 v2 8 Cores / 16 HT (Ebay $30)
      • CPU Cooler: DEEPCOOL GAMMAXX 400 (Amazon $19)
      • Video: ASUS Neon PCIe 1x with DMS-59 Splitter (Ebay $15)
      • Video Riser: PCI-E 1x to 16x Riser Adapter (Amazon $4)
      • DISK: 600GB SSD (Ebay $80 for 10 Drives)
      • Power Supply Adapter: Dual 8(4+4) Pin Male for Motherboard Power Adapter Cable (Amazon $11)
      • Power Supply Extension Cable: StarTech.com 8in 24 Pin ATX 2.01 Power Extension Cable (Amazon $9)
      • CableCreation Internal Mini SAS SFF-8643 to (4) 29pin SFF-8482 (Amazon $18)
      • Case: Rosewill RISE Glow EATX (Newegg $54)
    • Existing Items I’ll move over from the old 3 Hosts:
      • Power Supply’s
      • 200GB SAS SSD
      • 600GB SAS HHD
      • 2TB SATA HDD
      • 64GB USB Thumb Drive
      • IBM 5210 JBOD Disk Controller
      • CableCreation Internal Mini SAS SFF-8643 to (4) 29pin SFF-8482 connectors with SATA Power,1M
      • HP 593742-001 NC523SFP DUAL PORT SFP+ 10Gb SERVER ADAPTER W/ HIGH PROFILE BRACKET
      • HP 684517-001 Twinax SFP 10gbe 0.5m DAC Cable Assembly

The total cost for me to upgrade each server using purchased and existing items came out to ~$425 US Each.  If you built this configuration without existing items the cost would be around ~$850 US.  Clearly, you can see reusing my existing hardware and taking a step back with older Xeon/DDR3 RAM it saved quite a few dollars.

Next Steps for me is to finalize my orders and start the assembly process.  I’ll post up soon around my progress.Here are a few initial photos from the build.

For now here are a few pre-deployment pics- ~Enjoy!

 

 

If you like my ‘no-nonsense’ videos and blogs that get straight to the point… then post a comment or let me know… Else, I’ll start posting really boring content!

Home Lab GEN V: The Quest for More Cores! – First Look

Posted on

Over the past 10+ years I’ve posted all types of information around my home lab builds.  Check out my guide for more information. Most recently I decided to update my home lab to be able to support the resource demands of so many great vmware products.  I’m not quite ready to go into detail around my new home lab but  in this video blog I wanted to give you a first look at some of the components.  Soon I post up several blogs around all the details.  So, for now enjoy the first look, it’s a bit of a rough video but I’m a technologist not a Hollywood director :)

If you like my ‘no-nonsense’ videos and blogs that get straight to the point… then post a comment or let me know… Else, I’ll start posting really boring content!

Create an ESXi installation ISO with custom drivers in 9 easy steps!

Video Posted on Updated on

One of the challenges in running a VMware based home lab is the ability to work with old / inexpensive hardware but run latest software. Its a balance that is sometimes frustrating, but when it works it is very rewarding. Most recently I decided to move to 10Gbe from my InfiniBand 40Gb network. Part of this transition was to create an ESXi ISO with the latest build (6.7U3) and appropriate network card drivers. In this video blog post I’ll show 9 easy steps to create your own customized ESXi ISO and how to pin point IO Cards on the vmware HCL.

 

Here are the written steps I took from my video blog.  If you are looking for more detail, watch the video.

Before you start – make sure you have PowerCLI installed, have download these files,  and have placed these files in c:\tmp.

I started up PowerCLI and did the following commands:

1) Add the ESXi Update ZIP file to the depot:

Add-EsxSoftwareDepot C:\tmp\update-from-esxi6.7-6.7_update03.zip

2) Add the LSI Offline Bundle ZIP file to the depot:

Add-EsxSoftwareDepot ‘C:\tmp\qlcnic-esx55-6.1.191-offline_bundle-2845912.zip’

3) Make sure the files from step 1 and 2 are in the depot:

Get-EsxSoftwareDepot

4) Show the Profile names from update-from-esxi6.7-6.7_update03. The default command only shows part of the name. To correct this and see the full name use the ‘| select name’ 

Get-EsxImageProfile | select name

5) Create a clone profile to start working with.

New-EsxImageProfile -cloneprofile ESXi-6.7.0-20190802001-standard -Name ESXi-6.7.0-20190802001-standard-QLogic -Vendor QLogic

6) Validate the LSI driver is loaded in the local depot.  It should match the driver from step 2.  Make sure you note the name and version number columns.  We’ll need to combine these two with a space in the next step.

Get-EsxSoftwarePackage -Vendor q*

7) Add the software package to the cloned profile. Tip: For ‘SoftwarePackage:’ you should enter the ‘name’ space ‘version number’ from step 6.  If you just use the short name it might not work.

Add-EsxSoftwarePackage

ImageProfile: ESXi-6.7.0-20190802001-standard-QLogic
SoftwarePackage[0]: net-qlcnic 6.1.191-1OEM.600.0.0.2494585

8) Optional: Compare the profiles, to see differences, and ensure the driver file is in the profile.

Get-EsxImageProfile | select name   << Run this if you need a reminder on the profile names

Compare-EsxImageProfile -ComparisonProfile ESXi-6.7.0-20190802001-standard-QLogic -ReferenceProfile ESXi-6.7.0-20190802001-standard

9) Create the ISO

Export-EsxImageProfile -ImageProfile “ESXi-6.7.0-20190802001-standard-QLogic” -ExportToIso -FilePath c:\tmp\ESXi-6.7.0-20190802001-standard-QLogic.iso

That’s it!  If you like my ‘no-nonsense’ videos and blogs that get straight to the point… then post a comment or let me know… Else, I’ll start posting boring video blogs!

Going to 10Gbe from 40Gb with the MikroTik CRS309-1G-8S+IN

Posted on

As my Home Lab has evolved I’m always looking to make it better and at the same time learn something new. When I started work on Home Lab Gen IV (2016) I really wanted to have a high speed network to support vSAN. At that time 10Gbe networking was cost prohibitive and InfiniBand networking was faster, something that was new to me, and its cost was in the ball park I wanted to spend.

While I was able to get my InfiniBand network to work, it had a couple of the downsides – ageing technology, lack of support, difficult to work with, and power/noise was not optimal. Most of these I overcame but what I really needed was affordable 10Gbe Cards, Cables and a fan-less/low power Switch.

I know its a pipe dream, right? Well, this past week I got a new MikroTik CRS309-1G-8S+IN!  Its 8 Ports of 10Gbe SFP+, no fan, and only 17 watts. Best part, its in a price range for most home lab folks. I found it doing a few searches and some of my fellow Home Lab enthusiasts are using it too.

In this video I chat a bit about InfiniBand vs. 10Gbe, I unbox the device and in the weeks to come I’ll post up some more videos around its use. 

If you like my ‘no-nonsense’ videos that get straight to the point… then post a comment or let me know… Else, I’ll start posting boring video blogs!

FIX for Netgear Orbi Router / Firewall blocks additional subnets

Posted on Updated on

Last April my trusty Netgear Switch finally gave in.  I bought a nifty Dell PowerConnect 6224 switch and have been working with it off an on.  About the same time, I decided to update my home network with the Orbi WiFi System (RBK50) AC3000 by Netgear.  My previous Netgear Wifi router worked quite well but I really needed something to support multiple locations seamlessly.

The Orbi Mesh has a primary device and allows for satellites to be connected to it.  It creates a Wifi mesh that allows devices to go from room to room or building to building seamlessly.  I’ve had it up for a while now and its been working out great – that is until I decided to ask it to route more than one subnet.   In this blog I’ll show you the steps I took to over come this feature limitation but like all content on my blog this is for my reference – travel, use, or follow at your own risk.

To understand the problem we need to first understand the network layout.   My Orbi Router is the Gateway of last resort and it supplies DHCP and DNS services. In my network I have two subnets which are untagged VLANS known as VLAN 74 – 172.16.74.x/24 and VLAN 75 – 172.16.75.x/24.   VLAN 74 is used by my home devices and VLAN 75 is where I manage my ESXi hosts.  I have enabled RIP v2 on the Orbi and on the PC6224 switch.  The routing tables are populated correctly, and I can ping from any subnet to any host without issue.

Issue:  Hosts on VLAN 75 are not able to get to the internet.  Hosts on VLAN 75 can resolve DNS names (example: yahoo.com) but it cannot ping any host on the Inet, where VLAN 74 can ping Inet hosts and get to the internet.  I’d like for my hosts on VLAN 75 to have all the same functionally as my hosts on VLAN 74.

Findings:  By default, the primary Orbi router is blocking any host that is not on VLAN 74 from getting to the INET.  I believe Netgear enable this block to limit the number of devices the Orbi could NAT.  I can only guess that either the router just can’t handle the load or this was a maximum Netger tested it to.  I found this block out by logging into the routers CLI and looking at the IPTables settings.  There I could clearly see there was firewall rule blocking hosts that were not part of VLAN 74.

Solution:  Adjust the Orbi to allow all VLAN traffic (USE AT YOUR OWN RISK)

  1. Enable Telnet access on your Primary Orbi Router.
    1. Go to http://{your orbi ip address}/debug.htm
    2. Choose ‘Enable Telnet’ (**reminder to disable this when done**)
    3. Telnet into the Orbi Router
  2. I issued the command ‘iptables -t filter -L loc2net’. In the output of this command you can see where its dropping all traffic that is not (!) VLAN74
  3. Let’s remove this firewall rule. The one I want to target is 5th in the list, yours may vary.  This command will remove it ‘iptables -t filter -D loc2net 5’
  4. Next, we need to clean up some post routing issues ‘iptables -t nat -I POSTROUTING 1 -o brwan -j MASQUERADE’
  5. A quick test and I can now PING and get to the internet.
  6. Disconnect from Telnet and Disable it on your router.

Note:  Unfortunately, this is not a permanent fix.  Once you reboot your router the old settings come back.  The good news is, its only two to three lines to fix this problem.  Check out the links below for more information and a script.

If you like my ‘no-nonsense’ blog articles that get straight to the point… then post a comment or let me know… Else, I’ll start writing boring blog content.

REF: