Whitebox

Home Lab Gen IV – Part IV: Overcoming installation challenges

Posted on

One of the joys of working with a home lab is doing something that no one has done before. Sure, your configuration might be similar to others, but in a way your home lab is unique. However, with this uniqueness comes its share of installation challenges. My new lab was no exception, there were a few challenges and one major issue I uncovered while setting up this new environment. In this blog post I am going review the environment I am working on, break down some of the hardware layout placement challenges, fun using the MAC PowerBook to complete the installation, and finally overcoming ESXi installation challenges.

Here is my new environment:

  • Mac Powerbook with macOS Sierra (Used for remote connection into my environment, normally I use a PC)
  • Gigabyte MX31-BS0
  • Intel Xeon E3-1230 v5
  • 32GB DDR4 RAM
  • 1 x Mellanox Connectx InfiniBand HCA
  • 4 x 200GB SSD, 1 x 64GB USB (Boot)
  • 1 x IBM M5210 JBOD SAS Controller
  • 1 x Mini SAS SFF-8643 to (4) 29pin SFF-8482
  • 1 x 64GB USB Boot Stick:

Hardware layout/placement challanges:

32GB of RAM: Pay attention to the placement of the RAM. As Channel 1 for the RAM are the two closest slots to the CPU, channel 2 being the two farthest away. Normally you would place the RAM pairs in like colors however this Mobo is a bit different

Mellanox Connectx InfiniBand HCA: Placed it in the 16x slot right next to the CPU. The HCA requires an 8x slot so this slot should not slow it down. No BIOS changes were required and I could see this HCA in the BIOS.

IBM M5210 JBOD SAS Controller: Placed it in the 8x slot which goes through the C232 chipset on the motherboard. Next, I needed to update the firmware but this proved to be a challenge. Keep in mind the M5210 with NO cache will not allow you to enter its BIOS management page (aka MegaRAID webbios). This means you’ll need to use the command line or other software to update and view its information. Initially, I tried several command line options (UEFI Shell, DOS CLI, etc.) with the MegaRAID CLI but I just could not find the right combination to get it to work. My solution — I simply used an older SSD drive, installed Windows Server 2012 on it, and used the Windows exe to update the firmware. It worked perfectly with no issues.

After the update, I had some issues decoding the M5210 running firmware version vs. the vSAN HCL. As you know when running vSAN in a home lab the closer you are to the HCL and vSAN HCL the better. (NOTE: as I’m sure you know production environments MUST match the HCLs). The published firmware version on the vSAN HCL is 4.660.00-8218. However, when the M5210 boots it shows 24.16.0-0104.

Solution: When you are looking at the boot screen you are seeing the FW Package number not the Firmware of your controller. Simply look at the release notes for the ‘FW Package’ and you’ll find the correct MR FW versions that match the vSAN HCL.

Boot Screen

Release notes

200GB SSD: The Sonata cases I am using are a bit dated but they fully meet my needs so there is no need to replace them. There are 4 x 3.5″ bottom mount disk trays in each case. Bottom mount means you insert your 3.5″ drive into the tray and bolt it to the tray from the bottom. I bought several 3.5″ to 2.5″ converters which will allow me to mount my 2.5″ SSDs. However, the converters didn’t have bottom mount holes that lined up with the standard 3.5″ holes. Fix — I used a hole in the existing tray to secure the converter to the tray. I also made sure I mounted the converter as far back as I could to ensure the SAS cables would not be on the side of the case. This mount position moved the drives back about 1.5″(38mm). The red line in the PIC show where the original mount point was.

Mini SAS SFF-8643 to (4) 29pin SFF-8482: From the PIC above you can see the disk end of the SAS cables. What is nice about them is each one has a disk number labeled and has integrated power and all 4 drives go back to a single connector. The only downside to the cable I bought was they seemed a bit frail, so I’d recommend if you plan to mod your environment frequently look into a better-quality cable. If you interested more in SAS and the associated cables I would recommend this wiki page – https://en.wikipedia.org/wiki/Serial_Attached_SCSI

64GB USB Boot Stick: I decided to use the internal USB port freeing up the rear ports for other items. The USB stick I am using is the SanDisk Ultra Fit 64GB USB 3.0 Flash Drive. ESXi will only take up ~10GB of this stick, so is 64GB overkill? Keep in mind I plan to run vSAN 6.6.x and one of the benefits is the log files now write to RAM and in case of a system failure, they can write these logs to the USB stick. However, the default partition sizes (2.5GB for diags) might not be large enough. The vSAN team as released a nifty script that will estimate and resize you USB partitions. I’ll cover this topic in later posts and show you how to “auto-resize” your USB storage after you have installed vSAN.

Fun with the MAC:

Function Keys: One of the challenges was MAC keyboard mapping into the remote KVM. For some reason, the function keys on a MAC always assume you want to their special function vs. the F# key you are pushing. This proves to be a challenge when you are trying to pass standard function keys. Simple fix: System Preferences > Keyboard > Ensure ‘Use F1, F2, etc. as standard function keys’ is checked.

Another option for F# keys is to create a macro inside of the vKVM Viewer to pass the key. The screenshot below shows where you can setup user defined Macros and in the background is the MeregPoint console for one of my ESXi hosts.

Java: One of the joys of this motherboard is the use of vKVM viewer and VM Media. However, these functions need JAVA installed and up to date to function properly. If your JAVA is behind, trust me just update it’ll save you hours of pain. Here is the remaining gotcha. In the Mergpoint web page, you simply click on the ‘Launch Java vKVM Viewer’ button to start your host remote session. The webpage will download a .jnlp file. If you just click on this file you are presented with an error stating it can’t be opened because it is from an unidentified developer. Solution – After the java app downloads, click on the down arrow next to the file and choose ‘Show in finder’. When finder launches select that file by holding down the control key and right-clicking on it. A pop-up window will appear, release the control key and finally choose open. This allows you to override the ‘unidentified developer’ error and launch the viewer.

ESXi Installation:

Setting up the ESXi hosts had one big challenge – after the install of ESXi I could not see my SAS disks. I am using the ESXi 6.5U1 Rollup.iso to do my installs and my main goal was to install and boot ESXi from the 64GB USB stick and be able to access the 4 x 200GB SSD attached to the IBM M5210

Problem – During the install of ESXi, I booted the host using the ESXi6.5 ISO via virtual media console. The installer program would recognize the IBM M5210 controller, the attached 4 x SAS disks, and the 64GB USB stick. The installation would complete without issue. However, after ESXi booted the SAS disks and the controller would not appear but I could see the 64GB USB stick.

Other observations –

First, in the ESXi Log files I noticed the megasas was having firmware issues:

2017-09-21T10:26:31.310Z cpu5:66065)<6>megasas: Waiting for FW to come to ready state 2017-09-21T10:26:31.310Z cpu5:66065)<7>megasas: FW in FAULT state!!

ESC[7m2017-09-21T10:26:31.310Z cpu5:66065)WARNING: vmklinux: pci_announce_device:1486: PCI: driver megaraid_sas probe failed for device 0000:07:00.0ESC[0m 2017-09-21T10:26:31.310Z cpu5:66065)LinPCI: LinuxPCI_DeviceUnclaimed:257: Device 0000:07:00.0 unclaimed.

And… even though ESXi saw the M5210 as vmhba1, its status was unknown

vmhba1 Avago (LSI) MegaRAID SAS Invader Controller

vmhba1 0000:07:00.0 PCI 0:0:29:0 PCI 0:7:0:0 Slot1 UNKNOWN

Second, I use Partition Wizard bootable ISO to remove all partitions prior to installing ESXi. I noted that sometimes after I booted to it as virtual media it would see the 4 x SAS disks and other times it would not.

Third, Installation of ESXI onto SAS or SATA SSD as the boot disk worked perfectly. After booting I could see the M5210 and SAS disks but my goal of using the 64GB USB stick for the boot device was not achieved.

Fourth, occasionally when I booted the ESXi host to the USB stick it would work okay, but upon reboot would not

Final Solution – The core reason why I could not see the SAS disks with ESXi or Partition Wizard was the boot type was UEFI and not legacy. During boot time the boot order would sometimes change if I had virtual media connected, meaning sometimes it would boot the 64GB USB stick or Partition Wizard as UEFI and other times as legacy. Apparently, UEFI boot was giving the M5210 firmware issues not allowing the SAS disk to come online.

FIX – I went into the BIOS of the motherboard > Advanced > CSM Configuration > changed ‘Boot option filter’ to ‘Legacy Only’ and all my issues went away.

Summary – I spent a lot of after-hours and weekends working out all various installation tweaks but what can I say, this is the joy of setting up a home lab! My hopes are in some way this post helps you move your home lab forward too. In my next post, I’ll be going over how to enable the InfiniBand HCA in ESXi 6.5.

If you like my ‘no-nonsense’ blog articles that get straight to the point… then post a comment or let me know… Else, I’ll start writing boring blog content.

Home Lab Gen IV – Part II: Design Considerations

Posted on

I have decided to move my Home Lab away from Gen III into Gen IV. In doing this I am going to follow my best practices laid out in my ‘Home Lab Generations’ and ‘VMware Home Labs: a Definitive guide’. As you read through the “Home Lab Generations page” you should notice a theme around planning each generation and documenting its outcomes and unplanned items. In this blog post, I am going to start laying out Design Considerations which include the ‘Initial use case/goals and needed Resources as they relate to GEN IV.

Design Considerations:

Initial Use case / goals:

  • Support vSphere 6.5 and vSAN 6.6 All Flash Environment
  • Move networking vSAN and FT to  high-speed InfiniBand
  • Support headless ESXi Environment

Resources needed:

  • To meet the initial use case/goals I’m will be investing quite a bit into this total refresh.
  • Some initial GEN IV resource choices (Still in the works and not all proven)
    • Reuse the 3 x Antec Cases / Power Supplies (No cost)
    • BitFenix Case to support Windows 10 PC
    • Mobo: Gigabyte MX31-BSO ($140 x 3 Ebay)
    • RAM: 32GB DDR4 ($144 x 3 Jet.com)
    • CPU: E3-1230v5 Xeon ($245 x 3 Jet.com)
    • IBM M5210 SAS RAID (Found on Ebay $75 x 3)
    • Mini SAS SFF-8643 to (4) 29pin SFF-8482 (Amazon 3 x $18)
    • 12 x 200GB Enterprise Class SDD (Found on an Ebay lot deal)
    • InfiniBand (All on Ebay)
      • Mellanox IS5022 40Gb/s 8 Port Switch ($250) << Do not recommend, no Subnet Manager
      • Mellanox ConnectX HCA rebranded as HP INFINIBAND 4X DDR PCI-E HCA CARD 452372-001 ($35 x 3)
      • 6 x Infiniband Cables (Mellanox MCC4N26C-003 Cable ($60)

Notes around some of the choices above:

Mellanox IS5022 – I liked the footprint of this device over it not having built-in IB subnet manager.  An IB subnet manager is required to manage the “network paths” on an IB network. Without an active IB subnet manager available your IB HCAs will not connect. Since the IS5022 I chose does not have an IB Subnet manager I will need a place to supply this service.   I can choose an ESXi Host or a Windows Host.  Since my ESXi hosts will be going up and down I plan to use my Windows PC as my subnet manager as it is always on and available.

Mellanox ConnectX HCA rebranded as HP INFINIBAND 4X DDR PCI-E HCA CARD 452372-001 –  I initially choose these HCAs based on some other blog posts.  They are at an attractive price point but they are much older and no longer have driver support.  I was able to get them operational with ESXi 6.0 and will be soon working with them on ESXi 6.5   My advice is Mellanox has great products and support, however, I would recommend if you can afford it to go with a newer card that supports ESXi and save yourself the trouble of modifying ESXi software vibs.  I’ll post more on this topic as I start deploying them with ESXi 6.5

Windows PC – I repurposed my Gigabyte Z68XP-UD3, i7 CPU, 90GB SSD, 16GB of DDR3 and then bought a cheap BitFenix case to build out my Windows PC.  This PC will serve as my Plex Home Media Server and IB Subnet Manager.  I also plan to run Workstation 10 and set up various service VM’s (AD, vCenter Server / VUM, DNS, etc).  So far it’s working pretty well but this Mobo has been known to give me issues.

ESXi Hosts – I have 3 Antec Sonata cases, one that I have had since 2003 that I will reuse in this environment.   I choose the following parts to make up my new ESXi Hosts: Mobo: Gigabyte MX31-BSO, 32GB DDR4, E3-1230v5 Xeon, and 4 x 200GB SSDs per host. This mobo is a bit limited on the ports but so far it seems to be working out well.  For the boot disk, I plan to use the onboard USB port and a 64GB USB Stick.  However, the question should be – What am I going to so with those 6 x 1TB SATA disks from GEN III — I put them into my old IX4 

Here are a few PICs of the current build:

Next Post I’ll be going over the Gigabyte Motherboard.

If you like my ‘no-nonsense’ blog articles that get straight to the point… then post a comment or let me know… Else, I’ll start writing boring blog content.

Create ISO with ESXi 6.0U2 and Realtek 8168 Drivers

Posted on Updated on

ESXi no longer supports RealTek Network drivers. To be able to install ESXi with the drivers on the installable users will have to create a custom ISO. Keep in mind these are unsupported drivers by VMware, so use at your own risk. Normally I would use my trusty ESXi-Customizer GUI program but it is no longer supported for ESXi 6 there CLI is still supported. However, I’d rather rely on the VMware supported tools vs. 3rd party as you never know when 3rd party tools will go away. VMware enables users to create custom ISOs via PowerCLI and in this blog I’ll explain how I used PowerCLI create my ISO. Keep in mind these are the steps that worked for me, your environment may vary.

To get started you will need two files and PowerCLI Installed on a Windows PC.

  1. File 1: VMware 6.0U2 Offline Bundle ZIP >> www.vmware.com/download

2. File 2: RealTek 8186 Offline bundle >> https://vibsdepot.v-front.de/wiki/index.php/Net55-r8168

3. PowerCLI Download and install >> https://communities.vmware.com/community/vmtn/automationtools/powercli

Tip: If you don’t know PowerCLI try starting here

4. Next Place the files from Step 1 and 2 into c:\tmp folder

–POWERCLI COMMANDS— For each command below I included a screenshot and the actual command allowing to copy, paste, and edit into your environment.

  1. Add ESXi 6.0u2 and RealTek8186 products to the local Software Depot

Add-EsxSoftwareDepot C:\tmp\update-from-esxi6.0-6.0_update02.zip

Add-EsxSoftwareDepot C:\tmp\net55-r8168-8.039.01-napi-offline_bundle.zip

2. Confirm the products are in the depot

Get-EsxSoftwareDepot

3. List out the ESXi Image Profiles

Get-EsxImageProfile

4. Create a Clone Image to be modified – Ensure you are targeting the “ESXi…..standard” profile from step 3

New-EsxImageProfile -cloneprofile ESXi-6.0.0-20160302001-standard -Name “RealTek8186a”

Forward-Looking Tip: Whatever name you choose it will show up in your boot ISO

5. Set the Acceptance Level to Community Supported – Remember RealTek is unsupported by VMware

Set-EsxImageProfile -Name RealTek8186a -AcceptanceLevel CommunitySupported

For ImageProfile Enter – RealTek8186a

6. Ensure the RealTek net55-r8186 driver is loaded from the local depot (Screenshot shortened)

Get-EsxSoftwarePackage

7. Add the RealTek software package to the profile

Add-EsxSoftwarePackage

ImageProfile: RealTek8186a

SoftwarePackage[0]: net55-r8168 8.039.01-napi

Tip: You MUST enter the full name here, if you just use the short name it will not work

9. Validate the RealTek drivers are now part of the RealTek8186a Profile (Screenshot shortened)

(Get-EsxImageProfile “RealTek8186a”).viblist

10. Export the profile to an ISO

Export-EsxImageProfile -ImageProfile “RealTek8186a” -ExportToIso -FilePath c:\tmp\RealTek8186a.iso

And that’s it… From here you can burn your ISO to a media of your choice or boot the ISO via ilo, iDRAC, virtual media etc.

If you like my ‘no-nonsense’ blog articles that get straight to the point… then post a comment or let me know… Else, I’ll start writing boring blog content.

VMware Home Labs: A definitive guide

Posted on Updated on

For nearly 10 years, I’ve compiled lots information about my home labs. From time to time I would blog about items I was working. However, readers of my blog were starting to respond with – “We enjoy your blogs around home labs. However, it’s a bit hard contextualize this information from multiple blog posts.” And “where do I start with home labs” Most recently the VMware User Group asked me to co-record a session for home lab vs. HOL Of course, I did the Home Lab side of the recording but it got me thinking more about the listener and how I could really put a focus on Who, what, why, and how around home labs. After the recording, I realized my “Home Lab Generations page” (HLG)was in major need of an overhaul.

This is when I decided to change my Home Lags Guide to “HOME LABS: A DEFINITIVE GUIDE”. It took me a few months to transform the HLG into this new approach as it puts a strong focus on Home Lab design considerations.  From there I used these design considerations to document and measure my 5 Generations of home labs dating back to 2008. In doing this I got a sense this design guide could help others to measure and plan out their home labs too.

Next Steps for the Home Lab Definitive guide:

  • I’ll continue to update it as I progress my home labs
  • I’ll continue to improve it over time but I’d love your feedback too

My hopes are you find this new guide useful and of value. Please post up your suggestions to help its continuous growth!

Also, if you are interested in applying these design practices to your home lab or even if you are just starting out, then reach out to me as I’d love to feature your design here on my blog!

If you like my ‘no-nonsense’ blog articles that get straight to the point… then post a comment or let me know… Else, I’ll start writing boring blog content.

Home Lab: A List of uncommon or niche products

Posted on

Part of the joy of building out a home lab or virtualization workstation is finding those one-off items that enable you to build something great, cheap, and unique. Below is a list of some those niche items and distributors I’ve found along the way. I’ll continue to update this post as we go along and I encourage you post up some of your findings too!

Sybausa.com

This place is full of all types of unique adapters and gadgets to make your home lab or workstation PC better. What I like about their product line focus is the support of cards with a PCIe x1 slot. Various server based add on cards (example 2/4 port NIC cards) typically require a PCIe x4 or x8 port. However, most home labs typically have plenty of x1 slots and very little to no support for x4 and x8. Syba seems to make a “plethora” of add on cards that support x1. There only downside poor documentation / support.

Some products I like from them —

  • 2 Port Gbe PCIe x1 card (SY-PEX24028): I own and use several of these, they seem to work quite well. Dislikes – No Jumbo frames and it uses a Realtek 8111e chips set which means you must add these drivers to support ESXi
  • Another cool item they make is a M.2 to 4-port SATA III Adapter. This little RAID controller allows you to plug directly into a M.2 port and allow for 4 mort SATA devices. I think this would be handy for smaller systems (ie. NUC builds)

StarTech.com

StarTech is really becoming a great company with a very diverse and well supported / documented product line. I think they are really starting to give Blackbox a run for their money. I really like their cable and adapter card lines.

I’ve been using their Startech Null Modem DB9 to USB to run the CLI on my Netgear manage Switch since 2012 and have yet to have an issue with it.

William Lam has blogged many times around the use of NUC style home labs with StarTech Single and Dual USB 3.0 network adapters.

 

Winyao

Winyao is a “boutique” distributor specializing in NICs, Fibre adapters, and Transceivers. One item I find of value is their PCIe x1 Dual NIC with Intel or Broadcom chipset. Personally, I don’t know much about this company or own any of their products, but at $40-$60 per brand new adapter I wished I had found them before buying the Syba adapters.

 

Fractal Design

If you are looking for your next server, workstation, media, or top of the line PC case then take a peek at Fractal Design. Founded in 2007 and based out of Sweden they have really started to dominate the custom case design market. Their innovative designs blend elegance with flexibility, which I might add is a hard combination to find. I like their Arc Midi and Arc Mini R2 line of cases for home lab build outs. However, when or if my trusty Antec Sonata from 2003 lets me down, then Fractal will be next on my list. Here is a great blog post from Erik Bussink around his use of Fractal Design for his 2014 Home Lab.

 

 

** 09/06/2017 – Here are some updates to this list **

BitFenix – Cases and products

Came across this interesting case / mod company that builds all kinds of custom cases, cables, etc to mod your PC’s. I like the Prodigy Mini-ITX case, with 2 PCI Slots and a spare slot for Disk or other mods it could be a good fit for a NAS project. However I’m not fond of the excessive top and bottom ornaments.


ASUS

ASUS came out with a great M.2 to U.2
option allowing users to interface with SAS disks. They claim this option will help users to interface with SAS SSD and get extreme performance. There are some contrains around this (cables, disks, chipsets, etc) so read up on this before you buy.


 

If you like my ‘no-nonsense’ blog articles that get straight to the point… then post a comment or let me know… Else, I’ll start writing boring blog content.

 

Gigabyte Firmware / BIOS update for MergePoint Embedded Management Software and Motherboard

Posted on

You’d think by now manufactures would have a solid and concise process around updating their products. They are quick to warn users to not update their BIOS unless there is a problem and quick to state if there is a problem they usually won’t support it. This total cycle of disservice is a constant for low-end manufacturers, heck even some high server platforms have the same issues. I have these same concerns when I started to look into updating my current MX31-BS0 Motherboard (mobo).

What can soften this blow a bit? How about the ability to update your BIOS remotely? This is a great feature of the MX-31BS0 and in this blog post, I’ll show you how I updated the BIOS and the remote MergePoint EMS (MP-EMS) firmware too.

Initial Steps –

  • My system is powered off and the power supply can supply power to the mobo.
  • I have setup remote access to the MP-EMS site with an IP address and have access to it via a browser. Additionally, I have validated the vKVM function works without issue
  • I downloaded the correct Mobo BIOS and BMC or MP-EMS Firmware and have extracted these files
  • Steps below were completed on a Gigabyte MX31-BS0 from BIOS F01 > F10 and MP-EMS 8.01 > 8.41, your system may vary

1 – Access the MergePoint EMS site

Start out by going to the IP address for MP-EMS site. From the initial display screen, we can see the MP-EMS Firmware versions but not the Platform (or Mobo) BIOS Version. Why not you may ask? Well, the MP-EMS will only display Mobo information when the Mobo is powered on. Before you power on your Mobo I would recommend opening the vKVM session so that you can see the boot screen. When you power on your mobo (MP-EMS > Power > Control > Power On ) use the vKVM screen to halt at the ‘boot menu’ or even go into setup and disable all the boot devices.

In this PIC, we can see my Firmware for the MP-EMS is 8.01 and the BIOS is blank as the Mobo is not powered on.

2- Selecting the Mobo BIOS Update

I choose the following to update the Mobo BIOS. Start out by uploading the file: Update > ‘BIOS & ME’ > Choose File > Image.RBU > Upload

Once the upload is complete, click on ‘Update’ to proceed. NOTE: a warning dialog box appeared for me stating the system would be powered off to update the BIOS. Good thing I’m in the Boot Menu as the system will just directly power off with no regard of the system state

3 – Installing the Mobo BIOS Update: Be Patient for the BIOS install to complete

Once I saw the message the ‘BIOS firmware image has been updated successfully’ I then exited the browser session and vKVM .  Note: I’d recommend closing the browser out entirely and then reopening a new session.


Once I restarted my vKVM and MP-EMS sessions and then powered on my Mobo. This allowed the BIOS update to continue.

Here is the patience part – My system was going from BIOS F01 > F10 and it rebooted 2 times to complete the update. Be patient it will complete.

Here is the behavior I noted:

  • First Reboot – The system posted normally, it cleared the screen, and then white text stated a warning message about the BIOS booted to default settings. Very shortly after it rebooted again.
  • On the 2nd reboot, it posted normally and I pressed F10 to get back to the Boot menu. I did this because next, we’ll need to update the MP-EMS firmware.

Once the system had rebooted I then refreshed my MP-EMS screen and viola there it was BIOS Version F10.

4 – Selecting the MP-EMS Firmware

While the Mobo is booted and I’m in the boot menu, I went into the MP-EMS session and choose the following Update > BMC > Choose File > 841.img > upload


5 – Installing the MP-EMS firmware update

Once the file was uploaded I could see the Current and New versions. I then choose Update button which promptly disconnected my vKVM session and Status changed from None to a % Completed.

Again, be patient and allow the system to update. For my systems the % Complete seemed to hang a few times but the total process, for me, took about


At 100% complete my system did an auto-reboot. When I heard my system beep I then closed my MP-EMS session and started anew.


Shortly after the system booted I went into the MP-EMS and validated the firmware was no 8.41.


Wrapping this up…

Ever heard the saying “It really is a simple process we just make it complicated”? Recent BIOS updates and overall system management sometimes feel this way when trying to do simple processes. Not trying to date myself but BIOS/Firmware updates have been around for decades now. I’ve done countless updates where it was simply extracting an update to simple media and then it completes the update on its own. Now one could argue that systems are more complicated and local boot devices don’t scale well for large environments and I’d say both are very true but that doesn’t mean the process can’t be made more simple.

My recommendation to firmware / bios manufactures — invest in simplicity or make it a requirement for your suppliers. You’ll have happier customers, less service calls, and more $$ in your pocket but then again if you do, what would I have to blog about?

Am I happy with with the way I have to update this Mobo? Yes, I am happy with it. For the price I paid it’s really nice to have a headless environment that I can remotely update. I won’t have to do it very often so I’m glad I wrote down my steps in this blog.

If you like my ‘no-nonsense’ blog articles that get straight to the point… then post a comment or let me know… Else, I’ll start writing boring blog content.

Home Lab Gen IV – Part III: Best ESXi White box Mobo yet?

Posted on Updated on

Initially, when I decided to start this refresh my Home Lab to GEN IV I planned to wipe just the software, add InfiniBand.  I would keep most of the hardware. However, as I started to get into this transformation I decided it was time for a hardware refresh too including moving to All Flash vSAN.

In this post, I wanted to write a bit more about my new mobo and why I think it’s a great choice for a home lab. The past workhorse of my home lab has been my trusty MSI Z68MS-G45(B3) Rev 3.0 (AKA MSI-7676). I bought 3 MƒSI-7676 in 2012 and this mobo has been a solid performer and they treated me very well. However, they were starting to age a bit so I sold them off to a good buddy of mine and I used those resources to fund my new items.

My new workhorse –

Items kept from Home Lab Gen III:

  • 3 x Antec Sonata Gen I and III each with 500W PS by Antec: I’ve had one of these cases since 2003, now that is some serious return on investment

New Items:

  • 3 x Gigabyte MX31-BS0 – So feature rich, I found them for $139 each, and this is partly why I feel it’s the best ESXi white box mobo
  • 3 x Intel Xeon E3-1230 v5 – I bought the one without the GPU and saved some $$
  • 3 x 32GB DDR4 RAM – Nothing special here, just 2133Mhz DDR4 RAM
  • 3 x Mellanox Connectx InfiniBand cards (More to come on this soon)
  • 4 x 200GB SSD, 1 x 64GB USB (Boot)
  • 1 x IBM M5210 JBOD SAS Controller

Why I chose the Gigabyte MX31-BS0 –

Likes:

  • Headless environment: This Mobo comes with an AST2400 headless chipset environment. This means I no longer am tied to my KVM. With a java enabled browser, I can view the host screen, reboot, go into BIOS, BIOS updates, view hardware, and make adjustments as if I was physically at the box
  • Virtual Media: I now can virtually mount ISOs to the ESXi host without directly being at the console (Still to test ESXi install)
  • Onboard 2D Video: No VGA card needed, the onboard video controller takes care of it all. Why is this important? You can save money by choosing a CPU that doesn’t have the integrated GPU, the onboard video does this for you
  • vSphere HCL Support: Really? Yep, most of the components on this mobo are on the HCL and Gigabyte lists ESXi 6 as a supported OS, its not 100% HCL but for a white box its darn close
  • Full 16x PCIe Socket: Goes right into the CPU << Used for the Infiniband HCA
  • Full 8x PCIe Socket: Goes into the C232  << Used for the IBM M5210
  • M.2 Socket: Supporting 10Gb/s for SSD cards
  • 4 x SATA III ports (white)
  • 2 x SATA III can be used for Satadom ports (orange) with onboard power connectors
  • 2 x Intel i210 1Gbe (HCL supported) NICs
  • E3 v5 Xeon Support
  • 64GB RAM Support (ECC or Non-ECC Support)
  • 1 x Onboard USB 2.0 Port (Great for a boot drive)

Dislikes: (Very little)

  • Manual is terrible
  • Mobo Power connector is horizontal with the mobo, this made it a bit tight for a common case
  • 4 x SATA III Ports (White) are horizontal too, again hard to seat and maintain
  • No Audio (Really not needed, but would be nice)
  • For some installs, it could be a bit limited on PCIe Ports

Some PICS :

The pic directly below shows 2 windows: Window 1 has the large Gigabyte logo, this is the headless environmental controls. From here you can control your host and launch the video viewer (window 2). The video viewer allows you to control your host just as if you were physically there. In windows 2 I’m in the BIOS settings for the ESXi host.

This is a stock photo of the MX31-BS0. It’s a bit limited on the PCIe ports, however, I don’t need many ports as soon I’ll have 20Gb/s InfiniBand running on this board but that is another post soon to come!

If you like my ‘no-nonsense’ blog articles that get straight to the point… then post a comment or let me know… Else, I’ll start writing boring blog content.

Home Lab Gen IV – Part II: Lab Clean Up and Adding Realtek 8186 NIC Drivers to ESXi 6u2 ISO

Posted on Updated on

To prep my Home Lab for ESXi 6.0U2 with VSAN + IB. I wanted to ensure it was in pristine condition. It had been running ESXi 5.5 + VSAN for many years but it was in need of some updates. I plan to fully wipe my environment (no backups) and reinstall it all. Yes, that’s right I’m going to wipe it all – this means goodbye to those Windows 2008 VM’s I’ve been hanging on to for years now. Tip: If you’d like to understand my different Home lab generations please see my dedicated page around this topic.

In this post, I am going to focus on listing out my current to-do items, then describing how to flattening all SSD/HDD and finally building a custom ESXi 6.0U2 ISO with Realtek 8186 drivers.

Current to Do list –

Completed

  • PM the Hosts – While they are off it’s a good time to do some routine PM (Complete)
  • BIOS and Firmware – Check all MoBo BIOS, pNIC, and HDD/SDD firmware (Complete)
  • Netgear Switch BIOS – It’s doubtful but always worth a check (Complete)
  • Flatten all SDD / HDD with Mini-Partition Tool (This Post)
  • Create ISO with ESXi 6.0U2 and Realtek 8168 Drivers (This Post)
Still to do
  • Install Windows 2012 Server VM for DNS and AD Service (Local disk)
  • Install vCenter Server Appliance (Local Disk)
  • Get Infiniband Functional (Needs work)
  • Setup FT and VSAN Networks
  • Enable VSAN
  • Rebuild VM Environment

Flatten all SDD / HDD with Mini-Partition Tool

Installing VSAN fresh on to an environment requires the SDD / HDD’s to be free of data and partition information. The Mini-Partition tool is a FREE bootable software product allowing you to remove all the partitions on your ESXi Hosts and other PCs. You can download it here >> https://www.partitionwizard.com/partition-wizard-bootable-cd.html

Once I created the BOOT CD and allowed the product to boot. I was quickly able to see all the HDD / SDD’s in my Host.

I simply right clicked on each host and choose ‘Delete All Partitions’

After choosing ‘Delete All Partitions’ for all my disks I clicked on ‘Apply’ in the upper right-hand corner. The following window appeared, I choose ‘Yes’ to Apply pending changes, and it removed all my partitions on all my disks quite quickly.

Create ISO with ESXi 6.0U2 and Realtek 8168 Drivers

ESXi no longer supports RealTek Network drivers, so home lab users who need these drivers will have to create a custom ISO to add these drivers back in. Keep in mind these are unsupported drivers by VMware, so use at your own risk. My trusty ESXi-Customizer GUI program is no more for ESXi 6. It has moved to a CLI based product. However, PowerCLI has all the functionality I need to build my customer ISO. In this section, I’ll be using PowerCLI to create my ISO. Keep in mind these are the steps that worked for me, your environment may vary.

To get started you will need two files and PowerCLI Installed on a Windows PC.

  1. File 1: VMware Offline ZIP >> www.vmware.com/download

2. RealTek 8186 Offline bundle >> https://vibsdepot.v-front.de/wiki/index.php/Net55-r8168

3. PowerCLI Download and install >> https://communities.vmware.com/community/vmtn/automationtools/powercli

Tip: If you don’t know PowerCLI try starting here

4. Place the files from Step 1 and 2 into c:\tmp folder

–POWERCLI COMMANDS— For each command, I have included a screenshot and the actual command allowing to copy, paste, and edit into your environment.

  1. Add ESXi 6.0u2 and RealTek8186 products to the local Software Depot

Add-EsxSoftwareDepot C:\tmp\update-from-esxi6.0-6.0_update02.zip

Add-EsxSoftwareDepot C:\tmp\net55-r8168-8.039.01-napi-offline_bundle.zip

2. Confirm the products are in the depot

Get-EsxSoftwareDepot

3. List out the ESXi Image Profiles

Get-EsxImageProfile4

4. Create a Clone Image to be modified – Ensure you are targeting the “ESXi…..standard” profile from step 3

New-EsxImageProfile -cloneprofile ESXi-6.0.0-20160302001-standard -Name “RealTek8186a”

Forward-Looking Tip: Whatever name you choose it will show up in your boot ISO

5. Set the Acceptance Level to Community Supported – Remember RealTek is unsupported by VMware

Set-EsxImageProfile -Name RealTek8186a -AcceptanceLevel CommunitySupported

For ImageProfile Enter – RealTek8186a

6. Ensure the RealTek net55-r8186 driver is loaded from the local depot (Screenshot shortened)

Get-EsxSoftwarePackage

7. Add the RealTek software package to the profile

Add-EsxSoftwarePackage

ImageProfile: RealTek8186a

SoftwarePackage[0]: net55-r8168 8.039.01-napi

Tip: You MUST enter the full name here if you just use the short name it will not work

8. Validate the RealTek drivers are now part of the RealTek8186a Profile (Screenshot shortened)

(Get-EsxImageProfile “RealTek8186a”).viblist

9. Export the profile to an ISO

Export-EsxImageProfile -ImageProfile “RealTek8186a” -ExportToIso -FilePath c:\tmp\RealTek8186a.iso

And that’s it… now with my clean/updated hosts, flatten HDD/SDD’s, and a newly pressed custom ISO I am ready to install ESXi onto my systems. Next Steps for me will be to install ESXi, AD/DNS VM, and vCenter Server Appliance. However, my next post will be focused on getting InfiniBand running in my environment.

If you like my ‘no-nonsense’ blog articles that get straight to the point… then post a comment or let me know… Else, I’ll start writing boring blog content.

VSAN – The Migration from FreeNAS

Posted on Updated on

Well folks it’s my long awaited blog post around moving my Homelab from FreeNAS to VMware VSAN.

Here are the steps I took to migrate my Home Lab GEN II with FreeNAS to Home Lab GEN III with VSAN.

Note –

  • I am not putting a focus on ESXi setup as I want to focus on the steps to setup VSAN.
  • My home lab is in no way on the VMware HCL, if you are building something like this for production you should use the VSAN HCL as your reference

The Plan –

  • Meet the Requirements
  • Backup VM’s
  • Update and Prepare Hardware
  • Distribute Existing hardware to VSAN ESXi Hosts
  • Install ESXi on all Hosts
  • Setup VSAN

The Steps –

Meet the Requirements – Detailed list here

  • Minimum of three hosts
  • Each host has a minimum of one SSD and one HDD
  • The host must be managed by vCenter Server 5.5 and configured as a Virtual SAN cluster
  • Min 6GB RAM
  • Each host has a Pass-thru RAID controller as specified in the HCL. The RAID controller must be able to present disks directly to the host without a RAID configuration.
  • 1GB NIC, I’ll be running 2 x 1Gbs NICs. However 10GB and Jumbo frames are recommended
  • VSAN VMkernel port configured on every host participating in the cluster.
  • All disks that VSAN will be allocated to should be clear of any data.

Backup Existing VMs

  • No secret here around backups. I just used vCenter Server OVF Export to a local disk to backup all my critical VM’s
  • More Information Here

Update and Prepare Hardware

  • Update all Motherboard (Mobo) BIOS and disk Firmware
  • Remove all HDD’s / SDD’s from FreeNAS SAN
  • Remove any Data from HDD/SDD’s . Either of these tools do the job

Distribute Existing hardware to VSAN ESXi Hosts

  • Current Lab – 1 x VMware Workstation PC, 2 x ESXi Hosts boot to USB (Host 1 and 2), 1 x FreeNAS SAN
  • Desired Lab – 3 x ESXi hosts with VSAN and 1 x Workstation PC
  • End Results after moves
    • All Hosts ESXi 5.5U1 with VSAN enabled
    • Host 1 – MSI 7676, i7-3770, 24GB RAM, Boot 160GB HDD, VSAN disks (2 x 2TB HDD SATA II, 1 x 60GB SSD SATA III), 5 xpNICs
    • Host 2 – MSI 7676, i7-2600, 32 GB RAM, Boot 160GB HDD, VSAN disks (2 x 2TB HDD SATA II, 1 x 90 GB SSD SATA III), 5 x pNICs
    • Host 3 – MSI 7676, i7-2600, 32 GB RAM, Boot 160GB HDD, VSAN disks (2 x 2TB HDD SATA II, 1 x 90 GB SSD SATA III), 5 x pNICs
    • Note – I have ditched my Gigabyte z68xp-UD3 Mobo and bought another MSI 7676 board. I started this VSAN conversion with it and it started to give me fits again similar to the past. There are many web posts with bugs around this board. I am simply done with it and will move to a more reliable Mobo that is working well for me.

Install ESXi on all Hosts

  • Starting with Host 1
    • Prior to Install ensure all data has been removed and all disk show up in BIOS in AHCI Mode
    • Install ESXi to Local Boot HD
    • Setup ESXi base IP address via direct Console, DNS, disable IP 6, enable shell and SSH
    • Using the VI Client setup the basic ESXi networking and vSwitch
    • Using VI Client I restored the vCSA and my AD server from OVF and powered them on
    • Once booted I logged into the vCSA via the web client
    • I built out Datacenter and add host 1
    • Create a cluster but only enabled EVC to support my different Intel CPU’s
    • Cleaned up any old DNS settings and ensure all ESXi Hosts are correct
    • From the Web client Validate that 2 x HDD and 1 x SDD are present in Host
    • Installed ESXi Host 2 / 3, followed most of these steps, and added them to the cluster

Setup VSAN

  • Logon to the Webclient
    • Ensure on all the hosts
      • Networking is setup and all functions are working
      • NTP is working
      • All expected HDD’s for VSAN are reporting in to ESXi
    • Create a vSwitch for VSAN and attach networking to it
      • I attached 2 x 1Gbs NICs for my load that should be enough
    • Assign the VSAN License Key
      • Click on the Cluster > Manage > Settings > Virtual SAN Licensing > Assign License Key

  • Enable VSAN
    • Under Virtual SAN click on General then Edit
    • Choose ‘Turn on Virtual SAN’
    • Set ‘Add disks to storage’ to Manual
    • Note – for a system on the HCL, chances are the Automatic setting will work without issue. However my system is not on the any VMware HCL and I want to control the drives to add to my Disk Group.

       

  • Add Disks to VSAN
    • Under Virtual SAN click on ‘Disk Management’
    • Choose the ICON with the Check boxes on it
    • Finally add the disks you want in your disk group

  • Allow VSAN to complete its tasks, you can check on its progress by going to ‘Tasks’

  • Once complete ensure all disks report in as healthy.

  • Ensure VSAN General tab is coming up correct
    • 3 Hosts
    • 3 of 3 SSD’s
    • 6 of 6 Data disks

  • Check to see if the data store is online

 

Summary –

Migrating from FreeNAS to VSAN was relatively a simple process. I simply moved, prepared, and installed and the product came right up. My only issue was working with a faulty Gigabyte Mobo which I resolved by replacing it. I’ll post up more as I continue to work with VSAN. If you are interested in more detail around VSAN I would recommend the following book.

Geeks.com – Time to Say goodbye for now

Posted on Updated on

I was a bit shell shocked when I went to one of my favorite online stores, geeks.com, only to find out they had closed.

They had been open for 17 years and they were one of the first sites I trusted to buy quality products from new or used.

They had a lot of common items but every now and then they had something different or unique. It was one of the reasons why I kept coming back.

I had recommended geeks.com many times and everyone I sent there always let me know what excellent service and product they had.

Well Geeks.com – I salute you – you had a good run, I’m sorry to see you go, and I hope one day you return!

Just a quick note, if you liked geeks.com then check out http://www.pacificgeek.com/ they were very similar in product and layout.