InfiniBand
First Look GEN8 ESXi/vSAN ESA 8 Home Lab (Part 1)
I’m kicking off my next generation home lab with this first look in to my choice for an ESXi/vSAN 8 host. There will be more videos to come as this series evolves!
Create an ESXi installation ISO with custom drivers in 9 easy steps!
Video Posted on Updated on
One of the challenges in running a VMware based home lab is the ability to work with old / inexpensive hardware but run latest software. Its a balance that is sometimes frustrating, but when it works it is very rewarding. Most recently I decided to move to 10Gbe from my InfiniBand 40Gb network. Part of this transition was to create an ESXi ISO with the latest build (6.7U3) and appropriate network card drivers. In this video blog post I’ll show 9 easy steps to create your own customized ESXi ISO and how to pin point IO Cards on the vmware HCL.
** Update 06/22/2022 ** If you are looking to do USB NICs with ESXi check out the new fling (USB Network Native Driver for ESXi) that helps with this. This Fling supports the most popular USB network adapter chipsets ASIX USB 2.0 gigabit network ASIX88178a, ASIX USB 3.0 gigabit network ASIX88179, Realtek USB 3.0 gigabit network RTL8152/RTL8153 and Aquantia AQC111U. https://flings.vmware.com/usb-network-native-driver-for-esxi
NOTE – Flings are NOT supported by VMware
** Update 03/06/2020 ** Though I had good luck with the HP 593742-001 NC523SFP DUAL PORT SFP+ 10Gb card in my Gen 4 Home Lab, I found it faulty when running in my Gen 5 Home Lab. Could be I was using a PCIe x4 slot in Gen 4, or it could be the card runs to hot to touch. For now this card was removed from VMware HCL, HP has advisories out about it, and after doing some poking around there seem to be lots of issues with it. I’m looking for a replacement and may go with the HP NC550SFP. However, this doesn’t mean the steps in this video are only for this card, the steps in this video help you to better understand how to add drivers into an ISO.
Here are the written steps I took from my video blog. If you are looking for more detail, watch the video.
Before you start – make sure you have PowerCLI installed, have download these files, and have placed these files in c:\tmp.
- Download driver –
- LSI Driver: https://my.vmware.com/group/vmware/details?downloadGroup=DT-ESXI60-QLOGIC-QLCNIC-61191&productId=491
- Note: Extract the offline bundle from this package
- Download ESXi –
- ESXi Update ZIP File: vmware.com/downloads
- Note: make sure you download the Update ZIP file and not the ESXi ISO file
I started up PowerCLI and did the following commands:
1) Add the ESXi Update ZIP file to the depot:
Add-EsxSoftwareDepot C:\tmp\update-from-esxi6.7-6.7_update03.zip
2) Add the LSI Offline Bundle ZIP file to the depot:
Add-EsxSoftwareDepot ‘C:\tmp\qlcnic-esx55-6.1.191-offline_bundle-2845912.zip’
3) Make sure the files from step 1 and 2 are in the depot:
Get-EsxSoftwareDepot
4) Show the Profile names from update-from-esxi6.7-6.7_update03. The default command only shows part of the name. To correct this and see the full name use the ‘| select name’
Get-EsxImageProfile | select name
5) Create a clone profile to start working with.
New-EsxImageProfile -cloneprofile ESXi-6.7.0-20190802001-standard -Name ESXi-6.7.0-20190802001-standard-QLogic -Vendor QLogic
6) Validate the LSI driver is loaded in the local depot. It should match the driver from step 2. Make sure you note the name and version number columns. We’ll need to combine these two with a space in the next step.
Get-EsxSoftwarePackage -Vendor q*
7) Add the software package to the cloned profile. Tip: For ‘SoftwarePackage:’ you should enter the ‘name’ space ‘version number’ from step 6. If you just use the short name it might not work.
Add-EsxSoftwarePackage
ImageProfile: ESXi-6.7.0-20190802001-standard-QLogic
SoftwarePackage[0]: net-qlcnic 6.1.191-1OEM.600.0.0.2494585
8) Optional: Compare the profiles, to see differences, and ensure the driver file is in the profile.
Get-EsxImageProfile | select name << Run this if you need a reminder on the profile names
Compare-EsxImageProfile -ComparisonProfile ESXi-6.7.0-20190802001-standard-QLogic -ReferenceProfile ESXi-6.7.0-20190802001-standard
9) Create the ISO
Export-EsxImageProfile -ImageProfile “ESXi-6.7.0-20190802001-standard-QLogic” -ExportToIso -FilePath c:\tmp\ESXi-6.7.0-20190802001-standard-QLogic.iso
That’s it! If you like my ‘no-nonsense’ videos and blogs that get straight to the point… then post a comment or let me know… Else, I’ll start posting boring video blogs!
Cross vSAN Cluster support for FT
Home Lab Gen IV – Part V Installing Mellanox HCAs with ESXi 6.5
The next step on my InfiniBand home lab journey was getting the InfiniBand HCAs to play nice with ESXi. To do this I need to update the HCA firmware, this proved to be a bit of a challenge. In this blog post I go into how I solved this issue and got them working with ESXi 6.5.
My initial HCA selection was the ConnectX aka HP INFINIBAND 4X DDR PCI-E HCA CARD 452372-001, and Mellanox MHGA28-XTC InfiniHost III HCA these two cards proved to be a challenge when updating their firmware. I tried all types of operating systems, different drivers, different mobos, and MFT tools versions but they would not update or be OS recognized. Only thing I didn’t try was Linux OS. The Mellanox forums are filled with folks trying to solve these issues with mixed success. I went with these cheaper cards and they simply do not have the product support necessary. I don’t recommend the use of these cards with ESXi and have migrated to a ConnectX-3 which you will see below.
Updating the ConnectX 3 Card:
After a little trial and error here is how I updated the firmware on the ConnectX 3. I found the ConnectX 3 card worked very well with Windows 2012 and I was able to install the latest Mellanox OFED for Windows (aka Windows Drivers for Mellanox HCA card) and updated the firmware very smoothly.
First, I confirm the drivers via Windows Device Manager (Update to latest if needed)
Once you confirm Windows device functionality then install the Mellanox Firmware Tools for windows (aka WinMFT)
Next, it’s time to update the HCA firmware. To do this you need to know the exact model number and sometimes the card revision. Normally this information can be found on the back of your HCA. With this in hand go to the Mellanox firmware page and locate your card then download the update.
After you download the firmware place it in an accessible directory. Next use the CLI, navigate to the WinMFT directory and use the ‘mst status’ command to reveal the HCA identifier or the MST Device Name. If this command is working, then it is a good sign your HCA is working properly and communicating with the OS. Next, I use the flint command to update my firmware. Syntax is — flint -d <MST Device Name> -i <Firmware Name> burn
Tip: If you are having trouble with your Mellanox HCA I highly recommend the Mellanox communities. The community there is generally very responsive and helpful!
Installation of ESXi 6.5 with Mellanox ConnectX-3
I would love to tell you how easy this was, but the truth is it was hard. Again, old HCA’s with new ESXi doesn’t equal easy or simple to install but it does equal Home lab fun. Let me save you hours of work. Here is the simple solution when trying to get Mellanox ConnextX Cards working with ESXi 6.5. In the end I was able to get ESXi 6.5 working with my ConnectX Card (aka HP INFINIBAND 4X DDR PCI-E HCA CARD 452372-001) and with my ConnectX-3 CX354A.
Tip: I do not recommend the use of the ConnectX Card (aka HP INFINIBAND 4X DDR PCI-E HCA CARD 452372-001) with ESXi 6.x. No matter how I tried I could not update its firmware and it has VERY limited or non-existent support. Save time go with ConnectX-3 or above.
After I installed ESXi 6.5 I followed the following commands and it worked like a champ.
Disable native driver for vRDMA
- esxcli system module set –enabled=false -m=nrdma
- esxcli system module set –enabled=false -m=nrdma_vmkapi_shim
- esxcli system module set –enabled=false -m=nmlx4_rdma
- esxcli system module set –enabled=false -m=vmkapi_v2_3_0_0_rdma_shim
- esxcli system module set –enabled=false -m=vrdma
Uninstall default driver set
- esxcli software vib remove -n net-mlx4-en
- esxcli software vib remove -n net-mlx4-core
- esxcli software vib remove -n nmlx4-rdma
- esxcli software vib remove -n nmlx4-en
- esxcli software vib remove -n nmlx4-core
- esxcli software vib remove -n nmlx5-core
Install Mellanox OFED 1.8.2.5 for ESXi 6.x.
- esxcli software vib install -d /var/log/vmware/MLNX-OFED-ESX-1.8.2.5-10EM-600.0.0.2494585.zip
Ref Links:
After a quick reboot, I got 40Gb networking up and running. I did a few vmkpings between hosts and they ping perfectly.
So, what’s next? Now that I have the HCA working I need to get VSAN (if possible) working with my new highspeed network, but this folks is another post.
If you like my ‘no-nonsense’ blog articles that get straight to the point… then post a comment or let me know… Else, I’ll start writing boring blog content.
Home Lab Gen IV – Part II: Design Considerations
I have decided to move my Home Lab away from Gen III into Gen IV. In doing this I am going to follow my best practices laid out in my ‘Home Lab Generations’ and ‘VMware Home Labs: a Definitive guide’. As you read through the “Home Lab Generations page” you should notice a theme around planning each generation and documenting its outcomes and unplanned items. In this blog post, I am going to start laying out Design Considerations which include the ‘Initial use case/goals and needed Resources as they relate to GEN IV.
Design Considerations:
Initial Use case / goals:
- Support vSphere 6.5 and vSAN 6.6 All Flash Environment
- Move networking vSAN and FT to high-speed InfiniBand
- Support headless ESXi Environment
Resources needed:
- To meet the initial use case/goals I’m will be investing quite a bit into this total refresh.
- Some initial GEN IV resource choices (Still in the works and not all proven)
- Reuse the 3 x Antec Cases / Power Supplies (No cost)
- BitFenix Case to support Windows 10 PC
- Mobo: Gigabyte MX31-BSO ($140 x 3 Ebay)
- RAM: 32GB DDR4 ($144 x 3 Jet.com)
- CPU: E3-1230v5 Xeon ($245 x 3 Jet.com)
- IBM M5210 SAS RAID (Found on Ebay $75 x 3)
- Mini SAS SFF-8643 to (4) 29pin SFF-8482 (Amazon 3 x $18)
- 12 x 200GB Enterprise Class SDD (Found on an Ebay lot deal)
- InfiniBand (All on Ebay)
- Mellanox IS5022 40Gb/s 8 Port Switch ($250) << Do not recommend, no Subnet Manager
- Mellanox ConnectX HCA rebranded as HP INFINIBAND 4X DDR PCI-E HCA CARD 452372-001 ($35 x 3)
- 6 x Infiniband Cables (Mellanox MCC4N26C-003 Cable ($60)
Notes around some of the choices above:
Mellanox IS5022 – I liked the footprint of this device over it not having built-in IB subnet manager. An IB subnet manager is required to manage the “network paths” on an IB network. Without an active IB subnet manager available your IB HCAs will not connect. Since the IS5022 I chose does not have an IB Subnet manager I will need a place to supply this service. I can choose an ESXi Host or a Windows Host. Since my ESXi hosts will be going up and down I plan to use my Windows PC as my subnet manager as it is always on and available.
Mellanox ConnectX HCA rebranded as HP INFINIBAND 4X DDR PCI-E HCA CARD 452372-001 – I initially choose these HCAs based on some other blog posts. They are at an attractive price point but they are much older and no longer have driver support. I was able to get them operational with ESXi 6.0 and will be soon working with them on ESXi 6.5 My advice is Mellanox has great products and support, however, I would recommend if you can afford it to go with a newer card that supports ESXi and save yourself the trouble of modifying ESXi software vibs. I’ll post more on this topic as I start deploying them with ESXi 6.5
Windows PC – I repurposed my Gigabyte Z68XP-UD3, i7 CPU, 90GB SSD, 16GB of DDR3 and then bought a cheap BitFenix case to build out my Windows PC. This PC will serve as my Plex Home Media Server and IB Subnet Manager. I also plan to run Workstation 10 and set up various service VM’s (AD, vCenter Server / VUM, DNS, etc). So far it’s working pretty well but this Mobo has been known to give me issues.
ESXi Hosts – I have 3 Antec Sonata cases, one that I have had since 2003 that I will reuse in this environment. I choose the following parts to make up my new ESXi Hosts: Mobo: Gigabyte MX31-BSO, 32GB DDR4, E3-1230v5 Xeon, and 4 x 200GB SSDs per host. This mobo is a bit limited on the ports but so far it seems to be working out well. For the boot disk, I plan to use the onboard USB port and a 64GB USB Stick. However, the question should be – What am I going to so with those 6 x 1TB SATA disks from GEN III — I put them into my old IX4
Here are a few PICs of the current build:
Next Post I’ll be going over the Gigabyte Motherboard.
If you like my ‘no-nonsense’ blog articles that get straight to the point… then post a comment or let me know… Else, I’ll start writing boring blog content.
Gigabyte MX31-BS0 Firmware / BIOS update for MergePoint Embedded Management Software and Motherboard
** Blog Updates / Notes **
I wrote this post when I first got my MX31-BS0 (2017), since then I have updated my BIOS several times using this process. Here are my notes around my most recent update experiences and blog comments.
- 07/2019 – Updated 1 host with MX-31BS0 BIOS from R10 to R11 and MergePoint EMS 8.73 to 8.85. This update didn’t go smooth and I cannot recommend it.
- Why did I update — I recently installed Java 9 and the vKVM under EMS 8.73 would not work. After installing R11/8.85 Java 9 seemed to work but I had issues
- Notes / Issues:
- Noted EMS 8.85 Changes: I didn’t dig in deep but the install log screen got a new look, and all the rest of the screens seemed to be the same. I had some random quirkiness (as stated int he blog post below) during the update, but nothing out of the ordinary.
- vKVM/vMedia Issues:
- EMS 8.85 with the vKVM or the vMedia would disconnect upon reboot of the host OR it would do this randomly. This is with the ‘Keep PHY link up’ enabled.
- Tested on a system with Java 8 and had the same results
- Downgraded to EMS 8.73 with BIOS R11 all issues went away
- I did not test EMS 8.80 or 8.83
- 09/2018 – Mix of updates for my three hosts — Updated MX-31BS0 BIOS from R03 or R08 to R10 and 2 hosts MergePoint 8.58 to 8.73, as one host was on 8.73 already
- Noted behavior:
- After BIOS update was completed the Mobo powered off vs. rebooting as with previous updates. Had to power on the mobo to complete the BIOS install. Then the mobo rebooted one more time as expected.
- Even though the Mobo had been warm booted the BIOS Version in MergePoint web interface still showed the old version. However, the Boot BIOS screen reflected the update. A full power disconnect of the Mobo and a few ‘refreshes’ of the web browser allowed the MergePoint to report R10. I did not see this behavior with the MergePoint EMS BIOS update, it promptly reported 8.73 properly.
- Noted behavior:
- 05/2018 –Updated on host to MX-31BS0 BIOS from R03 to R08 and MergePoint 8.58 to 8.73.
- Blog readers noted issues going to R08 and could not connect to vKVM, I didn’t have any issues with update. Looks like it was a JAVA 8 Update issue (See post comments for more info)
- 09/2017 – Updated MX-31BS0 BIOS from F10 to R03 and MergePoint 8.41 to 8.58.
- 03/2017 – Original update documented below. Updated MX-31BS0 BIOS from F01 to F10 and MergePoint 8.01 to 8.41.
**** Blog Post ****
You’d think by now manufactures would have a solid and concise process around updating their products. They are quick to warn users to not update their BIOS unless there is a problem and quick to state if there is a problem they usually won’t support it. This total cycle of disservice is a constant for low-end manufacturers, heck even some high server platforms have the same issues. I have these same concerns when I started to look into updating my current MX31-BS0 Motherboard (mobo).
What can soften this blow a bit? How about the ability to update your BIOS remotely? This is a great feature of the MX-31BS0 and in this blog post, I’ll show you how I updated the BIOS and the remote MergePoint EMS (MP-EMS) firmware too.
Initial Steps –
- My system is powered off and the power supply can supply power to the mobo.
- I have setup remote access to the MP-EMS site with an IP address and have access to it via a browser. Additionally, I have validated the vKVM function works without issue
- I downloaded the correct Mobo BIOS and BMC or MP-EMS Firmware and have extracted these files
- Steps below were completed on a Gigabyte MX31-BS0 from BIOS F01 > F10 and MP-EMS 8.01 > 8.41, your system may vary
1 – Access the MergePoint EMS site
Start out by going to the IP address for MP-EMS site. From the initial display screen, we can see the MP-EMS Firmware versions but not the Platform (or Mobo) BIOS Version. Why not you may ask? Well, the MP-EMS will only display Mobo information when the Mobo is powered on. Before you power on your Mobo I would recommend opening the vKVM session so that you can see the boot screen. When you power on your mobo (MP-EMS > Power > Control > Power On ) use the vKVM screen to halt at the ‘boot menu’ or even go into setup and disable all the boot devices.
In this PIC, we can see my Firmware for the MP-EMS is 8.01 and the BIOS is blank as the Mobo is not powered on.
2- Selecting the Mobo BIOS Update
I choose the following to update the Mobo BIOS. Start out by uploading the file: Update > ‘BIOS & ME’ > Choose File > Image.RBU > Upload
Once the upload is complete, click on ‘Update’ to proceed. NOTE: a warning dialog box appeared for me stating the system would be powered off to update the BIOS. Good thing I’m in the Boot Menu as the system will just directly power off with no regard of the system state
3 – Installing the Mobo BIOS Update: Be Patient for the BIOS install to complete
Once I saw the message the ‘BIOS firmware image has been updated successfully’ I then exited the browser session and vKVM . Note: I’d recommend closing the browser out entirely and then reopening a new session.
Once I restarted my vKVM and MP-EMS sessions and then powered on my Mobo. This allowed the BIOS update to continue.
Here is the patience part – My system was going from BIOS F01 > F10 and it rebooted 2 times to complete the update. Be patient it will complete.
Here is the behavior I noted:
- First Reboot – The system posted normally, it cleared the screen, and then white text stated a warning message about the BIOS booted to default settings. Very shortly after it rebooted again.
- On the 2nd reboot, it posted normally and I pressed F10 to get back to the Boot menu. I did this because next, we’ll need to update the MP-EMS firmware.
Once the system had rebooted I then refreshed my MP-EMS screen and viola there it was BIOS Version F10.
** Note – Not every time but sometimes, I would notice the MP-EMS Screen would show the old BIOS Version #. However, in BIOS the updated BIOS Version # would be present. A cold boot didn’t always fix this, but eventually the MP-EMS would update and would reflect the correct BIOS #**
4 – Selecting the MP-EMS Firmware
While the Mobo is booted and I’m in the boot menu, I went into the MP-EMS session and choose the following Update > BMC > Choose File > 841.img > upload
5 – Installing the MP-EMS firmware update
Once the file was uploaded I could see the Current and New versions. I then choose Update button which promptly disconnected my vKVM session and Status changed from None to a % Completed.
Again, be patient and allow the system to update. For my systems the % Complete seemed to hang a few times but the total process, for me, took about 10 mins.
After the update was complete, the next screen let me know the update was successful, my system did an auto-reboot. When I heard my system beep I then closed my MP-EMS session and started a new browser session.
Shortly after the system booted I went into the MP-EMS and validated the firmware was now 8.41.
Wrapping this up…
Ever heard the saying “It really is a simple process we just make it complicated”? Recent BIOS updates and overall system management sometimes feel this way when trying to do simple processes. Not trying to date myself but BIOS/Firmware updates have been around for decades now. I’ve done countless updates where it was simply extracting an update to simple media and then it completes the update on its own. Now one could argue that systems are more complicated and local boot devices don’t scale well for large environments and I’d say both are very true but that doesn’t mean the process can’t be made more simple.
My recommendation to firmware / bios manufactures — invest in simplicity or make it a requirement for your suppliers. You’ll have happier customers, less service calls, and more $$ in your pocket but then again if you do, what would I have to blog about?
Am I happy with the way I have to update this Mobo? Yes, I am happy with it. For the price I paid it’s really nice to have a headless environment that I can remotely update. I won’t have to do it very often so I’m glad I wrote down my steps in this blog.
If you like my ‘no-nonsense’ blog articles that get straight to the point… then post a comment or let me know… Else, I’ll start writing boring blog content.
Home Lab Gen IV – Part III: Best ESXi White box Mobo yet?
Initially, when I decided to refresh my Home Lab to Generation IV I planned to wipe just the software and add InfiniBand. I would keep most of the hardware. However, as I started to get into this transformation I decided it was time for a hardware refresh too including moving to All Flash vSAN.
In this post, I wanted to write a bit more about my new motherboard (mobo) and why I think it’s a great choice for a home lab. The past workhorse of my home lab has been my trusty MSI Z68MS-G45(B3) Rev 3.0 (AKA MSI-7676). I bought 3 MSI-7676 in 2012 and this mobo has been a solid performer and they treated me very well. However, they were starting to age a bit so I sold them off to a good buddy of mine and I used those resources to fund my new items.
My new workhorse –
Items kept from Home Lab Gen III:
- 3 x Antec Sonata Gen I and III each with 500W PS by Antec: I’ve had one of these cases since 2003, now that is some serious return on investment
New Items:
- 3 x Gigabyte MX31-BS0 – So feature rich, I found them for $139 each, and this is partly why I feel it’s the best ESXi white box mobo
- 3 x Intel Xeon E3-1230 v5 – I bought the one without the GPU and saved some $$
- 3 x 32GB DDR4 RAM – Nothing special here, just 2133Mhz DDR4 RAM
- 3 x Mellanox Connectx InfiniBand cards (More to come on this soon)
- 4 x 200GB SSD, 1 x 64GB USB (Boot)
- 1 x IBM M5210 JBOD SAS Controller
Why I chose the Gigabyte MX31-BS0 –
Likes:
- Headless environment: This Mobo comes with an AST2400 headless chipset environment. This means I no longer am tied to my KVM. With a java enabled browser, I can view the host screen, reboot, go into BIOS, BIOS updates, view hardware, and make adjustments as if I was physically at the box
- Virtual Media: I now can virtually mount ISOs to the ESXi host without directly being at the console (Still to test ESXi install)
- Onboard 2D Video: No VGA card needed, the onboard video controller takes care of it all. Why is this important? You can save money by choosing a CPU that doesn’t have the integrated GPU, the onboard video does this for you
- vSphere HCL Support: Really? Yep, most of the components on this mobo are on the HCL and Gigabyte lists ESXi 6 as a supported OS, its not 100% HCL but for a white box its darn close
- Full 16x PCIe Socket: Goes right into the CPU << Used for the Infiniband HCA
- Full 8x PCIe Socket: Goes into the C232 << Used for the IBM M5210
- M.2 Socket: Supporting 10Gb/s for SSD cards
- 4 x SATA III ports (white)
- 2 x SATA III can be used for Satadom ports (orange) with onboard power connectors
- 2 x Intel i210 1Gbe (HCL supported) NICs
- E3 v5 Xeon Support
- 64GB RAM Support (ECC or Non-ECC Support)
- 1 x Onboard USB 2.0 Port (Great for a boot drive)
Dislikes: (Very little)
- Manual is terrible
- Mobo Power connector is horizontal with the mobo, this made it a bit tight for a common case
- 4 x SATA III Ports (White) are horizontal too, again hard to seat and maintain
- No Audio (Really not needed, but would be nice)
- For some installs, it could be a bit limited on PCIe Ports
Some PICS :
The pic directly below shows 2 windows: Window 1 has the large Gigabyte logo, this is the headless environmental controls. From here you can control your host and launch the video viewer (window 2). The video viewer allows you to control your host just as if you were physically there. In windows 2 I’m in the BIOS settings for the ESXi host.
This is a stock photo of the MX31-BS0. It’s a bit limited on the PCIe ports, however, I don’t need many ports as soon I’ll have 20Gb/s InfiniBand running on this board but that is another post soon to come!
If you like my ‘no-nonsense’ blog articles that get straight to the point… then post a comment or let me know… Else, I’ll start writing boring blog content.
Home Lab Gen IV – Part II: Lab Clean Up and Adding Realtek 8186 NIC Drivers to ESXi 6u2 ISO
To prep my Home Lab for ESXi 6.0U2 with VSAN + IB. I wanted to ensure it was in pristine condition. It had been running ESXi 5.5 + VSAN for many years but it was in need of some updates. I plan to fully wipe my environment (no backups) and reinstall it all. Yes, that’s right I’m going to wipe it all – this means goodbye to those Windows 2008 VM’s I’ve been hanging on to for years now. Tip: If you’d like to understand my different Home lab generations please see my dedicated page around this topic.
In this post, I am going to focus on listing out my current to-do items, then describing how to flattening all SSD/HDD and finally building a custom ESXi 6.0U2 ISO with Realtek 8186 drivers.
Current to Do list –
Completed
- PM the Hosts – While they are off it’s a good time to do some routine PM (Complete)
- BIOS and Firmware – Check all MoBo BIOS, pNIC, and HDD/SDD firmware (Complete)
- Netgear Switch BIOS – It’s doubtful but always worth a check (Complete)
- Flatten all SDD / HDD with Mini-Partition Tool (This Post)
-
Create ISO with ESXi 6.0U2 and Realtek 8168 Drivers (This Post)
- Install Windows 2012 Server VM for DNS and AD Service (Local disk)
- Install vCenter Server Appliance (Local Disk)
- Get Infiniband Functional (Needs work)
- Setup FT and VSAN Networks
- Enable VSAN
- Rebuild VM Environment
Flatten all SDD / HDD with Mini-Partition Tool
Installing VSAN fresh on to an environment requires the SDD / HDD’s to be free of data and partition information. The Mini-Partition tool is a FREE bootable software product allowing you to remove all the partitions on your ESXi Hosts and other PCs. You can download it here >> https://www.partitionwizard.com/partition-wizard-bootable-cd.html
Once I created the BOOT CD and allowed the product to boot. I was quickly able to see all the HDD / SDD’s in my Host.
I simply right clicked on each host and choose ‘Delete All Partitions’
After choosing ‘Delete All Partitions’ for all my disks I clicked on ‘Apply’ in the upper right-hand corner. The following window appeared, I choose ‘Yes’ to Apply pending changes, and it removed all my partitions on all my disks quite quickly.
Create ISO with ESXi 6.0U2 and Realtek 8168 Drivers
ESXi no longer supports RealTek Network drivers, so home lab users who need these drivers will have to create a custom ISO to add these drivers back in. Keep in mind these are unsupported drivers by VMware, so use at your own risk. My trusty ESXi-Customizer GUI program is no more for ESXi 6. It has moved to a CLI based product. However, PowerCLI has all the functionality I need to build my customer ISO. In this section, I’ll be using PowerCLI to create my ISO. Keep in mind these are the steps that worked for me, your environment may vary.
To get started you will need two files and PowerCLI Installed on a Windows PC.
- File 1: VMware Offline ZIP >> www.vmware.com/download
2. RealTek 8186 Offline bundle >> https://vibsdepot.v-front.de/wiki/index.php/Net55-r8168
3. PowerCLI Download and install >> https://communities.vmware.com/community/vmtn/automationtools/powercli
Tip: If you don’t know PowerCLI try starting here
4. Place the files from Step 1 and 2 into c:\tmp folder
–POWERCLI COMMANDS— For each command, I have included a screenshot and the actual command allowing to copy, paste, and edit into your environment.
- Add ESXi 6.0u2 and RealTek8186 products to the local Software Depot
Add-EsxSoftwareDepot C:\tmp\update-from-esxi6.0-6.0_update02.zip
Add-EsxSoftwareDepot C:\tmp\net55-r8168-8.039.01-napi-offline_bundle.zip
2. Confirm the products are in the depot
Get-EsxSoftwareDepot
3. List out the ESXi Image Profiles
Get-EsxImageProfile4
4. Create a Clone Image to be modified – Ensure you are targeting the “ESXi…..standard” profile from step 3
New-EsxImageProfile -cloneprofile ESXi-6.0.0-20160302001-standard -Name “RealTek8186a”
Forward-Looking Tip: Whatever name you choose it will show up in your boot ISO
5. Set the Acceptance Level to Community Supported – Remember RealTek is unsupported by VMware
Set-EsxImageProfile -Name RealTek8186a -AcceptanceLevel CommunitySupported
For ImageProfile Enter – RealTek8186a
6. Ensure the RealTek net55-r8186 driver is loaded from the local depot (Screenshot shortened)
Get-EsxSoftwarePackage
7. Add the RealTek software package to the profile
Add-EsxSoftwarePackage
ImageProfile: RealTek8186a
SoftwarePackage[0]: net55-r8168 8.039.01-napi
Tip: You MUST enter the full name here if you just use the short name it will not work
8. Validate the RealTek drivers are now part of the RealTek8186a Profile (Screenshot shortened)
(Get-EsxImageProfile “RealTek8186a”).viblist
9. Export the profile to an ISO
Export-EsxImageProfile -ImageProfile “RealTek8186a” -ExportToIso -FilePath c:\tmp\RealTek8186a.iso
And that’s it… now with my clean/updated hosts, flatten HDD/SDD’s, and a newly pressed custom ISO I am ready to install ESXi onto my systems. Next Steps for me will be to install ESXi, AD/DNS VM, and vCenter Server Appliance. However, my next post will be focused on getting InfiniBand running in my environment.
If you like my ‘no-nonsense’ blog articles that get straight to the point… then post a comment or let me know… Else, I’ll start writing boring blog content.
Home Lab Gen IV – Part I: To InfiniBand and beyond!
I’ve been running ESXi 5.5 with VSAN using a Netgear 24 Port Managed Gig switch for some time now, and though it has performed okay I’d like to step up my home lab to be able to support the emerging vSphere features (VSAN 6.x, FT-SMP, and faster vMotion). To support some of these features 10Gb/s is HIGHLY recommend if not fully required. Looking at 10Gbe switches and pNICS the cost is very prohibitive for a home lab. I’ve toyed around with InfiniBand in the past (See my Xsigo Posts here) and since then I’ve always wanted to use this SUPER fast and cost effective technology. Initially, the cost to do HPC (High-performance computing) has always been very expensive. However, in recent years the InfiniBand price per port has become very cost effective for the home lab.
Let’s take a quick peek at the speed InfiniBand brings. When most of us were still playing around with 100Mb/s Ethernet InfiniBand was able to provide 10Gb/s since 2001. When I state 10Gb/s I’m talking about each port being able to produce 10Gb/s and in most cases Infiniband switches have a non-blocking backplane. So a 24 Port InfiniBand Switch, 10Gb/s per port, Full duplex, Non-blocking switch will support 480Gb/s! Over time InfiniBand speed has greatly increased and over time the older switches have dropped in price making InfiniBand a good choice for a growing home lab. For most home labs a 40Gb/s per port QDR switch is financial achievable. Even the 20Gb/s DDR or 10Gb/s SDR switch give ample speed and are VERY cost effective. However, step above QDR and you’ll find the price point is a bit too steep for home lab use.
So let’s take a look at the price / speed comparisons for InfiniBand vs. 10Gb/s Ethernet.
| 10Gb/s | 20Gb/s | 40Gb/s | |
| InfiniBand HCA | 2 Port 10Gb/s ($15-$75) | 2 Port 20Gb/s ($20-$100) | 2 Ports 40GB/s ($30-$150) |
| InfiniBand Switch | 24 Ports SDR (~$30-$70) | 24 Ports DDR (~$70-$120) | 8-36 Ports QDR (~$250-$500) |
| InfiniBand Cable | CX4 (SFF-8470) ($15-$30) | CX4 (SFF-8470) ($15-$30) | QSFP (SFF-8436) ($15-$30) |
| Ethernet Switch | 8 Ports 10Gbe ($700-$900) | ||
| Ethernet pNIC | 2 Port 10Gbe ($300-$450) | ||
| Ethernet Cable | 1M / 3ft. CAT 6a ($5-$10) |
Let’s break this down a bit further. I used the high dollar from each line item above and figured 3 x HCAs or pNICs and 6 cables for my 3 hosts.
Ethernet 10Gb/s – (3 Host Total cost $2310)
- Cost Per Switch – $900 Switch / 8 Ports = $112 per port
- Cost to enable 3 Hosts with 3 pNICs and 2 Cables -(3 Hosts x $450 pNICS) + ((2 Cables x 3 Hosts) x $10 each) = $1410 for three hosts or $470 per Host
- Total Cost to enable 3 hosts and switch cost – $1410 + $900 = $2310
- Fully populated 8 Port switch supporting 4 hosts = $2776
InfiniBand SDR 10Gb/s – (3 Host Total Costs $385)
- Cost Per Switch Port – $70 / 24 Ports = $2.91 per port
- Host Costs – (3 Hosts x $75 HCA) + ((2 Cables x 3 Hosts) x $30 = $315 (Per Host $105)
- Total Cost to enable 3 hosts and switch cost – $315 + $70 = $385
- Fully populated 24 port switch supporting 12 hosts = $1330
InfiniBand DDR 20Gb/s – (3 Host Total Cost $510)
- Cost Per Switch Port – $120 / 24 Ports = $5 per port
- Host Costs – (3 Hosts x $100 HCA) + ((2 Cables x 3 Hosts) x $30 = $390 (Per Host $130)
- Total Cost to enable 3 hosts and switch cost – $390 + $120 = $510
- Fully populated 24 port switch supporting 12 hosts = $1680
InfiniBand QDR 40Gb/s – (3 Host Total Cost $1040)
- Cost Per Switch Port – $500 / 24 Ports = $20.83 per port
- Host Costs – (3 Hosts x $150 HCA) + ((2 Cables x 3 Hosts) x $30 = $540 (Per Host $180)
- Total Cost to enable 3 hosts and switch cost – $540 + $500 = $1040
- Fully populated 24 port switch supporting 12 hosts = $2660
From these costs you can clearly see that InfiniBand is TRULY the best value for speed and port price. Even if you got a great deal, let’s say 50% off on 10Gbe, it still would be slower and it would cost you more. Heck, for the price you could easily buy an extra switch as a backup.
With this in mind my plan it to replace my backend Gbe network with InfiniBand. Using IPoIB (IP over InfiniBand) for VSAN, vMotion, and FT traffic and my 1Gbe network for the VM’s and ESXi management traffic. However, without knowledge wisdom cannot be achieved. So, my next steps are to learn more about InfiniBand and review these great videos by Mellanox. Then come up with a plan to move forward using this technology.
Check out these Videos: InfiniBand Principles Every HPC Expert MUST Know!
vmexplorer 