ESXi

Tips for installing Windows 7 x32 SP1 on Workstation 16.1.2

Posted on Updated on

This past weekend I needed to install Windows 7 x32 to support some older software. After installing Windows 7 x32 I noticed VMware tools is grayed out.  I then tried to install VMware tools manually but it failed. There are a few tricks when installing Windows 7 x32 on to Workstation 16.1.2 and in this blog I’ll cover the the steps I took.

So what changed and why all these extra steps?

You may recall that Workstation 16.0.0 could install Windows 7SP1 x32 without any additionally intervention. Starting September 2019, Microsoft added SHA-2 algorithm requirements for driver signing.  As Workstation 16 released updates it too included updated VMtools that were complaint with the Microsoft SHA-2 requirements.  So if you deploy the Windows 7 SP1 x32 ISO (which doesn’t have the SHA-2 patch) the vmtools install will fail because it cannot validate the drivers. For a bit more information See VMware KB 78655.

What are options to fix this?

By default Windows 7×32 SP1 doesn’t include the needed SHA-2 updates.  Users have 2 choices for new installs

  1. Create an updated Windows 7SP1 ISO by slip streaming in the Convenience Rollup Patch (More details here) and then use this ISO to do the install on Workstation
  2. After Windows 7 SP1 is installed, then manually install the SHA-2 update, and then install VMtools.  See steps below.

Steps for Option 2:

  • First I created a new Workstation VM. When creating it I made sure the ISO path pointed to the Windows 7 SP1 ISO and Workstation adjusted the VM hardware to be compatible with Windows 7 SP1.  I allowed the OS installation to complete.
  • After the OS was installed I applied the following MS Patch.
  • After the rebooted, I went into Workstation and did the following:
    • Right clicked on the VM > Settings > CD/DVD
    • Made sure ‘Devices status’ was check for connected and connect at power on
    • Clicked on ‘Use ISO Image’ > Browse
    • Browsed to this folder ‘C:\Program Files (x86)\VMware\VMware Workstation’
    • Choose ‘windows.iso’
    • Choose OK to closed the VM Settings
    • Back in the Windows 7 VM I went into File Explorer opened up the CD and ran setup.exe
    • From there I followed the default steps to install VM Tools and rebooted
  • Screenshot of the final outcome

Home Lab Generation 7: Part 2 – New Hardware and Software Updates

Posted on Updated on

In the final part of this 2 part series, I’ll be documenting the steps I took to update my Home Lab Generation 7 with the new hardware and software changes.  There’s quite a bit of change going on and these steps worked well for my environment.

Pre-Update-Steps:

  1. Check Product Interoperability Matrix (VCSA, ESXi, NSX, vRNI, VRLI)
  2. Check VMware Compatibility Guide (Network Cards, JBOD)
  3. Ensure the vSAN Cluster is in a health state
  4. Backup VM’s
  5. Ensure your passwords are updated
  6. Document Basic Host settings (Network, vmks, NTP, etc.)
  7. Backup VCSA via the Management Console > Backup

Steps to update vCenter Server from 7U2d (7.0.2.00500) to 7U3a (7.0.3.00100):

  1. Downloaded VCSA 7U3a VMware-vCenter-Server-Appliance-7.0.3.00100-18778458-patch-FP.iso
  2. Use WinSCP to connect to an ESXi host and upload the update/patch to vSAN ISO-Images Folder
  3. Mount the ISO from step 1 to VCSA 7U2d VM
    • NOTE: A reboot of the VCSA my be necessary for it to recognize the attached ISO
  4. Went to VCSA Management Console > Update > Check Updates should auto-start
    • NOTE: It might fail to find the ISO. If so, choose CD ROM to detect the ISO
  5. Expanded the Version > Run Pre-Update checks
  6. Once it passed pre-checks, choose Stage and Install > Accept the Terms > Next
  7. Check ‘I have backed up vCenter Server…’
    • NOTE: Clicking on ‘go to Backup’ will Exit out and you’ll have to start over
  8. Click Finish and allow it to complete
  9. Once done log back into the Management console > Summary and validate the Version
  10. Lastly, detach the datastore ISO, I simple choose ‘Client Device’

Change Boot USB to SSD and upgrade to ESXi 7U3 on Host at a time:

  1. Remove Host from NSX-T Manager (Follow these steps)
  2. In vCenter Server
    1. Put Host 1 in Maintenance Mode Ensure Accessibility (better if you can evacuate all data | run pre-check validation)
    2. Shut down the host
    3. Remove Host from Inventory (NOTE: Wait for host to go to not responding first)
  3. On the HOST
    1. Precautionary step – Turn off the power supply on the host, helps with the onboard management ability to detect changes
    2. Remove the old USB boot device
    3. Install Dell HBA330 and M.2/NVMe PCIe Card w/ 240GB SSD into the Host
    4. Power On the Host and validate firmware is updated (Mobo, Disk, Network, etc.)
    5. During boot ensure the Dell HBA330 POST screen displays (optional hit CTRL-C to view its options)
    6. In the Host BIOS Update the boot disk to the new SSD Card
  4. ESXi Install 
    1. Boot the host to ESXi 7.0U3 ISO (I used SuperMicro Virtual Media to boot from)
    2. Install ESXi to the SSD Card, Remove ISO, Reboot
    3. Update Host boot order in BIOS for the SSD Card and boot host
    4. In the ESXi DUCI, configure host with correct IPv4/VLAN, DNS, Host Name, enable SSH/Shell, disable IPv6 and reboot
    5. From this ESXi host and from another connected device, validate you can ping the Host IP and its DNS name
    6. Add Host to the Datacenter (not vSAN Cluster)
    7. Ensure Host is in Maintenance mode and validate health
    8. Erase all partitions on vSAN Devices (Host > Configure > Storage Devices > Select devices > Erase Partitions)
    9. Rename the new SSD datastore (Storage > R-Click on datastore > Rename)
    10. Add Host to Cluster (but do not add to vSAN)
    11. Add Host to vDS Networking, could be multiple vDS switches (Networking > Target vDS > Add Manage Hosts > Add Hosts > Migrate VMKernel)
    12. Complete the Host configuration settings (NTP, vmks)
    13. Create vSAN Disk Groups (Cluster > Configure > vSAN > Disk Management)
    14. Monitor and allow to complete, vSAN Replication Objects (Cluster > Monitor > vSAN > Resyncing Objects)
    15. Extract a new Host Profile and use it to build out the other hosts in the cluster
  5. ESXi Install – Additional Hosts
    1. Repeat Steps 1, 2, 3, and only Steps 4.1-4.10
    2. Attach Host Profile created in Step 4.15
    3. Check Host Profile Compliance
    4. Edit and update Host Customizations
    5. Remediate the host (the remediation will to a pre-check too)
    6. Optional validate host settings
    7. Exit Host from Maintenance mode
    8. Before starting next host ensure vSAN Resyncing Objects is completed

Other Notes / Thoughts:

Host Profiles: You may be thinking “why didn’t he use ESXi Backup/Restore or Host Profiles to simply this migration vs. doing all these steps?”.  Actually, at first I did try both but they didn’t work due to the add/changes of PCIe devices and upgrade of the ESXi OS.  Backup/Restore and Host Profiles really like things to not change for them to work with out error.  Now there are adjustments one could make and I tried to adjust them but in the end I wasn’t able to get them to adjust to the new hosts.  They were just the wrong tool for the first part of this job.   However, Host Profiles did work well post installation after all the changes were made. vSAN Erase Partitions Step 4.8:  This step can be optional it just depends on the environment.  In-fact I skipped this step on the last host and vSAN imported the disks with out issue.  Granted most of my vm’s are powered off, which means the vSAN replicas are not changing.  In an environment where there are a lot of powered on VM’s vSAN doing step 4.8 might be best.  Again, it just depends on the environment state. If you like my ‘no-nonsense’ videos and blogs that get straight to the point… then post a comment or let me know… Else, I’ll start posting really boring content!

Home Lab Generation 7: Updating the Dell HBA330 firmware without a Dell Server

Posted on Updated on

In this quick video I review how I updated the Dell HBA330 firmware using a Windows 10 PC.

 

This video was made as a supplement to my 2 Part blog post around updating my Home Lab Generation 7.

See:

Blog >> https://vmexplorer.com/2021/11/10/home-lab-generation-7-part-1-change-rational-for-software-and-hardware-changes/

Firmware >> https://www.dell.com/support/home/en-ng/drivers/driversdetails?driverid=tf1m6

Quick NAS Topics Changing Storage Pool from RAID 1 to RAID5 with the Synology 1621+

Posted on

In this not so Quick NAS topic I cover how to expand a RAID 1 volume and migrate it to a RAID 5 storage pool with the Synology 1621+. Along the way we find a disk that has some bad sectors, run an extended test and then finalize the migration.

** Products / Links Seen in this Video **

Synology DiskStation DS1621+ — https://www.synology.com/en-us/products/DS1621+

Home Lab Generation 7: Part 1 – Change Rational for software and hardware changes

Posted on Updated on

Well its that time of year again, time to deploy new changes, upgrades, and add some new hardware.  I’ll be updating my ESXi hosts and vCenter Server to the latest vSphere 7 Update 3a from 7U2d. Additionally, I’ll be swapping out the IBM 5210 JBOD for a Dell HBA330+ and lastly I’ll change my boot device to a more reliable and persistent disk.  I have 3 x ESXi hosts with VSAN, vDS switches, and NSX-T.  If you want to better understand my environment a bit better check out this page on my blog.  In this 2 part blog I’ll go through the steps I took to update my home lab and some of the rational behind it.

There are two main parts to the blog:

  • Part 1 – Change Rational for software and hardware changes – In this part I’ll explain some of my thoughts around why I’m making these software and hardware changes. 
  • Part 2 – Installation and Upgrade Steps – These are the high level steps I took to change and upgrade my Home lab

Part 1 – Change Rational for software and hardware changes:

There are three key changes that I plan to make to my environment:

  • One – Update to vSphere 7U3a
    • vSphere 7U3 has brought many new changes to vSphere including many needed features updates to vCenter server and ESXi.  Additionally, there have been serval important bug fixes and corrections that vSphere 7U3 and 7U3a will address. For more information on the updates with vSphere 7U3 please see the “vSphere 7 Update 3 – What’s New” by Bob Plankers.  For even more information check out the release notes.   
    • Part of my rational in upgrading is to prepare to talk with my customers around the benefits of this update.   I always test out the latest updates on Workstation first then migrate those learnings in to Home Lab.  
  • Two – Change out the IBM 5210 JBOD
    • The IBM 5210 JBOD is a carry over component from my vSphere 6.x vSAN environment. It worked well with vSphere 6.x and 7U1.  However, starting in 7U2 it started to exhibit stuck IO issues and the occasional PSOD.  This card was only certified with vSphere/vSAN 6.x and at some point the cache module became a requirement.  My choices at this point are to update this controller with a cache module (~$50 each) and hope it works better or make a change.  In this case I decided to make a change to the Dell HBA330 (~$70 each).  The HBA330 is a JBOD controller that Dell pretty much worked with VMware to create for vSAN.  It is on the vSphere/vSAN 7U3 HCL and should have a long life there too.  Additionally, the HBA330 edge connectors (Mini SAS SFF-8643) line up with the my existing SAS break-out cables. When I compare the benefits of the Dell HBA330 to upgrading the cache module for the IBM 5210 the HBA330 was the clear choice.  The trick is finding a HBA330 that is cost effective and comes with a full sized slot cover.  Its a bit tricky but you can find them on eBay, just have to look a bit harder.

  • Three – Change my boot disk
    • Last September-2021, VMware announced boot from USB is going to change and customers were advised to plan ahead for these upcoming changes.   My current hosts are using cheap SanDisk USB 64GB memory sticks.  Its something I would never recommend for a production environment, but for a Home Lab these worked okay.  I originally chose them during my Home Lab Gen 5 updates as I need to do testing with USB booted Hosts.  Now that VMware has deprecated support for USB/SD devices it’s time to make a change. Point of clarity: the word deprecated can mean different things to different people.  However, in the software industry deprecated means “discourage the use of (something, such as a software product) in favor of a newer or better alternative”.  vSphere 7 is in a deprecated mode when it comes to USB/SD booted hosts, they are still supported, and customers are highly advised to plan ahead. As of this writing, legacy (legacy is a fancy word for vSphere.NEXT) USB hosts will require a persistent disk and eventually (Long Term Supported) USB/SD booted hosts will no longer be supported.  Customers should seek guidance from VMware when making these changes.

    • The requirement to be in a “Long Term Supported” mode is to have a ESXi host be booted from HDD, SSD, or a PCIe device.  In my case, I didn’t want to add more disks to my system and chose to go with a PCIe SSD/NVMe card. I chose this PCIe device that will support M.2 (SATA SSD) and NMVe devices in one slot and I decided to go with a Kingston A400 240G Internal SSD M.2  as my boot disk. The A400 with 240GB should be more than enough to boot the ESXi hosts and keep up with its disk demands going forward.   

 

Final thoughts and a important warning.  Making changes that affect your current environment are never easy but are sometimes necessary.  With a little planning it can make the journey a bit easier.  I’ll be testing these changes over the next few months and will post up if issues occur.  However, a bit of warning – adding new devices to an environment can directly impact your ability to migrate or upgrade your hosts.  Due to the hardware decisions I have made a direct ESXi upgrade is not possible and I’ll have to back out my current hosts from vCenter Server plus other software and do a new installation.  However, those details and more will be in Part 2 – Installation and Upgrade Steps.

Opportunity for vendor improvement – If backup vendors like Synology, asustor, Veeam, Veritas, naviko, and Arcoins could really shine.  If they could backup and restore a ESXi host to dislike hardware  or boot disks this would be a huge improvement for VI Admin, especially when they have tens of thousands of hosts the need to change from their USB to persistent disks.  This is not a new ask, VI admins have been asking for this option for years, now maybe these companies will listen as many users and their hosts are going to be affected by these upcoming requirements.

VMware vSphere 7.0 Update 1 | vCenter, ESXi, vSAN | Information

Posted on Updated on

VMware announced the GA Releases of the following:

  • VMware vCenter 7.0 Update 1
  • VMware ESXi 7.0 Update 1
  • VMware vSAN 7.0 Update 1

See the base table for all the technical enablement links, now including VMworld 2020 OnDemand Sessions

.

Release Overview
vCenter Server 7.0 Update 1 | ISO Build 16860138

ESXi 7.0 Update 1 | ISO Build 16850804

VMware vSAN 7.0 Update 1 | Build 16850804

What’s New vCenter Server
Inclusive terminology: In vCenter Server 7.0 Update 1, as part of a company-wide effort to remove instances of non-inclusive language in our products, the vSphere team has made changes to some of the terms used in the vSphere Client. APIs and CLIs still use legacy terms, but updates are pending in an upcoming release.

  • vSphere Accessibility Enhancements: vCenter Server 7.0 Update 1 comes with significant accessibility enhancements based on recommendations by the Accessibility Conformance Report (ACR), which is the internationally accepted standard.  Read more
  • vSphere Ideas Portal: With vCenter Server 7.0 Update 1, any user with a valid my.vmware.com account can submit feature requests by using the vSphere Ideas portal. Read more
  • Enhanced vSphere Lifecycle Manager hardware compatibility pre-checks for vSAN environments: vCenter Server 7.0 Update 1 adds vSphere Lifecycle Manager hardware compatibility pre-checks. Read more
  • Increased scalability with vSphere Lifecycle Manager: For vSphere Lifecycle Manager​ operations with ESXi hosts and clusters is up to:
    • 64 supported clusters from 15
    • 96 supported ESXi hosts within a cluster from 64. For vSAN environments, the limit is still 64
    • 280 supported ESXi hosts managed by a vSphere Lifecycle Manager Image from 150
    • 64 clusters on which you can run remediation in parallel, if you initiate remediation at a data center level, from 15
  • vSphere Lifecycle Manager support for coordinated upgrades between availability zones: With vCenter Server 7.0 Update 1, to prevent overlapping operations, vSphere Lifecycle Manager updates fault domains in vSAN clusters in a sequence. ESXi hosts within each fault domain are still updated in a rolling fashion. For vSAN stretched clusters, the first fault domain is always the preferred site.
  • Extended list of supported Red Hat Enterprise Linux and Ubuntu versions for the VMware vSphere Update Manager Download Service (UMDS): vCenter Server 7.0 Update 1 adds new Red Hat Enterprise Linux and Ubuntu versions that UMDS supports. For the complete list of supported versions, see Supported Linux-Based Operating Systems for Installing UMDS.
  • Silence Alerts button in VMware Skyline Health – With vCenter Server 7.0 Update 1, you can stop alerts for certain health checks, such as notifications for known issues, by using the Silence Alerts button.  Read more
  • Configure SMTP authentication: vCenter Server 7.0 Update 1 adds support to SMTP authentication in the vCenter Server Appliance to enable sending alerts and alarms by email in secure mode. Configure Mail Sender Settings.   Read more
  • System virtual machines for vSphere Cluster Services: In vCenter Server 7.0 Update 1, vSphere Cluster Services adds a set of system virtual machines in every vSphere cluster to ensure the healthy operation of VMware vSphere Distributed Resource Scheduler. For more information, see VMware knowledge base articles KB80472KB79892 and KB80483.
  • Licensing for VMware Tanzu Basic: With vCenter Server 7.0 Update 1, licensing for VMware Tanzu Basic splits into separate license keys for vSphere 7 Enterprise Plus and VMware Tanzu Basic. In vCenter Server 7.0 Update 1, you must provide either a vSphere 7 Enterprise Plus license key or a vSphere 7 Enterprise Plus with an add-on for Kubernetes license key to enable the Enterprise Plus functionality for ESXi hosts. In addition, you must provide a VMware Tanzu Basic license key to enable Kubernetes functionality for all ESXi hosts that you want to use as part of a Supervisor Cluster.
    When you upgrade a 7.0 deployment to 7.0 Update 1, existing Supervisor Clusters automatically start a 60-day evaluation mode. If you do not install a VMware Tanzu Basic license key and assign it to existing Supervisor Clusters within 60 days, you see some limitations in the Kubernetes functionality. For more information, see Licensing for vSphere with Tanzu and VMware knowledge base article KB80868.
  • For VMware vSphere with Tanzu updates, see VMware vSphere with Tanzu Release Notes.
Upgrade/Install Considerations vCenter
Before upgrading to vCenter Server 7.0 Update 1, you must confirm that the Link Aggregation Control Protocol (LACP) mode is set to enhanced, which enables the Multiple Link Aggregation Control Protocol (the multipleLag parameter) on the VMware vSphere Distributed Switch (VDS) in your vCenter Server system.

If the LACP mode is set to basic, indicating One Link Aggregation Control Protocol (singleLag), the distributed virtual port groups on the vSphere Distributed Switch might lose connection after the upgrade and affect the management vmknic, if it is on one of the dvPort groups. During the upgrade precheck, you see an error such as Source vCenter Server has instance(s) of Distributed Virtual Switch at unsupported lacpApiVersion.

For more information on converting to Enhanced LACP Support on a vSphere Distributed Switch, see VMware knowledge base article 2051311. For more information on the limitations of LACP in vSphere, see VMware knowledge base article 2051307.

Product Support Notices

  • vCenter Server 7.0 Update 1 does not support VMware Site Recovery Manager 8.3.1.
  • Deprecation of Server Message Block (SMB) protocol version 1.0
    File-based backup and restore of vCenter Server by using Server Message Block (SMB) protocol version 1.0 is deprecated in vCenter Server 7.0 Update 1. Removal of SMBv.1 is due in a future vSphere release.
  • End of General Support for ​VMware Tools 9.10.x and 10.0.x  VMware Product Lifecycle Matrix
  • Deprecation of the VMware Service Lifecycle Manager API
    VMware plans to deprecate the VMware Service Lifecycle Manager API (vmonapi service) in a future release. For more information, see VMware knowledge base article 80775.
  • End of support for Internet Explorer 11
    Removal of Internet Explorer 11 from the list of supported browsers for the vSphere Client is due in a future vSphere release.
  • VMware Host Client in maintenance mode
What’s New ESXi
What’s New

  • ESXi 7.0 Update 1 supports vSphere Quick Boot on the following servers:
    • HPE ProLiant BL460c Gen9
    • HPE ProLiant DL325 Gen10 Plus
    • HPE ProLiant DL360 Gen9
    • HPE ProLiant DL385 Gen10 Plus
    • HPE ProLiant XL225n Gen10 Plus
    • HPE Synergy 480 Gen9
  • Enhanced vSphere Lifecycle Manager hardware compatibility pre-checks for vSAN environments: ESXi 7.0 Update 1 adds vSphere Lifecycle Manager hardware compatibility pre-checks. The pre-checks automatically trigger after certain change events such as modification of the cluster desired image or addition of a new ESXi host in vSAN environments. Also, the hardware compatibility framework automatically polls the Hardware Compatibility List database at predefined intervals for changes that trigger pre-checks as necessary.
  • Increased number of vSphere Lifecycle Manager concurrent operations on clusters: With ESXi 7.0 Update 1, if you initiate remediation at a data center level, the number of clusters on which you can run remediation in parallel, increases from 15 to 64 clusters.
  • vSphere Lifecycle Manager support for coordinated updates between availability zones: With ESXi 7.0 Update 1, to prevent overlapping operations, vSphere Lifecycle Manager updates fault domains in vSAN clusters in a sequence. ESXi hosts within each fault domain are still updated in a rolling fashion. For vSAN stretched clusters, the first fault domain is always the preferred site.
  • Extended list of supported Red Hat Enterprise Linux and Ubuntu versions for the VMware vSphere Update Manager Download Service (UMDS): ESXi 7.0 Update 1 adds new Red Hat Enterprise Linux and Ubuntu versions that UMDS supports. For the complete list of supported versions, see Supported Linux-Based Operating Systems for Installing UMDS.
  • Improved control of VMware Tools time synchronization: With ESXi 7.0 Update 1, you can select a VMware Tools time synchronization mode from the vSphere Client instead of using the command prompt. When you navigate to VM Options > VMware Tools > Synchronize Time with Host, you can select Synchronize at startup and resume (recommended)Synchronize time periodically, or, if no option is selected, you can prevent synchronization.
  • Increased Support for Multi-Processor Fault Tolerance (SMP-FT) maximums: With ESXi 7.0 Update 1, you can configure more SMP-FT VMs, and more total SMP-FT vCPUs in an ESXi host, or a cluster, depending on your workloads and capacity planning.
  • Virtual hardware version 18: ESXi Update 7.0 Update 1 introduces virtual hardware version 18 to enable support for virtual machines with higher resource maximums, and:
    • Secure Encrypted Virtualization – Encrypted State (SEV-ES)
    • Virtual remote direct memory access (vRDMA) native endpoints
    • EVC Graphics Mode (vSGA).
  • Increased resource maximums for virtual machines and performance enhancements:
    • With ESXi 7.0 Update 1, you can create virtual machines with three times more virtual CPUs and four times more memory to enable applications with larger memory and CPU footprint to scale in an almost linear fashion, comparable with bare metal. Virtual machine resource maximums are up to 768 vCPUs from 256 vCPUs, and to 24 TB of virtual RAM from 6 TB. Still, not over-committing memory remains a best practice. Only virtual machines with hardware version 18 and operating systems supporting such large configurations can be set up with these resource maximums.
    • Performance enhancements in ESXi that support the larger scale of virtual machines include widening of the physical address, address space optimizations, better NUMA awareness for guest virtual machines, and more scalable synchronization techniques. vSphere vMotion is also optimized to work with the larger virtual machine configurations.
    • ESXi hosts with AMD processors can support virtual machines with twice more vCPUs, 256, and up to 8 TB of RAM.
    • Persistent memory (PMEM) support is up twofold to 12 TB from 6 TB for both Memory Mode and App Direct Mode.
Upgrade/Install Considerations ESXi
In vSphere 7.x, the Update Manager plug-in, used for administering vSphere Update Manager, is replaced with the Lifecycle Manager plug-in. Administrative operations for vSphere Update Manager are still available under the Lifecycle Manager plug-in, along with new capabilities for vSphere Lifecycle Manager.

The typical way to apply patches to ESXi 7.x hosts is by using the vSphere Lifecycle Manager. For details, see About vSphere Lifecycle Manager and vSphere Lifecycle Manager Baselines and Images.

You can also update ESXi hosts without using the Lifecycle Manager plug-in, and use an image profile instead. To do this, you must manually download the patch offline bundle ZIP file from the VMware download page or the Product Patches page and use the esxcli software profile command.
For more information, see the Upgrading Hosts by Using ESXCLI Commands and the VMware ESXi Upgrade guide.

What’s New vSAN
vSAN 7.0 Update 1 introduces the following new features and enhancements:

Scale Without Compromise

  • HCI Mesh. HCI Mesh is a software-based approach for disaggregation of compute and storage resources in vSAN. HCI Mesh brings together multiple independent vSAN clusters by enabling cross-cluster utilization of remote datastore capacity within vCenter Server. HCI Mesh enables you to efficiently utilize and consume data center resources, which provides simple storage management at scale.
  • vSAN File Service enhancements. Native vSAN File Service includes support for SMB file shares. Support for Microsoft Active Directory, Kerberos authentication, and scalability improvements also are available.
  • Compression-only vSAN. You can enable compression independently of deduplication, which provides a storage efficiency option for workloads that cannot take advantage of deduplication. With compression-only vSAN, a failed capacity device only impacts that device and not the entire disk group.
  • Increased usable capacity. Internal optimizations allow vSAN to no longer need the 25-30% of free space available for internal operations and host failure rebuilds. The amount of space required is a deterministic value based on deployment variables, such as size of the cluster and density of storage devices. These changes provide more usable capacity for workloads.
  • Shared witness for two-node clusters. vSAN 7.0 Update 1 enables a single vSAN witness host to manage multiple two-node clusters. A single witness host can support up to 64 clusters, which greatly reduces operational and resource overhead.

Simplify Operations

  • vSAN Data-in-Transit encryption. This feature enables secure over the wire encryption of data traffic between nodes in a vSAN cluster. vSAN data-in-transit encryption is a cluster-wide feature and can be enabled independently or along with vSAN data-at-rest encryption. Traffic encryption uses the same FIPS-2 validated cryptographic module as existing encryption features and does not require use of a KMS server.
  • Enhanced data durability during maintenance mode. This improvement protects the integrity of data when you place a host into maintenance mode with the Ensure Accessibility option. All incremental writes which would have been written to the host in maintenance are now redirected to another host, if one is available. This feature benefits VMs that have PFTT=1 configured, and also provides an alternative to using PFTT=2 for ensuring data integrity during maintenance operations
  • vLCM enhancements. vSphere Lifecycle Manager (vLCM) is a solution for unified software and firmware lifecycle management. In this release, vLCM is enhanced with firmware support for Lenovo ReadyNodes, awareness of vSAN stretched cluster and fault domain configurations, additional hardware compatibility pre-checks, and increased scalability for concurrent cluster operations.
  • Reserved capacity. You can enable capacity reservations for internal cluster operations and host failure rebuilds. Reservations are soft-thresholds designed to prevent user-driven provisioning activity from interfering with internal operations, such as data rebuilds, rebalancing activity, or policy re-configurations.
  • Default gateway override. You can override the default gateway for VMkernel adapter to provide a different gateway for vSAN network. This feature simplifies routing configuration for stretched clusters, two-node clusters, and fault domain deployments that previously required manual configuration of static routes. Static routing is not necessary
  • Faster vSAN host restarts. The time interval for a planned host restart has been reduced by persisting in-memory metadata to disk before the restart or shutdown. This method reduces the time required for hosts in a vSAN cluster to restart, which decreases the overall cluster downtime during maintenance windows.
  • Workload I/O analysis. Analyze VM I/O metrics with IOInsight, a monitoring and troubleshooting tool that is integrated directly into vCenter Server. Gain a detailed view of VM I/O characteristics such as performance, I/O size and type, read/write ratio, and other important data metrics. You can run IOInsight operations against VMs, hosts, or the entire cluster
  • Consolidated I/O performance view. You can select multiple VMs, and display a combined view of storage performance metrics such as IOPS, throughput, and latency. You can compare storage performance characteristics across multiple VMs.
  • VM latency monitoring with IOPS limits. This improvement in performance monitoring helps you distinguish the periods of latency that can occur due to enforced IOPS limits. This view can help organizations that set IOPS limits in VM storage policies.
  • Secure drive erase. Securely wipe flash storage devices before decommissioning from a vSAN cluster through a set of new PowerCLI or API commands. Use these commands to safely erase data in accordance to NIST standards
  • Data migration pre-check for disks. vSAN’s data migration pre-check for host maintenance mode now includes support for individual disk devices or entire disk groups. This offers more granular pre-checks for disk or disk group decommissioning.
  • VPAT section 508 compliant. vSAN is compliant with the Voluntary Product Accessibility Template (VPAT). VPAT section 508 compliance ensures that vSAN had a thorough audit of accessibility requirements, and has instituted product changes for proper compliance.

 Note: vSAN 7.0 Update 1 improves CPU performance by standardizing task timers throughout the system. This change addresses issues with timers activating earlier or later than requested, resulting in degraded performance for some workloads.

Upgrade/Install Considerations vSAN
For instructions about upgrading vSAN, see vSAN Documentation   Upgrading the vSAN Cluster   Before You Upgrade   Upgrading vCenter Server  Upgrading Hosts

Note: Before performing the upgrade, please review the most recent version of the VMware Compatibility Guide to validate that the latest vSAN version is available for your platform.

vSAN 7.0 Update 1 is a new release that requires a full upgrade to vSphere 7.0 Update 1. Perform the following tasks to complete the upgrade:

1. Upgrade to vCenter Server 7.0 Update 1. For more information, see the VMware vSphere 7.0 Update 1 Release Notes.
2. Upgrade hosts to ESXi 7.0 Update 1. For more information, see the VMware vSphere 7.0 Update 1 Release Notes.
3. Upgrade the vSAN on-disk format to version 13.0. If upgrading from on-disk format version 3.0 or later, no data evacuation is required (metadata update only).

 Note: vSAN retired disk format version 1.0 in vSAN 7.0 Update 1. Disks running disk format version 1.0 are no longer recognized by vSAN. vSAN will block upgrade through vSphere Update Manager, ISO install, or esxcli to vSAN 7.0 Update 1. To avoid these issues, upgrade disks running disk format version 1.0 to a higher version. If you have disks on version 1, a health check alerts you to upgrade the disk format version.

Disk format version 1.0 does not have performance and snapshot enhancements, and it lacks support for advanced features including checksum, deduplication and compression, and encryption. For more information about vSAN disk format version, see KB2145267.

Upgrading the On-disk Format for Hosts with Limited Capacity

During an upgrade of the vSAN on-disk format from version 1.0 or 2.0, a disk group evacuation is performed. The disk group is removed and upgraded to on-disk format version 13.0, and the disk group is added back to the cluster. For two-node or three-node clusters, or clusters without enough capacity to evacuate each disk group, select Allow Reduced Redundancy from the vSphere Client. You also can use the following RVC command to upgrade the on-disk format: vsan.ondisk_upgrade –allow-reduced-redundancy

When you allow reduced redundancy, your VMs are unprotected for the duration of the upgrade, because this method does not evacuate data to the other hosts in the cluster. It removes each disk group, upgrades the on-disk format, and adds the disk group back to the cluster. All objects remain available, but with reduced redundancy.

If you enable deduplication and compression during the upgrade to vSAN 7.0 Update 1, you can select Allow Reduced Redundancy from the vSphere Client.

Limitations

For information about maximum configuration limits for the vSAN 7.0 Update 1 release, see the Configuration Maximums  documentation.

Technical Enablement
Release Notes vCenter Click Here  |  What’s New  |  Earlier Releases  |  Patch Info  |  Installation & Upgrade Notes   |  Product Support Notices

Resolved Issues  |  Known Issues

Release Notes ESXi Click Here  |  What’s New  |  Earlier Releases  |  Patch Info  |  Product Support Notices  |  Resolved Issues  |  Known Issues
Release Notes vSAN Click Here  |  What’s New  |  VMware vSAN Community  |  Upgrades for This Release  |  Limitations  |  Known Issues
docs.vmware/vCenter Installation & Setup  |   vCenter Server Upgrade  |   vCenter Server Configuration
Docs.vmware/ESXi Installation & Setup  |  Upgrading   |   Managing Host and Cluster Lifecycle  |   Host Profiles  |   Networking  |   Storage  |   Security

Resource Management  |   Availability  |  Monitoring & Performance

docs.vmware/vSAN Using vSAN Policies  |  Expanding & Managing a vSAN Cluster  |  Device Management  |  Increasing Space Efficiency  |  Encryption

Upgrading the vSAN Cluster   Before You Upgrade   Upgrading vCenter Server  Upgrading Hosts

Compatibility Information Interoperability Matrix vCenter  |  Configuration Maximums vSphere (All)  |  Ports Used vSphere (All)

Interoperability Matrix ESXi  |  Interoperability Matrix vSAN  |  Configuration Maximums vSAN  |  Ports Used vSAN

Blogs & Infolinks What’s New with VMware vSphere 7 Update 1  |  Main VMware Blog vSphere 7    |  vSAN  |  vSphere  |   vCenter Server

Announcing the ESXi-Arm Fling  |  In-Product Evaluation of vSphere with Tanzu

vSphere 7 Update 1 – Unprecedented Scalability

YouTube A Quick Look at What’s New in vSphere 7 Update 1  |  vSphere with Tanzu Overview in 3 Minutes

VMware vSphere with Tanzu webpage  |  eBook: Deliver Developer-Ready Infrastructure Using vSphere with Tanzu

What’s New in vSAN 7 Update 1   |  PM’s Blog, Cormac vSAN 7.0 Update 1

Download vSphere   |   vSAN
VMworld 2020 OnDemand

(Free Account Needed)

Deep Dive: What’s New with vCenter Server [HCP1100]    |   99 Problems, But A vSphere Upgrade Ain’t One [HCP1830]

Certificate Management in vSphere [HCP2050]      |     Connect vSAN Capacity Across Clusters with VMware HCI Mesh [DEM3206]

Deep Dive: vSphere 7 Developer Center [HCP1211]    |

More vSphere & vSAN VMworld Sessions

VMworld HOL Walkthrough

(VMworld Account Needed)

Introduction to vSphere Performance [HOL-2104-95-ISM]

VMware vSphere – What’s New [HOL-2111-95-ISM]

What’s New in vSAN – Getting Started [HOL-2108-95-ISM]

Home Lab Generation 7: Upgrading vSAN 7 Hybrid capacity step by step

Posted on Updated on

My GEN5 Home Lab is ever expanding and the space demands on the vSAN cluster were becoming more apparent.  This past weekend I updated my vSAN 7 cluster capacity disks from 6 x 600GB SAS HDD to 6 x 2TB SAS HDD and it went very smoothly.   Below are my notes and the order I followed around this upgrade.  Additionally, I created a video blog (link further below) around these steps.  Lastly, I can’t stress this enough – this is my home lab and not a production environment. The steps in this blog/video are just how I went about it and are not intended for any other purpose.

Current Cluster:

  • 3 x ESXi 7.0 Hosts (Supermicro X9DRD-7LN4F-JBOD, Dual E5 Xeon, 128GB RAM, 64GB USB Boot)
  • vSAN Storage is:
    • 600GB SAS Capacity HDD
    • 200GB SAS Cache SDD
    • 2 Disk Groups per host (1 x 200GB SSD + 1 x 600GB HDD)
    • IBM 5210 HBA Disk Controller
    • vSAN Datastore Capacity: ~3.5TB
    • Amount Allocated: ~3.7TB
    • Amount in use: ~1.3TB

Proposed Change:

  • Keep the 6 x 200GB SAS Cache SDD Drives
  • Remove 6 x 600GB HDD Capacity Disk from hosts
  • Replace with 6 x 2TB HDD Capacity Disks
  • Upgraded vSAN Datastore ~11TB

Upgrade Notes:

  1. I choose to backup (via clone to offsite storage) and power off most of my VMs
  2. I clicked on the Cluster > Configure > vSAN > Disk Management
  3. I selected the one host I wanted to work with and then the Disk group I wanted to work with
  4. I located one of the capacity disks (600GB) and clicked on it
  5. I noted its NAA ID (will need later)
  6. I then clicked on “Pre-check Data Migration” and choose ‘full data migration’
  7. The test completed successfully
  8. Back at the Disk Management screen I clicked on the HDD I am working with
  9. Next I clicked on the ellipse dots and choose ‘remove’
  10. A new window appeared and for vSAN Data Migration I choose ‘Full Data Migration’ then clicked remove
  11. I monitored the progress in ‘Recent Tasks’
  12. Depending on how much data needed to be migrated, and if there were other objects being resynced it could take a bit of time per drive.  For me this was ~30-90 mins per drive
  13. Once the data migration was complete, I went to my host and found the WWN# of the physical disk that matched the NAA ID from Step 5
  14. While the system was still running, removed disk from the chassis, and replaced it with the new 2TB HDD
  15. Back at vCenter Server I clicked on the Host on the Cluster > Configure > Storage > Storage Devices
  16. I made sure the new 2TB drive was present
  17. I clicked on the 2TB drive, choose ‘erase partitions’ and choose OK
  18. I clicked on the Cluster > Configure > vSAN > Disk Management > ‘Claim Unused Disks’
  19. A new Window appeared and I choose ‘Capacity’ for the 2TB HDD, ‘Cache’ for the 200GB SDD drives, and choose OK
  20. Recent Task showed the disk being added
  21. When it was done I clicked on the newly added disk group and ensured it was in a health state
  22. I repeated this process until all the new HDDs were added

Final Outcome:

  • After upgrade the vSAN Storage is:
    • 2TB SAS Capacity HDD
    • 200GB SAS Cache SDD
    • 2 Disk Groups per host (1 x 200GB SSD + 1 x 2TB HDD)
    • IBM 5210 HBA Disk Controller
    • vSAN Datastore is ~11.7TB

Notes & other thoughts:

  • I was able complete the upgrade in this order due to the nature my home lab components.  Mainly because I’m running a SAS Storage HBA that is just a JBOD controller supporting Hot-Pluggable drives.
  • Make sure you run the data migration pre-checks and follow any advice it has.  This came in very handy.
  • If you don’t have enough space to fully evacuate a capacity drive you will either have to add more storage or completely remove VM’s from the cluster.
  • Checking Cluster>Monitor>vSAN>Resyncing Objects, gave me a good idea when I should start my next migration.  I look for it to be complete before I start. If you have an very active cluster this maybe harder to achieve.
  • Checking the vSAN Cluster Health should be done, especially the Cluster > Monitor > Skyline Health > Data > vSAN Object Health, any issues in these areas should be looked into prior to migration
  • Not always, but mostly, the disk NAA ID reported in vCenter Server/vSAN usually coincides with the WWN Number on the HDD
  • By changing my HDDs from 600GB SAS 10K to 2TB SAS 7.2K there will be a performance hit. However, my lab needed more space and 10k-15K drives were just out of my budget.
  • Can’t recommend this reference Link from VMware enough: Expanding and Managing a vSAN Cluster

 

Video Blog:

Various Photos:

If you like my ‘no-nonsense’ videos and blogs that get straight to the point… then post a comment or let me know… Else, I’ll start posting really boring content!

Create an ESXi installation ISO with custom drivers in 9 easy steps!

Video Posted on Updated on

One of the challenges in running a VMware based home lab is the ability to work with old / inexpensive hardware but run latest software. Its a balance that is sometimes frustrating, but when it works it is very rewarding. Most recently I decided to move to 10Gbe from my InfiniBand 40Gb network. Part of this transition was to create an ESXi ISO with the latest build (6.7U3) and appropriate network card drivers. In this video blog post I’ll show 9 easy steps to create your own customized ESXi ISO and how to pin point IO Cards on the vmware HCL.

** Update 03/06/2020 ** Though I had good luck with the HP 593742-001 NC523SFP DUAL PORT SFP+ 10Gb card in my Gen 4 Home Lab, I found it faulty when running in my Gen 5 Home Lab.  Could be I was using a PCIe x4 slot in Gen 4, or it could be the card runs to hot to touch.  For now this card was removed from VMware HCL, HP has advisories out about it, and after doing some poking around there seem to be lots of issues with it.  I’m looking for a replacement and may go with the HP NC550SFP.   However, this doesn’t mean the steps in this video are only for this card, the steps in this video help you to better understand how to add drivers into an ISO.

Here are the written steps I took from my video blog.  If you are looking for more detail, watch the video.

Before you start – make sure you have PowerCLI installed, have download these files,  and have placed these files in c:\tmp.

I started up PowerCLI and did the following commands:

1) Add the ESXi Update ZIP file to the depot:

Add-EsxSoftwareDepot C:\tmp\update-from-esxi6.7-6.7_update03.zip

2) Add the LSI Offline Bundle ZIP file to the depot:

Add-EsxSoftwareDepot ‘C:\tmp\qlcnic-esx55-6.1.191-offline_bundle-2845912.zip’

3) Make sure the files from step 1 and 2 are in the depot:

Get-EsxSoftwareDepot

4) Show the Profile names from update-from-esxi6.7-6.7_update03. The default command only shows part of the name. To correct this and see the full name use the ‘| select name’ 

Get-EsxImageProfile | select name

5) Create a clone profile to start working with.

New-EsxImageProfile -cloneprofile ESXi-6.7.0-20190802001-standard -Name ESXi-6.7.0-20190802001-standard-QLogic -Vendor QLogic

6) Validate the LSI driver is loaded in the local depot.  It should match the driver from step 2.  Make sure you note the name and version number columns.  We’ll need to combine these two with a space in the next step.

Get-EsxSoftwarePackage -Vendor q*

7) Add the software package to the cloned profile. Tip: For ‘SoftwarePackage:’ you should enter the ‘name’ space ‘version number’ from step 6.  If you just use the short name it might not work.

Add-EsxSoftwarePackage

ImageProfile: ESXi-6.7.0-20190802001-standard-QLogic
SoftwarePackage[0]: net-qlcnic 6.1.191-1OEM.600.0.0.2494585

8) Optional: Compare the profiles, to see differences, and ensure the driver file is in the profile.

Get-EsxImageProfile | select name   << Run this if you need a reminder on the profile names

Compare-EsxImageProfile -ComparisonProfile ESXi-6.7.0-20190802001-standard-QLogic -ReferenceProfile ESXi-6.7.0-20190802001-standard

9) Create the ISO

Export-EsxImageProfile -ImageProfile “ESXi-6.7.0-20190802001-standard-QLogic” -ExportToIso -FilePath c:\tmp\ESXi-6.7.0-20190802001-standard-QLogic.iso

That’s it!  If you like my ‘no-nonsense’ videos and blogs that get straight to the point… then post a comment or let me know… Else, I’ll start posting boring video blogs!

Cross vSAN Cluster support for FT

 

FIX for Netgear Orbi Router / Firewall blocks additional subnets

Posted on Updated on

Last April 2019 my trusty Netgear Switch finally gave in.  I bought a nifty Dell PowerConnect 6224 switch and have been working with it off an on.  About the same time, I decided to update my home network with the Orbi WiFi System (RBK50) AC3000 by Netgear.  My previous Netgear Wifi router worked quite well but I really needed something to support multiple locations seamlessly.

The Orbi Mesh has a primary device and allows for satellites to be connected to it.  It creates a Wifi mesh that allows devices to go from room to room or building to building seamlessly.  I’ve had it up for a while now and its been working out great – that is until I decided to ask it to route more than one subnet.   In this blog I’ll show you the steps I took to over come this feature limitation but like all content on my blog this is for my reference – travel, use, or follow at your own risk.

**2021-NOV Update**  Per the last Orbi Update that I deployed (Router Firmware Version V2.7.3.22) the telnet option is no longer available in the debug menu.  This means the steps below will not work unless you are a earlier router firmware version.  I looked for solutions but didn’t find any.  However, I solved this issue by using an additional firewall using NAT between VLAN74 and VLAN 75.  If you find a solution, please post a comment and I’ll be glad to update this blog.

To understand the problem we need to first understand the network layout.   My Orbi Router is the Gateway of last resort and it supplies DHCP and DNS services. In my network I have two subnets which are untagged VLANS known as VLAN 74 – 172.16.74.x/24 and VLAN 75 – 172.16.75.x/24.   VLAN 74 is used by my home devices and VLAN 75 is where I manage my ESXi hosts.  I have enabled RIP v2 on the Orbi and on the Dell 6224 switch.  The routing tables are populated correctly, and I can ping from any internal subnet to any host without issue, except when the Orbi is involved.

 

Issue:  Hosts on VLAN 75 are not able to get to the internet.  Hosts on VLAN 75 can resolve DNS names (example: yahoo.com) but it cannot ping any host on the Inet. Conversely VLAN 74 can ping Inet hosts and get to the internet.  I’d like for my hosts on VLAN 75 to have all the same functionally as my hosts on VLAN 74.

Findings:  By default, the primary Orbi router is blocking any host that is not on VLAN 74 from getting to the INET.  I believe Netgear enabled this block to limit the number of devices the Orbi could NAT.  I can only guess that either the router just can’t handle the load or this was a maximum Netgear tested it to.  I found this firewall block out by logging into the CLI of my Orbi and looking at the IPTables settings.  There I could clearly see there was firewall rule blocking hosts that were not part of VLAN 74.

Solution:  Adjust the Orbi to allow all VLAN traffic (USE AT YOUR OWN RISK)

  1. Enable Telnet access on your Primary Orbi Router.
    1. Go to http://{your orbi ip address}/debug.htm
    2. Choose ‘Enable Telnet’ (**reminder to disable this when done**)
    3. Telnet into the Orbi Router (I just used putty)
    4. Logon as root using your routers main password
  2. I issued the command ‘iptables -t filter -L loc2net’. Using the output of this command I can see where line 5 is dropping all traffic that is not (!) VLAN74.
  3. Let’s remove this firewall rule. The one I want to target is the 5th in the list, yours may vary.  This command will remove it ‘iptables -t filter -D loc2net 5’
    • NOTES:
    • Router Firmware Version V2.5.1.16 (Noted: 10.2020) — It appears that more recent firmware updates have changed the targeting steps.  I noticed in Router Firmware Version V2.5.1.16 I had to add 2 to the targeted line number to remove it with the ip tables command.  This my vary for the device that is being worked on.
    • Router Firmware Version V2.5.2.4  (Noted: Jan-2021) — It appears the targeting for steps are now fixed in this version.
    • Again, as with all my posts, blogs, and videos are for my records and not for any intended purpose. 
  4. Next, we need to clean up some post routing issues ‘iptables -t nat -I POSTROUTING 1 -o brwan -j MASQUERADE’
  5. A quick test and I can now PING and get to the internet from VLAN 75
  6. Disconnect from Telnet and disable it on your router.

Note:  Unfortunately, this is not a permanent fix.  Once you reboot your router the old settings come back.  The good news is, its only two to three lines to fix this problem.  Check out the links below for more information and a script.

Easy Copy Commands for my reference:

iptables -t filter -L loc2net

iptables -t filter -D loc2net 7  << Check this number

iptables -t nat -I POSTROUTING 1 -o brwan -j MASQUERADE

If you like my ‘no-nonsense’ blog articles that get straight to the point… then post a comment or let me know… Else, I’ll start writing boring blog content.

REF:

Home Lab Gen IV – Part V Installing Mellanox HCAs with ESXi 6.5

Posted on Updated on

The next step on my InfiniBand home lab journey was getting the InfiniBand HCAs to play nice with ESXi. To do this I need to update the HCA firmware, this proved to be a bit of a challenge. In this blog post I go into how I solved this issue and got them working with ESXi 6.5.

My initial HCA selection was the ConnectX aka HP INFINIBAND 4X DDR PCI-E HCA CARD 452372-001, and Mellanox MHGA28-XTC InfiniHost III HCA these two cards proved to be a challenge when updating their firmware. I tried all types of operating systems, different drivers, different mobos, and MFT tools versions but they would not update or be OS recognized. Only thing I didn’t try was Linux OS. The Mellanox forums are filled with folks trying to solve these issues with mixed success. I went with these cheaper cards and they simply do not have the product support necessary. I don’t recommend the use of these cards with ESXi and have migrated to a ConnectX-3 which you will see below.

Updating the ConnectX 3 Card:

After a little trial and error here is how I updated the firmware on the ConnectX 3. I found the ConnectX 3 card worked very well with Windows 2012 and I was able to install the latest Mellanox OFED for Windows (aka Windows Drivers for Mellanox HCA card) and updated the firmware very smoothly.

First, I confirm the drivers via Windows Device Manager (Update to latest if needed)

Once you confirm Windows device functionality then install the Mellanox Firmware Tools for windows (aka WinMFT)

Next, it’s time to update the HCA firmware. To do this you need to know the exact model number and sometimes the card revision. Normally this information can be found on the back of your HCA. With this in hand go to the Mellanox firmware page and locate your card then download the update.

After you download the firmware place it in an accessible directory. Next use the CLI, navigate to the WinMFT directory and use the ‘mst status’ command to reveal the HCA identifier or the MST Device Name. If this command is working, then it is a good sign your HCA is working properly and communicating with the OS. Next, I use the flint command to update my firmware. Syntax is — flint -d <MST Device Name> -i <Firmware Name> burn

Tip: If you are having trouble with your Mellanox HCA I highly recommend the Mellanox communities. The community there is generally very responsive and helpful!

Installation of ESXi 6.5 with Mellanox ConnectX-3

I would love to tell you how easy this was, but the truth is it was hard. Again, old HCA’s with new ESXi doesn’t equal easy or simple to install but it does equal Home lab fun. Let me save you hours of work. Here is the simple solution when trying to get Mellanox ConnextX Cards working with ESXi 6.5. In the end I was able to get ESXi 6.5 working with my ConnectX Card (aka HP INFINIBAND 4X DDR PCI-E HCA CARD 452372-001) and with my ConnectX-3 CX354A.

Tip: I do not recommend the use of the ConnectX Card (aka HP INFINIBAND 4X DDR PCI-E HCA CARD 452372-001) with ESXi 6.x. No matter how I tried I could not update its firmware and it has VERY limited or non-existent support. Save time go with ConnectX-3 or above.

After I installed ESXi 6.5 I followed the following commands and it worked like a champ.

Disable native driver for vRDMA

  • esxcli system module set –enabled=false -m=nrdma
  • esxcli system module set –enabled=false -m=nrdma_vmkapi_shim
  • esxcli system module set –enabled=false -m=nmlx4_rdma
  • esxcli system module set –enabled=false -m=vmkapi_v2_3_0_0_rdma_shim
  • esxcli system module set –enabled=false -m=vrdma

Uninstall default driver set

  • esxcli software vib remove -n net-mlx4-en
  • esxcli software vib remove -n net-mlx4-core
  • esxcli software vib remove -n nmlx4-rdma
  • esxcli software vib remove -n nmlx4-en
  • esxcli software vib remove -n nmlx4-core
  • esxcli software vib remove -n nmlx5-core

Install Mellanox OFED 1.8.2.5 for ESXi 6.x.

  • esxcli software vib install -d /var/log/vmware/MLNX-OFED-ESX-1.8.2.5-10EM-600.0.0.2494585.zip

Ref Links:

After a quick reboot, I got 40Gb networking up and running. I did a few vmkpings between hosts and they ping perfectly.

So, what’s next? Now that I have the HCA working I need to get VSAN (if possible) working with my new highspeed network, but this folks is another post.

If you like my ‘no-nonsense’ blog articles that get straight to the point… then post a comment or let me know… Else, I’ll start writing boring blog content.