vCenter Server

Home Lab Generation 7: Part 2 – New Hardware and Software Updates

Posted on Updated on

In the final part of this 2 part series, I’ll be documenting the steps I took to update my Home Lab Generation 7 with the new hardware and software changes.  There’s quite a bit of change going on and these steps worked well for my environment.

Pre-Update-Steps:

  1. Check Product Interoperability Matrix (VCSA, ESXi, NSX, vRNI, VRLI)
  2. Check VMware Compatibility Guide (Network Cards, JBOD)
  3. Ensure the vSAN Cluster is in a health state
  4. Backup VM’s
  5. Ensure your passwords are updated
  6. Document Basic Host settings (Network, vmks, NTP, etc.)
  7. Backup VCSA via the Management Console > Backup

Steps to update vCenter Server from 7U2d (7.0.2.00500) to 7U3a (7.0.3.00100):

  1. Downloaded VCSA 7U3a VMware-vCenter-Server-Appliance-7.0.3.00100-18778458-patch-FP.iso
  2. Use WinSCP to connect to an ESXi host and upload the update/patch to vSAN ISO-Images Folder
  3. Mount the ISO from step 1 to VCSA 7U2d VM
    • NOTE: A reboot of the VCSA my be necessary for it to recognize the attached ISO
  4. Went to VCSA Management Console > Update > Check Updates should auto-start
    • NOTE: It might fail to find the ISO. If so, choose CD ROM to detect the ISO
  5. Expanded the Version > Run Pre-Update checks
  6. Once it passed pre-checks, choose Stage and Install > Accept the Terms > Next
  7. Check ‘I have backed up vCenter Server…’
    • NOTE: Clicking on ‘go to Backup’ will Exit out and you’ll have to start over
  8. Click Finish and allow it to complete
  9. Once done log back into the Management console > Summary and validate the Version
  10. Lastly, detach the datastore ISO, I simple choose ‘Client Device’

Change Boot USB to SSD and upgrade to ESXi 7U3 on Host at a time:

  1. Remove Host from NSX-T Manager (Follow these steps)
  2. In vCenter Server
    1. Put Host 1 in Maintenance Mode Ensure Accessibility (better if you can evacuate all data | run pre-check validation)
    2. Shut down the host
    3. Remove Host from Inventory (NOTE: Wait for host to go to not responding first)
  3. On the HOST
    1. Precautionary step – Turn off the power supply on the host, helps with the onboard management ability to detect changes
    2. Remove the old USB boot device
    3. Install Dell HBA330 and M.2/NVMe PCIe Card w/ 240GB SSD into the Host
    4. Power On the Host and validate firmware is updated (Mobo, Disk, Network, etc.)
    5. During boot ensure the Dell HBA330 POST screen displays (optional hit CTRL-C to view its options)
    6. In the Host BIOS Update the boot disk to the new SSD Card
  4. ESXi Install 
    1. Boot the host to ESXi 7.0U3 ISO (I used SuperMicro Virtual Media to boot from)
    2. Install ESXi to the SSD Card, Remove ISO, Reboot
    3. Update Host boot order in BIOS for the SSD Card and boot host
    4. In the ESXi DUCI, configure host with correct IPv4/VLAN, DNS, Host Name, enable SSH/Shell, disable IPv6 and reboot
    5. From this ESXi host and from another connected device, validate you can ping the Host IP and its DNS name
    6. Add Host to the Datacenter (not vSAN Cluster)
    7. Ensure Host is in Maintenance mode and validate health
    8. Erase all partitions on vSAN Devices (Host > Configure > Storage Devices > Select devices > Erase Partitions)
    9. Rename the new SSD datastore (Storage > R-Click on datastore > Rename)
    10. Add Host to Cluster (but do not add to vSAN)
    11. Add Host to vDS Networking, could be multiple vDS switches (Networking > Target vDS > Add Manage Hosts > Add Hosts > Migrate VMKernel)
    12. Complete the Host configuration settings (NTP, vmks)
    13. Create vSAN Disk Groups (Cluster > Configure > vSAN > Disk Management)
    14. Monitor and allow to complete, vSAN Replication Objects (Cluster > Monitor > vSAN > Resyncing Objects)
    15. Extract a new Host Profile and use it to build out the other hosts in the cluster
  5. ESXi Install – Additional Hosts
    1. Repeat Steps 1, 2, 3, and only Steps 4.1-4.10
    2. Attach Host Profile created in Step 4.15
    3. Check Host Profile Compliance
    4. Edit and update Host Customizations
    5. Remediate the host (the remediation will to a pre-check too)
    6. Optional validate host settings
    7. Exit Host from Maintenance mode
    8. Before starting next host ensure vSAN Resyncing Objects is completed

Other Notes / Thoughts:

Host Profiles: You may be thinking “why didn’t he use ESXi Backup/Restore or Host Profiles to simply this migration vs. doing all these steps?”.  Actually, at first I did try both but they didn’t work due to the add/changes of PCIe devices and upgrade of the ESXi OS.  Backup/Restore and Host Profiles really like things to not change for them to work with out error.  Now there are adjustments one could make and I tried to adjust them but in the end I wasn’t able to get them to adjust to the new hosts.  They were just the wrong tool for the first part of this job.   However, Host Profiles did work well post installation after all the changes were made. vSAN Erase Partitions Step 4.8:  This step can be optional it just depends on the environment.  In-fact I skipped this step on the last host and vSAN imported the disks with out issue.  Granted most of my vm’s are powered off, which means the vSAN replicas are not changing.  In an environment where there are a lot of powered on VM’s vSAN doing step 4.8 might be best.  Again, it just depends on the environment state. If you like my ‘no-nonsense’ videos and blogs that get straight to the point… then post a comment or let me know… Else, I’ll start posting really boring content!

Home Lab Generation 7: Part 1 – Change Rational for software and hardware changes

Posted on Updated on

Well its that time of year again, time to deploy new changes, upgrades, and add some new hardware.  I’ll be updating my ESXi hosts and vCenter Server to the latest vSphere 7 Update 3a from 7U2d. Additionally, I’ll be swapping out the IBM 5210 JBOD for a Dell HBA330+ and lastly I’ll change my boot device to a more reliable and persistent disk.  I have 3 x ESXi hosts with VSAN, vDS switches, and NSX-T.  If you want to better understand my environment a bit better check out this page on my blog.  In this 2 part blog I’ll go through the steps I took to update my home lab and some of the rational behind it.

There are two main parts to the blog:

  • Part 1 – Change Rational for software and hardware changes – In this part I’ll explain some of my thoughts around why I’m making these software and hardware changes. 
  • Part 2 – Installation and Upgrade Steps – These are the high level steps I took to change and upgrade my Home lab

Part 1 – Change Rational for software and hardware changes:

There are three key changes that I plan to make to my environment:

  • One – Update to vSphere 7U3a
    • vSphere 7U3 has brought many new changes to vSphere including many needed features updates to vCenter server and ESXi.  Additionally, there have been serval important bug fixes and corrections that vSphere 7U3 and 7U3a will address. For more information on the updates with vSphere 7U3 please see the “vSphere 7 Update 3 – What’s New” by Bob Plankers.  For even more information check out the release notes.   
    • Part of my rational in upgrading is to prepare to talk with my customers around the benefits of this update.   I always test out the latest updates on Workstation first then migrate those learnings in to Home Lab.  
  • Two – Change out the IBM 5210 JBOD
    • The IBM 5210 JBOD is a carry over component from my vSphere 6.x vSAN environment. It worked well with vSphere 6.x and 7U1.  However, starting in 7U2 it started to exhibit stuck IO issues and the occasional PSOD.  This card was only certified with vSphere/vSAN 6.x and at some point the cache module became a requirement.  My choices at this point are to update this controller with a cache module (~$50 each) and hope it works better or make a change.  In this case I decided to make a change to the Dell HBA330 (~$70 each).  The HBA330 is a JBOD controller that Dell pretty much worked with VMware to create for vSAN.  It is on the vSphere/vSAN 7U3 HCL and should have a long life there too.  Additionally, the HBA330 edge connectors (Mini SAS SFF-8643) line up with the my existing SAS break-out cables. When I compare the benefits of the Dell HBA330 to upgrading the cache module for the IBM 5210 the HBA330 was the clear choice.  The trick is finding a HBA330 that is cost effective and comes with a full sized slot cover.  Its a bit tricky but you can find them on eBay, just have to look a bit harder.

  • Three – Change my boot disk
    • Last September-2021, VMware announced boot from USB is going to change and customers were advised to plan ahead for these upcoming changes.   My current hosts are using cheap SanDisk USB 64GB memory sticks.  Its something I would never recommend for a production environment, but for a Home Lab these worked okay.  I originally chose them during my Home Lab Gen 5 updates as I need to do testing with USB booted Hosts.  Now that VMware has deprecated support for USB/SD devices it’s time to make a change. Point of clarity: the word deprecated can mean different things to different people.  However, in the software industry deprecated means “discourage the use of (something, such as a software product) in favor of a newer or better alternative”.  vSphere 7 is in a deprecated mode when it comes to USB/SD booted hosts, they are still supported, and customers are highly advised to plan ahead. As of this writing, legacy (legacy is a fancy word for vSphere.NEXT) USB hosts will require a persistent disk and eventually (Long Term Supported) USB/SD booted hosts will no longer be supported.  Customers should seek guidance from VMware when making these changes.

    • The requirement to be in a “Long Term Supported” mode is to have a ESXi host be booted from HDD, SSD, or a PCIe device.  In my case, I didn’t want to add more disks to my system and chose to go with a PCIe SSD/NVMe card. I chose this PCIe device that will support M.2 (SATA SSD) and NMVe devices in one slot and I decided to go with a Kingston A400 240G Internal SSD M.2  as my boot disk. The A400 with 240GB should be more than enough to boot the ESXi hosts and keep up with its disk demands going forward.   

 

Final thoughts and a important warning.  Making changes that affect your current environment are never easy but are sometimes necessary.  With a little planning it can make the journey a bit easier.  I’ll be testing these changes over the next few months and will post up if issues occur.  However, a bit of warning – adding new devices to an environment can directly impact your ability to migrate or upgrade your hosts.  Due to the hardware decisions I have made a direct ESXi upgrade is not possible and I’ll have to back out my current hosts from vCenter Server plus other software and do a new installation.  However, those details and more will be in Part 2 – Installation and Upgrade Steps.

Opportunity for vendor improvement – If backup vendors like Synology, asustor, Veeam, Veritas, naviko, and Arcoins could really shine.  If they could backup and restore a ESXi host to dislike hardware  or boot disks this would be a huge improvement for VI Admin, especially when they have tens of thousands of hosts the need to change from their USB to persistent disks.  This is not a new ask, VI admins have been asking for this option for years, now maybe these companies will listen as many users and their hosts are going to be affected by these upcoming requirements.

Great VSAN 6.6 Network Primer Video!

Posted on Updated on

At VMworld 2017 Cormac Hogan and Andreas Scherr did a great job going over the basics and gotchas around VSAN 6.6 Networking. Additionally, towards the end of the video they went through a Demo on performance and talked about the different VSAN Network topologies. The video is about an hour long and I know finding the time to watch it all can be hard sometimes. However, I took the time to breakdown the video and I listed when each topic started at. (just incase you want to jump to a specific topic of interest)

What I found beneficial was the information around the Network Unicast and vCenter Server new role with VSAN host tracking. Both topics are well work a look and it starts @19:22 in the video.

Here is the link to: VMworld 2017 – STO1193BE – Closer Look at VMware vSAN Networking and Configuration Considerations

https://www.youtube.com/watch?v=h-Ad4OSzS1Y

Here is the topic breakdown if you want to go to a specific section.

  • @3:42     – Major Component overview
  • @5:09    — Ports and Firewall
    • Encryption need 3rd Party KMS provider
  • @6:54    — IPv6
    • Don’t rung IPv6 and IPv4 mixed mode, okay to run to migrate but not run over a long time
  • @7:57    – Min NIC Requirements
    • Great chart on min / Max, see attached screen shot
  • @10:00    – Discussion around vSS vs vDS
    • Major difference is vDS can use lag groups
  • @13:17    – Network IO Control with vDS
    • Can help with vMotion traffic over whelming VSAN track
  • @14:17 – NIC Teaming and Failover Options
    • Load balancing options are a bit weak
    • LAG tends to be the best for load balancing (vDS and Physical Switch config needed)
  • @15:55 – Multicast
  • @19:22 – Unicast
    • vCenter Server now tracks who is in the cluster and what core info
  • @22:15 – Upgrade / Mixed Cluster Considerations with Unicast
    • Great chart around upgrading to vSAN 6.6
  • @22:24 – Considerations for DHCP
    • Not a good idea to run DHCP
  • @26:22 – Unicast CLI Commands
  • @27:25 – NIC Teaming and Load Balancing
  • @28:07 – NIC Teaming Pros/Cons
  • @33:58    – Supported Network Topologies
  • @36:06 – Layer2, Single site, Single Rack
  • @36:55 – Layer2, Single Site, Multi Rack (pre-VSAN-6.6)
  • @37:51 – Layer2, Single Site, Multi Rack VSAN 6.6 and later Unicast
  • @38:38 – Stretch Cluster (SC) L2 Data, L3 Witness
  • @39:37 – SC Why not L2 only traffic?
  • @41:15 – 2 Node Robo
  • @42:08 – 2 Node Direct Connect and Witness Traffic Separation
  • @43:57 – VSAN and Network Performance (General Concept)
  • @46:46 – Host Network Performance
  • @48:05 – Network Latency Demo

If you like my ‘no-nonsense’ blog articles that get straight to the point… then post a comment or let me know… Else, I’ll start writing boring blog content.

Limited vCenter Server options with Windows 2016

Posted on Updated on

If you plan to update your vCenter Server to Windows 2016 then you might want to make sure you do your homework. Recently after reviewing the following KB its apparent that vCenter Server for Windows 2016 is only supported with vCenter Server 6.5. This might be a great time to consider moving to the vCenter Server Appliance (aka VCSA).

Here is the KB around the compatibility – https://kb.vmware.com/s/article/2091273?language=en_US

vSphere 6.0 / 6.5 Cross reference build release for ESXi, vSAN, and vCenter Server

Posted on Updated on

I love the Correlating build numbers and versions of VMware products (1014508). This one KB has made my job, and I’m sure yours too, so much easier. Before this KB was released it was a bit difficult to correlate build, patch, and update levels to vSphere Environments. Now with just a few clicks one can find out all this information and more. However, I really need the ability to correlate multiple core products. Typically, I work with — ESXi, vCenter Server, and vSAN. So, today I took the time today to align all this information.

It took me about 5 mins to build the chart below but it will save me loads of time. Can’t tell you how many times I’ve been asked which version of ESXi was related to which version of vSAN and Oh, what version of vCenter Server was released with it? Well with this cart below you can answer those questions and more.

~ Enjoy!

vSAN version

ESXi version

Release Date

Build Number

vCenter Server

Version

Release Date

Build Number

vSAN 6.6.1

ESXi 6.5 Update 1

7/27/2017

5969303

vCenter Server 6.5 Update 1

7/27/2017

5973321

       

vCenter Server 6.5 0e Express Patch 3

6/15/2017

5705665

vSAN 6.6

ESXi 6.5.0d

4/18/2017

5310538

vCenter Server 6.5 0d Express Patch 2

4/18/2017

5318154

vSAN 6.5 Express Patch 1a

ESXi 6.5 Express Patch 1a

3/28/2017

5224529

vCenter Server 6.5 0c Express Patch 1b

4/13/2017

5318112

vSAN 6.5 Patch 01

ESXi 6.5 Patch 01

3/9/2017

5146846

vCenter Server 6.5 0b Patch 1

2017-03-14

5178943

vSAN 6.5.0a

ESXi 6.5.0a

2/2/2017

4887370

vCenter Server 6.5 0a Express Patch 1

2/2/2017

4944578

vSAN 6.5

ESXi 6.5 GA

11/15/2016

4564106

vCenter Server 6.5 GA

11/15/2016

4602587

vSAN 6.2 Patch 5

ESXi 6.0 Patch 5

7/11/2017

5572656

     

vSAN 6.2 Express Patch 7c

ESXi 6.0 Express Patch 7c

3/28/2017

5251623

vCenter Server 6.0 Update 3b

4/13/2017

5318200/5318203

vSAN 6.2 Express Patch 7a

ESXi 6.0 Express Patch 7a

3/28/2017

5224934

vCenter Server 6.0 Update 3a

3/21/2017

5183549

vSAN 6.2 Update 3

ESXi 6.0 Update 3

2/24/2017

5050593

vCenter Server 6.0 Update 3

2/24/2017

5112527

vSAN 6.2 Patch 4

ESXi 6.0 Patch 4

11/22/2016

4600944

vCenter Server 6.0 Update 2a

11/22/2016

4541947

vSAN 6.2 Express Patch 7

ESXi 6.0 Express Patch 7

10/17/2016

4510822

     

vSAN 6.2 Patch 3

ESXi 6.0 Patch 3

8/4/2016

4192238

     

vSAN 6.2 Express Patch 6

ESXi 6.0 Express Patch 6

5/12/2016

3825889

     

vSAN 6.2

ESXi 6.0 Update 2

3/16/2016

3620759

vCenter Server 6.0 Update 2

3/16/2016

3634793

vSAN 6.1 Express Patch 5

ESXi 6.0 Express Patch 5

2/23/2016

3568940

     

vSAN 6.1 Update 1b

ESXi 6.0 Update 1b

1/7/2016

3380124

vCenter Server 6.0 Update 1b

1/7/2016

3339083

vSAN 6.1 Express Patch 4

ESXi 6.0 Express Patch 4

11/25/2015

3247720

     

vSAN 6.1 U1a (Express Patch 3)

ESXi 6.0 U1a (Express Patch 3)

10/6/2015

3073146

     

vSAN 6.1

ESXi 6.0 U1

9/10/2015

3029758

vCenter Server 6.0 Update 1

9/10/2015

3018524

vSAN 6.0.0b

ESXi 6.0.0b

7/7/2015

2809209

vCenter Server 6.0.0b

7/7/2015

2776511

vSAN 6.0 Express Patch 2

ESXi 6.0 Express Patch 2

5/14/2015

2715440

     

vSAN 6.0 Express Patch 1

ESXi 6.0 Express Patch 1

4/9/2015

2615704

vCenter Server 6.0.0a

4/16/2015

2656760

vSAN 6.0

ESXi 6.0 GA

3/12/2015

2494585

vCenter Server 6.0 GA

3/12/2015

2559268

 

If you like my ‘no-nonsense’ blog articles that get straight to the point… then post a comment or let me know… Else, I’ll start writing boring blog content.

2 VMTools Secrets your mother never told you about!

Posted on Updated on

These are pretty common asks amongst operators of ESXi – ‘Which VMtools version came with my ESXi Host’ and ‘Where can I view and download all the VMTools directly?’ The answers are below and the outputs aren’t pretty but they sure are useful!

1st – Check out the URL below to see all the ESXi Host build to released versions.

https://packages.vmware.com/tools/versions

2nd – Where can I view and download all the VMTools directly

https://packages.vmware.com/tools/esx/index.html

Finally, if you read this far then you are in luck here is the best tip — Watch this video and you’ll know more about VMtools than your mom :)

http://vmware.mediasite.com/mediasite/Play/6d33be3f5da840a19ec1997e220aedfe1d

If you like my ‘no-nonsense’ blog articles that get straight to the point… then post a comment or let me know… Else, I’ll start writing boring blog content.

 

vSAN – Working with the vSAN HCL Database

Posted on Updated on

The vSAN HCL DB is a local file enabling vCenter Server to validate your vSAN hardware deployment.   This local DB file contains information around the supported products on the VMware compatibility guides. Part of the vSAN Health checks is validating the age of the vSAN HCL DB file.  The initial release of the health feature ships with a copy of the vSAN HCL DB, which was current when released. This copy of the database will become outdated over time. The file can be updated via an internet connection or through manual download (See KB’s below). However, if the HCL DB file is not updated and is 90 days past you will see a warning and at 180 days past you’ll receive an error. These alerts in no way will affect your vSAN cluster as they are merely non-impactful alarms.

You can find this check by clicking on your vSAN Cluster > Monitor > Virtual SAN > Health and then expand Hardware compatibility (See the PIC below). Under Hardware compatibility, you will see various checks that validate your installation.   The ‘vSAN HCL DB up to date’ is the check that will alarm when needed.

You might be thinking –

“I validated my vSAN deployment against the HCL & VCL’s when it was initially built, so why do I need to recheck it over and over?” There are a few good reasons why this validation is important. First off – New firmware and drivers are validated on a routine basis, keeping on top of these will help to ensure your vSAN cluster is able to work optimally and is less problematic. Second – Just because a component was listed on the VGC, doesn’t necessarily mean it will stay on the VGC. Allowing vSAN to self-check itself not only will save you time but will identify any potential issues.

“My vSAN cluster doesn’t have an internet connection and I am pretty good about keeping up to date on the VGC. Do I really need these checks, and if not how can I disable them” Frist off I would not recommend disablement but there may be a need for this. It could be very true that your company does a good job of manually checking the VCG but automating these check would only help your efforts and would be more efficient. However, there are some deployments where automated checks may not be desirable. For those cases follow this guidance to disable: Cluster > Manage > Virtual SAN > General > Internet Connectivity > Disable Auto HCL update

For more information around the vSAN HCL DB, including how to disable and update, please see the following KB’s

In this PIC I’m showing where you can locate the vSAN HCL DB Check status.

Screen Shot 2017-04-20 at 5.14.57 PM

If you like my ‘no-nonsense’ blog articles that get straight to the point… then post a comment or let me know… Else, I’ll start writing boring blog content.

What new in vSphere 6.5

Posted on Updated on

Hey folks — This great video came my way today.  Watch Kevin Steil, (Southeast VMware Technical Account Manager (TAM) Team Lead) talk about HOL-1710-SDC-3 – vSphere with Operations Management: Product Deep Dive which introduces the cool, new features coming in vSphere 6.5.

P2V GOLD – Remove all Windows Non-Present devices at once via GUI and CLI!

Posted on Updated on

Issue >> If you’ve done any type of Windows P2V (Physical 2 Virtual) then you’d know all about the value in removing non-present or ghosted devices. Normally non-present devices are harmless but from time to time they can cause you an issue or two. P2V best practice is to remove non-present devices enabling a pristine OS. The issue with removing non-present devices is the time to complete the task. Currently, you have to go to command line, enter a few commands, and then manually remove each non-present device from device manager. If you have to remove 200+ non-present devices that could take several hours to complete. Until now…

Solution >> I located 3 great tools that remove all the non-present devices at once — Device Clean up Tool GUI based, Device Clean up Tool CLI based, and Ghostbuster GUI based. All the links are below.

Other Notes >> Personally, I used the Device Cleanup Tool GUI and I was able to remove 213 devices from my recent P2V. Not only did it clean up my OS but it also fixed a pesky USB issue I was having.

Device Cleanup Tool V0.5 – removes non-present devices from the Windows device management

Device Cleanup CMD

GhostBuster

If you like my ‘no-nonsense’ blog articles that get straight to the point… then post a comment or let me know… Else, I’ll start writing boring blog content.

ESXi Host NIC failure and the Web client vSwitch orange line doesn’t move? — The results are Shocking!

Posted on Updated on

Okay, the title was a bit dramatic, but it got your attention. Now keeping with my quest to deliver no-nonsense blog articles here is what the orange line means…

Question 1 – What is the function of the orange line when selecting a vmnic, port group, or vSwitch while viewing them in the Web client network settings?

The orange line is showing you the teaming order for the pNICs or vmnics based on their vSwitch or port group teaming policy. In this screenshot, the policy is Active / Active for both vmnic0 and 1.

The orange line will not move to the other pNIC’s unless they are marked as “active” in the teaming policy. “Active in the teaming policy” vs. “which pNIC is passing traffic” are two different things. The orange line is not a representation of the latter, “pNIC passing traffic”.

 

Question 2 – How can I tell which pNIC is currently passing traffic?

The Web or Thick client vSwitch display (aka the orange line) doesn’t display the pNIC which is currently passing network traffic. You need to use ESXTOP to determine the active pNIC.

Simply go into ESXTOP, Press N, find your vSwitch and it will lead you to the pNIC currently being used to pass traffic.

 

Question 3 – I had a pNIC failure why isn’t the Web client moving the orange line to the standby NIC?

Again… the orange line ONLY points to the Active pNIC in the teaming policy. In this screenshot below, the teaming policy is setup for vmnic3 as Active and vmnic2 as stand by.

Even though vmnic3 is down, traffic should be flowing through vmnic2. Use ESXTOP to determine this (See Question 2)

 

 

If you like my ‘no-nonsense’ blog articles that get straight to the point… then post a comment or let me know… Else, I’ll start writing boring blog content.