In this video I show how I installed ESXi 8 on to the ASRockRack EPC621D8A motherboard and discuss some of the caveats of doing so.
Links in the video:
ASRockRack EPC621D8A motherboard: https://www.asrockrack.com/general/productdetail.asp?Model=EPC621D8A#specifications
VMware Comptibility Guide: https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=server&productid=47030&deviceCategory=server&details=1&partner=600&page=1&display_interval=10&sortColumn=Partner&sortOrder=Asc
H3C NIC-GE-4P-360T-L3 NIC: https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=io&productid=50861&deviceCategory=io&details=1&VID=8086&DID=37d1&SVID=8086&page=1&display_interval=10&sortColumn=Partner&sortOrder=Asc
There aren’t many choices when it comes to Socket LGA 3647 CPU coolers and Noctua seems to have a solid, yet expensive, option. Based on their reputation in the industry alone I expected better instructions but this cooler soon let me down. After reading the instructions serval times I got it all together. I choose to make a video on the installation of this CPU cooler and share a few tips that might help others with their install.
It’s been a long wait for Part 8 but I was able to release it today! If you are interested on how to network performance test your storage environment this session might help. The purpose of this session is to show how to interconnect two MikroTik switches and ensure their performance is optimal when compared to a single switch. The two NAS devices in this session have different physical capabilities and by no means is this a comparison of their performance. The results are merely data points. Users should work with their vendor of choice to ensure best performance and optimization.
I’ve been building White box PCs since the early 90’s and if you seen my home lab blogs and videos its a passion that has continued on for so many years. When I look for a case, I’m usually looking for practicality and usability as it relates to the indented design. As a DIY home lab builder, using commodity cases is normal but unique cases for home labs are not always easy to find. When I do look for unique Home Lab case ideas, I usually run into lists of the gamer builds that are not so much meant for Home Labs. In this blog I wanted to compile a list of cases that are a bit more unique but someone might want to use for a home lab. For each case, I listed out some of my thoughts around home lab use cases. Of course, deeper research will be needed to determine if they fit your indented use.
USE CASE: Could be used for a stackable home lab or workstations
- ~1U Formfactor | ++ Coolness factor | Portability | Light Weight | Low Noise
- The slide out system tray makes for easy access to internal components, especially when stacked
- Tight form factor limiting options
- Sometimes limited SFX Power Supplies Options
- ITX Standard might be hard for Home Lab deployments
- Limited to 3 Drives
USE CASE: Sure this may look like a standard PC Case, but what’s unique about this case is the MANY ways it can be configured and re-configured. Because of this unique flexibility it would work well as a Workstation or ESXi Host.
- MANY case configurations options
- Want even more space? Look at the Define 7 XL
- Supports ATX and some E-ATX configurations
- Clean case design with 3 Color Options
- Horizontal and vertical PCI Slots
- Wire management
- Air Filters
- 9 Fan Connections
- Lots of Disk space
- No 5.25″ disk bays
- No front facing USB or external ports (all on top)
- It’s big and the XL even bigger
- Some options sold separately
USE CASE: With so many disk options could see this case being used for FreeNAS or even a vSAN cluster
- LOTS of disk space 5 x 3.5 and 1 2.5
- MINI-ITX / Small form factor
- PCI Low Profile slot
- Upright or Lie-down configurations
- Check out the manufacture site for more and similar case designs
- Does require SFX power supply
- The size may limit flexibility
- Only one PCI slot
- No 5.25″ disk bays
With so many 10Gbe Switch options out there for VMware Home Labs I thought I would take some time to create a list of some of the more common options.
Where did I get this data?
William Lam started the VMware Community Homelab project a few years ago. It allows Home Lab users to enter their information around their Home lab. To date the VMware Home Lab community have entered over 125 different VMware Home Labs. When a user registers they provide a URL link which leads to their home lab build-of-materials (BOM) or a description of the users home lab. Its a great resource when you are looking to see what others are doing. This was my primary data source for the results below.
On to the Results!
Over this past weekend, I took some time to review all VMware Community Homelab project links and specifically documented all the folks that noted their 10Gb Switch. I found where 25 users listed the use of a 10Gbe Switch. As I went to each link I documented the switch, its 10Gb Port count, who made it, the model, a current price, and a helpful link.
Here are the TOP 3 most popular and a curious switch:
#1 – With a user count of 7 the Ubiquity Unifi US-16-XG was the most used switch by a single model. Additionally, I noticed many of their other products in users home labs.
#2 – MikroTik with a user count of 8 across 4 different models. Their products are know to be very cost effective for 10Gbe so its no wonder they are in the top 3.
#3 – Our surprise result with a user count of 4 across 2 models is Netgear. But, its no surprise that Netgear has been making great home lab products for decades and they seem to be a bit popular in this 10Gbe arena.
Lastly, a curious switch I noted was the Brocade Communications BR-VDX6720-24-R VDX 6720. With 24 Ports of SFP+ 10Gbe its got me curious why you can find these on Ebay for ~$150. This is one switch I’ll have to look into.
This table contains to total results and extra information :
|Count||10Gb Ports||Ports||Manufacture||Product||USD Cost (05/2022)||Link||Notes|
|7||16||12 x 10G SFP+ ports | 4 x 10Gbe RJ45||Ubiquity||UniFi US-16-XG 10G||$600-800||https://store.ui.com/collections/unifi-network-switching/products/unifi-switch-16-xg|
|4||8||8 x 10 Gb SFP+ | 1 x 1Gbe RJ45||MikroTik||CRS309-1G-8S+IN||$269||https://mikrotik.com/product/crs309_1g_8s_in|
|2||8||8 x 10Gbe RJ45 | 2 RJ45/SFP+ Combo Ports||Netgear||Prosafe XS708T||$850||https://www.netgear.com/business/wired/switches/smart/xs708t/|
|2||16||16 x 10 Gb SFP+ | 1 x 1Gbe RJ45||MikroTik||CRS317-1G-16S+RM||$400||https://mikrotik.com/product/crs317_1g_16s_rm|
|2||8||8 x 10Gbe RJ45 | 1 x 10GB RJ45/SFP+ Combo Ports||Netgear||XS708E||EOL||https://www.netgear.com/support/product/XS708E.aspx||EOL|
|1||12||8 x 10Gbe RJ45| 4 x Combo (TP and SFP+) | 1 10/100Gbe RJ45||MikroTik||CRS312-4C+8XG-RM||$625||https://mikrotik.com/product/crs312_4c_8xg_rm|
|1||8||8 x 10Gbe RJ45||Buffalo||BS-XP20||EOL||https://www.buffalotech.com/resources/bs-mp20-10gbe-multi-gigabit-switch-replaces-the-bs-xp20-10gbe-switch||EOL|
|1||24||24 x 10 Gb SFP+||Lenovo||RackSwitch G8124E||EOL||https://lenovopress.lenovo.com/tips0787|
|1||24||24 x 10 Gb SFP+||Brocade Communications||BR-VDX6720-24-R VDX 6720||EOL $150-400||https://www.andovercg.com/datasheets/brocade-vdx-6720-switch-datasheet.pdf||Hard to find information on this switch|
|1||See Note||48 x 1Gbe RG45 | 4 x QSFP+ 40GB||Cisco||N3K-C3064PQ-10GX Nexus 3064||$1,200||https://www.cisco.com/c/en/us/products/collateral/switches/nexus-3000-series-switches/data_sheet_c78-651097.html||Looks like the 4 x 40GB QSFP+ can be spilt into mutiple 10Gb SFP|
|1||4||4 x 10Gb SFP+||MikroTik||CRS305-1G-4S+IN||$140||https://mikrotik.com/product/crs305_1g_4s_in|
|1||4||4 x 10GB SFP+ | 24 x 1Gbe RJ45||Cisco||3750-24P w/ Cisco C3KX-NM-10G 3K-X Network Module||https://www.cisco.com/c/en/us/products/collateral/switches/catalyst-3560-x-series-switches/data_sheet_c78-584733.html|
|1||8||4 x 10GB SFP+ | 4 x 10Gb RJ45/SFP+ Combo Ports||Qnap||QSW-804-4C||$500||https://www.qnap.com/en-us/product/qsw-804-4c|
Update: Here are a few switches that folks mentioned to me in their comments but were not part of the VMware Community HomeLab listing:
- Brocade icx-6610 ~$200 | 16x10G ports, 2x40G ports, 48x1G
- TP-Link TL-SG3428XMP $600 | 24xPOE 1G + 4xSFP+
- Netgear XS508M $600 | 8 x 10Gbe RJ45 1 x shared SFP+
- Ubiquity USW-Pro-Aggregation $899 | Layer 3 switch with (28) 10G SFP+ ports and (4) 25G SFP28 ports.
- Ubiquity USW-Aggregation $270 | L2 8x 10Gb SFP+
- Dell S4112T $1500-$5000 | Multi-function Rack Switch See docs
- D-LINK DX-1210-10TS $1200 | 8 x 10Gbe RJ50 2 x 10Gbe SFP+
- Dell X4012 ~$500-1500 | 12 x 10Gbe SFP+
It was a bit of a surprise the the following switch vendors were not mentioned by users: Linksys, Aruba (now HPE), Juniper, and Extreme Networks.
For a really good list of Network Switch and Router vendors check out this wiki page.
Lastly, it should be noted, there is a another way for Home lab users to enter their BOMs. Most recently a VMware fling known as Solution Designer is allowing Home lab users to enter their data. Here is a quick description of the new service:
The Solution Designer Fling provides a platform to manage custom VMware solutions. Building a custom VMware solution involves many challenging tasks. One of the most difficult is continuous manual verifications: checking the interoperability of multiple VMware products and performing compatible hardware validations. Solution Designer seeks to resolve these issues by automating repetitive manual steps and collecting scattered resources in a single platform.
Note: The only downside to this fling is you can only see your data and not others.
To sum it up, I’m sure this table is less then 100% accurate when it comes to VMware Home Labs. In viewing the listings on the VMware Community Home lab project, I found many dead user links and incomplete BOMs. The list above is more about how many folks are using which switch vs. the specifics of the switch. The specifics are something you might want to review at a deeper level. However, its a good start and the table above should come in handy if you are looking to compare some common 10Gbe switches for your home lab.
Thanks for reading and if I missed your switch, please do comment below and I’ll be glad to add it!
In this Quick NAS Topic video I go over how to install VirutalHere USB Server on the LOCKERSTOR 10 and its client on my Windows 10 PC. This enables the client to establish a link to the a USB NULL Model Cable which is connected directly into the NAS. Once established I’m able to use putty to create a serial SSH connection.
** Products in this Video **
- LOCKERSTOR 10 https://www.asustor.com/en/product?p_id=64
- Mikrotik CRS309-1G-8S+IN https://mikrotik.com/product/crs309_1g_8s_in
- VirtualHere https://www.virtualhere.com/home
- StarTech.com USB to Serial RS232 Adapter https://www.amazon.com/USB-Serial-Adapter-Modem-9-pin/dp/B008634VJY?th=1
In this Quick NAS Topic video and the steps further below, I use docker to create a ubuntu container with Linux tools and iperf3.
This video is a supplement for the 10Gbe Home NAS Lab Part 7. In Part 7 I show how to use these containers to network performance test the 3 NAS devices I have.
- Create your own Docker Account – https://hub.docker.com/signup
- My Docker Repository – https://hub.docker.com/u/vmexplorer
- A Blog around using a Docker Image that has Iperf3 but limits you to server mode only.
Docker Ubuntu/iperf3 Basic Steps: Items in-between [ ] and the brackets should be removed
- On the NAS:
- Ensure devices can access the inet OR not covered in this blog, you’ll need to manually import and export images, etc.
- Ensure Docker-ce and if needed Shell-in-a-box and portainer are installed and basic configuration is done. The Synology didn’t need shell in a box or portainter
- Test Docker Install
- docker -v << Shows the version
- docker images << Show the images that are available
- docker ps << Shows the running containers
- Elevate local privileges to run docker commands
- It may be necessary to use ‘sudo’ in front of docker commands to get them to execute, followed by the admin/root password. Example: sudo docker ps
- Download and run Ubuntu
- docker pull ubuntu << Image is located here https://hub.docker.com/_/ubuntu
- docker run -it ubuntu bash << Creates an instance of this image for us to modify and opens up the terminal
- Update the Ubuntu running container
- apt-get -y update
- apt-get install iproute2
- apt-get install net-tools
- apt-get install iputils
- apt-get install iputils-ping
- apt-get install -y iperf3
- Test with ping and iperf3 -v
- Do not exit
- Commit and push the new image
- docker ps -l << Check for the latest running container, and note the Container ID of the container that was just updated with these steps
- docker commit [Container ID] [repository name]/[insert-container-name]
- docker images << will validate that the image is now there
- docker push [repository name]/[Container you want to push]
- Testing Steps
- Check Product Interoperability Matrix (VCSA, ESXi, NSX, vRNI, VRLI)
- Check VMware Compatibility Guide (Network Cards, JBOD)
- Ensure the vSAN Cluster is in a health state
- Backup VM’s
- Ensure your passwords are updated
- Document Basic Host settings (Network, vmks, NTP, etc.)
- Backup VCSA via the Management Console > Backup
- NOTE: I used /n software SFTP server to receive backup files
Steps to update vCenter Server from 7U2d (7.0.2.00500) to 7U3a (7.0.3.00100):
- Downloaded VCSA 7U3a VMware-vCenter-Server-Appliance-7.0.3.00100-18778458-patch-FP.iso
- Use WinSCP to connect to an ESXi host and upload the update/patch to vSAN ISO-Images Folder
- Mount the ISO from step 1 to VCSA 7U2d VM
- NOTE: A reboot of the VCSA my be necessary for it to recognize the attached ISO
- Went to VCSA Management Console > Update > Check Updates should auto-start
- NOTE: It might fail to find the ISO. If so, choose CD ROM to detect the ISO
- Expanded the Version > Run Pre-Update checks
- Once it passed pre-checks, choose Stage and Install > Accept the Terms > Next
- Check ‘I have backed up vCenter Server…’
- NOTE: Clicking on ‘go to Backup’ will Exit out and you’ll have to start over
- Click Finish and allow it to complete
- Once done log back into the Management console > Summary and validate the Version
- Lastly, detach the datastore ISO, I simple choose ‘Client Device’
Change Boot USB to SSD and upgrade to ESXi 7U3 on Host at a time:
- Remove Host from NSX-T Manager (Follow these steps)
- In vCenter Server
- Put Host 1 in Maintenance Mode Ensure Accessibility (better if you can evacuate all data | run pre-check validation)
- Shut down the host
- Remove Host from Inventory (NOTE: Wait for host to go to not responding first)
- On the HOST
- Precautionary step – Turn off the power supply on the host, helps with the onboard management ability to detect changes
- Remove the old USB boot device
- Install Dell HBA330 and M.2/NVMe PCIe Card w/ 240GB SSD into the Host
- Power On the Host and validate firmware is updated (Mobo, Disk, Network, etc.)
- Here is how I updated the Dell HBA330 Firmware (SEE VIDEO HERE)
- During boot ensure the Dell HBA330 POST screen displays (optional hit CTRL-C to view its options)
- In the Host BIOS Update the boot disk to the new SSD Card
- ESXi Install
- Boot the host to ESXi 7.0U3 ISO (I used SuperMicro Virtual Media to boot from)
- Install ESXi to the SSD Card, Remove ISO, Reboot
- Update Host boot order in BIOS for the SSD Card and boot host
- In the ESXi DUCI, configure host with correct IPv4/VLAN, DNS, Host Name, enable SSH/Shell, disable IPv6 and reboot
- From this ESXi host and from another connected device, validate you can ping the Host IP and its DNS name
- Add Host to the Datacenter (not vSAN Cluster)
- Ensure Host is in Maintenance mode and validate health
- Erase all partitions on vSAN Devices (Host > Configure > Storage Devices > Select devices > Erase Partitions)
- Rename the new SSD datastore (Storage > R-Click on datastore > Rename)
- Add Host to Cluster (but do not add to vSAN)
- Add Host to vDS Networking, could be multiple vDS switches (Networking > Target vDS > Add Manage Hosts > Add Hosts > Migrate VMKernel)
- Complete the Host configuration settings (NTP, vmks)
- Create vSAN Disk Groups (Cluster > Configure > vSAN > Disk Management)
- Monitor and allow to complete, vSAN Replication Objects (Cluster > Monitor > vSAN > Resyncing Objects)
- Extract a new Host Profile and use it to build out the other hosts in the cluster
- ESXi Install – Additional Hosts
- Repeat Steps 1, 2, 3, and only Steps 4.1-4.10
- Attach Host Profile created in Step 4.15
- Check Host Profile Compliance
- Edit and update Host Customizations
- Remediate the host (the remediation will to a pre-check too)
- Optional validate host settings
- Exit Host from Maintenance mode
- Before starting next host ensure vSAN Resyncing Objects is completed
Other Notes / Thoughts:Host Profiles: You may be thinking “why didn’t he use ESXi Backup/Restore or Host Profiles to simply this migration vs. doing all these steps?”. Actually, at first I did try both but they didn’t work due to the add/changes of PCIe devices and upgrade of the ESXi OS. Backup/Restore and Host Profiles really like things to not change for them to work with out error. Now there are adjustments one could make and I tried to adjust them but in the end I wasn’t able to get them to adjust to the new hosts. They were just the wrong tool for the first part of this job. However, Host Profiles did work well post installation after all the changes were made. vSAN Erase Partitions Step 4.8: This step can be optional it just depends on the environment. In-fact I skipped this step on the last host and vSAN imported the disks with out issue. Granted most of my vm’s are powered off, which means the vSAN replicas are not changing. In an environment where there are a lot of powered on VM’s vSAN doing step 4.8 might be best. Again, it just depends on the environment state. If you like my ‘no-nonsense’ videos and blogs that get straight to the point… then post a comment or let me know… Else, I’ll start posting really boring content!
In this quick video I review how I updated the Dell HBA330 firmware using a Windows 10 PC.
This video was made as a supplement to my 2 Part blog post around updating my Home Lab Generation 7.
In this not so Quick NAS topic I cover how to expand a RAID 1 volume and migrate it to a RAID 5 storage pool with the Synology 1621+. Along the way we find a disk that has some bad sectors, run an extended test and then finalize the migration.
** Products / Links Seen in this Video **
Synology DiskStation DS1621+ — https://www.synology.com/en-us/products/DS1621+