Announcing my Generation 8 Super VMware Workstation!

Posted on Updated on

I’ve been using VMware Workstation for many years and its been a great tool to have in my home lab plan. Over the last couple of years, I had been using multiple computers to do various tasks (VMware Workstation, PC, PLEX, Video Editing, etc.) and they seemed to work okay. Most of these were older PCs I had around that I repurposed. However, a follower of my blog made a very generous donation of several key components. It was enough to really get me thinking about creating a SUPER Workstation box. I wanted to make an all in one workstation that could handle everything I currently supported plus more. I started to evaluate what I wanted to accomplish and created a simple goal list.

Goals:

  • Support for 3 different Networks:
    • I wanted to have a 1Gbe and 10Gbe into my Home lab
    • The other connection would be used for basic networking
  • Run VMware Workstation:
    • Support several PC VMs
    • Support nested ESXi/vSAN 8 VMs, without having to force the install
    • Have enough resources to run everything from one box
  • Run PLEX Server
    • I wanted to make sure I had enough HDD space to run my PLEX media
  • Video Editing
    • Support x16 PCIe Video Card
    • From time to time I create videos and I wanted a system that could be performant around this.

From there I inventoried the parts I already had, chose new components, and created my Generation 8 VMware Workstation.

BOM:

To better clarify some of my component choices I created this video and announced my DREAM #vmware #workstation

I’m sure it will be updated as time goes and my most recent Bill of Materials (BOM) can be found here. https://vmexplorer.com/home-lab-bom/

Working with Dell T7820 Disk trays

Posted on Updated on

As part of my Generation 8 ESXi/vSAN ESA Home lab build, in this video I go over how to work with the T7820 disk trays. They can be a bit tricky to work with, especially when adding or subtracting the 2.5″ carrier.

First Look GEN8 ESXi/vSAN ESA 8 Home Lab (Part 1)

Posted on Updated on

I’m kicking off my next generation home lab with this first look in to my choice for an ESXi/vSAN 8 host. There will be more videos to come as this series evolves!

Dell T7820: CPU Upgrade Issues *Solved*

Posted on Updated on

Quick Read: Did you buy a used barebones Dell T7820 or upgrade your CPU from Xeon Silver to Gold and now the T7820 won’t boot? Are you getting the error Code Memory/RAM failure: 2 amber blinks followed by a short pause, 4 white blinks, long pause, then repeats:?

Solution: Ensure your BIOS is at least 2.6.3 or later.

More Details:

I recently purchased 3 used Dell Precision Workstations model T7820 from eBay. They will be the replacements for my next generation Home Lab with vSphere 8. These used T7820 are barebone systems that didn’t come with a CPU or RAM. Other than a vNivida Video card they were pretty much empty. After inserting a Xeon 6252 and RAM modules, they all powered on but only one would post. The other two simply gave me a blink code (2 amber blinks followed by a short pause, 4 white blinks, long pause, then repeats). It was an odd error as their manuals stated it was related to a RAM issue but I was using supported RAM ( 2933Mhz) for the CPU I was inserting (Xeon Gold 6252).

Looking at the Dell 7820 manual I soon found Dell code description that matched the blink code: Memory/RAM failure. I swapped the RAM and CPU from my working T7820 to the non-working systems but still the errors persisted. So at this point, I knew my RAM and CPU were compatible with the T7820 but why would only 1 of 3 them work?

I now put a focus on the one system that worked. I first checked its BIOS level. It was running Dell BIOS 2.6.3 and funny enough the BIOS release notes specifically called out fixing a memory speed issue. It read: ‘Supports memory speed of 2933 MHz with two memory modules per channel’

It appears that when the T7820 posts, it queries the CPU for its max memory speed rating. In my case the Xeon 6252 has a max memory speed is 2933Mhz. So, even if I put in 2666Mhz RAM it still will not work, because its asking the CPU for is max speed rating. Dell BIOS 2.6.3 fixes this issue allowing the for faster memory speeds to be supported.

Knowing this, the fix should be simple. All I need to do is update my BIOS to 2.6.3 or later. However, the issue is I can’t update or even check the BIOS version on my 2 non-working systems. To do this I’m going to need a Xeon CPU whose max RAM frequency is slower than 2933Mhz, it’s something I don’t have. Enter my eBay Seller oztech llc. to the rescue!

I’ve heard of so many bad experiences when working with eBay sellers, but working with the eBay vendor oztech llc has been an absolute pleasure and I would highly recommend them. They are very responsive, helpful, and knowledgeable around the product they sell. They had not heard of this issue before but were willing to help and after a short call explaining my issue, they promptly shipped out a Xeon Silver 4114 CPU that supported a max RAM speed of 2666Mhz and matching RAM. This should allow me power on my T7820s and update to the latest Dell BIOS (2.29.0)

When the Xeon Silver CPU and RAM arrived I did the following:

  • Prepare the T7820
    • With the power off, I removed the existing Xeon Gold CPU and 2933 Mhz RAM
    • Plugged in the power to the system and then unplugged (don’t skip this step)
    • Cleared the CMOS
    • Installed the Xeon Silver CPU and 2666Mhz RAM in Slot one
    • Plugged in the T7820
    • It power cycled about 4 times as it adjusted the system settings, and then it booted!
  • Confirm Current BIOS Level
    • During boot I pushed F12 and went into setup
    • Confirmed its BIOS level to be 1.7.1
  • Updating the BIOS to 2.29.0
    • The T7820 BIOS update is designed to work with Windows 10.
    • I booted to Windows 10, and ran the 2.29.0 BIOS update
    • The update confirmed it was currently at 1.7.1
    • I ran the BIOS update
    • Once it was completed I checked the T7820 BIOS but it came up with BIOS 2.6.3
    • I ran the BIOS update again, it confirmed 2.6.3 was current and it would update the T7820 to 2.29.0
    • After a reboot I confirmed the BIOS to be 2.29.0
  • Finally, installing the Xeon Gold CPU
    • Powered down the T7820, removed the Xeon Silver CPU and RAM
    • Plugged in the power then removed, and cleared the CMOS
    • Inserted the Xeon Gold CPU / 2933Mhz RAM, powered on, and allowed it to adjust for the new components (reboots multiple)
    • Went into setup and confirmed it saw the Xeon Gold CPU
  • I repeated this process on my other T7820, only its BIOS was 1.4.1.
  • I had no issues with either system after I completed this process.

Why this odd BIOS Update behavior? Though I could not find any information in the 2.29.0 release notes, I can only assume that it had a requirement for systems to be on 2.6.3 to be able to update. This would explain why it updated the system to 2.6.3 first then it updated to 2.29.0

It took a bit of trial and error to figure out this issue but with the great help of oztech and their all out willingness to ensure my satisfaction I now have 3 working T7820s. Next I’m off to install ESXi 8 and update my Home Lab but that my readers will be a different blog post.

Installing Intel U.2 Optane Disk using an adapter

Posted on Updated on

In this quick video I installed a Intel U.2 Optane disk into my Windows system using the StarTech U.2 to PCIe Adapter and then speed tested it with ATTO. I’ll be using these drives and adapters in my Next Generation 8 Super VMware Workstation that I am currently working on. Note the drive that I show being installed on to the StarTech card is different from the one I ran ATTO with.

** Please do keep in mind that the average PCIe slot has a 25 Watt Power Rating. Some systems have a higher wattage rating and some could be lower. The P5800X draws about 30 Watts and the P4800X about 23 Watts. Take this into account when using this adapter and plan accordingly. **

Products Show in this video:

Drive that was tested with ATTO: https://ark.intel.com/content/www/us/en/ark/products/201861/intel-optane-ssd-dc-p5800x-series-400gb-2-5in-pcie-x4-3d-xpoint.html

Drive that was shown being installed on the Adapter: https://ark.intel.com/content/www/us/en/ark/products/129968/intel-optane-ssd-dc-d4800x-series-750gb-2-5in-pcie-2×2-3d-xpoint.html

StarTech Adapter: https://www.startech.com/en-us/hdd/pex4sff8639

Super VMware Workstation: Install tips for the Supermicro X11SPL-F

Posted on Updated on

In this quick video I go over a few of the installation tips when mounting this motherboard into a Phanteks Enthoo Pro case. I’m looking at this motherboard as a possible candidate for my next Super VMware Workstation / PLEX Server.

Motherboard: https://www.supermicro.com/en/products/motherboard/x11spl-f

Case: https://www.phanteks.com/Enthoo-Pro.html

Super VMware Workstation: Supermicro X11SPL-F First Look and Basic Overview

Posted on Updated on

In this video I give a first look at the motherboard I plan to possibly use for my new Super VMware Workstation. I got a lot of great plans for this server and it all starts here.

Supporting 18TB drives with Intel Virtual RAID on CPU (VROC)

Posted on Updated on

To compliment my vSphere Home Lab I use VMware Workstation all the time. It’s great for quickly spinning up VM’s and running nested ESXi. My current workstation (See specs here) was starting to show its age and wasn’t keeping up with my needs. To replace it, I recently bought a ASRock Rack EPC621D8A motherboard in hopes of paring it with a Intel Xeon 6252 CPU, 256GB RAM, GTX 1650 Super video card, Noctua NH-D9 DX-3647 cooler, and 5 x 18TB drives. My hopes were to use this mobo as the foundation for my new SuperWorkstation, Plex Server, and to process videos.

Looking for mobo that will fit a LGA 3647 CPU is pretty hard. The lowest end mobo will set you back at least $430 USD and your choices are very limited. I chose the ASRock Rack as it checked all the boxes – lowest cost, supported PCIe x16, quad Intel NICs, onboard sound, IPMI, Dual NVMe, and many other options, plus it supported the CPU+RAM I already had.

As a test, I did install ESXi 8 on it, and it worked well. However, when I stared my build with Windows 11/PLEX and I hooked up the 5 x 18TB HDD, only one disk would be seen in the VROC, but in Windows 11 all the disks were present. This meant I could not create a RAID 5 group with VROC.

I sent in my findings to ASRock support looking for help. Soon after the video stopped working, so I had to replace the mobo. However, the new ASRock replacement board had that same issue with the 18TB drives. At this point I reached out to ASRock support around this issue.

In the mean time, I decided to start working with the SuperMicro X11SPL-F. This mobo has a similar price point but is lacking in features when compared to the ASRock Rack. It only supports dual Intel NICs, though it looks like it has true x16 PCIe slots they are only a disappointing x8, no onboard audio, and single NVMe. The good news was, there was a current BIOS update that stated it supported my larger 18TB drives. I hooked up the SuperMicro mobo, updated its BIOS, and sure enough all the 18TB drives showed up. Next, I installed Windows 11 and with some quick Intel INF updates everything was accounted for and working.

Soon after I updated ASRock support letting them know my findings. Both the SuperMirco and the ASROCK RACK use the Intel C621 with VROC and with out the VROC update to the ASRock Mobo it just won’t support those larger drives. Their last ASRock mobo BIOS update was from 2019 and their VROC was clearly out of date. I’m hoping this information helps them to reconsider updating their BIOS which should allow for larger HDD support. I got a response back very quickly and they plan to look into this item. Additionally, throughout this issue the ASRock support person was very responsive, friendly, and communicated well through email. Kudos to their support team.

*Update Jan-01-2023 – I got word from ASRock support that they are looking into updating their BIOS and VROC. They asked me to test it out and I will be doing that shortly.*

*Update Jan-29-2023 – Over the past few weeks I’ve been testing the ASRock Beta BIOS and it seems to be working perfectly with my 18TB Drives. Last check I checked on their product page they have not released an update yet. *

Thanks for reading, and please do post up a comment!

*Update* vExpert and Intel NVMe Optane Giveaway

Posted on Updated on

A few months ago I was contacted by Intel as they wanted to engage deeper with the vCommunity and vExperts. My Intel contact, Scott Sherman, was a person I new from my Phoenix VMUG days. We met back in 2009 when Scott was working with Xsigo. He remembered the great blogs I did for Xsigo back in 2011 and wanted to chat more about enabling the vCommunity with Intel hardware.

Through a series of meetings Scott and I were able to come up with a framework worthy of presenting to the vExpert team. Scott was able to allocate 600 Intel Optane P4800 375GB NVMe drives in a PCIe or 2.5 Inch formfactor. The drives came in packs of 10 which would enable us to give them away to 60 vExperts. We chose 10 drives as this would enable the vExperts to create an all flash vSAN configuration or they could use the drives for other home lab fun. The only question now was how do we get these drives in the hands of the vExperts.

We presented our ideas to the vExpert Staff (Corey Romero) and they loved the idea. We came up with the idea to ask the vExperts to fill out a surrey explaining how they would use the disks and if selected how could they promote their outcomes. We’d present this program overview to the vExperts and after they filled out the survey we would review then select the lucky 60.

Over this past week we’ve been reviewing the surveys and there were a lot of great candidates. In fact, some of them even made special blog posts and videos!

It wasn’t easy to choose which vExperts would receive these disks. In reviewing their responses, Intel, VMware, and myself really looked at each entry and tried to surmise how they would use and promote these drives. We visited their social media platforms to get a feel of how they might plus we looked at their activity. We also considered other platforms and venues they shared with us and read each of their entries. We narrowed it down and discussed why each candidate would be chosen.

Now that we have our 1st round picks, we will be sending drives to them this week. They will be notified by the vExpert program and should see their drives the first week of January 2022. Additionally, we will be scheduling a Mid-January meeting to help enable them.

But wait there’s more! We will be sending out a “thanks for applying” email to those who were not selected in the 1st round. However don’t fret, We are working on a 2nd chance plan for those who applied but didn’t get selected. We are planning to setup a meeting to help these candidates become more viable. With a little work I have no doubt that many folks from this group will be selected.

But wait there’s even more! This is not fully confirmed but we may be able to open the 2nd round to folks in EMEA. We are working on some Intel magic to be able to ship to EMEA. Fingers crossed on this one and stay tuned!

It’s been a pleasure helping and enabling my fellow vExperts but please do keep in mind that all official communication should be coming from the vExpert team.

Why I didn’t choose a Noctua replacement fan

Posted on Updated on

We’ve all been there, we’ve picked out a new router, switch, or other device for our home lab and the fans are LOUD. First thing we do is to replace those fans with something a little more quiet. We hit up our favorite online store, maybe read some reviews, and choose a fan that fits. Sometimes that fan is an expensive Noctua fan because its promise of being quiet is so alluring. After the fan is replaced it is a bit more quiet but now the fan error lights are on or it malfunctions. Clearly it’s the wrong fan for our device.

In this blog I’ll go over some of the items you should look for when buying a replacement fan for your devices that can help you find a better fit and not break your wallet. Fair warning, the stock fans in these engineered devices were designed to be optimal for said device. Altering them in any way can be harmful to the device plus working on electronics without proper training is never advised.

First, identify the stock fan in your device and find its datasheet. You may need to remove the fan from your device. I recently replaced some fans in my Mellanox IS5022 InfiniBand Switch. The stock fan was made by Delta, the make #EPB0142VHD Subtype -R00, it has 3 wires, 12 Volt DC Brushless, and draws .18 AMPS. I underlined Subtype as it is very important when identifying your stock fan. In this case if I just search for the make I’ll get the wrong fan information. In fact EPB0142VHD with no subtype only has 2 wires.

Second, I review the stock fan specification datasheet. I already know the Voltage and Amp rating but here are the things I also need:

  • Fan Size – 40mm x 40mm x 20mm
  • Hole Mount Size – 32 mm between mount points
  • Hole diameter – 3.5 mm diameter.
  • Length of Wires – 330 mm
  • Identify the 3 wires and their purpose – 12v, Ground, and Lock Rotor
  • Db Noise rating – 32-36 Dba
  • RPM – 9000 RPM
  • CFM – 10

Not sure if you caught it but identifying the 3 wires on the stock fan is critical if you want to resolve these error lights. Most 3 wires fans are going to have 12v DC and Ground. It’s that 3rd wire that makes them unique and its one of the more important items you must find out to select the correct replacement fan.

The 3 most common types of 3 wire fans are:

  • Step RPM Speed – think of this like gears on a bike. The fan speed steps from one RPM to another. Most have between 3-5 steps in RPM.
  • PWM – Pulse Width Modulation, allows for granular speed control. Instead of instantly stepping to the next speed it is gradually sped up and down.
  • Locked Rotor (sometimes called alert) – This is a fan spin error detection. Normally, the fan will spin at one speed. 40 mm Locked Rotor fans seem to be the most common for routers, switches, and other similar devices.

Another item is the length of the wires. The datasheet shows 330 mm (+-10mm), however the fan you order could be shorter. It’s best just to measure the stock fan, and make sure the replacement fan you ordered has enough length or room to stash the wires if they are too long.

Third, now that I understand my stock fan I’m ready to choose a replacement fan that meets with my goal of reducing fan noise. In most cases, fan noise is reduced by slowing the RPM. Additionally, there are fans specifically designed to reduce noise but they can be expensive. I thoroughly looked at 40mm Noctua fans but none of them matched the voltage and Locked Rotor requirements. However, I still see a lot of folks buying Noctua 40mm fans and then complaining about the fan error lights or issues with it malfunctioning. Most just ignore these errors or alter the fan wires to send a false message to the device. Both I don’t recommend.

In this case I choose the Sunon MagLev KDE1204PKV3 MS.AR.GN 40x40x20mm 3pin Low-Speed 5200RPM 6.3CFM (Locked Rotor Alarm Signal). Cost is about $6.50 US, compared to a non-compliant Noctura $14 US

How do the stock and replacement fans compare:

Item (recommendation)Delta EPB0142VHD-R00Sunon KDE1204PKV3 MS.AR.GN
DC Volts (match)1212
Amps (do not exceed stock)0.180.03
Fan Size (match)40mm x 40mm x 20mm40mm x 40mm x 20mm
Hole Mount Size (match)32 mm32 mm
Hole Diameter (close match)3.5 mm4 mm
Length of Wires (match)330 mm300 mm
3 wire purpose (match)12v, Ground, Lock Rotor12v, Ground, Locked Rotor
Db Noise rating (reduce)32-36 Dba18 Dba
RPM (close match)90005200
CFM (close match)106.3

Fourth, Prepare the fan to be installed. One item I didn’t mention was the fan edge connector. Most data sheets do not come with information on the edge connector as device manufactures may customize this. In this size of fan the edge connectors seem to be a standard size with some variants.

Some fans will need their wire order changed to match the circuitry on the device. Aligning these pins is critical, if they are wrong you could damage your device. For example your replacement fan came with Pin 1 12v Red, Pin 2 Ground Black, and Pin 3 Motor Lock Yellow (Sometimes White or Blue) you might need to reordered them to match your device. Simply use a wire pin removal tool, light pressure down, and push the pin out. Then, reorder the pins to match your device and you are good to go.

Next the replacement fan mount hole might be a factor. Some replacement fans come with screws or bolts that you may be able to use. If not, you may be able to use the stock hardware or hardware you provide. Either way, depending on the hole size you may have to work this out a bit. In my case, the stock fan screws worked perfectly. Tip – Don’t over crank or force in screws, it may damage your fan.

If your stock fan had a protection sleeve over the wires you may want to reuse it as some devices have sharp metal edges that may cut into your wires. Fan vibration may also cause this too. As an alternative, you may want to consider adding heat shrink when you re-pin the fan.

Lastly, how did my selection perform? Basically, the Sunon is a very close replacement to the Delta. It has a reduced RPM and CFM which drops its Db noise by 20 Db. Since I choose a replacement fan that is not an exact match, I’ll need to monitor the device and ensure its temps are within normal thresholds.

Very unscientifically, I used a Db meter app in my smartphone to measure the Db for the Delta and Sunon Fans. The noise reduction was notable and best of all no fan error lights.

Summary, there is no doubt that Noctua makes a quality fan product but they can be expensive and sometimes do not meet the requirements of your stock fans. If you can find one that does, it may be worth the extra spend. However, by doing just a bit of research you are sure to land on a replacement fan that will meet your goals and not break your wallet. My goal was to reduce fan noise for my home lab and by doing my homework I hit a home run with the first fan I chose.

Thanks for reading and do feel free to leave a comment or suggestion.