The Error >> When running an Intel x710 NIC with the ESXi i40e driver you notice your vmkernel.log completely full of the error “WARNING: Link is up but PHY type 0x3 is not recognized”
The Solution >> Ensure X710 firmware is at 17.5.11 (aka 5.04 in ESXi) and ESXi i40e Driver to 1.4.26 or 1.4.28 and these errors stop
The Follow-up >> Check out your NIC on the VMWare HCL for the Correct driver/firmware guidance. This is the link I used.
Sending Millions of the PHY errors to your event logs could be causing other issues for your ESXi host. Look for local boot disk latency or Networking errors in your ESXi host event logs. Once you apply this solution these issues should stop. If not, then you may have other issues impacting your boot disks.
- After applying this solution we then noticed the vmkernel started to populate with ‘driver issue detected, PF reset issued’ the solution for this is to disable TSO/LRO. VMWare KB 205140.
- 04-10-2017 There is a new VMware driver listed for the X710, will be testing soon and will post up results. Release notes indicate fixes for the following:
– Fix duplicate mulicast packet issue
– Fix PSOD caused by small TSO segmentation
If you like my ‘no-nonsense’ blog articles that get straight to the point… then post a comment or let me know… Else, I’ll start writing boring blog content.
ESXi Host NIC failure and the Web client vSwitch orange line doesn’t move? — The results are Shocking!
Okay, the title was a bit dramatic, but it got your attention. Now keeping with my quest to deliver no-nonsense blog articles here is what the orange line means…
Question 1 – What is the function of the orange line when selecting a vmnic, port group, or vSwitch while viewing them in the Web client network settings?
The orange line is showing you the teaming order for the pNICs or vmnics based on their vSwitch or port group teaming policy. In this screenshot, the policy is Active / Active for both vmnic0 and 1.
The orange line will not move to the other pNIC’s unless they are marked as “active” in the teaming policy. “Active in the teaming policy” vs. “which pNIC is passing traffic” are two different things. The orange line is not a representation of the latter, “pNIC passing traffic”.
Question 2 – How can I tell which pNIC is currently passing traffic?
The Web or Thick client vSwitch display (aka the orange line) doesn’t display the pNIC which is currently passing network traffic. You need to use ESXTOP to determine the active pNIC.
Simply go into ESXTOP, Press N, find your vSwitch and it will lead you to the pNIC currently being used to pass traffic.
Question 3 – I had a pNIC failure why isn’t the Web client moving the orange line to the standby NIC?
Again… the orange line ONLY points to the Active pNIC in the teaming policy. In this screenshot below, the teaming policy is setup for vmnic3 as Active and vmnic2 as stand by.
Even though vmnic3 is down, traffic should be flowing through vmnic2. Use ESXTOP to determine this (See Question 2)
If you like my ‘no-nonsense’ blog articles that get straight to the point… then post a comment or let me know… Else, I’ll start writing boring blog content.
Well folks it’s my long awaited blog post around moving my Homelab from FreeNAS to VMware VSAN.
Here are the steps I took to migrate my Home Lab GEN II with FreeNAS to Home Lab GEN III with VSAN.
- I am not putting a focus on ESXi setup as I want to focus on the steps to setup VSAN.
- My home lab is in no way on the VMware HCL, if you are building something like this for production you should use the VSAN HCL as your reference
The Plan –
- Meet the Requirements
- Backup VM’s
- Update and Prepare Hardware
- Distribute Existing hardware to VSAN ESXi Hosts
- Install ESXi on all Hosts
- Setup VSAN
The Steps –
Meet the Requirements – Detailed list here
- Minimum of three hosts
- Each host has a minimum of one SSD and one HDD
- The host must be managed by vCenter Server 5.5 and configured as a Virtual SAN cluster
- Min 6GB RAM
- Each host has a Pass-thru RAID controller as specified in the HCL. The RAID controller must be able to present disks directly to the host without a RAID configuration.
- 1GB NIC, I’ll be running 2 x 1Gbs NICs. However 10GB and Jumbo frames are recommended
- VSAN VMkernel port configured on every host participating in the cluster.
- All disks that VSAN will be allocated to should be clear of any data.
Backup Existing VMs
- No secret here around backups. I just used vCenter Server OVF Export to a local disk to backup all my critical VM’s
- More Information Here
Update and Prepare Hardware
- Update all Motherboard (Mobo) BIOS and disk Firmware
- Remove all HDD’s / SDD’s from FreeNAS SAN
Remove any Data from HDD/SDD’s . Either of these tools do the job
Distribute Existing hardware to VSAN ESXi Hosts
- Current Lab – 1 x VMware Workstation PC, 2 x ESXi Hosts boot to USB (Host 1 and 2), 1 x FreeNAS SAN
- Desired Lab – 3 x ESXi hosts with VSAN and 1 x Workstation PC
End Results after moves
- All Hosts ESXi 5.5U1 with VSAN enabled
- Host 1 – MSI 7676, i7-3770, 24GB RAM, Boot 160GB HDD, VSAN disks (2 x 2TB HDD SATA II, 1 x 60GB SSD SATA III), 5 xpNICs
- Host 2 – MSI 7676, i7-2600, 32 GB RAM, Boot 160GB HDD, VSAN disks (2 x 2TB HDD SATA II, 1 x 90 GB SSD SATA III), 5 x pNICs
- Host 3 – MSI 7676, i7-2600, 32 GB RAM, Boot 160GB HDD, VSAN disks (2 x 2TB HDD SATA II, 1 x 90 GB SSD SATA III), 5 x pNICs
- Note – I have ditched my Gigabyte z68xp-UD3 Mobo and bought another MSI 7676 board. I started this VSAN conversion with it and it started to give me fits again similar to the past. There are many web posts with bugs around this board. I am simply done with it and will move to a more reliable Mobo that is working well for me.
Install ESXi on all Hosts
Starting with Host 1
- Prior to Install ensure all data has been removed and all disk show up in BIOS in AHCI Mode
Install ESXi to Local Boot HD
- After the install I experienced the ESXi hanging at Boot – ‘Starting up Services – Running usbarbitrator start’
- Solution – Stop usbarbitrator service
- Setup ESXi base IP address via direct Console, DNS, disable IP 6, enable shell and SSH
- Using the VI Client setup the basic ESXi networking and vSwitch
- Using VI Client I restored the vCSA and my AD server from OVF and powered them on
- Once booted I logged into the vCSA via the web client
- I built out Datacenter and add host 1
- Create a cluster but only enabled EVC to support my different Intel CPU’s
- Cleaned up any old DNS settings and ensure all ESXi Hosts are correct
- From the Web client Validate that 2 x HDD and 1 x SDD are present in Host
- Installed ESXi Host 2 / 3, followed most of these steps, and added them to the cluster
Logon to the Webclient
Ensure on all the hosts
- Networking is setup and all functions are working
- NTP is working
- All expected HDD’s for VSAN are reporting in to ESXi
Create a vSwitch for VSAN and attach networking to it
- I attached 2 x 1Gbs NICs for my load that should be enough
Assign the VSAN License Key
- Click on the Cluster > Manage > Settings > Virtual SAN Licensing > Assign License Key
- Under Virtual SAN click on General then Edit
- Choose ‘Turn on Virtual SAN’
- Set ‘Add disks to storage’ to Manual
Note – for a system on the HCL, chances are the Automatic setting will work without issue. However my system is not on the any VMware HCL and I want to control the drives to add to my Disk Group.
Add Disks to VSAN
- Under Virtual SAN click on ‘Disk Management’
- Choose the ICON with the Check boxes on it
- Finally add the disks you want in your disk group
- Allow VSAN to complete its tasks, you can check on its progress by going to ‘Tasks’
- Once complete ensure all disks report in as healthy.
Ensure VSAN General tab is coming up correct
- 3 Hosts
- 3 of 3 SSD’s
- 6 of 6 Data disks
- Check to see if the data store is online
Migrating from FreeNAS to VSAN was relatively a simple process. I simply moved, prepared, and installed and the product came right up. My only issue was working with a faulty Gigabyte Mobo which I resolved by replacing it. I’ll post up more as I continue to work with VSAN. If you are interested in more detail around VSAN I would recommend the following book.
I’ve been lucky enough to make it to every VMworld since 2008 and 2014 will be my 7th. time in a row. In this blog post I wanted to share with you a breakdown of some of the tips and tricks I’ve used to get to these events. Being the former Phoenix VMUG leader I’ve shared these tips with fellow VMUG users and now I’m sharing them with all of you. Users would tell me cost is the number one reason why they don’t go – “My Company sees value in this event but will not pay for it”. This breaks down to Food, Hotel, Travel, and the infamous golden ticket, aka the VMworld pass. So how do users overcome the cost to attend? This is what this blog post is all about…
Working with your employer –
Having your employer pick up the tab not only benefits them as a company but yourself too. As you know VMworld is full of great content and the socialization aspects are second to none. Chances are you’ll be asked to put together a total cost to attend and this cost can be quite high for some companies on a tight budget. My suggestion is if you are getting the big ‘No’ then work with your boss around the total costs. First find out why it’s a ‘No’ and look for opportunities to overcome this. Maybe your company will pay for some of the items. Example – They might be able to cover airfare, but the rest is on you. Don’t forget if your company has a VMware TAM (Technical Account Manager) reach out them. Even if you are not directly working with the TAM they are your best resource not only for VMware Technology but also for getting you to VMworld. They don’t have passes but they usually know the community very well and can assist.
Sometimes I hear “My employer will not allow me to accept gifts”. True your company may have a policy around the type of gifts you can receive and by all means follow this policy. However, keep in mind you may be able to take vacation time and represent yourself at this event not your employer. Then there is a possibility gifts could be accepted but on the premises you don’t represent your company. Some companies are okay with this but just make sure they are. If you are able to do this I would suggest you represent it as ‘personal development’.
How do I get a free VMworld Pass?
This can be your biggest challenge. However here are some ways to get your hands on one.
- I can’t tell you how many vendors have giveaways contests right now — hit them early and enter as many contests as you can find
- When you enter, find out who your local vendor contact is and let them know you entered. Then stay in contact with them.
- Keep in mind not all contests are the same, some are based on random drawing and others are not. This is why I say keep in contact with the vendor.
- How do I find give-a-ways >> Google ‘VMworld getting there for free’
Get the word out
- Tell your boss, workmates, vendors, and partners. Post on Twitter, Linked-In, etc. and Repeat again and again. By doing this you let others know about your strong interest in getting there, in turn they might get a lead for you.
- Most importantly, reach out to your local VMUG leader and ask them for tips in your area. They are usually well connected and might have a lead for you as well.
Follow Twitter and Linked-In – You never know who is going to post up “I have a pass and need to give it to someone”. Yes that is right, before the event you can transfer a pass to someone.
- New to Twitter, need contacts? It’s a pretty simple to get started. Simply find the #VMworld hastag, see who is posting to it and start following them. Then look at all their contacts and follow them too, soon you’ll have a gaggle of folks.
- This sound like work. Why do all this? Simple, distributed coverage model. The more people know the more likely they are to help and in turn the more likely you’ll succeed
Don’t forgo an Expo-Only or Solutions Exchange Pass
- If you get offered this pass take it. I can’t tell you how many vendors have these passes and have trouble giving them away, seriously this is gold but folks don’t know how to leverage them.
- First off this pass has great value, there is a TON of value here.
Second this pass can get you on to the Solutions Exchange floor where all the vendors and partners are.
- Once there start talking to all the vendors, fellow attendee, all those folks you meet on Twitter, etc. as you never know who has a full pass they couldn’t get rid of, take it and upgrade yours.
- Third, while you are there with an Expo Pass use Twitter and the VMworld hash tags to let folks know you’re here and you are looking for a full pass.
- Stop by the VMUG booth on the Expo floor, you never know who will be there and you never know if users there might be able to help you.
Vendors and Partners
Find out who is sponsoring VMworld this year, and then…
- Start calling the ones you know well, ask them for support getting there.
- Don’t forget to call the ones you don’t know so well too.
- If you have an upcoming deal on the table with a vendor, inquire if they will throw in passes, travel, etc.
What about Food, Hotel, and Travel Costs?
- There will be free food everywhere, in-fact feel free to give some to the homeless I usually do.
- If you get a pass then lunch and usually breakfast are included.
- For dinner, find out where the nightly events are as they usually have food.
- Talk with Vendors as they might take you out, you never know.
- Ask a Vendor to pay for just the room or ask them to gift hotel points to you.
- Room Share with someone at the event << Think about it, you won’t be in the room that often and chances are from 7AM till 10PM you’ll be out of your room.
Use travel sites to cut down the cost.
- Secret Hotels: Best Western Carriage Inn and The Mosser. Good if you’re on a budget but chances are they are full this year (2014).
- Use your hotel or other travel points to book the hotel for free.
- Get a low cost hotel away from the event, but watch your travel costs.
- Ask a vendor to pay for just the airfare, or maybe they have points they can gift you.
- Use your own travel points to pay for this.
Rideshare to the event
- See if one of your connections are driving to the event, offer to split fuel costs.
- You drive someone to the event, and they pick up the hotel or vise versa.
Use the following –
- VMworld Shuttle
- Once again hit up those vendors, they might have a way to get you around for free
Finally here is a breakdown of how I got to so many events and how/who paid for it….
|2008 VMworld||Vendor Sponsor – Full Pass||Employer Paid||Vendor / Event||Employer Paid|
|2009 VMworld||VMUG Sponsored – Full Pass||Vendor paid for Airfare with Miles||Vendor / Event||Employer Paid|
|2010 VMworld||VMUG Sponsored – Full Pass||Vendor paid for Airfare with Miles||Vendor / Event||Vendor Sponsored|
|2011 VMworld||Vendor Sponsor – Expo Pass but I got an upgrade to Full by asking others||I drove two others and I paid for the fuel||Vendor / Event||Travel Companion paid for room|
|2012 VMworld||Employee Labs||Employer Paid||Employer Paid||Employer Paid|
|2013 VMworld||Employee TAM||Employer Paid||Employer Paid||Employer Paid|
|2014 VMworld||Employee TAM||Employer Paid||Employer Paid||Employer Paid|
Summing it up…
My take is this, if you REALLY want to go you’ll get there but sometimes it takes effort to do so and if you do it right it might not cost you a thing. Don’t let anything stop you and find your way there.
Finally, after you’ve been to the event don’t forget about the folks who got you there and say ‘Thank you’. Then over the next year continue to build this relationship, as you never know if you’ll need help again, or you want to help someone else get there.
This is by far and away my favorite VMware Poster and I use it as a reference quite often.
- Clear representation of all the TCP/IP Ports needed by VMware Products and their dependences
- PDF has a very clear diagram that can be printed on to a large format
- PDF Diagram has reference numbers that correlate to further slides for more information
You can find the PDF Here >>http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2054806
PIC of the Reference Link –
Part of a PIC of the Port Diagram
I was a bit shell shocked when I went to one of my favorite online stores, geeks.com, only to find out they had closed.
They had been open for 17 years and they were one of the first sites I trusted to buy quality products from new or used.
They had a lot of common items but every now and then they had something different or unique. It was one of the reasons why I kept coming back.
I had recommended geeks.com many times and everyone I sent there always let me know what excellent service and product they had.
Well Geeks.com – I salute you – you had a good run, I’m sorry to see you go, and I hope one day you return!
Just a quick note, if you liked geeks.com then check out http://www.pacificgeek.com/ they were very similar in product and layout.
Check out this great network port diagram for vSphere 5.x. It was recently released on kb.vmware.com
This is a must have for those doing architecture and design around VMware – Enjoy!
Recently I updated my home lab with a freeNAS server (post here). In this post, I will cover my iSCSI setup with freeNAS and ESXi 5.1.
Keep this in mind when reading – This Post is about my home lab. My Home Lab is not a high-performance production environment, its intent is to allow me to test and validate virtualization software. Some of the choices I have made here you might question, but keep in mind I’ve made these choices because they fit my environment and its intent.
Click on these links for more information on my lab setup…
- ESXi Hosts – 2 x ESXi 5.1, iCore 7, USB Boot, 32GB RAM, 5 x NICS
- freeNAS SAN – freeNAS 8.3.0, 5 x 2TB SATA III, 8GB RAM, Zotac M880G-ITX Mobo
- Networking – Netgear GSM7324 with several VLAN and Routing setup
Here are the overall goals…
- Setup iSCSI connection from my ESXi Hosts to my freeNAS server
- Use the SYBS Dual NIC to make balanced connections to my freeNAS server
- Enable Balancing or teaming where I can
- Support a CIFS Connection
Here is basic setup…
Create 3 networks on separate VLANs – 1 for CIFS, 2 x for iSCSI < No need for freeNAS teaming
The CIFS settings are simple. I followed the freeNAS guide and set up a CIFS share.
Create 2 x iSCSI LUNS 500GB each
Setup the basic iSCSI Settings under “Servers > iSCSI”
- I used this doc to help with the iSCSI setup
- The only exception is – Enable both of the iSCSI network adapters in the “Portals” area
Setup your iSCSI vSwitch and attach two dedicated NICS
Setup two VMKernel Ports for iSCSI connections
Ensure that the First VMKernel Port group (iSCSI72) goes to ONLY vmnic0 and vice versa for iSCSI73
Enable the iSCSI LUNs by following the standard VMware instructions
Note – Ensure you bind BOTH iSCSI VMKernel Ports
Once you have your connectivity working, it’s time to setup round robin for path management.
Right click on one of the LUNS, choose ‘Manage Paths…’
Change the path selection on both the LUNS to ‘Round Robin’
Tip – After the fact if you make changes to your iSCSI settings, then ensure you check your path selection as it may go back to default
Notes and other Thoughts…
Browser Cache Issues — I had issues with freeNAS updating information on their web interface, even after reboots of the NAS and my PC. I moved to Firefox and all issues went away. I then cleared my cache in IE and these issues were gone.
Jumbo Frames — Can I use Jumbo Frames with the SYBA Dual NICs SY-PEX24028? – Short Answer is NO I was unable to get them to work in ESXi 5.1. SYBA Tech support stated the MAX Jumbo frames for this card is 7168 and it supports Windows OS’s only. I could get ESXi to accept a 4096 frame size but nothing larger. However, when enabled none of the LUNS would connect, once I moved the frame size back to 1500 everything worked perfectly. I beat this up pretty hard, adjusting all types of ESXi, networking, and freeNAS settings but in the end, I decided the 7% boost that Jumbo frames offer wasn’t worth the time or effort.
These settings will enable my 2 ESXi Hosts to balance their connections to my iSCSI LUNS hosted by freeNAS server without the use of freeNAS Networking Teaming or aggregation. By far it is the simplest way to setup and the out of the box performance works well.
My advice is — go simple with these settings for your home lab and save your time to beat up more important issues like “how do I shutdown windows 8” J
I hope you found this post useful and if you have further questions or comments feel free to post up or reach out to me.
At some point in your VMware administrator career you discover you need information around the correct settings to deploy a VM properly.
You find that you need to answer questions like –
What is the supported network adapter for my Guest OS?
Are Paravirtualization adapters supported for my Guest OS?
Can I do Hot memory add?
A few years ago the default standard was the Guest Operating System Installation Guide.
It gave you all the information you needed to setup the virtual hardware or confirm what recommend virtual hardware should be by the OS Type.
Recently the compatibility and OS installation guides have come online and they can lead you to best practices around settings and KB’s too.
In this blog post I’m going to step you through how to find basic information around a Windows 2008 server.
Start here – http://partnerweb.vmware.com/GOSIG/home.html#other
This link will take you to the Guest Operating System Installation Guide.
Select your OS – In this case I choose Windows 2008 Server
Here are the base install instructions for the Guest OS, note at the bottom the KB Articles and Guest OS Compatibility Guide.
The Guest OS Compatibility Guide can tell you what network drivers etc are support for the guest OS..
Click here to go to the VMware Compatibility Guide, Select your OS Family, OS Type and choose Update & Review…
Then select your ESX / ESXi version to see the details…
Here are the results… Also from this page you can choose a different product like ESXi 5.0U1 or other…
Most recently I expanded my home lab to include a Layer 3 switch.
Why would I choose a Layer 3 switch and what/how would I use it is the topic of this blog post.
Here are my requirements for my home lab –
I would like to setup my home network to support multiple VLANs and control how they route.
This will enable me to control the network traffic and segment my network to allow for different types of testing.
I’d also like to be able to run all of these VM’s on Workstation 9, support remote access, and ESXi Hosts.
Frist thing I did was come up with a drawing of what I wanted. It included all my wants and needs…
This was my chance to brain storm a bit and I just wrote down everything I wanted or really needed.
From this drawing I came up with this list…
- Support Remote Access
- VLAN Routing
- VLAN Tagging
- ESXi Host with 5 NICs
- Workstation 9 Host with 5 NICs
- Support 5 Different VLANs
- Support Internet Access for VM’s
- Local Storage support for home files
- Printer / Scanner need to be on the network
- I’m going to need a switch with 24 Ports or better
- Design the network so that I can power down the test lab and allow home devices to print and access the Inet.
Second thing – What do I currently have to work with…
- Windows 7×64, Workstation 9, 32GB RAM, iCore 7, 2 x SATA3 2TB 6gbs HD, 2 x SATA3 SSD (60 & 120), 1 NIC
- IO Mega IX4 with Dual NICs
- Older Netgear 16 Port Gig Layer 2 Switch unmanaged
- Netgear N900 with Guest Network Support
Based on these lists I came up with my shopping list…
- I need a Layer 3 Switch to support all this
- I need some Multiport Giga Bit NICs
Let’s start with the switch…Here is what I looked for in a Switch –
Must Have –
- Layer 3 Routing
- VLAN Routing
- Quiet – It is a must for home networking as I work from home and am frequently on calls.
- Cost effective – keep it below a few hundred
Nice to have –
- Quality Brand
- Community behind the product
- Non-Blocking Switch
- OSPF or RIP
Basically most good Layer 3 switches achieve the requirements for 1-4. However these switches usually run in a Data center or Networking closet and are quite loud
I did some looking around for different switches, mostly used Cisco and Extreme Networks. These are switches that I am familiar with and would fit my home lab. However I’ve seen my share of their innards, I know their fans are loud and cannot easily be replaced. When I was at VMworld 2012 I chatted quite a bit with the Netgear folk about their products and I remember talking with them about their products and how they fit SMB to Enterprise quite nicely. I started to look on Ebay and I found an affordable Netgear Switch. I did some research on line and found how others were modifying the fans to help them run more quietly.
My choice was the Netgear GSM7324. It is a 24 Port Layer 3 Managed Switch from 2008. It meets all my must have needs and it fulfilled all of the “Nice To haves”
I also bought the following to support this switch –
Startech Null Modem DB9 to USB to run the CLI on the Switch
Sunon MagLev HA40201V4-0000-C99, 40x20mm,Super Silent FAN for $10 apiece, they fit perfectly and they run the switch at a tolerable noise level
TIP – And this is important… I had to move the PIN outs on these fans to meet the PIN outs on the Switch. If I didn’t it could have damaged the switch…
Next I started looking for Multi-Port Gigabit NICS…
What do I have to work with?
I’m using the Gigabyte Z68XP-UD3 for Workstation 9 and MSI Z68MA-G45B3 for my ESXi 5.x Host.
What are the Must haves for the NIC’s?
- Dual Gigabit
- VLAN support
- Jumbo Frames
- Support for ESXi and Windows 7×64
- I need about 4 of these cards
I choose the SYBA SY-PEX24028. It’s a Dual NIC Gigabit card that meets my requirements. I found it for $39 on Newegg .
Tip – When choosing a network card I needed to ensure the card will fit into my motherboards, not all x1 PCIe slots are the same and when looking at Dual Gigabit NICs most only work in server class hardware.
I achieved what I was looking to accomplish and with some good design work I should have a top notch home lab. All in all I spent about ~$400 to upgrade my home lab. Which is not a bad deal considering most Layer 3 switches cost $400+. All my toys have now come in and I’m off to rework my lab…. But that my friend is a different blog post…