vmexplorer

HP c7000 Mid-Plane Replacement

Posted on July 19, 2010 Updated on July 25, 2020

If you’re reading this blog, I’m hoping you never have to do this procedure. However the other day we had a c7000 Mid-plane go out.

We first noticed some odd issues with Ethernet NICs randomly disconnecting. Then, our chassis started to fail entire Ethernet pass through modules, it came back online and then a few weeks later just one Ethernet port stayed in a failed state.  Working with HP and our local VAR we tried all the troubleshooting steps and found it was the mid-plane had failed.

This blog post is merely my notes and some of my findings during the replacement of the mid-plane.  I worked with an HP certified technician and he had a punch list from HP to complete this repair. I recommend a certified technician do the repair as you never know what you’re going to run into and we did run into a few gotchas that stumped us all.

Quick Disclaimer – This is not a guide for repairing this device but merely my notes…

First thing – The Plan:

• 7:00    Migrate VM’s to BC2 (BC = blade chassis) and shut down ESX hosts on BC1
• 8:00    Shutdown remaining blades in BC1
• 8:20    Start Repair on BC1
• 8:40    Test repair on BC1
• 8:50    bring up Blades on BC1
• 9:00    TEST applications
• 9:10    Migrate VM’s from BC2 to BC1
• 10:00    Finish

The Plan with Notes:

7:00    Migrate VM’s to BC2 (BC = blade chassis) and shut down ESX hosts in BC1

• No issues, vMotion worked without issues & shutdown 6 ESX hosts via vCenter Server

8:00    Shutdown remaining blades in BC1

• No issues, shut down blade servers
  

8:20    Start Repair on BC1

Power down the c7000 chassis, remove all the cards, power supplies and blades

This is a pic where we removed the rear cards.  One thing we were able to do was remove the Ethernet & Fibre Pass through cards without having to disconnect the cables.

Second shot of the removal

Remove the blades and Power supplies

To remove the mid-plane there were four bolts holding in it in place.  We removed them and then pull it till it stops.  The retaining clips needed depressed to allow the unit to be removed and we pressed them in and it removed with ease.  Note: The manual fails to mention this.

Here is a shot of the retaining clips. They help to hold the mid-plane in. At first we could not figure out why the mid-place would not come out.  This held us up a bit as the instructions failed to mention depressing them to allow the mid-plane removal.

Inside the blade chassis – Rear

After removing the mid-plane chassis there are several bolts that hold in this together.  One item you have to do is remove the foam tape in the screen shot below.

This will allow you to separate the mid-plane board from the chassis. This was another gotcha that the instructions failed to mention this step.

Our package came with replacement tape, yours may vary.

After replacing the mid-plane board we reassemble the chassis in this order..

• Reinserted the Mid-Plane chassis
• ilo Control Modules
• Power Supplies
• Powered on device, wait
• Inserted pass though modules, wait
• Inserted Blades and powered them on

9:00    Ensure the Blades on BC1 are reporting in

• No issues, everything is now working!

   

9:15     TEST applications

• No issues, all blades and apps came right up

   

9:30    Migrate VM’s from BC2 to BC1

• No Issues, all VM’s migrated perfectly

10:00    Finish

Gotchas / Notes

2 Items slowed us down

1. The retaining clips on the lower right and left hand sides were not noted in the documentation. We had lots of cables on the left and right side so it made it hard to clearly see where they were.
2. The tape holding on the mid-plane board wasn’t documented. It took us a bit to figure out that a simple piece of foam tape could hold together, but it did.

Next time I mount a c7000 chassis I’ll remember to mount it about 4U from the base of a rack. The issue we were having was the cabling and PDU power cords were getting in the way of the mid-plane and this made it harder to remove.

Other than these minor issues, the repair went smooth, and it was fun to see the “guts” of a c7000.