Category Archives: VMware Vsphere 5

Proliant DL380 embedded NICs missing after firmware update

I ran into a very strange issue today when I went to redeploy an old Proliant DL380 G5. The first thing I did was use the most current service pack DVD to update the firmware. The most current is from 2/2014 and has the number 2013.02.0. After installing ESXi 5.1 U1 I noticed that I was only showing 4 NICs and not the 6 I started with.

The two embedded NICs were missing!!

After a quick google search or twelve I stumbled upon an HP discussion with exactly the same problem that I was having. I followed the instructions from the HP discussion and here is what it took to fix (Most of this is copied from user hase3d’s post).

1. Download all necessary tools
     – download FreeDOS
     – download XDIAG.exe 
     – download bc08c740.bin 
     – read all information in setup.txt

2. Prepare the FreeDOS.iso
     -After downloading open the iso with a tool like UltraISO. I used Magic ISO.
-Add the XDIAG.exe and the bc08c740.bin to the iso – I these files to the          root so that I wouldn’t need to add a path later.
-Save the iso with a new name.
-Burn it or mount it with ilo.

3. Boot from FreeDOS
     -Select Install to harddisk
     -Press 1
     -Select your language and press Enter
    -Press ESC
    -Select run FreeDOS from CD-ROM

4. Mine booted to f:\freedos. Do a cd\ to get back to the root of f:
5. Run xdiag in engineering mode by typing xdiag -b06eng
6. type device 1
7. nvm fill 0 0x600 0
8. nvm upgrade -bc bc08c740.bin
9. nvm cfg
     -Press q
     -Type default
     -Press q again
     -Type 16=10 wich sets the BAR size to 32
     -Press q for the third time
     -Type save and then exit out to the main menu

10. Type device 2  and repeat steps 7-9, run the command 1=00:00:18:xx:xx:xx <— change the last digit for different mac on device 2.

I did not do anything else from the setup.txt file.

I powered down the host and then when I rebooted I had 6 NICs again!

The authentication server returned an unexpected error

I came in this morning only to be greeted by my web client telling me that I can’t login because it can’t create SAML 2.0. I am not sure that I really want it creating SAML 2.0….I don’t know SAML 1.0. Ok, bad joke. Here was the message…

I found KB2034798 at which point I remoted into my SSO server and checked the imsTrace.log for “NetUserGetLocalGroups”. I didn’t find it…so the KB didn’t apply to me…L

After some more googling I found this blog post that indicated that references KB2043070. The idea is that there is a local identity source within SSO that it is trying to authenticate the users to. You have to login with the admin@system-domain account and password. Hopefully you saved this when setting up your SSO server. The only problem I had was that I didn’t have this local identity source to remove.

I thought to myself, that there might be a stale identity source on the list that it is authenticating to. I was talking to a coworker and they mentioned that there was a domain that was deleted the day before. AHAH!! I clicked on the identity source of the domain that had been removed and then clicked “Test Connection”. There was an error that didn’t tell me much.

3-12-2014 2-42-32 PMI cancelled out and was back at my list of identity sources. I selected the identity source that had been removed from AD and I hit the red X, “Delete Identity Source”. You will get a prompt asking for you to confirm. One thing to note is that the identity source that I deleted was not one of the default domains at the bottom. If you haven’t set a default domain up, I would do that now. I am wondering if there might be a bug that uses the identity source at the top of the list instead of the default at the bottom. After deleting the state Identity Source I was able to login again.


vSphere HA detected that host is in a different network partition than the master

Target: Host
Previous Status: Green
New Status: Red
Alarm Definition:
([Event alarm expression: vSphere HA agent on a host has an error; Status = Red] OR [Event alarm expression: vSphere HA detected a network isolated host; Status = Red] OR [Event alarm expression: vSphere HA detected a network-partitioned host; Status = Red] OR [Event alarm expression: vSphere HA detected a host failure; Status = Red] OR [Event alarm expression: Host has no port groups enabled for vSphere HA; Status = Red] OR [Event alarm expression: vSphere HA agent is healthy; Status = Green])
Event details:
vSphere HA detected that host (host) is in a different network partition than the master (Cluster) in Datacenter

I had been getting this message randomly over the last couple months on some of my datacenter hosts. These alerts didn’t seem to be causing any problems within the cluster, but I wanted to get to the bottom of this. I opened a ticket with VMware and uploaded the logs from both the host and vCenter, but they didn’t see anything out of the ordinary. On the second webex with VMware I noticed a couple strange things with the management network that might be the cause.

  1. The first thing I noticed was that the NICs were set for “Auto Negotiate”. I originally set up our environment on ESXi 4 before upgrading to ESXi 5.1. When I initially set this up I hard coded (KB1004089) these to 1000GB/Full. I am wondering if at some point during the upgraded that they defaulted back. On our switches it was set at 1000GB/Full so it is important that we set this on the host NICs to 1000GB/Full as well.
  2. The second thing that I noticed that in the Management network that I had the Load Balancing set to “Route based on IP hash”. The problem here is that for this to work correctly you need a port channel configured (I do not have this configured this way). This might be the cause of the HA problem if the traffic is going across these NICs is getting confused because of the Load Balancing configuration. I changed this to “Route based on the originating virtual port ID”, which makes the traffic go out on the port that it came in on. There is a good read found here…http://blogs.vmware.com/kb/2013/03/troubleshooting-network-teaming-problems-with-ip-hash.html.

This case is still ongoing with VMware and I should know in the next couple weeks if this solves my problem; my gut tells me it will.

Creating a template for Server 2012 R2 – Part 2

Now that our VM is created, we will install VMware Tools, upgrade the Hardware Version, and make some helpful Windows changes. If you are going to use the BGinfo program, please make sure you go and download that.

Finishing VMware Changes and Configuring the OS

  1. First thing we need to do is upgrade the VMware Tools on the machine. This will install critical drivers for both Network and Video and will make for a better all-around experience. Select your VM and then click “Guest” and then “Install/Upgrade VMware Tools”.
  2. Click “OK” on the window that comes up.
  3. Go back to your console screen for the VM and you should see the D: drive change to “VMware Tools”. Double click this to start the install.
  4. Click “Next” on the tools welcome screen.
  5. I used to do a custom install and removed the shared folders, but from all my reading I don’t think this is needed anymore. Choose the “Typical” radio button and then click “Next”.
  6. To continue with the install click “Install”.
  7. Click “Finish”.
  8. Click “No” when it asks for you to reboot.
  9. Instead we are going to Shut the VM down so that we can update the hardware. Select your VM in VMware then choose “VM” from the menu bar then “Power” and then “Shut Down Guest”.
  10. Once the VM is shut down, right click on it in vCenter Server and then choose “Upgrade Virtual Hardware”. The hardware version for the server will change to 9.
  11. Right click on the VM again and this time click on “Edit Settings”.
  12. Click the CD/DVD drive 1 and change the Device Type to “Client Device”.
  13. Click on the Floppy drive 1 and then click “Remove”.
  14. Click the “Options” tab and then click “Boot Options” check the box for “Force BIOS Setup”. Click “OK”.
  15. Power on the VM again using the “Power On” button.
  16. Arrow down to “Legacy Diskette A:” and hit the + key until is says “disabled”.
  17. Arrow to the right so that “Advanced” is highlighted. Then arrow down 4 times until “I/O Device Configuration” is highlighted and then press “Enter”. Go down the list changing “Serial port A:, Serial port B:, Parallel port:, and Floppy disk controller to “Disabled”.
  18. Press “ESC” twice and then press “Enter” to Exit Saving Changes. Press “Enter” again when it asks for confirmation.
  19. Logon using your administrator password that you created in part 1.
  20. I like to get the time right on the server before I do anything else. Do this by right clicking the time in the lower right corner and choose “Adjust date/time”. Click the “Change time zone” button. Make the appropriate change for your location and then click “OK”, and then “OK” again.
  21. The first thing I like to do is to rename the server and add it to the domain if needed (I try to not add it to the domain if I don’t have to). The Server Manager should open automatically for you. Click “Local Server” on the left side and then click the “Computer Name” Change the name, but you won’t be able to add it to the domain yet because it has not been IPed.
  22. Right click on the Network icon in the task bar and choose “Open Network and Sharing Center”.

  23. Click “Change adapter settings”.
  24. Click “Properties”.
  25. The default lists the following items.
  26. Click “QoS Packet Scheduler” and then click “Uninstall”. Do the same for both “Link-Layer Topology” items. QoS Packet Scheduler is not needed unless you are doing QoS at the Windows layer instead of the L2/L3 switch layer. Please read about Link Layer Topology here and determine if you want/need it in your environment – http://en.wikipedia.org/wiki/Link_Layer_Topology_Discovery

  27. Finally, uncheck “Internet Protocol Version 6 (TCP/IPv6). WARNING – DO NOT uninstall IPv6 as this might cause problems.

  28. The first thing we are going to do with Server Manager is to change its behavior on startup. Open up Server Manager if it is not already done. Click “Manage” and then click “Server Manager Properties”.
  29. Check the “Do not start Server Manager automatically at logon”.
  30. In Server Manager click on the “Internet Explorer Enhanced Security Configuration” and set both Administrators and Users to “Off”.
  31. Now we are going to add some important items to the desktop. From the main Server Manager Dashboard page click “Add roles and features”. Click “Next” on the Before you begin page, leave the radio button on “Role-based or feature based installation” and click “Next.
  32. On the Server Selection screen leave everything default and then click “Next”. Click “Next” again to bypass the Server Roles and move to “Features”.
  33. Expand both .NET Framework 3.5 and 4.5 and then check to have both installed.
  34. Next click the check box for “User Interface and Infrastructure”. This is going to allow us to add some missing desktop icons. Click “Add Features” when the required features window comes up.
  35. We use SNMP for server monitoring so I check the box for “SNMP Service” and then click “Add Features” when the required features window comes up, then click “Install”.
  36. Right click on your desktop background and click “Personalize”. Click the “Change desktop icons” link and then check the boxes for “Computer, Recycle Bin, and Control Panel”.

  37. Right click on the Desktop again, and under “View”, set icon size to “Small”, and set Auto Arrange and Sort By options according to your preference.


  38. Right click the task bar and click “Properties”.
  39. Check the box “Use small taskbar buttons”.
  40. Click on the “Navigation” tab and then check the box “When I sign in or close all apps on a screen, go to the desktop instead of Start”. I also check the boxes “Show the Apps view automatically when I go to Start” and “Search everywhere instead of just my apps when I search from the Apps view”.
  41. Create a new folder on C: called BGInfo. Place all your BGinfo files into this folder. Edit the BGInfo.bgi file if you want to customize the BGinfo settings. Create a .bat file called bginfolaunch.bat in the BGinfo folder. I have included what I have in my batch file.
  42. Right click on the start button and choose “Run”. Then type Regedit in the open box.
  43. Adding the following entry into the registry will cause BGInfo to automatically refresh BGInfo every time you log onto the server. Add a reg key (string value) called BGInfo with the value of C:\BGInfo\bginfolaunch to HKLM\Software\Microsoft\Windows\CurrentVersion\Run.

  44. Right click on the start button again and this time choose “Control Panel”. Change the view to “Small Icons”. Click on “Power Options”.
  45. Change the power settings to “High performance”. And then click “Change plan settings”. Set both “Turn off the display” and “Put the computer to sleep” to “Never”.

  46. Right click on the Powershell icon and the select “Run as Administrator”. Type powercfg –h off and press “Enter”.
  47. Click “File Explorer” on the task bar. Click “View” then “Options” and then “Change folder and search options”
  48. Check “Display the full path in the title bar area” and click the radio button for “Show hidden files, folders, and drives”. Uncheck “Hide protected operating system files”.
  49. Right click the start button and click “Run”. Type gpedit.msc in the run box. When the group policy window comes up go to Computer Configuration/Windows Settings/Security Settings/Local Policies/Security Options. Set “User Account Control: Run all administrators in Admin Approval Mode” to disabled.

  50. Then change “User Account Control: Behavior of the elevation prompt for administrators” to Elevate without prompting.
  51. The final thing is to include Logoff and Disk Manager Icons on the desktop. Create them and place them in C:\Users\Public\Desktop.

Creating a template for Server 2012 R2 – Part 1

I have borrowed items from http://www.boche.net/blog/index.php/2012/08/16/microsoft-windows-server-2012-tips/ to create this post. I encourage you to take a moment to check out that post.

This is the step by step document that I used to build my 2012 and 2012 R2 VMs.

VM values will start at:

Hardware: Value:
Memory 4 GB
CPU’s 1
Video card Auto-detect video settings
VMCI device None
SCSI Controller 0 VMware Paravirtual
Hard disk 1 40 GB, Thin
CD/DVD Drive 1 Client Device
Floppy Drive 1 Removed (when done)
Network Adapter 1 VMXNET3
General Options OS: Microsoft Windows Server 2012 and 2012 R2
VMware Tools Default Settings
Virtual Machine Version 9

Creating the VM

  1. In vCenter click “File” then “New” then “Virtual Machine”
  2. Choose the “Custom” radio button and then “Next”.
  3. Name the SM and choose the folder location.
  4. Choose the datastore for the VM and then click “Next”.
  5. Make sure that the “Virtual Machine Version: 8” radio button is selected. (5.1 is using version 9 so I am not sure why this can’t be selected here. We will change this later.
  6. Select the “Windows” radio button and then choose “Microsoft Windows Server 2012 (64-bit).
  7. Take the default of 1 virtual socket and 1 core per virtual socket.
  8. Take the default of 4GB memory and click “Next”
  9. Choose your Network and change the adapter to “VMXNET 3” then click “Next”.
  10. Change the SCSI controller to “VMware Paravirtual” and then click “Next”.
  11. Select the “Create new virtual disk” radio button and then “Next”.
  12. Take the default of 40GB and click “Next”.
  13. Take the default virtual device node. For the system partition you want this to be SCSI 0:0. Click “Next”.
  14. On the summary screen click the “Edit the virtual machine settings before completion” box and then click “Continue”.
  15. Click on the Video Card and then change the radio button to “Auto-detect settings”.
  16. Click on the CD/DVD and then choose the datastore location that you have the 2012 R2 install ISO. Make sure under Device Status that “Connect at power on” is checked. Now click “Finish”.
  17. Right click on your newly created VM and click “Edit Settings”.
  18. Click on the Floppy drive 1 and choose the “Use existing floppy image in datastore” radio button. Then click “Browse”. At the bottom of the datastores you should see a folder called “vmimages”. Double click this folder. (For some reason until the VM is created this folder does not show up and that is why we had to create the VM and then go back into the settings to change this).
  19. Double click on the “floppies” folder.
  20. Choose the “pvscsi-Windows2008.flp” and then “OK”.
  21. The Floppy drive 1 settings should look like this and then click “OK”.
  22. On the list of VMs click the one you are building and then click the “Power On” button.
  23. Now click the “Open Console” button.
  24. The VM should boot into the 2012 R2 setup screen. Choose your language and then “Next”.
  25. Click “Install”.
  26. Choose the version of server that you are using. We use the Datacenter here. Then click “Next”.
  27. Agree to give up your first born to Microsoft by clicking the “I accept the license terms” box and then click “Next”.
  28. Choose “Custom: Install Windows only (advanced).
  29. Uh oh, there is no location to install Windows. Luckily you configured the floppy drive 1 earlier right? Click the “Load Driver” button.
  30. Click “Browse”.
  31. Look for the Floppy Disk Drive and then double click “amd64”.
  32. Select the “VMware PVSCSI Controller (A:\amd64\pvscsi.inf) and click “Next”.
  33. Hey look there is our drive!! Click “Next”.
  34. Windows should now be installing.
  35. Enter a password for your admin account. Do not lose this password!
  36. You should have the login screen now.

Installing APC Network Shutdown for ESXi – Part 3

In Part 3 we are going to install Powerchute Network Shutdown on the OVA that we deployed, then we are going to configure it to shut down the VMs in case of a problem.

  • See APC pdf FA159776. Open Putty.exe, insert the name or IP of the VMA you just deployed, and then click “Open“. Click “Yes” if you get a security alert. Login with vi-admin and your password that you set earlier.
  • Create a temp directory in opt using the command (You will be prompted for the vi-admin password): sudo mkdir /opt/temp
  •   Next we need to change the permissions to this temp directory: sudo chmod 777 /opt/temp
  • Now to check the permissions: ls -la /opt The permissions should now read drwxrwxrwx
  • Now using WINSCP we need to transfer the .tar.gz file that we downloaded earlier up to the ESXi host. Enter the appropriate information and then click “Login“. Click “Yes” or “Proceed” if prompted with a security warning.
  • Check the “Never show this banner again” box and then click “Continue“. You should now see a screen with two windows. The window on the left is your local computer and the screen on the right is the VMA. Navigate on the left window until you find the .tar.gz file.
  • On the right window the drop down where it says “vi-admin“. Change this to /<root>. Then navigate to “opt–>temp 
  • Drag the .tar.gz file from the left window to the right window. Click “Copy” when prompted.
  • Verify that the file has been copied successfully.
  • Now go back to Putty.exe and we are going to uncompress the file. The commands are: gunzip pcnsname.tar.gz then: tar -xvf pcnsname.tar
  • Use the ls-la command and you should see a new ESXi folder. Use the command cd ESXi to change to this folder.
  • List the contents of ESXi with the ls -la command. We need to change the permissions for the installation file: sudo chmod 777 install_en.sh
    Now do another ls -la to see that the permissions have changed to rwxrwxrwx.
  • Now we are ready to install PCNS. Use the command: sudo ./install_en.sh
    Press “Enter” and then use the “z” key to scroll to the end of the agreement. If you agree then type “yes” and then press “Enter“.
  • Accept the default installation path (or insert a different one if you prefer). Press “Enter“. Type “yes” and “Enter” that you are sure about the path.
  • Take the default for the java directory. Press “Enter“.
  • Next the installation looks for the ESXi host that will be shut down. First add the IP of the host and then it will ask for the username and password for the host to make this change.  Update:  Almost all of the deployments failed to add the ESXi host here, so I would choose “q” to skip and then at the command line do: sudo vifp addserver <hostname/ IP address of ESXi host>
  • Verify that the server has been added with the command: vifp listservers
  • To ensure Powerchute can shutdown the VMs on the host, we need to add the ESXi host to the fasspass. Use the command: vifptarget -s <server name or ipaddress>
    Now type the command: vicfg-nics -l
    You should see a list of nics on the ESXi host.
  • One the server has been added you should be able to open a browser and go to the powerchute configuration wizard: https://vmahostnameorip:6547
  • Click “Next” and you should see the Configuration Wizard: Security page. Insert the username and password and the authentication phrase. This must match the card in your APC device. By default this is apc/apc with the passphrase: “admin user phrase” then click “Next“.
  • On the UPS Electrical Configuration page choose the correct configuration for your company and then click “Next“.
  • On the UPS Details page choose the protocol, port, and IP for the APC network card.
  • On the Miscellaneous page check the box for “Automatically check for updates to PCNS” and then click “Next“.
  • Confirm the details and then click “Apply“.
  • Hopefully you see that the computer is now protected. Click “Next“.
  •  You should now see that the wizard is complete, now click “Finish”.
  • You will now see the main page for the Network Shutdown. Click “Configure Events” and then click the check box for “Shutdown System” on “UPS: On Battery“.
  • The “Shut Down Operating System” page will display and input 300 into the “Shut down the PCNS operating system only when the event lasts this long (seconds)
  • Finally, we need to set up the virtual machine shutdown options on the ESXi host. Open the vSphere Client, select the host, and then choose the “Configuration” tab. Under the “Software” pane click on “Virtual Machine Startup/Shutdown“.
  • In the top right corner click “Properties“. Click the box “Allow virtual machines to start and stop automatically with the system“. Set the shutdown delay (120 default) and then set the shutdown action to “Guest Shutdown“.
  • Leaving VMs under the Manual startup will make it so when the host turns back on, the VMs will not start up by themselves. Usually you want to make sure power is restored and stable before bringing up VMs. You can change your VMs to start automatically if you really wanted to. 

THAT’S IT!!

Installing APC Network Shutdown for ESXi – Part 2

In Part 2 we are going to configure the OVA that we just deployed

  • Click on your new VMA and then click the “Open Console” button.
  •  There should be a Network Configuration menu. I have found that if I set the gateway first that it will not save when I set the IP. I am going to set that last. Choose option “3” to set the hostname.
  •   Make your hostname match your VM name.
  •  Select option “4” to set the DNS servers. Type the appropriate primary (Server 1) DNS IP and then press “Enter“. If used, also add your secondary (Server 2) DNS IP and press “Enter” again.
  • Select option “6” to set the IP for eth0. I only use IPv4 so type “n” to not configure IPv6, then “y” to configure IPv4, and then “n” to not use DHCP. Type the IP and Subnet for your VMA and then “y” to confirm it is correct.
  • Now I set the Gateway. Choose option “2” and the press “Enter” to set the gateway for eth0. Type the IP of your IPv4 Default Gateway and the press “Enter“. Press “Enter” again to skip the gateway for IPv6.
  • Choose option “1” to “Exit this program“. This will boot the VMA with the network settings that we just configured.
  • Next the VMA will ask for the old password for the vi-admin account. Press “Enter” for the Old Password. Then type your new password “Enter” and then retype it when prompted. “Enter” again.
  • The VMA should boot and you should see the following screen. Browse to https://VMA-IP:5480 to verify connectivity. 

Great, now you have configured the new VMA, it is now time to install Powerchute in Part 3.