VMworld 2015 Schedule Builder is now live!

Time to sign up for the Ask the Expert vBloggers session first! #VMworld 2015 Schedule Builder is now live!

VMworld 2015 Schedule Builder is now live!

Search from over 400 unique sessions, sign up early for your favorites, and view customized recommendations based on your attendee profile! Are you ready for any?


VMware Advocacy

Who’s ready for the VMworld Party Already…

Looks like VMware is heading back to ATT stadium this year.

Join the VMworld community at AT&T park, home of the San Francisco Giants, for an evening of fun, feasting, and fantastic entertainment! VMworld will transform the park with exciting carnival rides and interactive midway games with the stunning backdrop of the San Francisco Bay. The party starts Wednesday, September 2 at 7:30PM .


VMware Advocacy

VMworld 2015 Content Catalog is now live!

What sessions will you attend? The #VMworld 2015 Content Catalog is now live!

VMworld 2015 Content Catalog is now live!

The Content Catalog allows prospective VMworld attendees access to the VMworld agenda, with the ability to peruse breakouts and note sessions of interest. You can search and filter to your heart’s content—by track, category, session format, industry, role, technical level, speaker name, location (US or Europe), and keyword search. You cannot schedule sessions in the catalog.


VMware Advocacy

Change Active Adapters and Load Balancing using ESXi Shell

Anyone ever had a network administrator make your hosts lose connectivity and the only option you have is to use the ESXi shell through an HP ILO port? I have. Here are the changes that were helpful to me at the time.

freakout

esxcli network vswitch standard list — This lists all of the vswitches on the host.
listvswitch

esxcli network vswitch standard uplink add -u vmnic1 -v vSwitch0 — Add an uplink to the vswitch. Just switch “add” to “remove” to get rid of a vmnic from the vswitch.
adduplinktovswitch

esxcli network vswitch standard policy failover get -v vSwitch0 — Get the load balancing policy for the vSwitch.
getvswitchpolicy

esxcli network vswitch standard policy failover set -a vmnic1 -v vSwitch0 — Be careful because when changing the active vmnics on a vswitch it will only add ones that you specify and if there are active ones that you specify it will move them to unused.
Incorrect
setactivevmniconvswitchfail
Correctsetactivevmniconvswitch

esxcli network vswitch standard portgroup policy failover get -p “Management Network” — This shows the policy information for the portgroup “Management Network”
getportgrouppolicy

esxcli network vswitch standard portgroup policy failover set -a vmnic1 -p “Management Network” — Change the active adapter on the Management Network.
setactivevmniconportgroup

esxcli network vswitch standard policy failover set -l iphash -v vSwitch0 — Set the load balancing on the vSwitch0 to iphash. Please be aware that this might change the portgroups to iphash if they are set to get their settings from the vswitch.
setloadbalanceonvswitch

esxcli network vswitch standard portgroup policy failover set -l iphash -p “Management Network” — Sets the load balancing policy on the Management Network portgroup to iphash. This will cause the load balancing policy on the vswitch to no longer apply to the Management Network portgroup.
setloadbalanceonportgroup

PowerCLI shutdown/startup script for remote sites.

In my environment I have 11 Remote Locations that each have one ESXi host.  I had a problem with ILO firmware updates the other day where I had to reboot these remote hosts.  I really didn’t want to use the VIC to click on each of the VMs to shut them down, so using a lot of help from mpreston’s site I came up with a script that will shut down all of my VMs and then when ready, start the VMs again.  My script shuts down “Front End” VMs first, then DB VMs, and then finally DCs.  I have a modified version that allows me to do the same for my Datacenters that I will share later.  You might have to edit the script if your database and or DCs are named differently.

add-pssnapin VMware.VimAutomation.Core
connect-viserver servername #Insert your vcenter server

#Variables
$datacenter = “datacenter”  #Add all datacenters here with a comma and quotations between each example “datacenter1”, “datacenter2”
$filename = “c:\vmshutdown\poweredonvms.csv”  #You must create this file first…it won’t create it for you.

#Get a list of the hosts being rebooted and set them to maintenance mode.  This keeps the VMs from powering on again.
Get-VMHost -location $datacenter | set-vmhost -State Maintenance

#Get a list of powered on VMs
get-vm -location $datacenter | Where-Object {$_.PowerState -eq “PoweredOn”} | Select Name | Export-CSV $filename

#Get a list of VMs that are not Database servers or DCs.  
Write-Host “”
Write-Host “Retrieving a list of powered on VMs…” -ForegroundColor Green
Write-Host “”
$poweredonguests = Get-VM -Location $datacenter | Where-Object {$_.PowerState -eq “PoweredOn”} | where {$_.Name -notlike “*DB*”} | where {$_.Name -notlike “*DC*”}

#Power off servers that are not Database and DCs
foreach ($guest in $poweredonguests)
{
Write-Host “Processing $guest…” -ForegroundColor Green
Write-Host “Checking for VMware Tools Install” -ForegroundColor Green
$guestinfo = get-view -Id $guest.Id
if ($guestinfo.config.Tools.ToolsVersion -eq 0)
{
Write-Host “No VMware tools detected in $guest, these will be powered off” -ForegroundColor Yellow
Stop-VM $guest -Confirm:$false
}
else
{
Write-Host “VMware tools detected. I will attempt to gracefully shutdown $guest”
$vmshutdown = $guest | Shutdown-VMGuest -Confirm:$false
Sleep 5
}
}

#Waiting for these shutdowns to complete
Write-Host “”
Write-Host “Giving VMs 3 minutes for VMs that are not DB or DC to shutdown.”
Write-Host “”
Sleep 180

#Shut down DB Servers at remote sites
Write-Host “”
Write-Host “Shutting Down DB Servers”
Write-Host “”
#get our list of DB servers that are powered on…
$poweredondbs = Get-VM -Location $datacenter | Where-Object {$_.PowerState -eq “PoweredOn”} | where {$_.Name -like “*DB*”} | where {$_.Name -notlike “*DC*”}
foreach ($dbs in $poweredondbs)
{
Write-Host “Processing $dbs…” -ForegroundColor Green
Write-Host “Checking for VMware Tools Install” -ForegroundColor Green
$guestinfo = get-view -Id $dbs.Id
if ($guestinfo.config.Tools.ToolsVersion -eq 0)
{
Write-Host “No VMware tools detected in $dbs, these will be powered off” -ForegroundColor Yellow
Stop-VM $dbs -Confirm:$false
}
else
{
Write-Host “VMware tools detected. I will attempt to gracefully shutdown $dbs”
$vmshutdown = $dbs | Shutdown-VMGuest -Confirm:$false
Sleep 5
}
}

#Waiting for these shutdowns to complete
Write-Host “”
Write-Host “Giving VMs 3 minutes for Database VMs to shutdown.”
Write-Host “”
Sleep 180

#Shut down DCs
Write-Host “”
Write-Host “Shutting Down DCs”
Write-Host “”
#get our list of DB servers that are powered on…
$poweredondcs = Get-VM -Location $datacenter | Where-Object {$_.PowerState -eq “PoweredOn”} | where {$_.Name -like “*DC*”}
foreach ($dcs in $poweredondcs)
{
Write-Host “Processing $dcs…” -ForegroundColor Green
Write-Host “Checking for VMware Tools Install” -ForegroundColor Green
$guestinfo = get-view -Id $dcs.Id
if ($guestinfo.config.Tools.ToolsVersion -eq 0)
{
Write-Host “No VMware tools detected in $dcs, these will be powered off” -ForegroundColor Yellow
Stop-VM $dcs -Confirm:$false
}
else
{
Write-Host “VMware tools detected. I will attempt to gracefully shutdown $dcs”
$vmshutdown = $dcs | Shutdown-VMGuest -Confirm:$false
Sleep 5
}
}

#Waiting for these shutdowns to complete
Write-Host “”
Write-Host “Giving VMs 3 minutes for DCs to shutdown.”
Write-Host “”
Sleep 180

#Time to restart the hosts
get-vmhost -location $datacenter | restart-vmhost -runasync -confirm $false

Now that you have rebooted your hosts, it is time to start all the VMs again.  You are going to use the dame variable that we created above.

#Take hosts out of maintenance mode.  
get-vmhost -location $datacenter | set-vmhost -state connected

#Power On DCs
Write-Host “”
Write-Host “Powering On DCs”
Write-Host “”
#get our list of DCs servers that are powered off…
$poweredoffdcs = Get-VM -Location $datacenter | Where-Object {$_.PowerState -eq “PoweredOff”} | where {$_.Name -like “*DC*”}
foreach ($dcs in $poweredoffdcs)
{
Start-VM $dcs -Confirm:$false
Sleep 5
}

#Power On DB Servers
Write-Host “”
Write-Host “Powering On DBs”
Write-Host “”
#get our list of DB servers that are powered off…
$poweredoffdbs = Get-VM -Location $datacenter | Where-Object {$_.PowerState -eq “PoweredOff”} | where {$_.Name -like “*DB*”}
foreach ($dbs in $poweredoffdbs)
{
Start-VM $dbs -Confirm:$false
Sleep 5
}

#Power On the rest
Write-Host “”
Write-Host “Powering On Remaining VMs”
Write-Host “”
#get our list of VMs that are still powered off…
$poweredoffguest = Get-VM -Location $datacenter | Where-Object {$_.PowerState -eq “PoweredOff”}
foreach ($guest in $poweredoffguest)
{
Start-VM $guest -Confirm:$false
Sleep 5
}

 You remote locations should be back up!

False alerts after upgrading ILO firmware from 1.55 to 1.8

There are two separate alerts that I have experience from these upgrades.  Please note that after the update the ILO cards were rebooted, but our esxi hosts running vsphere 5.1 were not rebooted.

Error1:

Out of 31 hosts that had firmware updated, this error has appeared on 6 so far.  Some of them took a couple days before they presented with the alert.  The alarm triggered is: Host Memory Status.  Under the “Hardware Status” tab the alert shows for the System Board 8 Memory: Uncorrectable ECC.

memoryalarm

uncorrectable

 

Error2:

The second problem that I have encountered is the filling up of the IPMI SEL Log.  I am able to go in an clear the log which gets rid of the alert for a short time, but the log fills up again.  The alert shows as Host IPMI System Event Log Status.  Under the “Hardware Status” tab the “System Event Log” and “IPMI SEL” show as Unknown.  You can click on “Show Event Log” and then “Reset Event Log” and it will clear for a while…but the alert will return.  Notice the future date of 12/31/9998 which I am guessing might be when the world ends.

hostipmistatus

ipmisellog
thefuturelogs

How do I fix these problems?  After calling HP and VMware I was told that I needed to put each host into Maintenance Mode and then run a “Detailed Hardware Diagnostic”.  According to VMware this was the only way to clear the error (especially the memory one).  The solution that ended up working for me was to just reboot the host…:)  The VMware couldn’t believe that worked, but it did.  I know it isn’t a difficult fix, but maybe this might help others that get this alert.

Please let me know if you have encountered similar alerts from upgrading the firmware on your ILO ports.

Failing/Failed Disk And Detailed Information From HPACUCLI

Today I was alerted by vCenter with a yellow, “Host Storage Status” alarm.
diskalert

Clicking the ESXi host in the alert I find from the description that I can check the Hardware Status tab of the host for more details
diskalertdetail

I see that the disk is failing and that RAID is in “Interim Recovery”.  This doesn’t give me much information about the failing disk though.  Maybe in HP ILO I can find more information?  Nope, ILO doesn’t seem to have any information about the disks in the host.  According to the ILO System Information, everything looks just fine.
nodiskilo

I called to open a ticket with HP and while on the phone the support technician mentioned a diagnostic tool called HPACUCLI.  If you installed your host a HP customized ESXi image, then this should be there for you.

SSH into your host then type cd opt\hp\hpacucli\bin.  If you type ls you should see the script called “hpacucli”.  Type ./hpacucli and then press “Enter”.  You should see something about detecting controllers and you should be at a new prompt.
hpacucli1

Type controller all diag file=/tmp/adu1.zip and then press “Enter”.
hpacucli2

Now you will need a program like WinSCP that will get the newly created zip file off the host.  Connect to the host with WinSCP and then navigate to the /tmp folder.  Click on the adu1.zip file and the click “Download”.
tmpadu

Looking into the zip file we see four files; double click on “ADUReport.htm”.  The report should open into your browser and should have a lot of information.  Towards the top of my report I see that the drive has failed and if I scroll down I can actually get some in depth information about the drive.  Things like the model and Serial.
Drivefaileddrivefailed2