Category Archives: VMware Vcenter Server

PowerCLI shutdown/startup script for remote sites.

In my environment I have 11 Remote Locations that each have one ESXi host.  I had a problem with ILO firmware updates the other day where I had to reboot these remote hosts.  I really didn’t want to use the VIC to click on each of the VMs to shut them down, so using a lot of help from mpreston’s site I came up with a script that will shut down all of my VMs and then when ready, start the VMs again.  My script shuts down “Front End” VMs first, then DB VMs, and then finally DCs.  I have a modified version that allows me to do the same for my Datacenters that I will share later.  You might have to edit the script if your database and or DCs are named differently.

add-pssnapin VMware.VimAutomation.Core
connect-viserver servername #Insert your vcenter server

#Variables
$datacenter = “datacenter”  #Add all datacenters here with a comma and quotations between each example “datacenter1”, “datacenter2”
$filename = “c:\vmshutdown\poweredonvms.csv”  #You must create this file first…it won’t create it for you.

#Get a list of the hosts being rebooted and set them to maintenance mode.  This keeps the VMs from powering on again.
Get-VMHost -location $datacenter | set-vmhost -State Maintenance

#Get a list of powered on VMs
get-vm -location $datacenter | Where-Object {$_.PowerState -eq “PoweredOn”} | Select Name | Export-CSV $filename

#Get a list of VMs that are not Database servers or DCs.  
Write-Host “”
Write-Host “Retrieving a list of powered on VMs…” -ForegroundColor Green
Write-Host “”
$poweredonguests = Get-VM -Location $datacenter | Where-Object {$_.PowerState -eq “PoweredOn”} | where {$_.Name -notlike “*DB*”} | where {$_.Name -notlike “*DC*”}

#Power off servers that are not Database and DCs
foreach ($guest in $poweredonguests)
{
Write-Host “Processing $guest…” -ForegroundColor Green
Write-Host “Checking for VMware Tools Install” -ForegroundColor Green
$guestinfo = get-view -Id $guest.Id
if ($guestinfo.config.Tools.ToolsVersion -eq 0)
{
Write-Host “No VMware tools detected in $guest, these will be powered off” -ForegroundColor Yellow
Stop-VM $guest -Confirm:$false
}
else
{
Write-Host “VMware tools detected. I will attempt to gracefully shutdown $guest”
$vmshutdown = $guest | Shutdown-VMGuest -Confirm:$false
Sleep 5
}
}

#Waiting for these shutdowns to complete
Write-Host “”
Write-Host “Giving VMs 3 minutes for VMs that are not DB or DC to shutdown.”
Write-Host “”
Sleep 180

#Shut down DB Servers at remote sites
Write-Host “”
Write-Host “Shutting Down DB Servers”
Write-Host “”
#get our list of DB servers that are powered on…
$poweredondbs = Get-VM -Location $datacenter | Where-Object {$_.PowerState -eq “PoweredOn”} | where {$_.Name -like “*DB*”} | where {$_.Name -notlike “*DC*”}
foreach ($dbs in $poweredondbs)
{
Write-Host “Processing $dbs…” -ForegroundColor Green
Write-Host “Checking for VMware Tools Install” -ForegroundColor Green
$guestinfo = get-view -Id $dbs.Id
if ($guestinfo.config.Tools.ToolsVersion -eq 0)
{
Write-Host “No VMware tools detected in $dbs, these will be powered off” -ForegroundColor Yellow
Stop-VM $dbs -Confirm:$false
}
else
{
Write-Host “VMware tools detected. I will attempt to gracefully shutdown $dbs”
$vmshutdown = $dbs | Shutdown-VMGuest -Confirm:$false
Sleep 5
}
}

#Waiting for these shutdowns to complete
Write-Host “”
Write-Host “Giving VMs 3 minutes for Database VMs to shutdown.”
Write-Host “”
Sleep 180

#Shut down DCs
Write-Host “”
Write-Host “Shutting Down DCs”
Write-Host “”
#get our list of DB servers that are powered on…
$poweredondcs = Get-VM -Location $datacenter | Where-Object {$_.PowerState -eq “PoweredOn”} | where {$_.Name -like “*DC*”}
foreach ($dcs in $poweredondcs)
{
Write-Host “Processing $dcs…” -ForegroundColor Green
Write-Host “Checking for VMware Tools Install” -ForegroundColor Green
$guestinfo = get-view -Id $dcs.Id
if ($guestinfo.config.Tools.ToolsVersion -eq 0)
{
Write-Host “No VMware tools detected in $dcs, these will be powered off” -ForegroundColor Yellow
Stop-VM $dcs -Confirm:$false
}
else
{
Write-Host “VMware tools detected. I will attempt to gracefully shutdown $dcs”
$vmshutdown = $dcs | Shutdown-VMGuest -Confirm:$false
Sleep 5
}
}

#Waiting for these shutdowns to complete
Write-Host “”
Write-Host “Giving VMs 3 minutes for DCs to shutdown.”
Write-Host “”
Sleep 180

#Time to restart the hosts
get-vmhost -location $datacenter | restart-vmhost -runasync -confirm $false

Now that you have rebooted your hosts, it is time to start all the VMs again.  You are going to use the dame variable that we created above.

#Take hosts out of maintenance mode.  
get-vmhost -location $datacenter | set-vmhost -state connected

#Power On DCs
Write-Host “”
Write-Host “Powering On DCs”
Write-Host “”
#get our list of DCs servers that are powered off…
$poweredoffdcs = Get-VM -Location $datacenter | Where-Object {$_.PowerState -eq “PoweredOff”} | where {$_.Name -like “*DC*”}
foreach ($dcs in $poweredoffdcs)
{
Start-VM $dcs -Confirm:$false
Sleep 5
}

#Power On DB Servers
Write-Host “”
Write-Host “Powering On DBs”
Write-Host “”
#get our list of DB servers that are powered off…
$poweredoffdbs = Get-VM -Location $datacenter | Where-Object {$_.PowerState -eq “PoweredOff”} | where {$_.Name -like “*DB*”}
foreach ($dbs in $poweredoffdbs)
{
Start-VM $dbs -Confirm:$false
Sleep 5
}

#Power On the rest
Write-Host “”
Write-Host “Powering On Remaining VMs”
Write-Host “”
#get our list of VMs that are still powered off…
$poweredoffguest = Get-VM -Location $datacenter | Where-Object {$_.PowerState -eq “PoweredOff”}
foreach ($guest in $poweredoffguest)
{
Start-VM $guest -Confirm:$false
Sleep 5
}

 You remote locations should be back up!

False alerts after upgrading ILO firmware from 1.55 to 1.8

There are two separate alerts that I have experience from these upgrades.  Please note that after the update the ILO cards were rebooted, but our esxi hosts running vsphere 5.1 were not rebooted.

Error1:

Out of 31 hosts that had firmware updated, this error has appeared on 6 so far.  Some of them took a couple days before they presented with the alert.  The alarm triggered is: Host Memory Status.  Under the “Hardware Status” tab the alert shows for the System Board 8 Memory: Uncorrectable ECC.

memoryalarm

uncorrectable

 

Error2:

The second problem that I have encountered is the filling up of the IPMI SEL Log.  I am able to go in an clear the log which gets rid of the alert for a short time, but the log fills up again.  The alert shows as Host IPMI System Event Log Status.  Under the “Hardware Status” tab the “System Event Log” and “IPMI SEL” show as Unknown.  You can click on “Show Event Log” and then “Reset Event Log” and it will clear for a while…but the alert will return.  Notice the future date of 12/31/9998 which I am guessing might be when the world ends.

hostipmistatus

ipmisellog
thefuturelogs

How do I fix these problems?  After calling HP and VMware I was told that I needed to put each host into Maintenance Mode and then run a “Detailed Hardware Diagnostic”.  According to VMware this was the only way to clear the error (especially the memory one).  The solution that ended up working for me was to just reboot the host…:)  The VMware couldn’t believe that worked, but it did.  I know it isn’t a difficult fix, but maybe this might help others that get this alert.

Please let me know if you have encountered similar alerts from upgrading the firmware on your ILO ports.

Extending VMware VMDKs with System Partitions

hard-disk-drive
I have had to deal with extending System Partition VMDKs within our environment, but this can get tricky especially with 2003 Server and below.  Hope this helps…Note that Data drives can be extended without issue on 2012 and below.

2008 and above
1.  Open your VIC and edit the settings of the VM that you would like to modify.
Webeditsettings
2.  Change the system disk to the new size (Usually Hard disk 1).
Disk Change
3.   Log into your VM using the console or RDP.
4.  There are two ways to do the extend.  If it is 2008 R2 and above I usually do option 1.  2008 Standard and below I use option 2.

Option 1:
a.  Open Server Manager.
ServerManager
b.  Click Disk Management.
c.  Usually you will have to do a rescan to see the unallocated space.  Right click Disk Management and click Rescan Disks.
Rescandisks
d.  Right click the drive that you wish to expand (should show unallocated space) and click Extend Volume.
ExtendVolume
e.  Click OK on the Extend Volume Wizard and then again click OK on the Select Disks screen.  Now click Finish to complete the wizard.
f.  You should now see the drive is larger.

Option 2:
a.  Click Start –> Run and then type cmd.  Right click on the command prompt application and choose Run as Administrator.
runadmincmd
b.  Type diskpart.
diskpartsmall
c.  Type list volume.
diskpartlistvolume
d.  Select the system partition by typing select volume #.
diskpartselectvolume
e.  You should see an asterisk by the selected volume.  Type extend.
diskpartextend
f.  The drive should now be extended.  Type exit.

2003 Server and Below
1.   Have a 2008 R2 Helper machine that you will use to expand the drive.
2.  Power down the 2003 Server VM.
3.  Edit the settings and add space to the system partition.
Disk Change
4.  Power on the 2003 Server VM.
5.  Click Start –>All Programs –> Administrative Tools –> Computer Management.
ServerManager03
6.  Click Disk Management.  You should see the unallocated space, but if you do not then right click Disk Management and click Rescan Disks.   Just verify…don’t do anything with the space yet.
Rescandisks
7.  Shutdown the 2003 Server VM.
8.  Edit the 2003 Server VM within the VIC and note the location of the disk on the datastore.  Click remove on the drive that you would like extended by clicking on the X next to the hard disk.  DO NOT CHOOSE DELETE FILES FROM DATASTORE or REMOVE FROM VIRTUAL MACHINE AND DELETE FILES FROM DISK.  This is different wording depending on the Full VIC or the Web VIC.  Click OK.
RemoveDisk
diskremoved
9.  Now find your 2008 R2 helper VM within the VIC and then edit the settings.
10.  Where it says “New Device” choose Existing Hard Disk from the menu.  Click Add.
Addexistingdisk
11.  Using the information that you noted before, find the location of the 2003 Server disk that you removed before, select the disk, click OK.
choosevmdk
12.  Cick OK and this will mount the 2003 Server disk on your 2008 R2 helper VM.
13.  Log into the 2008 R2 helper VM with the console or RDP.
14. Click Start –> Run and then type cmd.  Right click on the command prompt application and choose Run as Administrator.
runadmincmd
15.  Type diskpart.
diskpartsmall
16.  Type list volume.
diskpartlistvolume
17.  Select the system partition by typing select volume #.
diskpartselectvolume
18.  You should see an asterisk by the selected volume.  Type extend.
diskpartextend
19.  The drive should now be extended.  Type exit.
20.  Go back into the VIC and edit the 2008 R2 helper VM.
21.  Click the X next to the Hard disk that you added earlier to this machine.  Again, DO NOT CHOOSE DELETE FILES FROM DATASTORE or REMOVE FROM VIRTUAL MACHINE AND DELETE FILES FROM DISK. 
RemoveDisk
diskremoved

22.  Edit the 2003 Server and then from the “New device” menu click Existing Hard Disk and then click Add.
Addexistingdisk
23.  Find the vmdk that you just extended then click OK.  Click again on the settings screen.
choosevmdk
24.  Power on the 2003 Server and you should see the additional space.  You might get a message indicating that a reboot is needed.  If requested, do the reboot.

Random Issues I Have Run Across
I did have an issue the other day with one our our 2008 Standard Servers.  When I went to extend the drive it gave me a weird error.  The parameter is incorrect.
DiskGrowParameterIncorrect

I give thanks to the Lord every day for the googles.  I found a great post by Jonathan Medd that gave the simple fix for this.  When you are in diskpart, select the volume # that you are extending and then use the command extend filesystem.

Enable Copy and Paste Through the VMware Console

I was setting up a new host the other day and I received a call from one of my admins letting me know that they could not copy/paste within the console; but they can copy/paste in RDP.
nocopypast

This is a simple fix found in KB1026437.  You can make the change on an individual VM, but I think it is best to change it on the host (which applies to all VMs).  I really wish the default would have this enabled.

Open a Putty session…if you don’t have putty then get it here.
putty

1. Log into the ESXi host that you want to change.
2. Type vi /etc/vmware/config
3. Arrow down to the last line and type which stands for “insert”.
4. Add the lines:
         vmx.fullpath = “/bin/vmx”
         isolation.tools.copy.disable=”FALSE”
         isolation.tools.paste.disable=”FALSE”
5. Press the ESC key and then type :wq which stands for “write and quit”.
addlines

The next time each VM is power cycled it will enable the copy/paste functionality.  Keep in mind that if you ever upgrade this host to a new ESXi version that this setting will go back to the default of disabled and you will have to add this line again.

The authentication server returned an unexpected error

I came in this morning only to be greeted by my web client telling me that I can’t login because it can’t create SAML 2.0. I am not sure that I really want it creating SAML 2.0….I don’t know SAML 1.0. Ok, bad joke. Here was the message…

I found KB2034798 at which point I remoted into my SSO server and checked the imsTrace.log for “NetUserGetLocalGroups”. I didn’t find it…so the KB didn’t apply to me…L

After some more googling I found this blog post that indicated that references KB2043070. The idea is that there is a local identity source within SSO that it is trying to authenticate the users to. You have to login with the admin@system-domain account and password. Hopefully you saved this when setting up your SSO server. The only problem I had was that I didn’t have this local identity source to remove.

I thought to myself, that there might be a stale identity source on the list that it is authenticating to. I was talking to a coworker and they mentioned that there was a domain that was deleted the day before. AHAH!! I clicked on the identity source of the domain that had been removed and then clicked “Test Connection”. There was an error that didn’t tell me much.

3-12-2014 2-42-32 PMI cancelled out and was back at my list of identity sources. I selected the identity source that had been removed from AD and I hit the red X, “Delete Identity Source”. You will get a prompt asking for you to confirm. One thing to note is that the identity source that I deleted was not one of the default domains at the bottom. If you haven’t set a default domain up, I would do that now. I am wondering if there might be a bug that uses the identity source at the top of the list instead of the default at the bottom. After deleting the state Identity Source I was able to login again.


vSphere HA detected that host is in a different network partition than the master

Target: Host
Previous Status: Green
New Status: Red
Alarm Definition:
([Event alarm expression: vSphere HA agent on a host has an error; Status = Red] OR [Event alarm expression: vSphere HA detected a network isolated host; Status = Red] OR [Event alarm expression: vSphere HA detected a network-partitioned host; Status = Red] OR [Event alarm expression: vSphere HA detected a host failure; Status = Red] OR [Event alarm expression: Host has no port groups enabled for vSphere HA; Status = Red] OR [Event alarm expression: vSphere HA agent is healthy; Status = Green])
Event details:
vSphere HA detected that host (host) is in a different network partition than the master (Cluster) in Datacenter

I had been getting this message randomly over the last couple months on some of my datacenter hosts. These alerts didn’t seem to be causing any problems within the cluster, but I wanted to get to the bottom of this. I opened a ticket with VMware and uploaded the logs from both the host and vCenter, but they didn’t see anything out of the ordinary. On the second webex with VMware I noticed a couple strange things with the management network that might be the cause.

  1. The first thing I noticed was that the NICs were set for “Auto Negotiate”. I originally set up our environment on ESXi 4 before upgrading to ESXi 5.1. When I initially set this up I hard coded (KB1004089) these to 1000GB/Full. I am wondering if at some point during the upgraded that they defaulted back. On our switches it was set at 1000GB/Full so it is important that we set this on the host NICs to 1000GB/Full as well.
  2. The second thing that I noticed that in the Management network that I had the Load Balancing set to “Route based on IP hash”. The problem here is that for this to work correctly you need a port channel configured (I do not have this configured this way). This might be the cause of the HA problem if the traffic is going across these NICs is getting confused because of the Load Balancing configuration. I changed this to “Route based on the originating virtual port ID”, which makes the traffic go out on the port that it came in on. There is a good read found here…http://blogs.vmware.com/kb/2013/03/troubleshooting-network-teaming-problems-with-ip-hash.html.

This case is still ongoing with VMware and I should know in the next couple weeks if this solves my problem; my gut tells me it will.

Creating a template for Server 2012 R2 – Part 1

I have borrowed items from http://www.boche.net/blog/index.php/2012/08/16/microsoft-windows-server-2012-tips/ to create this post. I encourage you to take a moment to check out that post.

This is the step by step document that I used to build my 2012 and 2012 R2 VMs.

VM values will start at:

Hardware: Value:
Memory 4 GB
CPU’s 1
Video card Auto-detect video settings
VMCI device None
SCSI Controller 0 VMware Paravirtual
Hard disk 1 40 GB, Thin
CD/DVD Drive 1 Client Device
Floppy Drive 1 Removed (when done)
Network Adapter 1 VMXNET3
General Options OS: Microsoft Windows Server 2012 and 2012 R2
VMware Tools Default Settings
Virtual Machine Version 9

Creating the VM

  1. In vCenter click “File” then “New” then “Virtual Machine”
  2. Choose the “Custom” radio button and then “Next”.
  3. Name the SM and choose the folder location.
  4. Choose the datastore for the VM and then click “Next”.
  5. Make sure that the “Virtual Machine Version: 8” radio button is selected. (5.1 is using version 9 so I am not sure why this can’t be selected here. We will change this later.
  6. Select the “Windows” radio button and then choose “Microsoft Windows Server 2012 (64-bit).
  7. Take the default of 1 virtual socket and 1 core per virtual socket.
  8. Take the default of 4GB memory and click “Next”
  9. Choose your Network and change the adapter to “VMXNET 3” then click “Next”.
  10. Change the SCSI controller to “VMware Paravirtual” and then click “Next”.
  11. Select the “Create new virtual disk” radio button and then “Next”.
  12. Take the default of 40GB and click “Next”.
  13. Take the default virtual device node. For the system partition you want this to be SCSI 0:0. Click “Next”.
  14. On the summary screen click the “Edit the virtual machine settings before completion” box and then click “Continue”.
  15. Click on the Video Card and then change the radio button to “Auto-detect settings”.
  16. Click on the CD/DVD and then choose the datastore location that you have the 2012 R2 install ISO. Make sure under Device Status that “Connect at power on” is checked. Now click “Finish”.
  17. Right click on your newly created VM and click “Edit Settings”.
  18. Click on the Floppy drive 1 and choose the “Use existing floppy image in datastore” radio button. Then click “Browse”. At the bottom of the datastores you should see a folder called “vmimages”. Double click this folder. (For some reason until the VM is created this folder does not show up and that is why we had to create the VM and then go back into the settings to change this).
  19. Double click on the “floppies” folder.
  20. Choose the “pvscsi-Windows2008.flp” and then “OK”.
  21. The Floppy drive 1 settings should look like this and then click “OK”.
  22. On the list of VMs click the one you are building and then click the “Power On” button.
  23. Now click the “Open Console” button.
  24. The VM should boot into the 2012 R2 setup screen. Choose your language and then “Next”.
  25. Click “Install”.
  26. Choose the version of server that you are using. We use the Datacenter here. Then click “Next”.
  27. Agree to give up your first born to Microsoft by clicking the “I accept the license terms” box and then click “Next”.
  28. Choose “Custom: Install Windows only (advanced).
  29. Uh oh, there is no location to install Windows. Luckily you configured the floppy drive 1 earlier right? Click the “Load Driver” button.
  30. Click “Browse”.
  31. Look for the Floppy Disk Drive and then double click “amd64”.
  32. Select the “VMware PVSCSI Controller (A:\amd64\pvscsi.inf) and click “Next”.
  33. Hey look there is our drive!! Click “Next”.
  34. Windows should now be installing.
  35. Enter a password for your admin account. Do not lose this password!
  36. You should have the login screen now.