Category Archives: VMware Vsphere 5

Enabling EVC (Enhanced vMotion Compatibility)

It has been a while since I have had to enable EVC, but I needed to the other day in the office.  I created a cluster with a HP DL380 G7 and an older HP DL380 G5.  When I tried to turn EVC on for the cluster, I ran into this error.  “The host cannot be admitted to the cluster’s current Enhanced vMotion Compatibility mode.  Powered-on or suspended virtual machines on the host may be using CPU features hidden by that mode.”  This message is telling you that the current machines that are powered on are using the technology from the newer processor and in order to turn on EVC for the cluster, the VMs need to be powered off.  So…I powered off all of my VMs on the DL380 G7 (newer host).EVC1

After all VMs are powered down, right click on the cluster and select Edit Settings.
evc2

Click the VMware EVC on the left pane and the click Change EVC Mode… button.EVC3

I have Intel processors so I selected Enable EVC for Intel Hosts.  Now I get a green check under the Compatibility pane.  Looking good!EVC4

Now depending on the processor generation, you have to change the EVC Mode.  For mine, I chose the Intel “Penryn” Generation” and I still had a green check box.  If your hosts don’t support the EVC mode, it will let you know in the Compatibility pane.  The processor support documentation can be found here.EVC5

We now see that Intel “Penryn” Generation is my EVC mode.   The only thing left to do is power on the VMs and start your migrations!EVC6

 

PowerCLI shutdown/startup script for remote sites.

In my environment I have 11 Remote Locations that each have one ESXi host.  I had a problem with ILO firmware updates the other day where I had to reboot these remote hosts.  I really didn’t want to use the VIC to click on each of the VMs to shut them down, so using a lot of help from mpreston’s site I came up with a script that will shut down all of my VMs and then when ready, start the VMs again.  My script shuts down “Front End” VMs first, then DB VMs, and then finally DCs.  I have a modified version that allows me to do the same for my Datacenters that I will share later.  You might have to edit the script if your database and or DCs are named differently.

add-pssnapin VMware.VimAutomation.Core
connect-viserver servername #Insert your vcenter server

#Variables
$datacenter = “datacenter”  #Add all datacenters here with a comma and quotations between each example “datacenter1”, “datacenter2”
$filename = “c:\vmshutdown\poweredonvms.csv”  #You must create this file first…it won’t create it for you.

#Get a list of the hosts being rebooted and set them to maintenance mode.  This keeps the VMs from powering on again.
Get-VMHost -location $datacenter | set-vmhost -State Maintenance

#Get a list of powered on VMs
get-vm -location $datacenter | Where-Object {$_.PowerState -eq “PoweredOn”} | Select Name | Export-CSV $filename

#Get a list of VMs that are not Database servers or DCs.  
Write-Host “”
Write-Host “Retrieving a list of powered on VMs…” -ForegroundColor Green
Write-Host “”
$poweredonguests = Get-VM -Location $datacenter | Where-Object {$_.PowerState -eq “PoweredOn”} | where {$_.Name -notlike “*DB*”} | where {$_.Name -notlike “*DC*”}

#Power off servers that are not Database and DCs
foreach ($guest in $poweredonguests)
{
Write-Host “Processing $guest…” -ForegroundColor Green
Write-Host “Checking for VMware Tools Install” -ForegroundColor Green
$guestinfo = get-view -Id $guest.Id
if ($guestinfo.config.Tools.ToolsVersion -eq 0)
{
Write-Host “No VMware tools detected in $guest, these will be powered off” -ForegroundColor Yellow
Stop-VM $guest -Confirm:$false
}
else
{
Write-Host “VMware tools detected. I will attempt to gracefully shutdown $guest”
$vmshutdown = $guest | Shutdown-VMGuest -Confirm:$false
Sleep 5
}
}

#Waiting for these shutdowns to complete
Write-Host “”
Write-Host “Giving VMs 3 minutes for VMs that are not DB or DC to shutdown.”
Write-Host “”
Sleep 180

#Shut down DB Servers at remote sites
Write-Host “”
Write-Host “Shutting Down DB Servers”
Write-Host “”
#get our list of DB servers that are powered on…
$poweredondbs = Get-VM -Location $datacenter | Where-Object {$_.PowerState -eq “PoweredOn”} | where {$_.Name -like “*DB*”} | where {$_.Name -notlike “*DC*”}
foreach ($dbs in $poweredondbs)
{
Write-Host “Processing $dbs…” -ForegroundColor Green
Write-Host “Checking for VMware Tools Install” -ForegroundColor Green
$guestinfo = get-view -Id $dbs.Id
if ($guestinfo.config.Tools.ToolsVersion -eq 0)
{
Write-Host “No VMware tools detected in $dbs, these will be powered off” -ForegroundColor Yellow
Stop-VM $dbs -Confirm:$false
}
else
{
Write-Host “VMware tools detected. I will attempt to gracefully shutdown $dbs”
$vmshutdown = $dbs | Shutdown-VMGuest -Confirm:$false
Sleep 5
}
}

#Waiting for these shutdowns to complete
Write-Host “”
Write-Host “Giving VMs 3 minutes for Database VMs to shutdown.”
Write-Host “”
Sleep 180

#Shut down DCs
Write-Host “”
Write-Host “Shutting Down DCs”
Write-Host “”
#get our list of DB servers that are powered on…
$poweredondcs = Get-VM -Location $datacenter | Where-Object {$_.PowerState -eq “PoweredOn”} | where {$_.Name -like “*DC*”}
foreach ($dcs in $poweredondcs)
{
Write-Host “Processing $dcs…” -ForegroundColor Green
Write-Host “Checking for VMware Tools Install” -ForegroundColor Green
$guestinfo = get-view -Id $dcs.Id
if ($guestinfo.config.Tools.ToolsVersion -eq 0)
{
Write-Host “No VMware tools detected in $dcs, these will be powered off” -ForegroundColor Yellow
Stop-VM $dcs -Confirm:$false
}
else
{
Write-Host “VMware tools detected. I will attempt to gracefully shutdown $dcs”
$vmshutdown = $dcs | Shutdown-VMGuest -Confirm:$false
Sleep 5
}
}

#Waiting for these shutdowns to complete
Write-Host “”
Write-Host “Giving VMs 3 minutes for DCs to shutdown.”
Write-Host “”
Sleep 180

#Time to restart the hosts
get-vmhost -location $datacenter | restart-vmhost -runasync -confirm $false

Now that you have rebooted your hosts, it is time to start all the VMs again.  You are going to use the dame variable that we created above.

#Take hosts out of maintenance mode.  
get-vmhost -location $datacenter | set-vmhost -state connected

#Power On DCs
Write-Host “”
Write-Host “Powering On DCs”
Write-Host “”
#get our list of DCs servers that are powered off…
$poweredoffdcs = Get-VM -Location $datacenter | Where-Object {$_.PowerState -eq “PoweredOff”} | where {$_.Name -like “*DC*”}
foreach ($dcs in $poweredoffdcs)
{
Start-VM $dcs -Confirm:$false
Sleep 5
}

#Power On DB Servers
Write-Host “”
Write-Host “Powering On DBs”
Write-Host “”
#get our list of DB servers that are powered off…
$poweredoffdbs = Get-VM -Location $datacenter | Where-Object {$_.PowerState -eq “PoweredOff”} | where {$_.Name -like “*DB*”}
foreach ($dbs in $poweredoffdbs)
{
Start-VM $dbs -Confirm:$false
Sleep 5
}

#Power On the rest
Write-Host “”
Write-Host “Powering On Remaining VMs”
Write-Host “”
#get our list of VMs that are still powered off…
$poweredoffguest = Get-VM -Location $datacenter | Where-Object {$_.PowerState -eq “PoweredOff”}
foreach ($guest in $poweredoffguest)
{
Start-VM $guest -Confirm:$false
Sleep 5
}

 You remote locations should be back up!

False alerts after upgrading ILO firmware from 1.55 to 1.8

There are two separate alerts that I have experience from these upgrades.  Please note that after the update the ILO cards were rebooted, but our esxi hosts running vsphere 5.1 were not rebooted.

Error1:

Out of 31 hosts that had firmware updated, this error has appeared on 6 so far.  Some of them took a couple days before they presented with the alert.  The alarm triggered is: Host Memory Status.  Under the “Hardware Status” tab the alert shows for the System Board 8 Memory: Uncorrectable ECC.

memoryalarm

uncorrectable

 

Error2:

The second problem that I have encountered is the filling up of the IPMI SEL Log.  I am able to go in an clear the log which gets rid of the alert for a short time, but the log fills up again.  The alert shows as Host IPMI System Event Log Status.  Under the “Hardware Status” tab the “System Event Log” and “IPMI SEL” show as Unknown.  You can click on “Show Event Log” and then “Reset Event Log” and it will clear for a while…but the alert will return.  Notice the future date of 12/31/9998 which I am guessing might be when the world ends.

hostipmistatus

ipmisellog
thefuturelogs

How do I fix these problems?  After calling HP and VMware I was told that I needed to put each host into Maintenance Mode and then run a “Detailed Hardware Diagnostic”.  According to VMware this was the only way to clear the error (especially the memory one).  The solution that ended up working for me was to just reboot the host…:)  The VMware couldn’t believe that worked, but it did.  I know it isn’t a difficult fix, but maybe this might help others that get this alert.

Please let me know if you have encountered similar alerts from upgrading the firmware on your ILO ports.

Extending VMware VMDKs with System Partitions

hard-disk-drive
I have had to deal with extending System Partition VMDKs within our environment, but this can get tricky especially with 2003 Server and below.  Hope this helps…Note that Data drives can be extended without issue on 2012 and below.

2008 and above
1.  Open your VIC and edit the settings of the VM that you would like to modify.
Webeditsettings
2.  Change the system disk to the new size (Usually Hard disk 1).
Disk Change
3.   Log into your VM using the console or RDP.
4.  There are two ways to do the extend.  If it is 2008 R2 and above I usually do option 1.  2008 Standard and below I use option 2.

Option 1:
a.  Open Server Manager.
ServerManager
b.  Click Disk Management.
c.  Usually you will have to do a rescan to see the unallocated space.  Right click Disk Management and click Rescan Disks.
Rescandisks
d.  Right click the drive that you wish to expand (should show unallocated space) and click Extend Volume.
ExtendVolume
e.  Click OK on the Extend Volume Wizard and then again click OK on the Select Disks screen.  Now click Finish to complete the wizard.
f.  You should now see the drive is larger.

Option 2:
a.  Click Start –> Run and then type cmd.  Right click on the command prompt application and choose Run as Administrator.
runadmincmd
b.  Type diskpart.
diskpartsmall
c.  Type list volume.
diskpartlistvolume
d.  Select the system partition by typing select volume #.
diskpartselectvolume
e.  You should see an asterisk by the selected volume.  Type extend.
diskpartextend
f.  The drive should now be extended.  Type exit.

2003 Server and Below
1.   Have a 2008 R2 Helper machine that you will use to expand the drive.
2.  Power down the 2003 Server VM.
3.  Edit the settings and add space to the system partition.
Disk Change
4.  Power on the 2003 Server VM.
5.  Click Start –>All Programs –> Administrative Tools –> Computer Management.
ServerManager03
6.  Click Disk Management.  You should see the unallocated space, but if you do not then right click Disk Management and click Rescan Disks.   Just verify…don’t do anything with the space yet.
Rescandisks
7.  Shutdown the 2003 Server VM.
8.  Edit the 2003 Server VM within the VIC and note the location of the disk on the datastore.  Click remove on the drive that you would like extended by clicking on the X next to the hard disk.  DO NOT CHOOSE DELETE FILES FROM DATASTORE or REMOVE FROM VIRTUAL MACHINE AND DELETE FILES FROM DISK.  This is different wording depending on the Full VIC or the Web VIC.  Click OK.
RemoveDisk
diskremoved
9.  Now find your 2008 R2 helper VM within the VIC and then edit the settings.
10.  Where it says “New Device” choose Existing Hard Disk from the menu.  Click Add.
Addexistingdisk
11.  Using the information that you noted before, find the location of the 2003 Server disk that you removed before, select the disk, click OK.
choosevmdk
12.  Cick OK and this will mount the 2003 Server disk on your 2008 R2 helper VM.
13.  Log into the 2008 R2 helper VM with the console or RDP.
14. Click Start –> Run and then type cmd.  Right click on the command prompt application and choose Run as Administrator.
runadmincmd
15.  Type diskpart.
diskpartsmall
16.  Type list volume.
diskpartlistvolume
17.  Select the system partition by typing select volume #.
diskpartselectvolume
18.  You should see an asterisk by the selected volume.  Type extend.
diskpartextend
19.  The drive should now be extended.  Type exit.
20.  Go back into the VIC and edit the 2008 R2 helper VM.
21.  Click the X next to the Hard disk that you added earlier to this machine.  Again, DO NOT CHOOSE DELETE FILES FROM DATASTORE or REMOVE FROM VIRTUAL MACHINE AND DELETE FILES FROM DISK. 
RemoveDisk
diskremoved

22.  Edit the 2003 Server and then from the “New device” menu click Existing Hard Disk and then click Add.
Addexistingdisk
23.  Find the vmdk that you just extended then click OK.  Click again on the settings screen.
choosevmdk
24.  Power on the 2003 Server and you should see the additional space.  You might get a message indicating that a reboot is needed.  If requested, do the reboot.

Random Issues I Have Run Across
I did have an issue the other day with one our our 2008 Standard Servers.  When I went to extend the drive it gave me a weird error.  The parameter is incorrect.
DiskGrowParameterIncorrect

I give thanks to the Lord every day for the googles.  I found a great post by Jonathan Medd that gave the simple fix for this.  When you are in diskpart, select the volume # that you are extending and then use the command extend filesystem.

SRM Testing…What do I do with my Domain Controllers? Part 1

For the most part I love SRM for providing a relatively simple way to back up and recover our  VMware environment.  The problem that I have with SRM is that when it comes to testing failover, what do you do with your domain controllers?  According to VMware’s documentation found here, you should never use SRM to backup your domain controllers because you should already be using Active Directory replication to handle recovery situations.

When testing in the past I have added the DCs to my test recovery plan.  The problem was that most of the time the cloned DCs (in a test bubble) took forever to come up, and because of this, testing would take way longer than it should.  What is the solution?

The solution I have developed is to use scheduled powercli task that will clone my primary DC on a daily basis, change the network port group to a test bubble, and then power it on.  Doing this gives me a domain controller in my test bubble that should be ready to use whenever I want to test SRM.  I am still deciding whether or not I need to power down the primary DC first before cloning.  For now, I am cloning live.  I have a feeling I will hear from you guys on this one…:)

Preparation:

1.  The first thing I have is an old physical desktop to run my scripts.  You could run your scripts through a VM, however, I will be running my UPS Emergency shutdown script through this machine as well.

2.  I created a service account in AD that has very specific rights within vCenter.  Give this AD account admin rights on the physical machine that you will run your scripts from.

3.  Install PowerCLI on the machine using the AD account you created earlier.  Some good installation instructions can be found here.  Make sure that your set the execution policy to remotesigned.  While you still have PowerCLI open, add the credentials of the AD account to the PowerCLI credential store.  Information on this can be found here.  In our example in PowerCLI you would type New-VICredentialStoreItem -Host <vcenter server name/ip> -User <AD Script Account> -Password <Password for the script account>.  This will store the credentials within the user profile directory and it is encrypted!  This way there is no username and password in clear text going over the wire.  Note that if you try to test the script right now it will not work because there are no rights for the account in vCenter.

4.  I am going to be good and use the Web VIC to configure the permissions, sometimes I feel like this though…FullVIC

Now that I have gotten that out of my system, log into the Web VIC as an administrator and click Administration –> Roles.  Click GreenPlus to create a new role.  Name this role something like “clone” or cloneandchangenetwork”.  Check the following boxes:

Datastore –> Allocate Space
Network –> Assign Network
Network –> Configure
Resource –> Assign Virtual Machine to Resource Pool
Virtual Machine –> Configuration –> Add New Disk
Virtual Machine –> Configuration –> Add or Remove Device
Virtual Machine –> Configuration –> Modify Device Settings
Virtual Machine –> Interaction –> Power On
Virtual Machine –> Interaction –> Power Off
Virtual Machine –> Inventory –> Create from existing
Virtual Machine –> Inventory –> Create New
Virtual Machine –> Inventory –> Register
Virtual Machine –> Inventory –> Remove
Virtual Machine –> Provisioning –> Clone Virtual Machine

Now that our role was created, we need to assign it to the locations that we want this role to have rights.  To keep this simple, but not very secure, you could assign the role to the vCenter Server level.  The rights will propagate down the entire tree across any datacenters that you might set up.  Most people don’t want this because it would allow the cloning of any server, so we are going to assign rights exactly where they need to go.  But first we need to create a new folder where our cloned VM will eventually live.

Folder Rights
Click Home –> VMs and Templates.  Using the arrows, expand the list on the left pane.  Right click Discovered virtual machine –> New Folder and then name your new folder something like DCClone.  Click Manage –> Permissions.  Click GreenPlus and then from the drop down menu select the role that you created earlier and then click Add.  Choose your domain from the Domain drop down and then find the user that we created earlier in the list.  Click OK.  You should now see the AD user and the assigned role.  Now uncheck the Propagate to children box and then Click OK.

Cluster Rights
Click Home –>  Hosts and Clusters.  Then click on the Cluster where the DC resides that you will be cloning.  Click Manage –> Permissions.  Click GreenPlus and then from the drop down menu select the role that you created earlier and then click Add.  Choose your domain from the Domain drop down and then find the user that we created earlier in the list.  Click OK.  You should now see the AD user and the assigned role.  Now uncheck the Propagate to children box and then Click OK.  This keeps the script account from having rights down the tree.
Propigate

DC Rights
Click Home –>  Hosts and Clusters.  Then click on the DC that needs to be cloned.  Click Manage –> Permissions.  Click GreenPlus and then from the drop down menu select the role that you created earlier and then click Add.  Choose your domain from the Domain drop down and then find the user that we created earlier in the list.  Click OK.  You should now see the AD user and the assigned role.  Click OK.

Datastore Rights
Now we need to set the role for the host that will house the cloned VM.  Click Home –> Storage, and then on the left hand side click the datastore that the will be home to the clone.  Click the Manage –> Permissions.  Click GreenPlus and then from the drop down menu select the role that you created earlier and then click Add.  Choose your domain from the Domain drop down and then find the user that we created earlier in the list.  Click OK.  You should now see the AD user and the assigned role.  Click OK.

Host Rights
Now we need to set the role for the host that we are going to clone to.  Click the Home –>  Hosts and Clusters, and then on the left hand pane click the host  that the will be home to the clone.  Click the Manage –> Permissions.  Click GreenPlus and then from the drop down menu select the role that you created earlier and then click Add.  Choose your domain from the Domain drop down and then find the user that we created earlier in the list.  Click OK.  You should now see the AD user and the assigned role.  Now uncheck the Propagate to children box and then Click OK.  This keeps the script account from having rights down the host.
Propigate

Network Rights
Finally, we are going to set the role for the network.  Click Home –> Networking.  On the left pane select the test network that  you have in your environment.  Mine is called Test Bubble.  Click the Manage –> Permissions.  Click GreenPlus and then from the drop down menu select the role that you created earlier and then click Add.  Choose your domain from the Domain drop down and then find the user that we created earlier in the list.  Click OK.  You should now see the AD user and the assigned role.  Click OK.

By the time you are done, your script service account should have rights in vCenter for the Cluster (no propagate), host where the clone will live (no propagate), the Datastore, Network, Folder, and DC that will be cloned.
Rights

WAY TO GO!!  Are you ready for part 2?rejoicing

 

Enable Copy and Paste Through the VMware Console

I was setting up a new host the other day and I received a call from one of my admins letting me know that they could not copy/paste within the console; but they can copy/paste in RDP.
nocopypast

This is a simple fix found in KB1026437.  You can make the change on an individual VM, but I think it is best to change it on the host (which applies to all VMs).  I really wish the default would have this enabled.

Open a Putty session…if you don’t have putty then get it here.
putty

1. Log into the ESXi host that you want to change.
2. Type vi /etc/vmware/config
3. Arrow down to the last line and type which stands for “insert”.
4. Add the lines:
         vmx.fullpath = “/bin/vmx”
         isolation.tools.copy.disable=”FALSE”
         isolation.tools.paste.disable=”FALSE”
5. Press the ESC key and then type :wq which stands for “write and quit”.
addlines

The next time each VM is power cycled it will enable the copy/paste functionality.  Keep in mind that if you ever upgrade this host to a new ESXi version that this setting will go back to the default of disabled and you will have to add this line again.

Deploying VMware Support Assistant v5.5

The other day I set up and configured VMware’s new Support Assistant 5.5.  I have used older versions to do things like open tickets and pull log files, but the new version has proactive support built in.  You configure when you want your log files sent to VMware and they compare them to know issues.  VMware even includes the ability to scrub the files before sending them out.

Here is how to deploy.

  1. Download the .OVA from www.myvmware.com. I am using build 1549662.
  2. Open up your VIC.
  3. Click File and then Deploy OVF Template.
  4. Select the OVA that you downloaded and click Open.
    1. Click Next on the Source Location window and then Next again on the details screen.
    2. Click Accept and then Next.
    3. Name your new support appliance and choose a folder if applicable then click Next.
    4. Select the host that the appliance will run on and then click Next.
    5. If applicable, choose a resource pool for this appliance and then click Next.
    6. Choose the Destination Network and then click Next.
    7. Enter in the Gateway, DNS, IP, and Subnet Mask and then click Next.
    8. On the Ready to Complete screen check the Power on after deployment box and then click Finish.
    9. You should see a progress bar indicating the status of the deployment.
  5. Open a console screen and you should see the appliance boot and eventually tell you to browse to the appliance IP to finish configuring. Open up a browser and go to that address.
  6. Accept the EULA…actually relatively short. Click Next.
  7. The lookup service address is the SSO server. Enter your SSO server and then click Next.
  8. Enter your SSO credentials which will usually be either admin@System-Domain if you installed SSO in 5.1 or administrator@vsphere.local if you upgraded from 5.1 to 5.5. Click Finish.
  9. Add an account that has rights to vCenter and then check the box to Assign log collection permissions for the following vCenter Server instances. Click Next.
  10. Enter your proxy information if needed. This allows the appliance to talk with VMware and send the log files. Click the Test Connectivity button to ensure things are working properly and then click Next.
  11. Add an email address to receive update about your environment then click Finish.
  12. Hopefully you see that the Service is ready…log gathering is disabled though. We will fix that in just a minute, but first let’s not forget to change the root password. Click VA Settings. You will see a place to put in the current password and then a new password. This is for the root account. The default password is vmare and make it something that you will remember. Click Save.
  13. The Support Assistant only works in the new Web VIC. Open a browser and navigate to your vCenter server and login.
  14. You should see a new icon now that looks like a life preserver called vCenter Support Assistant. Double click the icon.
  15. Click the link Configure data collection.
  16. Here you can change when your appliance will upload logs to VMware for analysis. The default is never, but that won’t help us! In the example mine will upload every Monday at 10AM. Click Entity Selection.
  17. Select the vCenter servers (linked mode supported) and hosts that you want logs from that will be sent to VMware. Click Data Scrubbing.
  18. This is great that VMware includes the ability to scrub the log files. Note that if you check these boxes, it will have an effect on the virtual appliance as it looks through and redacts the log files before sending to VMware. Check the boxes that you require and then click OK.
  19. Click the Monitor tab. This tab shows the status of your support uploads. Click the Manage tab.
  20. This gives a summary of your proactive support settings, which you just changed. Click the Support Requests button.
  21. After logging in with your Myvmware.com username and password, you can check the status of open and closed tickets and upload logs if needed.