28 August 2012

Even Faster PowerCLI Code with Get-View, UpdateViewData() and LinkedViews

We are always looking for speed, speed, speed in our PowerShell / PowerCLI scripts.  When properly used, the Get-View cmdlet is known to be far faster than using other PowerCLI cmdlets for getting info (such as Get-VM, Get-VMHost, Get-Datastore, etc.).  It can take quite a bit longer to write a script using Get-View versus using other cmdlets, but the speed payoff is usually well worth the additional upfront time investement.  VMware has provided a way to speed things up even more.

A while back (Aug 2011), the official PowerCLI blog had a post about how to "Optimize the performance of PowerCLI's views".  The technique is centered around using the UpdateViewData() method of a "view" of a managed object (".NET View object") to retrieve nested views of managed objects.

The UpdateViewData() method had already been able to selectively update properties of the given managed object view, but one would have to use Get-View again to retrieve views of nested managed objects that are properties of that parent managed object.  "So, what?"  So, speed!

Example 0:  Get guest IP info for VMs using given VM network

Say you wanted to get the IPs configured on VMs that are connected to particular VM networks.  This example goes about doing so by getting said network(s), getting the VMs connected to them, and then getting guest IP info.

You could use the following standard way using multiple Get-View calls:
## the network name pattern to use for getting vitual networks
$strNetworkNamePattern = "MyVMNetwork1711"
## get the virtual networks whose name match this pattern
$arrNetworkViews = Get-View -ViewType Network -Property Name,VM -Filter @{"Name" = $strNetworkNamePattern}

## get the VM managed objects with a couple of properties
$arrVMsUsingThis = $arrNetworkViews | ?{$_.Vm -ne $null} | %{Get-View -Property Name,Guest.Net $_.Vm}

## get info about IPs used on this/these network(s)
$arrVMsUsingThis | %{
   $strVMName = $_.Name
   ## for each Guest.Net where the network name is like the given network name
   $_.Guest.Net | ?{$_.Network -like "*$strNetworkNamePattern*"} | %{
       ## get IPs
       $_.IpAddress | %{
           New-Object -TypeName PSObject -Property @{
               VMName = $strVMName
               IPAddr = $_
           } ## end new-object
       } ## end %
   } ## end %
} ## end %

Or, using the UpdateViewData() method, the code would be:
## the network name pattern to use for getting vitual networks
$strNetworkNamePattern = "MyVMNetwork1711"
## get the virtual networks whose name match this pattern
$arrNetworkViews = Get-View -ViewType Network -Property Name -Filter @{"Name" = $strNetworkNamePattern}

## get a couple of properties for the VMs using this network
$arrNetworkViews | %{$_.UpdateViewData("VM.Name","VM.Guest.Net")}

## get info about IPs used on this/these network(s)
$arrNetworkViews | %{$_.LinkedView.Vm} | %{
   $strVMName = $_.Name
   ## for each Guest.Net where the network name is like the given network name
   $_.Guest.Net | ?{$_.Network -like "*$strNetworkNamePattern*"} | %{
       ## get IPs
       $_.IpAddress | %{
           New-Object -TypeName PSObject -Property @{
               VMName = $strVMName
               IPAddr = $_
           } ## end new-object
       } ## end %
   } ## end %
} ## end %
The only slight differences in the code in the two ways:
  • Line 04 in each:  the first way gets an additional property, "VM", for later use
  • Line 07 in each:  the first way uses and additional Get-View call, whereas the second way uses the UpdateViewData() method
  • Line 10 in each:  both access an array of VMs connected to the given VM network, but getting to those VM arrays is a touch different (the "LinkedView" property in the second way is due to having used the UpdateViewData() method)
This runs reasonably quickly both ways when just getting information for VMs on one VM network.  But, to really illustrate the potential speed increases, I tested the main difference (line 07) with not just one or two networks, but with about 30 networks and with about 160 networks.  The results are impressive.  Again, this was measuring just the central point of each way:  the amount of time for the code on line 07 in each way above to get the VM info from vCenter (first via an additional Get-View, and then via UpdateViewData()).

Get-View vs. UpdateViewData() -- Speed testing results in a couple of different environment sizes:
Method usedNum. networksNum. netwk connectionsTime to run (sec)Total Get-View calls
additional Get-View call3343295.334
UpdateViewData()334327.751
additional Get-View call1593705811.4160
UpdateViewData()159370539.611

Mhmm.  7.75 seconds vs. 95.3 seconds in a small-ish environment -- more than twelve (12) times faster for UpdateViewData().  And 40 seconds vs. 811 seconds in a larger environment -- more than twenty (20) times faster!

Example 1:  Get VMHosts' FirewallSystems to refresh firewall info

Another example.  AC recently had needed to refresh the VMHost firewall info on hosts.  This stemmed from a bug that AC ran into in which the host firewall info is gathered during the host boot process before the firewall services are fully initiated.

Next is the overall code for each way, and then some speed comparisons of each method.  Note, we originally wrote this to only refresh firewall info for VMHosts in one cluster at a time, but to have a bit larger set of operations so as to better illustrate the speed differences, the following code performs the actions for all clusters, one cluster at a time.

Traditional way, using multiple Get-View calls:
## get the Cluster managed objects
Get-View -ViewType ClusterComputeResource -Property Name,Host | %{
   ## get all of this cluster's HostSystem managed objects
   $arrHostViews = Get-View $_.Host -Property ConfigManager.FirewallSystem
   ## for each host in the given cluster, get the FirewallSystem managed object
   foreach ($viewHost in $arrHostViews) {
       $viewFirewallSystem = Get-View $viewHost.ConfigManager.FirewallSystem -Property FirewallInfo.DefaultPolicy
       ## call RefreshFirewall() method for this host's FirewallSystem
       $viewFirewallSystem.RefreshFirewall()
   } ## end %
} ## end %

And, the UpdateViewData() way:
## get the Cluster managed objects
$arrClusterViews = Get-View -ViewType ClusterComputeResource -Property Name
## update the .NET View data with a select FirewallSystem sub-property
$arrClusterViews | %{$_.UpdateViewData("Host.ConfigManager.FirewallSystem.FirewallInfo.DefaultPolicy")}
## for each host in the given cluster, call RefreshFirewall() method
$arrClusterViews | %{$_.LinkedView.Host} | % {$_.ConfigManager.LinkedView.FirewallSystem.RefreshFirewall()}

The differences in code between the two ways:
  • Lines 04-07 in 1st way vs. Line 04 in 2nd way:  The first way uses the standard way of additionally calling Get-View for each cluster to get the HostSystems, and then again for each HostSystem to get the FirewallSystem objects.  The second way uses one UpdateViewData() call to make the FirewallSystem managed objects available from the original array of cluster managed objects.
To see the speed differences for the main info-gathering portion of each way, I measured these two pieces mentioned above:  lines 04-07 for the first way, and line 04 for the second way.  UpdateViewData() wins.

Get-View vs. UpdateViewData() -- Speed testing results in a couple of different environment sizes:
Method usedNum. clustersNum. HostSystemsTime to run (sec)Total Get-View calls
additional Get-View5219.4727
UpdateViewData()5211.391
additional Get-View 3619170.85228
UpdateViewData()361918.201


The speed gain comes, in part, from the sheer number of calls being made that require a round trip request to the given vCenter server(s).  To get additional managed object info using Get-View, there is a Get-View call involved for every [group of] managed object references (MORefs).  Conversely, there is but one (1) call to communicate with vCenter for all "child" managed objects involved when UpdateViewData() is used.

This is not the most obvious route at first, but eventually you can see how to use the method to update managed object view data, and then access the nested managed objects' views using "LinkedView" objects that then reside "in" the parent managed object as properties.

The reward for successfully employing the technique:  speed.  The PowerCLI blog post reports a 100x speed increase in the given example versus using multiple Get-View calls to get nested managed object views.  100x -- that is speeding up a script that takes five (5) minutes to take only three (3) seconds -- FaF!  This speed increase varies based on how many Get-View calls are saved/avoided, what they are retrieving, etc, but that kind of potential speed gain demands attention.

For some further info:  I used this technique in a VMware communities thread about VM portgroup inventory (yes -- December 2011; it has taken me a while to get this post together).  It shows other examples of using the resulting LinkedView properties.  And, there is good info in the PowerCLI lab guide from the 2011 VMworld labs.  The PowerCLI Blog post about PowerCLI Lab at VMWorld 2011 links to the lab manual PDF.  See pages 91-93.

How much faster is your script about to be?

6 comments:

  1. Thnak you for a very well written post, one of great quality and very usefull. I d have a question that is not directly related to this subject, is there a way to add an extent to a VMFS using powerCLI?

    ReplyDelete
    Replies
    1. Hello, Anonymous-

      Seems like using the method "AttachVmfsExtent()" might be a winner. I mention a bit about it at https://communities.vmware.com/message/2302911. In that same thread, the original poster notes that they used vmkfstools for the job (not a PowerCLI solution). Check out those options.

      Delete
  2. How would you use this approach to find all virtual machines meeting specific criteria? I have a need to find all virtual machines within a cluster where the guest OS is "*linux*".

    ReplyDelete
    Replies
    1. Hello, aenagy-

      For such a thing it would probably be better to use the -Filter parameter to Get-View to find VirtualMachines with a *Linux* value for the regular expression. Something like:

      Get-View -ViewType VirtualMachine -Property Name -Filter @{"Config.GuestId" = "Linux|SLES|Ubuntu"}

      That will get all VirtualMachines that are configured with a GuestID like Linux, SLES, or Ubuntu. Though, that GuestID is one of these does not necessarily mean that those are the OSes installed, of course. But, should generally be on the spot. Another property to check in that filter would be "Guest.GuestFamily". When populated (by VMware Tools), this should be more accurate. But, potentially not super helpful if the VM is off (may not be populated), or does not have Tools installed/running.

      Delete
  3. Thanks a lot for this helpful post! This has helped me a lot in collecting Inventory Data quickly! Much appreciated!

    ReplyDelete
    Replies
    1. Yes, you're welcome -- glad that you found it useful. And, glad to hear that you were able to increase speed by another 10%, for a total of about 90% on your inventory code that you were working on!

      Delete