Checking for “Dead” paths on HBAs with PowerCLI

Posted: 27/08/2011 in PowerShell, Virtualisation

This week I have been tasked with assisting a storage engineer while he replaces a couple of Fibre Channel switches. My task was simple, check the HBAs for any “Dead” paths (See below screen shot for an example) across all the ESX hosts every hour for a weekend. We have a fair few VMHosts so this gave me the opportunity to use PowerCLI (Think PowerShell with a VMware snapin).

It didn’t take long at all to knock up a little script that can report on path states, this is what I came up with:

$VMHosts = Get-VMHost  | ? { $_.ConnectionState -eq "Connected" } | Sort-Object -Property Name
$results= @()

foreach ($VMHost in $VMHosts) {
Get-VMHostStorage -RescanAllHba -VMHost $VMHost | Out-Null
[ARRAY]$HBAs = $VMHost | Get-VMHostHba -Type "FibreChannel"

	foreach ($HBA in $HBAs) {
    $pathState = $HBA | Get-ScsiLun | Get-ScsiLunPath | Group-Object -Property state
    $pathStateActive = $pathState | ? { $_.Name -eq "Active"}
    $pathStateDead = $pathState | ? { $_.Name -eq "Dead"}
    $pathStateStandby = $pathState | ? { $_.Name -eq "Standby"}
    $results += "{0},{1},{2},{3},{4},{5}" -f $VMHost.Name, $HBA.Device, $VMHost.Parent, [INT]$pathStateActive.Count, [INT]$pathStateDead.Count, [INT]$pathStateStandby.Count
    }

}
ConvertFrom-Csv -Header "VMHost","HBA","Cluster","Active","Dead","Standby" -InputObject $results | Ft -AutoSize

The output is pretty simple but does the job:

VMHost                            HBA    Cluster        Active Dead Standby
------                            ---    -------        ------ ---- -------
d1-vmesx-001.*************.uk    vmhba2 Cluster05      40     0    24
d1-vmesx-001.*************.uk    vmhba3 Cluster05      40     0    24
d2-vmesx-001.****************.uk vmhba2 Cluster02      98     0    80
d2-vmesx-001.****************.uk vmhba3 Cluster02      96     0    80
d2-vmesx-011.****************.uk vmhba2 Cluster03      76     0    88
d2-vmesx-011.****************.uk vmhba3 Cluster03      75     0    85
d2-vmesx-012.****************.uk vmhba2 Cluster03      76     0    88
d2-vmesx-012.****************.uk vmhba3 Cluster03      75     0    85
d2-vmesx-013.****************.uk vmhba1 Cluster03      83     0    85
d2-vmesx-013.****************.uk vmhba2 Cluster03      83     0    85
d2-vmesx-014.****************.uk vmhba1 Cluster03      83     0    85
d2-vmesx-014.****************.uk vmhba2 Cluster03      83     0    85
d2-vmesx-015.****************.uk vmhba1 Cluster03      83     0    85
d2-vmesx-015.****************.uk vmhba2 Cluster03      83     0    85
d2-vmesx-016.****************.uk vmhba1 Cluster03      83     0    85
d2-vmesx-016.****************.uk vmhba2 Cluster03      83     0    85
d2-vmesx-017.****************.uk vmhba1 MSCS Cluster01 44     0    308
d2-vmesx-017.****************.uk vmhba2 MSCS Cluster01 44     0    308
d2-vmesx-018.****************.uk vmhba1 MSCS Cluster01 44     0    308
d2-vmesx-018.****************.uk vmhba2 MSCS Cluster01 44     0    308
d2-vmesx-019.****************.uk vmhba1 Cluster04      68     0    52
d2-vmesx-019.****************.uk vmhba2 Cluster04      68     0    52
d2-vmesx-021.****************.uk vmhba1 Cluster04      68     0    52
d2-vmesx-021.****************.uk vmhba2 Cluster04      68     0    52

If you wanted to work with this information further then I would suggest creating a custom object and populating the properties with Add-Member instead of doing the format string method above, I just did that as it’s quick. Well this is my first PowerCLI script, I hope to share some more.

Bye for now.

jfrmilner

Advertisements
Comments
  1. […] me a simple report of the status of each of the LUN paths to each of the HBAs on my hosts. I found John Milner’s post to be very helpful, and it gave me exactly the results that I wanted. However, it took forever to […]

  2. sam says:

    Hi, any chance i could reset / rescan-all that ‘dead’ link on a FibreChannel connection? on our dev-box here, when i tried the Get-VMhost $ServerVM | Get-VMHostStorage -RescanAllHba, i keep getting SoftwareIScsiEnabled = False

    • jfrmilner says:

      Sam,

      The SoftwareIScsiEnabled message is informational and you should see in the Windows GUI the “Rescan all HBAs” with a Completed message in resent tasks, you could also try Right-Clicking on one of your VMHosts and selecting “Rescan for Datastores”.

      Hope this helps.

      Jfrmilner.

  3. Matt K says:

    Hello,
    How can I run this script against a single ESX host? Say I have this:

    connect-viserver -Server “vcdev.ndcvc.com” -protocol https

    $VMHostname = Read-Host “Please enter a hostname”

    Can the script be run against JUST $VMHostname?

    Thanks!

    • Matt K says:

      One other thing, when I run the script I get this error:

      out-lineoutput : The object of type “Microsoft.PowerShell.Commands.Internal.For
      mat.FormatStartData” is not valid or not in the correct sequence. This is likel
      y caused by a user-specified “format-table” command which is conflicting with t
      he default formatting.
      + CategoryInfo : InvalidData: (:) [out-lineoutput], InvalidOperat
      ionException
      + FullyQualifiedErrorId : ConsoleLineOutputOutOfSequencePacket,Microsoft.P
      owerShell.Commands.OutLineOutputCommand
      PowerCLI C:\scripts>

      Any insight into that error? Is it possible to dump the output to a .htm file?

      Thanks!

      • jfrmilner says:

        I’m unable to replicate the error your getting but I suspect that in your effort to copy and paste the code into your script file you may have picked up some extra html formatting, try using the “copy to clipboard” or “view source” feature on the code block and try again.

        Dumping to html is possible with the ConvertTo-Html cmdlet, please check the PowerShell help documentation for further details.

        Regards,

        jfrmilner

    • jfrmilner says:

      Matt,

      Yes, the first line collects the server(s) in the $VMHosts variable and this can be a single host or a collection.

      Regards,

      jfrmilner

  4. […] me a simple report of the status of each of the LUN paths to each of the HBAs on my hosts. I found John Milner’s post to be very helpful, and it gave me exactly the results that I wanted. However, it took forever to […]

  5. wojcieh says:

    Really useful script. Thanks!

  6. StuartCz says:

    To run it against one host just add a hostname after the get-vmhost. The below format checks for a specific vendor, runs a bit faster (no rescans), and exports it to a gridview. It also should run cleanly as a script

    connect-viserver **myvcenter**
    $VMHosts = get-cluster **mycluster** | Get-VMHost | ? { $_.ConnectionState -eq “Connected” } | Sort-Object -Property Name
    $allresults= @()

    foreach ($VMHost in $VMHosts) {
    write-host “Working on host $VMHost”
    [ARRAY]$HBAs = $VMHost | Get-VMHostHba -Type “FibreChannel”

    foreach ($HBA in $HBAs) {
    $results = “” | select Name, Device, Cluster, ActiveP, DeadP, StandbyP
    $pathState = $HBA | Get-ScsiLun | ? { $_.Vendor -match “EMC”} | Get-ScsiLunPath | Group-Object -Property state
    $results.Name = $VMHost.Name
    $results.Device = $HBA.Device
    $results.Cluster = $VMHost.Parent
    $results.ActiveP = [INT]($pathState | ? { $_.Name -eq “Active”}).Count
    $results.DeadP = [INT]($pathState | ? { $_.Name -eq “Dead”}).Count
    $results.StandbyP = [INT]($pathState | ? { $_.Name -eq “Standby”}).Count
    $allresults += $results
    }

    }

    # Display the results in Gridview
    $allresults | out-gridview
    # Or export to a CSV
    # $allresults | export-csv “myfile”

    disconnect-viserver **myvcenter** -confirm:$false

  7. […] jfrmilner: Check for “Dead” paths on HBAs with PowerCLI […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s