Read performance counter data remotely - OS

There was a day when there were a couple of boxes with sporadically and abnormally loaded CPU and had to catch when the CPU went up for a couple of minutes on any of them.

Instead of logging into each via RDP or open Perfmon against each host manually, here is a quick command line to get the aggregate CPU load from multiple hosts in one go.

PS C:\> while (1) { (("host1,host2,host3").split(",")) | % { get-counter "\\$_\Processor(_total)\% Processor Time" -maxsamples 1 | select -exp countersamples | select path,cookedvalue }| sort CookedValue -desc; sleep 60; clear}

Some explanation and other ideas:

•  (("host1,host2,host3").split(","))
  create an array from the comma delimited list of hosts
• get-counter "\\$_\Processor(_total)\% Processor Time" -maxsamples 1
  read counter from each host remotely. The counter can be replaced i something else needs to be monitored, other than CPU
• select -exp countersamples | select path,cookedvalue
  Expand what's in the countersamples property and take only the 'path' and 'cookedvalue'
• sort CookedValue -desc
  put  the largest value to the top
• sleep 60; clear
  wait a minute, clear the screen and do it again in a while loop

It's not quick, not sophisticated, no bells and whistles around it but it does the job and gives you a sample of the CPU load periodically.

Hope it's useful.
t

Check DNS server IPs on NICs remotely - OS

This will be a quick post. I've come across an issue - while you assume that configuration management on loads of hosts is important and usually managed well, you can never be sure. Let's say I have some DNS servers and I'm replacing them with new ones, hopefully all other servers - in one way or another - will pick up the new DNS IPs and drop the old ones on all of their network interfaces. I just had a bad feeling about this...felt tremor in The Force.

I wanted a quick report on a list of servers and see what DNS IPs they used. At the same time - if I was writing some PowerShell code already why not check if those DNS IPs were alive.

So here is what needed to be done:

Enumerate DNS IPs of all interfaces

This is the easy bit, just use WMI and take all data from all adapters which are IP enabled and have a default gateway defined.
[array]$InterfaceList = gwmi win32_networkadapterconfiguration -ComputerName $srv | ?{$_.ipenabled -ieq "true" -and $_.DefaultIPGateway}

Read DNS IPs from each interface

This will be in the DNSServerSearchOrder property and you can assume it will be an array with as many IPs as many DNS servers the interface has. In my case, I know I have 3 everywhere so I might just hard code it into a loop to look at 3 elements:
for ($i = 1; $i -le 3; $i++){
$script:obj.("DNS_" + $i) = $interface.DNSServerSearchOrder[$i]
}

Verify the given DNS IP to see if it's a live DNS server

This is the tricky bit. Not impossible though.. there are ugly solutions and some uglier ones. Why? The nice native method would be to perform a DNS lookup against the DNS server by using the System.Net.Dns class. The problem is that this class will always use the DNS server set on your local host and you can't specify one like with nslookup.

So an ugly solution would be to run nslookup against the given DNS server and parse its text output. Or you could also change your DNS server on the local host on the fly and use System.Net.Dns. Another one is to see if UDP port 53 is open, it's at least 30 lines of code because as UDP is not a handshaking protocol, you need to setup a System.Net.Sockets.Udpclient, set encoding, initiate connection, handle errors, false positives...etc.
I decided to go with an easier one. By default, all DNS servers have the TCP port 53 open as well for large queries and zone transfers.  That's much easier to test:
$Socket = New-Object System.Net.Sockets.TCPClient
$Connect = $Socket.BeginConnect($srv,53,$null,$null)
$Wait = $Connect.AsyncWaitHandle.WaitOne(3000,$false)

The below script takes a list of hosts from the pipe, enumerates 3 DNS IPs of each interface and checks if those DNS servers are alive and then lists the output in an object collection.

 param ( [string] $log = "",  
  [string] $hosts = "")  
   
   
   
 #### Function for checking if a TCP port is opened on a host  
 function TCPportCheck ([string]$fsrv, [int]$port) {  
  $Socket = New-Object System.Net.Sockets.TCPClient  
  $Connect = $Socket.BeginConnect($fsrv,$port,$null,$null)  
  Start-Sleep 1  
  if($Connect.IsCompleted){  
  $Wait = $Connect.AsyncWaitHandle.WaitOne(3000,$false)    
  if(!$Wait){  
   $Socket.Close()   
   return $false   
  }   
  else{  
   $Socket.EndConnect($Connect)  
   $Socket.Close()  
   return $true  
  }  
  }  
  else{  
  return $false  
  }  
 }  
   
   
 $script:objColl = @()  
   
 #### Collate the host list.  
 $hostlist = @($Input)  
 if ($hosts) {  
  if($hosts -imatch " "){  
  $hostsArr = @($hosts.split(" "))  
  $hostlist += $hostsArr  
  }  
  else{  
  $hostlist += $hosts  
  }  
 }  
   
 foreach ($srv in $hostlist) {  
    
  # read list of interfaces  
  [array]$InterfaceList = gwmi win32_networkadapterconfiguration -ComputerName $srv | ?{$_.ipenabled -ieq "true" -and $_.DefaultIPGateway}  
    
  # go through each interface and read DNS IPS  
  if($InterfaceList.Count -gt 0){  
  foreach ($interface in $interfaceList){  
     
   #then check the DNS accessibility and lookup the DNS server name.  
   $script:obj = "" | select ComputerName,Access,InterfaceName,DNS_1,DNS_1_check,DNS_2,DNS_2_check,DNS_3,DNS_3_check,Result  
   $script:obj.ComputerName = $srv  
   $script:obj.Access = "OK"  
   $script:obj.InterfaceName = $interface.Description  
   $tmpResult = $null  
     
   for ($i = 0; $i -le 2; $i++){  
   $tmpIP = $tmpName = $null  
   $tmpIP = $interface.DNSServerSearchOrder[$i]  
     
   $tmpName = [System.Net.Dns]::GetHostByAddress($tmpIP).HostName  
   $script:obj.("DNS_"+($i+1)) = $tmpIP + " ($tmpName)"  
     
   if((TCPportCheck $tmpName 53)){  
    $script:obj.("DNS_"+($i+1)+"_Check") = "DNS port open"  
   }  
   else{  
    $script:obj.("DNS_"+($i+1)+"_Check") = "DNS port closed"  
    $tmpResult = "One or more DNS servers are not accessible"  
   }  
   }  
   if($tmpResult) {$script:obj.Result = $tmpResult}  
   else  {$script:obj.Result = "OK"}  
   $script:objColl += $script:obj  
  }  
  }  
 }  
   
 $script:objColl  
   

t

Validate PATH environment variable entries - OS

Following on from the post where I wrote about managing the entries in the PATH environment variable to avoid OS and application misbehaviour, I thought I'd share another story on PATH which can also show you why we (IT guys) should never forget the basics and that the simplest idea is the best idea!

I've come across some hosts where the PATH was more than 2048 characters long which made applications stop working. OK, no problem (I thought), I just run my script which removes duplicates from the PATH and it should go well under 2048 characters. I'm not an overconfident guy, but I was surprised when I saw the length still well above 2000. Now what? Then it hit me, why do I take it for granted that all these entries in the PATH are needed? Maybe there are PATH entries which don't even exist on the file system anymore! Simple, isn't it, just get rid of garbage from the environment variable.

Let's run through the PATH entries one by one and check if those folders exist on the box at all - important: if you do this on a cluster node where one of the PATH entries points to one of the clustered drives, you can incorrectly identify that folder as non existent, but in reality the folder exists, it's just on a disk which is active on another node of the cluster at that time, so be careful.

Of course we want to do this remotely:
$srv = "r2d2"
$pathString = [Microsoft.Win32.RegistryKey]::OpenRemoteBaseKey('LocalMachine', $srv).OpenSubKey("SYSTEM\CurrentControlSet\Control\Session Manager\Environment").getvalue("path")

$pathstring.split(";") | %{
$obj = "" | Select Item,Accessible
$obj.Item = $_.trim()
$obj.Accessible = test-path ("\\$srv\" + $obj.item.replace(":","$"))
$obj}

Make it shorter by creating the output object on the fly:
$srv = "r2d2"
$pathString = [Microsoft.Win32.RegistryKey]::OpenRemoteBaseKey('LocalMachine', $srv).OpenSubKey("SYSTEM\CurrentControlSet\Control\Session Manager\Environment").getvalue("path")

 $pathstring.split(";") | %{
New-Object psobject -Property @{test=(test-path $_); item=$_}
} | ft item,test -auto

Or you can do it locally on a host (which makes it simpler and it can even be a oneliner:
(Get-ItemProperty "HKLM:SYSTEM\CurrentControlSet\Control\Session Manager\Environment").path.split(";") | %{new-object psobject -Property @{test=(test-path $_); item=$_}} | ft item,test -auto

This will show you something like:

You can make this a script by adding parameters, get the list of hosts from $input...etc.

t

Validate Domain Controller certificates - AD

This is a specific post about Domain Controller Authentication certificates but the problem and the solution can be applied to any type of certificate you have on your servers.

By default, a domain controller uses LDAP to provide your clients data from Active Directory (TCP port 389).  For example when a client wants to check if a user is member of a group, everything goes through the network in clear text.
If you want to provide LDAP over SSL in your domain to make the LDAP traffic secured, you need to have a so called Domain Controller Authentication certificate (which is in fact a template that describes a certificate for Client and Server authentication plus smart card logon) added to the DCs personal certificate container and taaadaaam, LDAPS will be available (TCP port 636), you should see on your DC something like this:

To make sure the certificate is always valid and does not expire, you can setup auto enrolment via GPO if you have a nice AD integrated PKI infrastructure. However, auto enrolment can sometimes fail if for example someone messes up the permissions on the CA server or folder permissions on domain controllers and if that's done at the wrong time, your DC certificate can expire and bang, there's your outage on a Sunday afternoon when some applications stop working because they can't access AD via LDAPS.

The best solution is to put some monitoring in place, e.g. via SCOM or anything similar which checks certificates periodically and if they are about to expire, sends an alert.

However, if you just want to query your DCs to see how those certificates are at a point in time or you want periodic report on them, it's easier to simply write a couple of lines in PowerShell.

Enumerate certificates on remote hosts

It's easy to get a list of certificates from a remote host:
$srv = "c3podc1"
$certStore = New-Object System.Security.Cryptography.X509Certificates.X509Store("\\$srv\My", "LocalMachine")
$certStore.Open("ReadOnly")
$certStore.certificates


But this does not give you all the fields you want to read, e.g. you may have many certs installed on your DC but you only want to know about the Domain Controller Authentication one, so you need to somehow enumerate the Cert Template names as well (see screen shot above), here is the trick to list all certificates' template names:
$certStore.certificates | %{($_.extensions | ?{$_.oid.friendlyname -match "template"}).format(0) -replace "(.+)?=(.+)\((.+)?", '$2'}

Basically, you need to go through each certificate's 'extensions' and see if the 'oid.friedlyname' contain template, if it is, then use the format method of the X509Extension object to get the name of the template. You will get it with a lot of junk, like this:

Template=Domain Controller Authentication(1.3.6.1.4.1.311.21.8.13987996.9101750.1067918.14758690.631985.210.1.28), Major Version Number=110, Minor Version Number=0

You can use the -replace operator to pick out the string which comes after the first '=':

List Domain Controller Authentication certificates

Now we can list all certificates, we can even pick up the one with Domain Controller Authentication template, we just need to read the date when it expires and then mark it with some RAG (red /amber / green) status based on how close it is to be expired -for me I mark it RED if it is to expire within 30 days because based on my cert template auto enrolment should renew the cert in the last 6 weeks:

Here is the simplified script (you can add function to send mails, log actions...etc., based on some of the previous posts in this blog):

 $hostlist = @($Input)  
   
 foreach($srv in $hostlist){  
    $certStore = New-Object System.Security.Cryptography.X509Certificates.X509Store("\\$srv\My", "LocalMachine")  
    $certStore.Open("ReadOnly")  
    $certStore.certificates | %{  
       $obj = "" | Select Subject,Template,ValidUntil,RAG  
       $obj.Subject = ($_.extensions | ?{$_.oid.friendlyname -match "Subject Alternative Name"}).format(0) -replace "^.+=", ""  
       $obj.Template = ($_.extensions | ?{$_.oid.friendlyname -match "template"}).format(0) -replace "(.+)?=(.+)\((.+)?", '$2'  
       $obj.ValidUntil = $_.NotAfter  
   
       if($obj.Template -ieq "Domain Controller Authentication"){  
          if((get-date($obj.ValidUntil)) -gt (Get-Date).adddays(30)){  
             $obj.RAG = "GREEN"  
          }  
          else{  
             $obj.RAG = "RED"  
          }  
          $obj     
       }     
    }  
 }  
   

t

List SCCM servers in your infrastructure - SCCM

When you have a large infrastructure, you have lots of moving parts, so you might not want to keep track of your servers on  PostIt or in an xls file. Why? It's just overhead and someone always forgets to update it when pulling a server out or installing a new one.

Let's make the inventory dynamic and base it on the infrastructure component itself... meaning, if you have e.g. an SCCM infrastructure with many sites, many servers with different roles (Management points, Site Servers, Fallback status points...etc.) try to gather it from the SCCM infrastructure itself, instead of having a manual list in an xls file somewhere.
Happened one day that I wanted to get the list of SCCM servers in all the sites (sub-sites) with all their roles (SCCM 2007 and SCCM 2012 as well. If you start browsing in the WMI root of SCCM on the site server and look through some of the tables in the database, you will the way eventually, here is the SQL query which lists all SCCM servers and their roles (with some garbage):
Select distinct SiteSystem, SiteCode, Role FROM dbo.Summarizer_SiteSystem ORDER BY SiteCode

To get rid of the garbage and only see the server names, let's do a regex match and some replacing:
[string]$tmpstr = [regex]::Match($sccmSrvObj.sitesystem, \\\\(\w|\.)+\\$)
$sccmServerName = $tmpstr.replace("\", "")

If you want the roles listed as well:
$sccmQueryResult | ?{$_.SiteSystem -imatch $sccmServerName}| %{$sObject.Role += $_.role + ","}

If you have SCCM 2012, be aware that the database naming changed from SMS_sitecode to CM_sitecode, however they also made the DB name available via WMI:
$sccmDB = (gwmi -ComputerName $sccmsrv -Namespace ROOT\SMS\site_$site -Class SMS_SCI_SiteDefinition).SQLDatabaseName

Here is an example script which takes an SCCM server name and a Site code and lists all servers from that site (and sub-sites) with their SCCM roles - as always, you can add logging, error handling...etc.
If you want to make it more sophisticated, you could just get an Active Directory domain name, read the list of SCCM sites registered and then list all SCCM servers... I may put it into another article...

 

 param(     [string] $sccmSrv = "",  
           [string] $site = "")  
   
   
 #### Function for executing a SQL query with integrated authentication    
 function execSQLQuery ([string]$fSQLServer, [string]$db, [string]$query){    
      $objConnection = New-Object System.Data.SqlClient.SqlConnection    
      $objConnection.ConnectionString = "Server = $fSQLServer; Database = $db; trusted_connection=true;"    
      $SqlCmd = New-Object System.Data.SqlClient.SqlCommand $query, $objConnection    
      trap {Write-Host -ForegroundColor 'red' "($sqlsrv/$db not accessible)";continue}    
      $SqlCmd.Connection.Open()    
   
      if ($SqlCmd.Connection.State -ine 'Open') {    
           $SqlCmd.Connection.Close()    
           return    
      }    
      $dr = $SqlCmd.ExecuteReader()    
   
      #get the data    
      $dt = new-object "System.Data.DataTable"    
      $dt.Load($dr)    
      $SqlCmd.Connection.Close()    
      $dr.Close()    
      $dr.Dispose()    
      $objConnection.Close()    
      return $dt    
 }    
   
 $objCollection = @()  
 $sccmCompquery = gwmi -q "Select distinct SiteSystem, Role, SiteCode FROM SMS_SiteSystemSummarizer where role = 'SMS SQL Server' and siteCode = '$site' ORDER BY SiteCode" -namespace "root\sms\site_$site" -ComputerName $sccmSrv  
   
 [string]$tmpstr = [regex]::Match($sccmCompquery.sitesystem, "\\\\(\w|\.)+\\$")  
 $sccmSQLServer = $tmpstr.replace("\", "")  
 $sccmDB = (gwmi -ComputerName $sccmsrv -Namespace ROOT\SMS\site_$site -Class SMS_SCI_SiteDefinition).SQLDatabaseName  
   
 if(!$sccmDB){  
      $sccmDB = "SMS_" + $site  
 }  
   
 $sccmCompquery = "Select distinct SiteSystem, SiteCode, Role FROM dbo.Summarizer_SiteSystem ORDER BY SiteCode"  
 $sccmQueryResult = execSQLQuery.NET $sccmSQLServer $sccmDB $sccmCompquery  
   
 foreach($sccmSrvObj in $sccmQueryResult){  
      [string]$tmpstr = [regex]::Match($sccmSrvObj.sitesystem, "\\\\(\w|\.)+\\$")  
      $sccmServerName = $tmpstr.replace("\", "")  
   
      # if we haven't recorded the given server  
      if(!($objCollection | ?{$_.SiteServer -ieq $sccmServerName})){  
   
           $sObject = new-Object -typename System.Object  
           $sObject | add-Member -memberType noteProperty -name SiteServer -Value $sccmServerName  
           $sObject | add-Member -memberType noteProperty -name SiteCode -Value $sccmSrvObj.SiteCode  
           $sObject | add-Member -memberType noteProperty -name Role -Value ""  
   
           # go through the rows in the query result and pick up each row where our server is listed  
           $sccmQueryResult | ?{$_.SiteSystem -imatch $sccmServerName}| %{$sObject.Role += $_.role + ","}  
           $sObject.Role = $sObject.Role -replace ",$",""  
        
           $objCollection += $sObject  
      }  
 }  
   
 $objCollection  
   

t

Parse OU location from DistinguishedName - AD

This post is just a bit of breadcrumb of Powershell bits. I've got some scripts which run regularly and have to analyse 100 000+ AD objects. It can take hours to run them, so every bit of code that can make one iteration in the loop a couple of milliseconds quicker can pay significant dividends when running against many objects.

As I was looking through my 3 years old code, I noticed an ugly solution (we all do these things, don't we). I needed to get the OU location of each object, so I decided to take the DistinguishedName attribute and drop the name of the object from the beginning of string therefore I end up with the full LDAP formatted path of the object (could have taken the CanonicalName attibute in reverse order and replace '\' with 'cn=' or 'dn=' or 'ou=', but then I would have to lookup each of those elements to figure if they are OUs or containers...etc.)

Let's take an example, the dinstinguishedName of an object is "CN=DroidServer,OU=ChalmunsCantina,OU=MosEisley,DC=tatooine,DC=com", so the LDAP path of the object can be determined by dropping the first part of this string before the first comma which leaves us with: "OU=ChalmunsCantina,OU=MosEisley,DC=tatooine,DC=com".

First attempt - original code in my script

Easy, lets split the string based on commas, put the elements into an array and drop the first element, then join the elements into a string again (now without the cn=objectname piece):

 $distinguishedName = "CN=DroidServer,OU=ChalmunsCantina,OU=MosEisley,DC=tatooine,DC=com"  
 $arrDN = New-Object System.Collections.ArrayList  
 $tmparr = $distinguishedName.Split(",")  
 $tmparr | %{[void]$arrDN.add($_)}  
 $arrDN.RemoveAt(0)  
 $accLocation = [string]::join(",",$arrDN)  
 $accLocation  

This will take 96.5 milliseconds on my machine.

96 milliseconds, fair enough, it's quicker than me doing this on paper.

Second attempt

Let's get rid of the foreach-object (%) when adding elements to $tmpArr and use the .AddRange method of the ArrayList instead - this will just add all elements in one go instead of going through element by element:

 $distinguishedName = "CN=DroidServer,OU=ChalmunsCantina,OU=MosEisley,DC=tatooine,DC=com"  
 $arrDN = New-Object System.Collections.ArrayList  
 $tmparr = $distinguishedName.Split(",")  
 [void]$arrDN.addrange($tmparr)  
 $arrDN.RemoveAt(0)  
 $accLocation = [string]::join(",",$arrDN)  
 $accLocation  

25 milliseconds, not bad, 4 times quicker.

 

Third attempt

To see if it can be even quicker, we'll need to "thinking outside the box" and see if there's any simpler solution than working with arrays and instead do this in one step and drop the first bit of the string which we don't need.
It's not obvious in PowerShell because the -replace operator does not support the regular expressions which refer only to the first occurrence in a string. What we can do is make it drop all characters which are not commas and they are followed by a comma, that would make sure the "cn=computername," string is dropped and we end up with the full LDAP path of the object:

 $distinguishedName = "CN=DroidServer,OU=ChalmunsCantina,OU=MosEisley,DC=tatooine,DC=com"  
 $accLocation = $distinguishedName -creplace "^[^,]*,",""  
 $accLocation  

Explanation for the regex pattern:

• ^       start of the string
• [^,]*   match one or more non-comma characters
• ,       match a comma character

 

0.4669 milliseconds!

200 times quicker than the first solution! With 100 000 objects, originally it takes 160 minutes (obviously in real life it will be less because of caching...etc.) and with the 3rd solution it should take a bit less than a minute. Maybe it can be quicker with some better trick, but I'm not greedy, I've shaved off ~2.5 hours runtime, it's good enough for me... for today...

t