Split Array into smaller arrays - Scripting

Have you had a case when you had a long list of things and you wanted to split it up? Like a long shopping list which you really didn't want to tackle alone. Or you have a long list of IP addresses or host names, or object in Active Directory...whatever. And you want to split the long list to many smaller lists to run things in parallel or to stagger the running of a script (make sure you only run a script on a smaller batch at a time instead of running it on the full list).

So let's split a long list of host names to 6 pieces. You can do this in many ways, but I use this oneliner: 

$NumberOfArrays = 6; $NewArrays = @{}; $i = 0;  gc c:\hostlist.txt | %{$NewArrays[$i % $NumberOfArrays] += @($_); $i++}; $NewArrays

It uses a few tricks.

The first one is the modulus operator (%) which spits out the reminder part when you divide two integers. That will mean if you do a mod operation on a running number (from 1 to the max count of the list) with the number you want to split the list to, it will repeat a sequence of numbers. From 0 until it reaches the number one less than the number we are dividing by. Like this:

Lift of index numbers we can use for splitting the list

The second trick is that this generated repeating sequence of numbers are used as index (or keys) of a hash table. Each key in the hash table has a value, in this case the values of keys will the split list of hosts. The hash table here is $NewArrays which has the arrays of the smaller lists:

The list of smaller lists (arrays) in a hash table

Note the 'Name' column which is the list of keys.

To pull each list one by one, you can address each key with the .Item parameterized property.



Add caption

Or like this:

0..5 | %{$NewArrays.item($_)}


t


Random password generator - Scripting

These days everything in shops are handcrafted. It's fashionable to buy handcrafted yogurt, honey, jam...beer... why not have your very own homemade password generator?!

It's actually useful to put into scripts which create user accounts in e.g. Active Directory. You can have the given user phone the support team to reset the password when they want to start using the account - so no one knows the password until then.

You really want long (15 characters long) complex passwords because you never know how long a newly created account will sit around waiting for the user to reset the password of it. You can read many books about it why complex and random passwords are needed.

I tell you the whole code upfront and then explain the bits, like in every Columbo episode, they show you the buildup and the murder and then Columbo solves the puzzle in front of your eyes.... Peter Falk was awesome in that character!

The random password generator itself:
[string]::join("",((48..57) + (65..90) + (97..122) | Get-Random -count 15 | %{[char]$_}))

The bits of the oneliner

Passwords need characters, the easiest way to generate some is from the ASCII table. In Powershell you can generate a list of numbers on the fly when you define an array, these can be the ASCII codes of characters:

• ASCII codes for all lowercase letters: (97..122)
• ASCII codes for all uppercase letters: (65..90)
• ASCII codes for all numbers from 0 to 9: (48..57)

You can see there are gaps in the list of ASCII codes of lowercase, uppercase letters and numbers. To make these 3 arrays of numbers look like one array, just add them up together:

(48..57) + (65..90) + (97..122)

Array of ASCII numbers for random password

Nice, we have a list of ASCII codes for all characters we want to chose from, let's pick random ones, 15 of them:

Get-Random -count 15

Then pipe these through to a foreach loop and convert the numbers to characters:
%{[char]$_}

Taking 15 random numbers of the ASCII arrays and converting to characters

Good, we have a random list of characters in an array, but I need it in one long string so I can use it as a real password. For this purpose, we can use the join function of the [string] type with no delimiter:
[string]::join("",...

There you go, you now have your handcrafted random password generator.

Object output for Powershell scripts - Scripting

Through previous articles I kept writing about object output and how good that is for further processing in Powershell. What is an object... well... I'm not a developer, so I won't be able to explain it with nice sophisticated phrases. The object is a thing :)

For me the easiest way to imagine an object as a row of a table. The table has several columns which will be the object's properties and you can put multiple objects into an object collection or array, that will form the table.



Listing a custom object collection

 

Why is this a good thing? First of all, if you want to store more than one type of data about a computer within your script, you can create those properties for your object and just add the data into them as you can see on the picture above. I stored information about 3 servers such as whether the servers are pingable, accessible and put some different test results as well.
Also, you can use filtering on your object collection, e.g. if I want to list rows (objects) where the 'Ping' column (property) is Error:
$objColl | where{$_.ping -eq "Error"}

Filtering a custom object collection

There are multiple ways to create an object.

• System.Object with Add-Member - it is slower than other methods, however, you can specify the order of properties, you can add values to the properties when creating them:
  $myObject = new-Object -typename System.Object
  $myObject | add-Member -memberType noteProperty -name ComputerName -Value $srvname
  $myObject | add-Member -memberType noteProperty -name Ping -Value "N/A"
  $myObject | add-Member -memberType noteProperty -name Accessible -Value "N/A"
  $myObject | add-Member -memberType noteProperty -name Test1_Result -Value "N/A"
  $myObject | add-Member -memberType noteProperty -name Test2_Result -Value "N/A"
  
  
• PSobject with Select - it is quicker than add-member and you can specify the order of properties, however you cannot add values to the properties when creating them and you assume all properties will have string type. It's not a big deal but not very elegant:
  $myObject = "" | Select ComputerName,Ping,Accessible,Test1_Result,Test2_Result


• You can store property names in an array and pass it to the Select-Object filter:
  $arrProperties = @("ComputerName","Ping","Accessible","Test1_Result","Test2_Result")
  $myObject = "" | Select $arrProperties
  
  
• PSObject defined via hash table - it's quick, you can add values to the properties in one go, however, as with hash tables in general, you cannot define the order of the properties, so it can be odd when you see the result and the columns are in random order
  $myObject = New-Object PSObject -Property @{ComputerName = $srvname; Ping = "OK"; Accessible = "N/A"; Test1_Result = "N/A"; Test2_Result = "N/A"}

The object is created, as you can see above, depending on how you create it, you can add values to properties on the fly or when the object is ready and can access the properties easily just referring to them with a dot:

More than one object

When you have more than one object, you can add them into an array and form a 'table':
$objColl = @()
$objColl += $myObject

Then you can modify the objects in the collection if you want to, e.g. if I want to make sure the Test1_Result column contains only a string of aaaa, I can simply do this:

There's one more good trick with object collections. I usually write scripts for large number of objects, e.g. check out 5000 servers, get data of 100 000 AD objects...etc. Therefore the object collection is big and non-indexed. Even though it's in the memory during runtime, it takes a while to search and find objects. One of my colleague had a good idea and I've been using it since then: put the objects into a hash table with the key that you want to search for. E.g. if you know that you want to search for the ComputerName and you want to read a property of that object from your collection you can make the ComputerName the hash key and the object itself the hash value:
$objCollHash = @{}
$objColl | %{$objCollHash.add($_.ComputerName, $_)}

Now we can test how much time it takes to find an object in a collection of 9999 objects with 3 attributes, first with normal filter:

900+ millisecond, not bad, so less than a second to find an object from 9999.

Ok, let's see the hash table:

Ughm... less than a millisecond! 900+ times quicker than the previous one. I guess it's worth the effort to put those 2 lines into the script.

That's all for today. Hope this helps people to understand why objects and pipe'ing is useful.

t

Useful Script functions - Scripting

As you can see in previous posts, I publish code snippets in the articles which of course are not my full scripts. There are good reasons for that, e.g. if I publish full scripts which are usually 500 - 2000 lines long, the main point of the article gets lost. The proper scripts contain much more "frippery" like parameter evaluation, logging, remote host checks...etc.

These things are quite important when you design a script for scalability and to run in a large environment, i.e. if you don't perform some sort of accessibility check (with well thought timeout interval) against the 1000+ servers your script works on, that script will run forever. OK, I'm exaggerating, but it feels like forever when people are nagging you on the phone to get results during an outage and it takes 10 mins to get them instead of 3 because the script waits for the default WMI interval for each and every query on offline hosts instead of just timing out once (and quickly) when it detects that the remote host is not accessible... and perspiration sprang from your brow.

Now, here are some important functions you want to have in your script - of course there can be many different ways of implementation for these, but you get the idea...:

Ping remote hosts:
#### Function for checking if a host is alive on the network
function PingServer ([string]$srv){
   $ping = new-object System.Net.Networkinformation.Ping
   Trap {Continue} $pingresult = $ping.send($srv, 3000, [byte[]][char[]]"z"*16)
   if($pingresult.Status -ieq "Success"){$true} else {$false}
}

Check if you can access remote hosts (e.g. RPC is not dead and check if you have admin rights). I usually check WMI. Why WMI? Because most of the remote actions you want to perform will use WMI, or something that builds on WMI. If WMI is broken, you might as well skip that host from the list and mark it as unhealthy. Another reason is that remote WMI is a good indicator for checking administrator access or if RPC is alive on the box.

However, do not use the native Get-WMIObject cmdlet, because it does not have a timeout value. I've seen Windows 2003 servers almost every week in the past which were in a sort of "half-hung" state. It is usually an indication of a paged or non-paged pool leak of a driver when the box is going downhill but is not quite there yet, so you can still RDP to it, but any remote query (registry, WMI, SMB...etc) will be taking hours to run or time out. Literally hours... so you need pre-defined timeout:
#### Function for checking WMI health
function checkWMI ([string]$srv){
   $checkwmi = $null
   $timeout = new-timespan -seconds 15
   $Scope = new-object System.Management.ManagementScope "\\$srv\root\cimv2", $options
   $Scope.Connect()
   $query = new-object System.Management.ObjectQuery "SELECT * FROM Win32_OperatingSystem"
   $searcher = new-object System.Management.ManagementObjectSearcher $scope,$query
   $SearchOption = $searcher.get_options()
   $SearchOption.set_timeout($timeout)
   $searcher.set_options($SearchOption)
   $checkwmi = $searcher.get()
   $lastBoot = $checkwmi | %{$_.lastbootuptime}

   if($lastBoot){
      return $true
   }
   else{
      return $false
   }
}


Logging the actions of your script. In Powershell you either write to the standard output (e.g. Write-Host) or into a file (e.g. Out-File). But what if you want to do both at the same time?  And you want to do color coding of different messages on the screen? And you want to have time stamps for each message? Here is one option:

$script:logfile = New-Item -type file "c:\temp\Test_logfile.log" -force

#### Function for creating log entries in a logfile and on standard output
function writeLog ([int]$type, [string]$message, [string]$modifier) {
   # usage: writeLog 0 "info or error message"
   # $modifier: <nonewline | extendline>
   # Value nonewline: writes the output the the console and the logfile without carriage return
   # Value extendline: writes the message to the output without date and
   # both values can be used e.g. for writing to the console and logfile and put a status message at the end of the line as a second step

   $date = get-date -uformat "%Y/%m/%d %H:%M:%S"
   if($modifier -eq "extendline"){
      switch ($type) {
         "0" {$color = "Green"}
         "1" {$color = "Yellow"}
         "2" {$color = "Red"}
      }
   }
   else{
      switch ($type) {
         "0" {$message = $date + ", INF, " + $message; $color = "Green"}
         "1" {$message = $date + ", WAR, " + $message; $color = "Yellow"}
         "2" {$message = $date + ", ERR, " + $message; $color = "Red"}
      }
   }

   if($modifier -eq "nonewline"){
      write-host $message -ForegroundColor $color -NoNewLine
      # need to encode with bytes to be able to drop the carriage return character in the output file
      $bytes = [text.encoding]::ascii.GetBytes($message)
      $bytes | add-content $script:logfile -enc byte
   }
   else{
      write-host $message -ForegroundColor $color
      Add-Content $script:logfile $message
   }
}

If you create log files, you want to maintain those so they don't get out of control and consume too much disk space. An option is to rotate them based on their date:
#### Function for deleting logfiles older than X days
function logRotate ([string]$logfolder, [string]$strScriptname){
   if($strScriptname){
      $lastWriteLimit = [DateTime]::Now.AddDays(-7)

      if($logfolder){
         $tobedeleted = Get-ChildItem "$logfolder" -force -filter "$strScriptname*" | where{(!$_.PSIsContainer) -and ($_.LastWriteTime -le "$lastWriteLimit")}

         if($tobedeleted){
            foreach($delLogfile in $tobedeleted){
               $strDelLogfile = $dellogfile.name + " - " + $dellogfile.lastWriteTime
               writelog 1 "Old log to be deleted: $strDelLogfile"
               $strDellogfile = ""
            }            Get-ChildItem "$logfolder" -force -filter "$strScriptname*" | where{(!$_.PSIsContainer) -and ($_.LastWriteTime -le "$lastWriteLimit")} | Remove-Item -Force
         }
      }
   }
}

Obviously, these are just my way of performing the checkouts and doing logging, but the principal is there: always perform quick checks if you handle large number of hosts, not just because you want to timeout on hosts quickly but because you want to have a proper output and useful info on all hosts, even if it says: "not pingable".

Clipboard friendly code:

 #### Function for checking if a host is alive on the network  
 function PingServer ([string]$srv){  
      $ping = new-object System.Net.Networkinformation.Ping  
      Trap {Continue} $pingresult = $ping.send($srv, 3000, [byte[]][char[]]"z"*16)  
      if($pingresult.Status -ieq "Success"){$true} else {$false}  
 }  
   

 #### Function for checking WMI health  
 function checkWMI ([string]$srv){  
      $checkwmi = $null  
      $timeout = new-timespan -seconds 15  
      $Scope = new-object System.Management.ManagementScope "\\$srv\root\cimv2", $options  
      $Scope.Connect()  
      $query = new-object System.Management.ObjectQuery "SELECT * FROM Win32_OperatingSystem"  
      $searcher = new-object System.Management.ManagementObjectSearcher $scope,$query  
      $SearchOption = $searcher.get_options()  
      $SearchOption.set_timeout($timeout)  
      $searcher.set_options($SearchOption)  
      $checkwmi = $searcher.get()  
      $lastBoot = $checkwmi | %{$_.lastbootuptime}  
        
      if($lastBoot){  
           return $true  
      }  
      else{  
           return $false  
      }  
 }  
   

 $script:logfile = New-Item -type file "c:\temp\Test_logfile.log" -force  
   
 #### Function for creating log entries in a logfile and on standard output  
 function writeLog ([int]$type, [string]$message, [string]$modifier) {   
      #usage: writeLog 0 "info or error message"  
      # $modifier: <nonewline | extendline>  
      # Value nonewline: writes the output the the console and the logfile without carriage return  
      # Value extendline: writes the message to the output without date and   
      # both values can be used e.g. for writing to the console and logfile and put a status message at the end of the line as a second step  
   
      $date = get-date -uformat "%Y/%m/%d %H:%M:%S"  
      if($modifier -eq "extendline"){  
           switch ($type) {  
                "0"     {$color = "Green"}  
                "1"     {$color = "Yellow"}  
                "2"     {$color = "Red"}  
           }  
      }  
      else{  
           switch ($type) {  
                "0"     {$message = $date + ", INF, " + $message; $color = "Green"}  
                "1"     {$message = $date + ", WAR, " + $message; $color = "Yellow"}  
                "2"     {$message = $date + ", ERR, " + $message; $color = "Red"}  
           }  
      }  
        
      if($modifier -eq "nonewline"){  
           write-host $message -ForegroundColor $color -NoNewLine  
           $bytes = [text.encoding]::ascii.GetBytes($message)  
           $bytes | add-content $script:logfile -enc byte  
      }  
      else{  
           write-host $message -ForegroundColor $color  
           Add-Content $script:logfile $message  
      }  
 }  
   

 #### Function for deleting logfiles older than X days  
 function logRotate ([string]$logfolder, [string]$strScriptname){  
      if($strScriptname){  
           $lastWriteLimit = [DateTime]::Now.AddDays(-7)  
           if($logfolder){  
                $tobedeleted = Get-ChildItem "$logfolder" -force -filter "$strScriptname*" | where{(!$_.PSIsContainer) -and ($_.LastWriteTime -le "$lastWriteLimit")}   
                if($tobedeleted){  
                     foreach($delLogfile in $tobedeleted){  
                          $strDelLogfile = $dellogfile.name + " - " + $dellogfile.lastWriteTime  
                          writelog 1 "Old log to be deleted: $strDelLogfile"  
                          $strDellogfile = ""  
                     }  
                     Get-ChildItem "$logfolder" -force -filter "$strScriptname*" | where{(!$_.PSIsContainer) -and ($_.LastWriteTime -le "$lastWriteLimit")} | Remove-Item -Force  
                }  
           }       
      }  
 }  
   

May The Force...
t