Faster filtering in PowerShell than using Where-Object

In a project I'm using a PowerShell script to read in a lot of .csv files and then do some lookups between these in order to get the wanted output. This all works well, but has been a bit slow lately due to a lot more data in the files.

I narrowed the speed issue down to a few lines where I iterate over some files in order to find a few rows that I need. Basically, the Where-Object cmdlet is the culprit.

A simplified example is below:

$mycsvfile = Import-Csv .\mydata.csv
$dataiwant = $mycsvfile | Where-Object {$_.idno -eq 5}

I had a few similar lines in the script, making the time to select the data add upp to 9 seconds, which was far to long in this case.

I found this post that show a few variants on filtering collections, and by simply using the PS v4 .Where() notation to do the same thing, I could bring this down to a single second.

$mycsvfile = Import-Csv .\mydata.csv
$dataiwant = $mycsvfile.Where({$_.idno -eq 5})

So lesson learned: PowerShell is evolving quickly and what I thought was a nice way to do something might very well be just fine, but there might also be a quicker way just around the corner. In this example, I'm sacrificing the streaming capabilities, but gain a lot of performance, just by changing a few characters.

XSD in Visual Studio and the warning message "Request for the permission of type 'System.Security.Permissions.FileIOPermission"

When working in a project in Visual Studio that had a schema repository added, the warning message "Request for the permission of type 'System.Security.Permissions.FileIOPermission" showed up in various places where I had schema imports. The warning did not show up everywhere though and the project built just fine.

It turned out that the reason was that some of the xsd files where downloaded or copied from a network share and were therefore blocked in Windows, leading up to the warning message.

The warnings could be removed by simply unblocking the offending files, and restarting Visual Studio.

This also lead me to the quick way to unblock a lot of files at once. Simply use PowerShell to iterate over the files and use the Unblock-File cmdlet:
gci -r | Unblock-File

Slow performance in PowerShell Get-ChildItem over UNC paths

I got a ticket submitted to me that proved to be quite interesting. The basics where that a specific maintenance script were running for ages on a server causing some issues.

The script itself was quite simple, it did a recursive gci on a path and performed an action on each item matching a where clause that identified empty folders. Nothing wrong there. However, even if the specified path contained a massive amount of folders and files, it still shouldn't have been running for days (literally) causing the ticket to be created.

When looking into the matter, I found this blog post on the Windows PowerShell blog that goes into detail why Get-ChildItem has slow performance at times. The main culprit is the .NET APIs that is too chatty and causes a lot of overhead traffic over the network when trying to query for files. This was fixed in PowerShell 3.0 that uses new API:s.

A quick test using a folder with 700 subfolders and a total of 40 000 files where I execute the script line

(gci d:\temp -Recurse | where-object {$_.PSIsContainer}) | Where-object {($_.GetFiles().Count + $_.GetDirectories().Count) -eq 0} | out-file d:\dir.txt

reveals the following execution time numbers:

Using PowerShell 2.0 and accessing the path as an UNC: 100s
Using PowerShell 3.0 and accessing the path as an UNC: 33s
Using PowerShell 2.0 and accessing the path locally: 6s
Using PowerShell 3.0 and accessing the path locally: 5s

In my case, the server was both running PowerShell 2.0 and accessing the path as an UNC causing a major performance hit. The solution is simply to both run the script locally on the server where the task has to be performed as well as upgrading to at least PowerShell version 3.0. As can be seen from my quick test, the best performance gain is clearly to run the script locally giving about 17 times better performance.

While no rocket science that accessing tons of files locally has to be faster then doing it over the network, it is still fairly common to see scripts executed on application servers and performing tasks on file shares.

This also proves that it is important in our business to stay on top of new versions of tools and frameworks in order to catch these types of improvements. I can be certain that there is quite a number of people out there bashing gci heavily for it's slow performance when in fact it has improved a lot between PowerShell 2.0 and 3.0.

The difference between single and double quotes in PowerShell

I had a colleague ask me the other day about whether there was a difference between using single or double quotes when defining a string variable in PowerShell. For that specific questions, the answer was "no". However, there is an important distinction to make between the two.

With double quotes, the text within the quotes will be parsed. For instance, the following code will output "Hello Marcus":

$name = 'Marcus'
$output = "Hello $name"
write-host $output

However, when changing the second line to using single quotes as so:

$output = 'Hello $name'

The output will instead be "Hello $name"

The parsers handling of the different quotes also means that if you are to use escape characters, for instance `n, you have to put the string within double quotes for the escape characters to work. If you want to use a double quote within such a string, you can write it using the escape character:

$mystring = "This is a variable called `"mystring`""

Having the possibility of using the two different quotes can also allow for creating strings that contain one of the two quote characters, for instance when building query strings:

$name = 'Marcus'
$query = "SELECT Id, City from Customers WHERE Name LIKE '%$name%'"

Knowing this difference between single and double quotes is quite crucial for being able to build PowerShell scripts and not get into a mess when trying to create dynamic execution of other scripts or programs that require parameters to be passed to them.

Script BizTalk Application export to .msi with options using PowerShell

I've had the need to export BizTalk Applications using scripts, but with the requirement to not export web directories or global parties to the .msi file.

This can be done using the GUI:

However, it cannot be done (that I know of) using the btstask command with the exportapp parameter.

I opted for creating a PowerShell script that more or less extends the btstask exportapp command. The script takes a few parameters making it a bit more versatile:

$appName is the name of the BizTalk application to export
$exportPath is the path to write the .msi file to
$exportWebDir is a boolean indicating whether to also export web directories or not
$removeDefaultBinding is a boolean controlling whether the default bindings will be written to the .msi file or not. This is handy to remove in those cases you want to just keep bindings for dev/test/production environments.
$exportGlobalParties is a boolean controlling whether to include the Global Parties in the binding file or not. These will of course not be written if for instance you only have the default bindings and set the parameter $removeDefaultBinding to false and $exportGlobalParties to true, since the binding file never will be written at all.

The script basically dumps a ResourceSpecification file which then gets modified depending on the parameters entered. This ResourceSpecification is then used as input to the btstask exportapp command creating an .msi file according to our needs.

The full script looks like this:

param($appName, $exportPath, $exportWebDir, $removeDefaultBinding, $exportGlobalParties)

Write-Output "Exporting ResourceSpec..."
BTSTask ListApp /ApplicationName:$appName /ResourceSpec:$pwd\ResourceSpecTemp.xml

If (!($?))
{
    throw "Could not export resource specification. Verify application name."
}

Write-Output "Reading ResourceSpec..."
$xmlResource = [xml] (Get-Content $pwd\ResourceSpecTemp.xml)

If ($exportWebDir -eq $false)
{
    Write-Output "Removing web directories..."
    $delnodes = $xmlResource.SelectNodes("/*[local-name()='ResourceSpec' and namespace-uri()='http://schemas.microsoft.com/BizTalk/ApplicationDeployment/ResourceSpec/2004/12']/*[local-name()='Resources' and namespace-uri()='http://schemas.microsoft.com/BizTalk/ApplicationDeployment/ResourceSpec/2004/12']/*[local-name()='Resource' and namespace-uri()='http://schemas.microsoft.com/BizTalk/ApplicationDeployment/ResourceSpec/2004/12'][@Type='System.BizTalk:WebDirectory']")

    ForEach($delnode in $delnodes)
    {
        [void]$xmlResource.ResourceSpec.Resources.RemoveChild($delnode)
    }
}

If ($removeDefaultBinding -eq $true)
{
    Write-Output "Removing Default binding info..."
    $delnodes = $xmlResource.SelectNodes("/*[local-name()='ResourceSpec' and namespace-uri()='http://schemas.microsoft.com/BizTalk/ApplicationDeployment/ResourceSpec/2004/12']/*[local-name()='Resources' and namespace-uri()='http://schemas.microsoft.com/BizTalk/ApplicationDeployment/ResourceSpec/2004/12']/*[local-name()='Resource' and namespace-uri()='http://schemas.microsoft.com/BizTalk/ApplicationDeployment/ResourceSpec/2004/12'][@Type='System.BizTalk:BizTalkBinding'][@Luid='Application/$appName']")

    ForEach($delnode in $delnodes)
    {
        [void]$xmlResource.ResourceSpec.Resources.RemoveChild($delnode)
    }
}

Write-Output "Saving modified ResourceSpec..."
$xmlResource.Save("$pwd\ResourceSpecTemp.xml")

Write-Output "Exporting application $appName..."

If ($exportGlobalParties -eq $true)
{
    $globalPartiesParam = "/G"
}
else
{
    $globalPartiesParam = ""
}

BTSTask ExportApp /ApplicationName:$appName /Package:$exportPath\$appName.msi /ResourceSpec:$pwd\ResourceSpecTemp.xml $globalPartiesParam

If (!($?))
{
    throw "Could not export application. Verify application name and parameters."
}
       
Write-Output "Cleaning up..."
Remove-Item $pwd\ResourceSpecTemp.xml

Write-Output "DONE!"

Exit 0

And will be called like so:

ExportBtsMsi.ps1 BtsDemo1 "C:\" false false false

Which will result in output like this:

"Clever" uninstall of msi packages/applications using PowerShell

When I created my own PowerShell script library for BizTalk deployment automation I ran across the need to uninstall applications, both BizTalk applications and non-BizTalk ones, by only knowing their name.

At first, I solved it using a WMI query like so:

$name = "application name"

$product = Get-WmiObject -Class Win32_Product -Filter "name='$name'" -ComputerName "localhost"

[void]$product.Uninstall()

This is not a recommended way of doing it though. It is both very slow and also causes a bit of spamming in the Windows Eventlog since the query in fact does a reconfigure of ALL applications installed. This reconfiguration can also cause a bit of other issues in some cases.

I then created another way of trying to uninstall applications in a more failsafe and secure way by using msiexec with the uninstall flag. The tricky part was to find a way to get the product key in order to be able to use msiexec since it requires this for uninstalling an application. The result can be found below.

The script function will take the application name as argument. It will then via the registry (note that this is configured for x64, so change the path if you are running x86) look up the application settings. If the application can be found via name (it should), we extract the product key. Then this is used as a parameter to msiexec.

If the product key cannot be found (it happens), we will instead try to read the uninstall string that is set when installing the application. Windows will run this string when you choose to uninstall an application, so why do not we use it? If found, we extract the product key and do our msiexec call.

If all fail, we throw an exception to be caught in the real part of the script.

This is the full script function:

Function Uninstall-Program([string]$name)
{
    $success = $false

    # Read installation information from the registry
    $registryLocation = Get-ChildItem "HKLM:\Software\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall\"
    foreach ($registryItem in $registryLocation)
    {
        # If we get a match on the application name
        if ((Get-itemproperty $registryItem.PSPath).DisplayName -eq $name)
        {
            # Get the product code if possible
            $productCode = (Get-itemproperty $registryItem.PSPath).ProductCode
           
            # If a product code is available, uninstall using it
            if ([string]::IsNullOrEmpty($productCode) -eq $false)
            {
                Write-Host "Uninstalling $name, ProductCode:$code"
           
                $args="/uninstall $code"

                [diagnostics.process]::start("msiexec", $args).WaitForExit()
               
                $success = $true
            }
            # If there is no product code, try to read the uninstall string
            else
            {
                $uninstallString = (Get-itemproperty $registryItem.PSPath).UninstallString
               
                if ([string]::IsNullOrEmpty($uninstallString) -eq $false)
                {
                    # Grab the product key and create an argument string
                    $match = [RegEx]::Match($uninstallString, "{.*?}")
                    $args = "/x $($match.Value) /qb"

                    [diagnostics.process]::start("msiexec", $args).WaitForExit()
                   
                    $success = $true
                }
                else { throw "Unable to uninstall $name" }
            }
        }
    }
   
    if ($success -eq $false)
    { throw "Unable to find application $name" }
}

Using PowerShell to remove all old files in a folder structure

A common task when working with BizTalk installations is to clean out old data from folder structures. May it be old log files, automatically saved messages from a send port, database backups or any other sort of data that is obsolete after a certain period of time.

When I started working with BizTalk, the norm was to use bat files or VB Script. These can still be seen in many places since they a) work just fine, and b) the developers/administrators hasn't learned a new way to script these kind of things, mostly because a) they work just fine.

I myself as mentioned earlier prefer to use PowerShell nowadays.

In order to recursivly delete old files and folders, one line of PowerShell script is all that is needed:

get-childitem -path 'c:\temp\test' -recurse | where -FilterScript {$_.LastWriteTime -le [System.DateTime]::Now.AddDays(-30)} | Remove-Item -recurse -force

This script simply traverses the specified folder (why not change it to take the folder and date as a parameter) and does so recursivly. All files older than 30 days in this case is then deleted. By using the -force parameter, locked and hidden files will not cause an error.

In order to have this run automatically on a server, it needs some adjustments. Add a Scheduled Task and set the Program/script to run to Powershell and then add the arguments as so:

-noninteractive -command "& {get-childitem -path 'c:\temp\test' -recurse | where -FilterScript {$_.LastWriteTime -le [System.DateTime]::Now.AddDays(-30)} | Remove-Item -recurse -force}"

Set the security options for the task to "Run whether user is logged on or not" and also check the "Run with highest privileges" checkbox.

Why I created my own PowerShell scripts for BizTalk deployment

Most of us working with BizTalk has at one point or another started scripting our deployments instead of doing them manually. I have during the years seen many variants of automating the deployment process. At one place the "old" way of using bat-files calling BTSTASK where used extensively. At another, a custom .Net program with a GUI was developed that would call the different API:s available. I myself resorted to using PowerShell to create a fully automatic deployment routine.

My scripts has two parts. One with different functions that I wrote and one part that encapsulates the functions and executes them in the correct order. As input I create an XML file that specifies which .msi packages that are to be deployed, which applications to remove/update/add, settings for the IIS virtual directories and so on. Since it is quite common during complicated deployments that something will err, I added functions to test run the scripts in advance. The test run simply checks for dependencies in the BizTalk configuration, making sure that applications can be modified as configured. It also checks that all files needed are provided in the setup package.

When creating the scripts, I briefly looked at using the BizTalk Deployment Framework and other extensions to PowerShell, but opted on developing it all in my own scripts. This due to several reasons, each quite strong in my case:

• I would learn more about PowerShell as well as the API:s and tools found in BizTalk
• I would know exactly where things could go wrong and what to do about it.
• It allowed me a greater flexibility since nothing was hidden in dll files or needed installation. As long as PowerShell was installed on the servers, the scripts would work. Nothing else is needed. This is also something that administrators liked. Some of them are a bit weary about installing "some stuff I found on the net". 

This post is the theoretical background to posts that are to follow that will showcase and explain my scripts in individual blocks. I have already started in advance with the post Recycle IIS application pools using PowerShell, and more snippets are to come.

Script updates of contacts in Outlook

I wanted to have all of my colleagues contact information in my iPhone so I thought of the simplest way to handle it. I quickly decided that the best way would be to simple add the appropriate domain accounts as contacts in Outlook which in a few seconds work gave me about 240 new contacts with the correct information entered as they appear in our Active Directory.

I then realised that they all had the internal speed dial phone number entered as their work number. This is a four digit number which is the same as the last four digits in the externally used number. Hence I needed a simple way to automatically add a prefix to the number for each contact, but only those that had four digits in the field since my address book is filled with external contacts as well.
After fiddling briefly with macros, csv-files and other means to get this to work, I had an epiphany and thought of PowerShell.

Three rows of PowerShell code written, tested and executed in a few minutes solved my problem

$outlook = new-object -com outlook.application
$contacts = $outlook.Session.GetDefaultFolder(10)
$contacts.Items | % { if($_.BusinessTelephoneNumber.Length -eq 4) { $_.BusinessTelephoneNumber = "123-45" + $_.BusinessTelephoneNumber; $_.save() } }

Using PowerShell to automatically update lots and lots of data will most likely be the first I think of the next time a similar task appears. This was almost ridiculously easy.

Convert C# to PowerShell code

Yesterday I saw a link on our company's intranet linking to a colleague's blog. It proved to be a very nice read. Among the .Net Reflector addins available on CodePlex, there's one for PowerShell allowing you to convert C# code to PowerShell. That will come in handy a lot of times since I definately write C# code faster than PowerShell code. Thanks for that tip!
The only downside is that Reflector for the last year requires a license fee and I have since switched over to ILSpy. With this and the other plugins available, it's either time to switch over to Reflector again, or rewrite some of the plugins to work with ILSpy.

Recycle IIS application pools using PowerShell

I have been working on a collection of deployment scripts for BizTalk solutions using PowerShell. One functionality is installation of services hosted in the IIS. After installation, the application pool need a restart in order to properly pick up the new/updated services.

At first, I just recycled all available application pools by the following command line

& $env:windir\system32\inetsrv\appcmd list apppools /xml | & $env:windir\system32\inetsrv\appcmd recycle apppools /in

First we list all available application pools and then we pipe this list as input into the recycle command.

While nice, it is unnecessary to recycle those application pools that are unaffected by our installation.

The main finesse in my scripts are that they are picking up all information on what to do from a configuration file. This XML file includes among other things the application pool for each service to install in IIS. Based on this I updated my recycle command to recycle each and every application pool that were specified in the configuration file. In order to not restart an application pool more than once, I pipe the foreach with sort-object and get-unique.

# List to hold the apppools
$appPools = New-Object System.Collections.ArrayList

# Loop through the objects in the xml file
# and extract the apppool name and add to the list
ForEach($wcfSetup in $xmlFile.DeployConfiguration.WcfSetups.WcfSetup)
{
    [void]$appPools.Add($wcfSetup.ApplicationPool.Trim())
}

# Recycle each unique apppool in the list
foreach($appPool in $appPools | sort-object | get-unique)
{
    & $env:windir\system32\inetsrv\appcmd recycle apppool /apppool.name:"$appPool"
}

Creating a multidimensional strongly typed array in Powershell

When getting stuck for a short while trying to figure out how to create a strongly typed multidimensional array in Powershell I tried to find an example on the net just to find that there simply just is no example of it to be found. Maybe it's too easy? The Technet page on the New-Object cmdlet gave me what I needed.

The way I later on created my two dimensional array was like this:

$d = New-Object 'Object[,]' 10, 20

This is however created as an Object array while I needed int. I changed the code to this:

$d = New-Object 'Int32[,]' 10, 20

Then I thought that it should be possible to streamline it a bit so I got this:

$d = Int32[,] 10, 20

Simple enough. When I see it, I wonder why I couldn't figure it out quicker.