Run Microsoft pre-configured VHD evaluations without Hyper-V using VirtualBox

I needed to run the BizTalk 2010 VHD for demo purposes, but lacked access to a Windows 2008 Server with Hyper-V in order to run the virtual machine.

My first attempt was to simply mount the VHD file in a newly configured Virtual PC machine. This failed with the virtual machine rebooting shortly after start.

Next I tried to create a new virtual machine in VirtualBox, giving me pretty much the same result, but with a bluescreen flashing past just before reboot.

In the bluescreen that quickly flashes by, it is possible to see the error code 0x0000007B, also known as INACCESSIBLE_BOOT_DEVICE. This is an indication of a possible issue with the boot sector, device driver and similar things. It led me to take a peek at the storage settings for the virtual machine that showed me that the IDE controller had no disks attached (but a CD/DVD drive) and my added VHD file was attached to a SATA controller.

I removed the VHD file from the SATA controller and removed the controller completely. I then added a new harddisk to the IDE controller and assigned the VHD file to it, thinking that this most likely was how the VHD file was set up initially.

And indeed it was so. This made the virtual machine start up ok!

I did run into issues when logging on though since the admin password includes the character "@" which I was unable to type due to some issue with VirtualBox and the Alt Gr key on my keyboard that was needed to type the character in. This was however quickly remedied by using the ALT-sequence instead, typing ALT+064 for the "@" character.

After a day of testing, I cannot see any issues in the VM, but everything is running just fine! For those that are using VMWare instead, the same solution should work just as well.

"Clever" uninstall of msi packages/applications using PowerShell

When I created my own PowerShell script library for BizTalk deployment automation I ran across the need to uninstall applications, both BizTalk applications and non-BizTalk ones, by only knowing their name.

At first, I solved it using a WMI query like so:

$name = "application name"

$product = Get-WmiObject -Class Win32_Product -Filter "name='$name'" -ComputerName "localhost"

[void]$product.Uninstall()

This is not a recommended way of doing it though. It is both very slow and also causes a bit of spamming in the Windows Eventlog since the query in fact does a reconfigure of ALL applications installed. This reconfiguration can also cause a bit of other issues in some cases.

I then created another way of trying to uninstall applications in a more failsafe and secure way by using msiexec with the uninstall flag. The tricky part was to find a way to get the product key in order to be able to use msiexec since it requires this for uninstalling an application. The result can be found below.

The script function will take the application name as argument. It will then via the registry (note that this is configured for x64, so change the path if you are running x86) look up the application settings. If the application can be found via name (it should), we extract the product key. Then this is used as a parameter to msiexec.

If the product key cannot be found (it happens), we will instead try to read the uninstall string that is set when installing the application. Windows will run this string when you choose to uninstall an application, so why do not we use it? If found, we extract the product key and do our msiexec call.

If all fail, we throw an exception to be caught in the real part of the script.

This is the full script function:

Function Uninstall-Program([string]$name)
{
    $success = $false

    # Read installation information from the registry
    $registryLocation = Get-ChildItem "HKLM:\Software\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall\"
    foreach ($registryItem in $registryLocation)
    {
        # If we get a match on the application name
        if ((Get-itemproperty $registryItem.PSPath).DisplayName -eq $name)
        {
            # Get the product code if possible
            $productCode = (Get-itemproperty $registryItem.PSPath).ProductCode
           
            # If a product code is available, uninstall using it
            if ([string]::IsNullOrEmpty($productCode) -eq $false)
            {
                Write-Host "Uninstalling $name, ProductCode:$code"
           
                $args="/uninstall $code"

                [diagnostics.process]::start("msiexec", $args).WaitForExit()
               
                $success = $true
            }
            # If there is no product code, try to read the uninstall string
            else
            {
                $uninstallString = (Get-itemproperty $registryItem.PSPath).UninstallString
               
                if ([string]::IsNullOrEmpty($uninstallString) -eq $false)
                {
                    # Grab the product key and create an argument string
                    $match = [RegEx]::Match($uninstallString, "{.*?}")
                    $args = "/x $($match.Value) /qb"

                    [diagnostics.process]::start("msiexec", $args).WaitForExit()
                   
                    $success = $true
                }
                else { throw "Unable to uninstall $name" }
            }
        }
    }
   
    if ($success -eq $false)
    { throw "Unable to find application $name" }
}

Running an Integration Competence Center for small businesses

A while ago I saw a post from an old colleague on LinkedIn where he linked to an old article called Too small for an Integration Competency Center? Think again. I was intrigued to read the article since I am in the middle of establishing an ICC at the organization I'm at, and we are quite small in that regard.

The four building blocks involved are:

• Adopt common integration practices
• Standardize on a common integration platform
• Create a decentralized group that specializes in integration
• Form a centralized ICC

This has all been done at the organization I am at, with the experience I have from larger organization as a basis for the setup. In my case, with an IT department of a whopping four people, forming an ICC is no big challenge in that regard.

I have opted to include external parties in the ICC to form a team that is not bound to a strict organizational hierarchy, but rather be agile and flexible to the demands from the organization at stake.

The basis of the ICC is in my role as integration architect/supervisor in the IT department. Coupled to this I have included an external integration architect to have a form of sounding board. Most, if not all, integration development work is carried out by consultants which the external integration architect is responsible to coordinate. As needed, I also include internal resources in the ICC to help forming the ICC and drive the integration development forward. So basically, the ICC is in it's simple form just two persons, with several others coming and going as needed.

So far, this is working very well. The agility in the ICC is a strong factor to consider. A lot of times, people are included full time even if they are not needed full time. Since the external architect is contracted on an as-needed-basis, costs can be brought down. Of course, that puts a lot of burden on me.

However, I have strived to create a simple and effective process of the entire integration life cycle. Document templates are available for everything from requesting an integration (done by the organization) to detailed specification and testing of the completed development. The development guidelines and deliverance specifications are to be as standardized as possible to not bind anything to a single person or organization (except our own). I aim to be able to both replace the consultants involved as well as myself without having to explain anything about the setup and behavior of the ICC and involved parts.

As for the four building blocks above, I feel that all (almost) of them can be ticked off in our case. The platform is BizTalk, best practices are established through documentation and processes based on experience from me and consultants, a decentralized group is formed and the ICC is formed and handling most of the integration needs. What is left is the funding that is still divided and handled by separate organizational units rather than taken care of completely by a central organization. This is currently the biggest issue to handle since it hamper a true value-for-money path regarding integration.

But to summarize, in accordance to the article linked in the beginning of this post, you cannot have a too small of an organization for an ICC to be formed. By including resources on a need-to basis as well as including external consultants where needed, it can be handled in a very effective way. The important part is to have a clear process of the entire life cycle as well as clear boundaries between different parties involved. My role in the whole ICC is most importantly to coordinate the different tasks between the resources, and as long as that is done appropriately, everything is flowing smoothly. Having a centralized funding of the work is not necessary, but very (very!) preferable.

Simple dos and don'ts for versioning webservice interfaces (and similar integration interfaces)

Earlier this week I went to a user group gathering for the ERP system we utilize at work. The topic for the day was "integration". At the end of the day during an open Q and A session, I approached the ERP representatives with questions on how they worked with the web service API exposed from the ERP system. I asked them if using the current web services guaranteed that they would work without changes and need for deep regression testing when upgrading the system. I also asked them how new functionality to existing services would be implemented, from an integration point of view.

The answer was that current implementations would require normal regression testing after upgrading, but that there could be changes in the backend making the services work a bit different in the new versions, depending on the changes made in the business logic and GUI.

This made me a bit quezy and fidgety. In the best of worlds, I want a web service to be seen as a hard written contract between two parties, in this case the ERP system on one side and the integration platform on the other. By exposing a service, you should guarantee that the specification for the service is not to be changed and that the underlying logic is to be set in stone as well. That seemed to not be the case.

The discussion then evolved into the realm of versioning and planned changes of existing services. As of now, new functionality is added to the service without changing its name, version or namespace. Fields are simply added as optional.

This method is something I have several issues with, and which I also expressed at the session (maybe a bit too harsh, but it is an important subject to me). First off, by adding new functionality as optional fields, you will break one of the (many) benefits with web service contracts: hard typing of them! By doing it this way, the developers will make future maintenance of the services harder, for both themselves as well as those utilizing the services. Also, the clients will have to upgrade the integrations whether they want to or not as soon as the ERP system is to be upgraded and the services change.

I instead proposed that they should version their services and operations. When changing an operation, create a new version of it and as long as it is possible, make sure that the previous version(s) are still functional. If a service operation is to be removed from the system, mark it as obsolete and let the clients know which version of the ERP system it will be removed in, and preferably how to utilize other services and operations to enable the needed functionality. I myself think that this is a very obvious way of handling the dilemma. I have grown to understand that it is very rare that versioning is used though and I do my best preaching the above statements.

So please, make sure that you use some form of versioning strategy when building your interfaces. It will make my (and many others, I guarantee) life a lot easier.

BizTalk Server 2013 (formerly BizTalk Server 2010R2) is in beta

Late last night I saw some tweets about the new version of BizTalk getting to beta status. A quick look, and sure enough, BizTalk 2010R2 is renamed to BizTalk 2013 and a beta version is available for download.

I must say that this release is very intriguing. Among the more interesting new features for me is the REST capabilities, the new SharePoint adapter and the ESB toolkit integration (especially since I loathe the way the first version was to be setup and handled).

The new dependency function to check out how artifacts are dependent of eachother is also a nice addition.

Go check it out and hope for a quick release of the final product!

The BizTalk File adapter, least permissions required for a Receive Location

Every now and then a questions pops up regarding BizTalks FILE adapter permissions. If a Receive Location is set up to poll a folder where the account used (by default the BizTalk host account) does not have appropriate permissions, the following error messages will be written in the event log.

File transport does not have read/write privileges for receive location "D:\BizTalk\In\".

The Messaging Engine failed to add a receive location "Receive Location1" with URL "D:\BizTalk\In\*.xml" to the adapter "FILE". Reason: "File transport does not have read/write privileges for receive location "D:\BizTalk\In\". ".

The receive location "Receive Location1" with URL "D:\BizTalk\In\*.xml" is shutting down. Details:"The Messaging Engine failed while notifying an adapter of its configuration. ".

Usually, the solution is to just give the BizTalk host account full permissions to the folder and be done with it.

I myself try to never give full permissions to any account as long as it is not needed. It is however more common than not to see applications and users all use the SA login to SQL Server, be local (or domain) admins, and be put as a member in all of the groups used in BizTalk. *shudder*

Anyhow, for the FILE adapter, the first attempt a user does if not going the "full permission route" is to give the account used read+write permissions as so:

This will not work but give the same errors again. Neither will adding the "Modify" permission do any good. It is at this point most users simply click the last checkbox available, giving Full permissions. And voila, it will work!

What one rather should do is go to the Advanced permission properties and add the "Delete subfolders and files" permission which in reality is what is missing. BizTalk has been able to properly read the file, but was unable to delete it, resulting in the error.

Notice that the "Delete" permission is not set. Basically, the account used for reading the files in the folder most likely should NOT have the permission to delete the folder itself, but just the contents inside of it.

Be careful though! While experimenting, I noticed that if I had correct permissions according to the screenshot above, but removed for example the "Write extended attributes" permission, the Receive Location would NOT read the files in the folder, neither would it give ANY type of errors or warnings. Everything will in that case look perfectly fine, with the Receive Location up and running, but no files will be read and processed!

Using PowerShell to remove all old files in a folder structure

A common task when working with BizTalk installations is to clean out old data from folder structures. May it be old log files, automatically saved messages from a send port, database backups or any other sort of data that is obsolete after a certain period of time.

When I started working with BizTalk, the norm was to use bat files or VB Script. These can still be seen in many places since they a) work just fine, and b) the developers/administrators hasn't learned a new way to script these kind of things, mostly because a) they work just fine.

I myself as mentioned earlier prefer to use PowerShell nowadays.

In order to recursivly delete old files and folders, one line of PowerShell script is all that is needed:

get-childitem -path 'c:\temp\test' -recurse | where -FilterScript {$_.LastWriteTime -le [System.DateTime]::Now.AddDays(-30)} | Remove-Item -recurse -force

This script simply traverses the specified folder (why not change it to take the folder and date as a parameter) and does so recursivly. All files older than 30 days in this case is then deleted. By using the -force parameter, locked and hidden files will not cause an error.

In order to have this run automatically on a server, it needs some adjustments. Add a Scheduled Task and set the Program/script to run to Powershell and then add the arguments as so:

-noninteractive -command "& {get-childitem -path 'c:\temp\test' -recurse | where -FilterScript {$_.LastWriteTime -le [System.DateTime]::Now.AddDays(-30)} | Remove-Item -recurse -force}"

Set the security options for the task to "Run whether user is logged on or not" and also check the "Run with highest privileges" checkbox.

Why I created my own PowerShell scripts for BizTalk deployment

Most of us working with BizTalk has at one point or another started scripting our deployments instead of doing them manually. I have during the years seen many variants of automating the deployment process. At one place the "old" way of using bat-files calling BTSTASK where used extensively. At another, a custom .Net program with a GUI was developed that would call the different API:s available. I myself resorted to using PowerShell to create a fully automatic deployment routine.

My scripts has two parts. One with different functions that I wrote and one part that encapsulates the functions and executes them in the correct order. As input I create an XML file that specifies which .msi packages that are to be deployed, which applications to remove/update/add, settings for the IIS virtual directories and so on. Since it is quite common during complicated deployments that something will err, I added functions to test run the scripts in advance. The test run simply checks for dependencies in the BizTalk configuration, making sure that applications can be modified as configured. It also checks that all files needed are provided in the setup package.

When creating the scripts, I briefly looked at using the BizTalk Deployment Framework and other extensions to PowerShell, but opted on developing it all in my own scripts. This due to several reasons, each quite strong in my case:

• I would learn more about PowerShell as well as the API:s and tools found in BizTalk
• I would know exactly where things could go wrong and what to do about it.
• It allowed me a greater flexibility since nothing was hidden in dll files or needed installation. As long as PowerShell was installed on the servers, the scripts would work. Nothing else is needed. This is also something that administrators liked. Some of them are a bit weary about installing "some stuff I found on the net". 

This post is the theoretical background to posts that are to follow that will showcase and explain my scripts in individual blocks. I have already started in advance with the post Recycle IIS application pools using PowerShell, and more snippets are to come.

BizTalk Map Viewer: Graphical interface for the BizTalk Map Documenter

Quite a while ago during an assignment we needed a way to display the logic in BizTalk maps for developers that didn't have BizTalk installed on their workstations. We quickly turned to the BizTalk Map Documenter which simply is an XSLT file built to take a standard BizTalk map file (.btm) and output it in a human readable format as an HTML page.

In order to use the XSLT file you need to execute it, preferably with the utility msxsl.exe which can take the .btm file and XSLT file as arguments and then output the result to be viewed by the user. However, this caused a bit too much command line hacking in our case and I took an evening in the hotel bar to quickly whip together a GUI for it and thus, the BizTalk Map Viewer was born.

The application is very simple and in it's basic form simply allows you to open a BizTalk map file which then will be parsed by the BizTalk Map Documenter XSLT file and the result will be shown in the application window. I added functionality to save, print and copy the result. I also added functionality to register the application as a valid right-click option for .btm files. If enabled, you will get the option to "Open with BizTalk Map Viewer" when right-clicking on BizTalk map files which then will open the selected file in the application. Just be sure to have placed the application executable in a good folder before registering it since the current path will be used.

Feel free to download the code and executable.

Use it as you please.

A take on documenting integration platforms using Visio

Earlier this year I wrote a post about having Visio update a field on a shape automatically when editing any other field, making it possible to always know when the metadata was last changed. In the comments I was asked to share the Visio sheet which it now is time to do. This is the result of that request, in some way documenting my attempt to use Visio to document an integration platform. Hopefully you can gain some ideas to use in your documentation endeavours.


In almost all of my encounters of different integration platforms at various companies, there has been a high level view of the entire integration setup showing the various message flows available. This map has usually been constructed in Visio. There has also been two types of layouts, lets call them the structured layout and the unstructured layout.

The following is an example of the unstructured layout. It's common and does a fairly good job at describing the full environment. It is often seen when there has not been a true approach to working with integrations.

Most of the time they are filled with objects, with varying descriptions, showing the entire network.You can see that two systems interact, but you do not know in which way. It is also often hard to see how everything is connected if you as in the example has an integration hub in the middle, doing some message routing.

Hence we get the structured layout as is seen below. In this case, there is a very solid structure with the integration platform in the middle, and the integrated systems on both sides with arrows showing each and every message flowing through.

These maps are good at showing the message flows as every message is entered, but it doesn't give you the overall picture that the unstructured layout gives you since a lot of fluff is stripped away. Usually there has also been an improvement from the unstructured layout in that each message flow has been given an identity which then maps to more detailed documentation in a file archive or document portal.


So, earlier this year I did a test with a variant of the structured layout, seeing if I could document the current state of integrations at my current job. The map I was given from the beginning was according to the unstructured layout. A lot of information was out of date, systems had been taken out, others taken in, versions changed, message flows altered and so on. I had to create an up to date picture of the integrations, so that I could commence working with them.

I took a brand new Visio document and started documenting. I created a "system" shape that had the following shape data:

• System name
• System name 2
• System version
• Server name
• Database
• Database version

In my Swedish Visio it looks like this:

In order to make it easier updating and changing these properties/metadata for all shapes, I changed the Shape Sheet to make the form data window open when doubleclicking on the shape. You can do this by editing the Shape Sheet property "EventDblClick" under "Events" to read "=DOCMD(1312)".

I then did the same thing for the arrow shape adding metadata that I want to know for each message:

• ID
• Object name
• Status (i.e. Planned, Under development, In use, To be removed)
• Transport (i.e. Web service, File, FTP, WCF...)
• Type (i.e. Xml, Flat file...)
• Frequency
• Automatic (i.e. a drop down giving the option of Yes or No, indicating whether the message transfer is completely automatic)
• Sync/Async
• One way/Request-response
• From system
• To system
• Comment

Many of the fields are defined in the form data editor as dropdown menus as to make the data more consistent and easier to edit. I also added the "From system" and "To system" in order to get good documentation automatically created by exporting the form data to Excel. As with the System shapes, I added the doubleclick event to the shape.

I also altered the shape so that the choices made in the form data will be visible in the shape. Depending on the "Status", the arrow will be yellow (planned), green (under development), black (in use) or red (to be removed). This is done by editing the Data Graphic and setting that each value will match a specific color.

Also, setting the "Automatic" to "No" will show a little man besides the ID and Object name. This can be set in the Shape Sheet as well under "Text fields" and "Value". In my case, the formula is: =IF(STRSAME(Prop.Automatic,"No",TRUE)," 웃","")&" "&Prop.ID&". "&Prop.Objectname&" ". But is of course dependent on the name you give each property in the form data.

In the same manner, I have a setting that will change the arrows on the shape so that I get an indication if it is a one way transfer or a request-response one, by adding a little unfilled arrow in the other direction of the data flow.

I have also adjusted where the text appears on the connector according to this post from the Visio Guy, so that the text is always at the end of the line instead of in the middle.

So, is this a good way of documenting message flows?

Well, in this case, it has been very helpful. I have a single document that shows me quite a lot of the current setup and state. It eases on the job of manually editing shapes to indicate whether it is a planned message flow, if it is request-response, a manual job and so on. Instead, I just select the desired property and let Visio format the shape accordingly, giving me a neat view of the state.

After documenting all message flows in this Visio document, I will most likely strip it down. I will remove a lot of the metadata in the connection shapes, and have this solely in the integration specification document that I will create for each message. I do not like to have duplicate information in my documentation, and I feel that that sort of data will belong better in the written documentation than in the Visio sheet. I will still keep the color coding depending on Status, as well as some other visual cues that is helpful when looking at the overall view of the integration platform.