Monday, 22 January 2018

Tips on dealing with WinRM and remoting using the Test Agent

Despite the push we’ve seen in the last few years, the Hosted Build Service might not be the right product for you for whatever reason.

Then, if you are in a situation where your agents aren’t running in the same domain as Team Foundation Server’s and you want to use the Test Agent then you really risk opening the Pandora’s box, courtesy of WinRM and PowerShell remoting.

And to be completely clear – I have nothing against them Smile the only downside is that they need to be approached in the right way, otherwise the can-of-worms effect is just behind the corner.

First and foremost, remember that whenever you target a machine for Test Agent deployment you only need to consider the Build Agent-Test Agent relationship. All the errors you will get are going to be from the Test box, not the build box.

So when you need to configure WinRM, the Test box is the machine that is going to be accepting the connections. While it sounds straightforward, sometimes things happen and one is tempted to look at the Build box first: don’t.

Also, if you really want to use HTTP and WinRM, remember that this is the trickiest combination – so think twice before going down that route!

Then in terms of errors – you will likely face WinRM errors of all sorts. The most common is this:


If you are outside a domain then REMEMBER about Shadow Accounts – it is the only way to keep identity issues to a minimum. You’ll also need to set the TrustedHosts value to the machines pushing the agent.

Then this:


Remember that passwords need to match, and that mixing users at setup time isn’t really a good idea if you are going down the workgroup/non-trusted domain route.

Always triple check passwords, and I recommend to use the same account for both provisioning and execution, at least as a baseline. This will make sure you have a safety net incase things don’t pan out as expected.

Eventually there is this error, that really puzzles me:


This is actually an aggregated exception:


Look at UAC and execution context for this – it always happens when you are not running stuff as Administrator when that’s supposed to be elevated. It always drives me mad.

Wednesday, 10 January 2018

Don’t overlook the details during a TFS outage

Two weeks ago I dealt with a head-scratching outage. A few minutes of downtime, for a very stupid reason.

So, we are starting in a situation of total outage. All the services which rely on the production AlwaysOn Availability Group cannot connect to the server. People start screaming, emails flow in at a rate of tens-per-second… well it wasn't that bad, but you get the idea 😊 outages are always annoying.

So off we go with the usual stuff – the TFS Management Console does not load any data from the Data Tier, so the first point of call is checking the database servers.

Which are humming along totally fine. What the hell?! The network stack works as expected, I can ping all the machines involved!

When checking the database servers, I can see that the Availability Group is totally fine – everything is green, synchronised and with no issues. While this is very good on its own (no backups to restore, nothing to sweat too much about), it still does not explain why the Application Tier cannot talk to the Data Tier.

Then the awakening – whenever I try to connect to the AlwaysOn Listener I get a network error, while going directly to the database server works without problems. There it is!

Pinging the Listener does not work indeed. But why? All the cluster resources were green, online. 
But for some reason the affected resource failed to perform its duties.

Given that all the other moving pieces were perfectly fine, a manual AlwaysOn failover solved the problem. The lesson learned here is that in a complex architecture there is always something unnoticeable but critical – it’s like breaking a malleolus. 

Thursday, 21 December 2017

Help! My TFS automatically redirects HTTPS to HTTP!

If you are deploying HTTPS to TFS and you want to keep HTTP for a while, you might experience a strange behaviour: despite following the documentation, every time you browse to your TFS Web Access with HTTPS you are redirected to HTTP.

Fairly simple answer to this, and it is not because of any extra tool or technology: it has to do with the Public URL (it used to be called Notification URL) you set on the TFS itself.

If you leave the HTTP version over there you will always get the HTTP version – even with HTTPS  – otherwise if you switch to an HTTPS URL there you will get the HTTPS version by default, and explicitly browsing the HTTP version keeps you on the non-secured protocol so you can keep them side by side.

Tuesday, 19 December 2017

The impact of an upgrade to Team Foundation Server 2018

Usually upgrading Team Foundation Server is smooth and easy, but there is an exception this time around. It is not strictly related to TFS per-se, instead the issue lies with the new deprecations introduced with version 2018.

If you have a local server with just a few users it might not be a huge deal, if you have hundred of users spanning across timezones all over the world things can get hot pretty quickly. I am not going to cover pre-requisites like versions of Windows, SQL Server, etc. but I want to focus on which are the deprecated features of the product.

Let’s take a look at what these deprecations are, and how to approach them to minimise service disruption.

XAML Team Build

The elephant in the room is clearly about saying farewell to the XAML Build. After 12 years it is time to say goodbye to the old system and move to the new Team Build.

While this is good news for some, it can be troublesome for others, especially if you have a complex Build Definition which hasn’t been updated for years.

The way to go here is to have a clear idea of what the build process does and try to replace it with as many OOB tasks as possible. Second choice would be to cherry-pick from the Marketplace, and eventually write PowerShell scripts.

You might be tempted to reuse existing scripts or automations – which is good – but my suggestion is about maintenance: less legacy scripts mean less problems when it comes to maintaining the process, and using OOB tasks makes easy to take advantage of the constant flow of upgrades you will get from the tasks.

SharePoint integration

SharePoint and TFS 2018 is no longer tightly coupled with SharePoint. This does not mean you will lose any of your Document Library, but only that some features won’t work anymore.

The impacted features are:

· SharePoint site creation when requesting a new Team Project

· Web Parts integration

· The Documents pane within Team Explorer

While this is a major change in the history of the product, it is not unexpected. In an era of APIs and extensibility, relying on some opaque integration which isn’t really fit for purpose anymore – it does not work with Office 365… – isn’t the way to go. But remember: the hyperlink field in Work Item Types is not changing, so you can still link your (now external) documents to Work Items.

The existing Excel reports and the Reporting Services reports are not affected by this change, they will still work normally as they leverage on different features of the product. What is going away is the tailor-made integration which nobody else can use, really.

Lab Management deprecation

Lab Management in its current form will be deprecated and removed. This does not mean you cannot use virtual testing environments anymore – there are a couple of choices. You can use Deployment Groups or the SCVMM task if you want to keep using what you are used to.

It is also worth mentioning that the MTM integration for Lab is going away too – not a huge surprise given how good the web interface is, but it is worth mentioning if you are a heavy user of Lab Management.

Team Rooms

This is pretty much the only feature that is removed with no replacement – again, for good reason: Teams, Slack and others are such good collaboration tools, which focus only on having a good communication experience so they are the prime candidates for adoption.

Monday, 4 December 2017

An unusual scenario for Release Management, part 2: production SonarQube upgrades

In the previous post we saw how you can automate SonarQube test upgrades, but now it is time for production.

As mentioned, my artifacts here will be the TFVC repository, and I am going to have just one environment as target: Production.


There are three phases here, two Deployment Group phases and an Agentless phase. This is also where Tags come into play:


The Release Definition itself is going to be fairly straightforward – after all, I am assuming testing already happened so this automation is just aimed at saving time:


The first phase will prepare the environment for the upgrade itself. It is going to copy the zip file to the location we’d like to use, unblock and extract the ZIP file, copy the plugins in use to the new instance and so on.

I am also using these variables to keep the process reusable, and I am taking the ALM Rangers template as a starting point, so everything will happen into C:\sq\:




The Start SonarQube Interactive launches StartSonar.bat from PowerShell to make sure it is not going to make the task hang and run indefinitely.


The Agentless phase is required because once you launch StartSonar.bat you will have to browse to the /setup URL and start the upgrade process.

I am going to get a notification (it could be anyone else here, even a group) and I am going to start the process. You could automate that, but it is surely better to do so manually IMHO.


Once the upgrade is completed (and you stopped the StartSonar batch job, again I could automate this too but I am happy to have it included in the manual interaction for now) you can resume the pipeline with the second Deployment Group phase, which is going to remove the old service, install the new one and start it using the OOB batch scripts in your SonarQube instance.

This saves time (remember you can schedule this Release Definition too Smile) but the most important takeaway here is the value you get from automating the process and versioning your configurations.

It might be unorthodox, but it works quite well IMHO.

Friday, 1 December 2017

An unusual scenario for Release Management, part 1: testing SonarQube upgrades

Among the many things I do I manage a SonarQube instance. Not a big deal to be fair with you, but it is a valuable tool and it has its quirks. You need to spend time on it.

So I thought about automating this process a little bit. It is a bit unusual, but it brings some value, so why not!

The result is a TFS or VSTS Team Project with a TFVC repository (TFVC is perfect for handling binary files!) and two Release Definitions, one for Test and one for Production.

The reason why there are two Definitions is because – oddly enough – the Test one came after the Production one (which is easier, you’ll discover why later on). I might revamp the whole thing in the future to have sequential environments, but this is it as of now Smile

In the TFVC repository you are going to find folders for each SonarQube version I deployed on my server, together with the relevant file filled with the values you want, and a scripts folder with some utility scripts.


The reason why I am not automating the configuration file creation (via a find-and-replace operation for example) is because you are explicitly told by SonarSource not to just replace this file with an existing version but to start from scratch.

While testing your configuration you will need to work on it anyway, so it is a good idea to put it in a repository, and you will get versioning for free as well. Bonus.

Both my Release Definitions feature a Deployment Group: guess what, it contains my SonarQube server SmileI also leveraged on Tags, in case I might want to have completely separate enviromnents, as the Deployment Group phases are marked to run only on machines that sport the right tag for the Release Definition. It isn’t the case for now though.


Now it comes the fun part, let’s start with the test upgrades. My process for testing SonarQube is as it follows:

  • Restore a backup of the production database
  • Get the new SonarQube version on the VM that hosts its services
  • Extract the new version, set the right values in the file (like different ports and java switches)
  • Check that the upgrade runs successfully
  • Verify all the involved plugins

Like I said I am not going to automatically find and replace values in the file, and the latter steps aren’t really worth scripting, but the first two steps can benefit from an automated process.

This is what my testing pipeline looks like:


Nothing too fancy, but it saves time.

The cool bit here IMHO is the Azure PowerShell script I am running to restore the database: given the Resource Group, Server, Database and SonarQube version (which is used to form the name) I can check if I already have a testing database – if not it starts restoring a backup copy from ten minutes before.


If this prerequisite check fails, I integrated the error handling so it stops the task immediately and marks the release as failed:



How? Like this:


The Write-Error statement stops the task execution and raises the error message, the Write-Host statement with the specific ##vso line marks the task result as failed and the exit 1 line terminates the sessions so that whatever is next (the database restore!) is not executed.

Eventually at the end there is an Agentless phase which is just manual intervention with the required things to do:


I will go through the production pipeline in the next post, as it is different Smile

Monday, 20 November 2017

Old but still very good, SlowCheetah with VSTS!

I was asked if there is a way of transforming configuration files at build time. Nothing else than the good, old and reliable SlowCheetah if you ask me Smile

Just install the Extension, and you will be able to add your transformations by right clicking the .config file and selecting Add Transform:


These are going to be based off the BuildConfiguration you defined in your Solution. Once this is done you can define your own settings, like this:


where a transformation can be something like this:


There are so many examples on how to do this, so please do not shoot on the pianist Smileall of that goes into a Version Control System of sorts, and it can be built by VSTS or TFS or any other Build Engine.

Your Build Definition needs to specify a Configuration, either at Queue time or embedded into the Build Definition itself.


The result? As expected:


This isn’t really about VSTS or TFS per-se, but it is always a valuable approach to configuration management, and it was worth a refresher Smile