Sunday, 10 February 2019

The continuous quest for automation in DevOps, and the Azure ML Studio example

When you think about it, most of the tasks you carry on in a DevOps environment revolve around automation.

Infrastructure as Code? It’s automation, right? Cool. Testing? As automated as possible. Integration with 3rd party systems (SonarQube, WhiteSource, Azure DevOps, Jira, anything you retrieve data from or send data to) – it is automated.

You might think I live in a DevOps bubble. Let’s expand then – SRE? Data processing? How many things we rely on in modern development that have automation to their core? Excellent, you got your answer.

Mind that, it is not a Cloud exclusive. At the end of the day it’s technology we are talking about – hence my mantra “A technology problem is never really a problem. Technology can be bent at will.

Now, I am working upon a client at the moment which has many efforts going on, and I was asked if I know of a way of automating deployment of Azure Machine Learning Experiments from Azure ML Studio.

Being completely oblivious to the technology I spent some time on it, and despite being a machine learning tool it is quite WYSIWYG. Hence no automation whatsoever, at the moment. Being a cloud-based product there isn’t the shadow of a doubt that eventually it will be implemented (given enough demand, obviously), but it is manual today. And I need to do this today, so there was no other option than going down a custom route.

Let’s pretend for a minute that I am not living under a rock, I don’t know how to use a search engine. The first step here would be to fire up Fiddler and see what happens when you press your buttons in the tool’s UI.

It is a modern web-based tool so there is communication between the UI and some API layer. There will be some sort of authentication involved. You can work your way around it. Eventually, you will be able to replicate this interaction with a script, and put it in your pipeline.

Given I do not live under a rock and I know how to use a browser the first search result would be the excellent Azure ML PS. Using it in a set of PowerShell scripts, stored in a Git repository, to be then consumed by an Azure PowerShell task in Azure DevOps Pipelines is really trivial. Again, this is a valid proposition for both on-premise systems as well as cloud-based systems.

Pipelines are brilliant for this – and I mean it. If you have something ready, or something that cannot use a task, just throw it within a PowerShell or a Bash script and you are in business. Use a Windows agent or a Unix agent, I don’t really care to be fair – as long as you can interact with the target system with a script everything is good in my view.

Sure, there will be situations where automating gets really difficult. Sometimes you can’t avoid doing manual things. Some other times you will be better off rewriting the whole thing. But usually, we are in a quest for continuous automation and it is what keeps us going. Keep automating!

Friday, 1 February 2019

How to reorder your stages in Azure Pipelines

It might seem funny, it might seem stupid... but this might happen to someone else, so it can be handy for the future. How do you re-order your stages in a pipeline?

Let's take this example:

Now, if you want to move Stage 2 to the end, you will try and click on the Stage UI or in the Tasks and you will not find how this is possible.

The answer is in the Pre-deployment conditions:

There you can find the triggers that can be linked to a Stage, which in this case is the completion of  a previous stage:

Changing that value will make your re-ordering needs a reality 😊

Wednesday, 23 January 2019

My road to an open source project: Azure Traffic Manager extension for Azure DevOps

As I mentioned in my last few posts, I have been working in my spare time to convert the existing scripts I had for Azure Traffic Manager into an Azure DevOps extension.

I finally managed a critical point to publish the code on GitHub! I am not a massive code person, it might feel like a drop in the ocean but it is something really big for me :-) It was also a great exercise on the actual end to end process of creating a working extension, which I found extremely interesting and valuable.

The extension does four things at the moment: enable and disable an Endpoint, promote and demote a priority-based Endpoint. It is fairly simple, but again this code stems from demos, labs and community efforts I did over the last 12 months so it is not something that was used by a company in production.

It is available in Public Preview here, and I am still ironing out a few issues both on the pipeline side and the code. Still, given the lack of an official Azure Traffic Manager extension I feel it can be a decent stopgap - at least with these basic features. I will try to make it even more complete over time so it can be even better!

Thursday, 17 January 2019

The URL - can I move away from the old URL?

Plenty of reasons behind this, but you might want to move away from your <organisation> URL and onto the new<organisation> URL as a matter of standards.

It is fairly easy to do - straight from the Overview Settings page of the organisation:

If you configured your Build Agents on-premise with a PAT and the old URL you are covered, the configuration will still work, plan for this in advance anyway and make sure you are prepared 👀

Friday, 11 January 2019

The importance of the .gitignore file

A few days ago I spent a some time reviewing some old repositories of mine (mostly demos) and I realised how cluttered they were.

You know why? Because they were created years ago, went unmaintained for ages and they were lacking a .gitignore file.

Early in the days of learning Git, one of the things I left behind was - guess what - a proper usage of .gitignore files. Moving on I realised how important they are: they prevent your repository from becoming filled with temporary files you don't really need. If you go back to old repos you created when you were not as proficient as you are today, you are guaranteed to stumble into that like I did.

For example: if you create an empty MVP Application using ASP.NET Core and you run a git add --all you will surely add files like sqlite's db.lock file, which is utterly useless in a Version Control System.

Or the whole .vs folder, .suo files, the obj folders and so don't need this stuff in a Git repository!

At the end of the day a .gitignore file is literally a list of files to ignore in order to maintain your repository lean and nimble. Spend five minutes (tops!) on it, and you will get the rewards way down the line.

If you are *that* lazy and you think you can't spend five minutes on it, there is a nice service (what isn't commoditised today?!) called that will create a default one for you based off your favourite IDE or environment.

Really, do it! 😁

Friday, 28 December 2018

Recycle your PowerShell scripts in a custom Azure DevOps task

I am a huge PowerShell fan, and pretty much anybody who knows me is aware of that.

Over time I developed my set of scripts for demos and conferences, and some of them are in use in my homelab on a regular basis. One of them is for managing Azure Traffic Manager, and I realised that there is no Azure DevOps task for managing Traffic Manager on the marketplace! Thanks to the brilliant session Utkarsh did at the London Microsoft DevOps Meetup I decided to give a go at at converting that script.

So this is what I did to recycle my script in a task - beware: this post is a crash course on how to do that, and I don't think it is actually anything special. It is just an easy way of productively recycle some existing scripts in a good way. I haven't finished yet - as a good Product Owner I started with an MVP and I see there is more and more to add to it 👀 - but it is slowly coming together and I hope it can be useful to someone in the near future.

What you need to do is create the scaffolding with the tfx cli. from which you can start customising the script.

Then remove any reference to the JavaScript portion of the task - you are just recycling a PowerShell script so you don't need it! It will also save you time when debugging it as otherwise the runner defaults to the JavaScript entry-point. Needless to say, the name of the target script is going to be the name of your script.

Add the VstsTaskSdk module otherwise you are going to get odd errors out. Also, remove any versioning reference. The structure should be .../ps_modules/VstsTaskSdk. This is true for every module you are going to use:

Code-wise, keep the scaffolding as clean as you can, and add your script inside the try statement after the Trace-VstsEnteringInvocation $MyInvocation. You should be able to recycle your script with little changes, of course some adjustment might be needed. If you need to get input parameters you can use the Get-VstsInput command.

Wednesday, 19 December 2018

How to authenticate with the Azure subscription you select in a custom PowerShell Azure DevOps task

I will cover more about my custom PowerShell Azure DevOps task, but I wanted to share this quick tip as I struggled as much as the next guy about it...

Say that you want to re-use your PowerShell scripts, and instead of having inline PowerShell tasks you want to package them properly for cross-organisational re-use. It is a relatively easy and straightforward task, but chances are you are going to target an Azure subscription with it and you want to have the nice dropbox and the integrated experience you get with first party tasks.

It is not really difficult, but for some reasons I could not really find an easy walkthrough about that - a limited set of steps, but somehow hard to find...

What you need to do is to import the VstsAzureHelpers  module and include it in the PowerShell modules consumed by your task. Once that is done, you need to make sure you can select the Azure Subscription in the parameters - easy to do as you need to edit the task.json file:

After this you can finally use the Initialize-Azure command in your PowerShell script, which is going to automatically pick the subscription from the parameter and initialise the relevant cmdlets so that you can use stuff like Set-AzureRM... without running Login-AzureRmAccount.

Easier done than said 😀 I hope it is going to be helpful!