Showing posts with label IaaS. Show all posts
Showing posts with label IaaS. Show all posts

Reading Notes #524

A blue kayak on the side of a blue and yellow tent in the wood.

Good Monday, time to share my reading notes. Those are a curated list of all the articles, blog posts, podcast episodes, and books that caught my interest during the week and that I found interesting. It's a mix of the actuality and what I consumed.

You think you may have interesting content, share it!

Cloud

Programming


~frank


Don't install your software yourself

I don't know for you, but I don't like losing time. This is why a few years ago I started using scripts to install all the software I need on my computer. Got a new laptop? N You just need to execute this script, go grab a coffee and when I'm back all my favorite (and required) softwares are all installed. On Linux, you could use apt-get, and on Windows, my current favorite is Chocolatey. Recently I needed to use more virtual machine (VM) in the cloud and I deceided that I should try using a Chocolatey script during the deployment. This way once the VM is created the softwares, I need is already installed! This post is all about my journey to get there, all scripts, issues and workarounds will be explained.

The Goal


Creating a new VM on premises applying the OS update and installing all the tools you need (like Visual Stutio IDE) will takes hours... This solution should be done under 10 minutes (~7min in my case).
Once the VM is available, it should have Visual Studio 2017 Enterprise, VSCode, Git and Node.Js installed. In fact, I would like to use the same Chocolatey script I use regularly.
# Install Chocolatey
Set-ExecutionPolicy Bypass -Scope Process -Force; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))

# Install Software
choco install visualstudiocode -y
choco install git -y 
choco install nodejs-lts  -y

(Available on gist.github)

The Tools


In this post I will use Azure CLI, because it will works on any environment. However, PowerShell can also be use only a few command will be different. The VM will be deploy with an Azure resource Manager (ARM) template. To create and edit the ARM template I like to use VSCode, you don't need it but it's so much easier with it! I use two extension.
The first one Azure Resource Manager Snippets will help by generating the schema for our needs. In a JSON file you just need to type arm en voila! You ahave a long list of ARM template!

armSnippets

The second is Azure Resource Manager Tools. This extension provides language support for ARM and some validate. Very useful...

toolvalidation

Creating the ARM Template


To Get started create a new JSon file. Then type arm and select the first option; to get an empty skeleton. Then add an extra line in resources and type again arm. This time scroll until you see arm-vm-windows.

step2Here

A multi-cursor will allow you to edit the name of your VM everywhere in the file in one shot. Hit Tab to navigate automatically to the userName, and Tab again to go to the password.

createARM
Now we have a functional ARM template that we could deploy. However, let's add a few things first.

Searching the Image SKUs by Code


One of my favorite VM images for a DevBox is the one that includes Visual Studio pre-installed. One thing to know is those images are only deployable in an MSDN subscription. To specify wich image you want to use you need to pass a publisher, offer, and sku.
Here how to do it with Azure CLI commands
# List all the Publishers that contain VisualStudio (It's case sensitive)
az vm image list-publishers --location eastus --output table --query "[?contains(name,'VisualStudio')]"

# List all offers for the Publisher MicrosoftVisualStudio
az vm image list-offers --location eastus --publisher MicrosoftVisualStudio  --output table

# List all availables SKUs for the Publisher MicrosoftVisualStudio with the Offer VisualStudio
az vm image list-skus --location eastus --publisher MicrosoftVisualStudio --offer VisualStudio --output table


Now that all the information is found, search in the ARM template and replace the current values by the one found. In my case, here are the new values.

"imageReference": {
                    "publisher": "MicrosoftVisualStudio",
                    "offer": "VisualStudio",
                    "sku": "VS-2017-Ent-Win10-N",
                    "version": "latest"
                }

Adding our Custom Script


Great now we have a VM with Visual Studio but our applications are still not installed. That will be done by adding the Custom Script Extension for Windows to our template. documentation page, a sample schema is there ready to be use.
The last node of your template is currently another extension. For the purpose of this blog post let's remove it. You should have something like this.

newExtensionPlace

We will copy/ paste the snippet from the documentation page a change a few little things. Change the type (thank to our VSCode Extension for that catch). Update the dependencies to reflet our demo.

To use the extension your script needs to be available online. It could be in a blob storage (with some security) or just publicly available. In this case, the script is publicly available from my gist.github page. I created a variable in the variables section that contains the RAW URL of my script, and a reference to that varaibale is used in the fileUris.

The extension will download the script and then execute a function locally. Change the commandToExecute to call our script with unrestricted execution policy.

You have a timed window of ~30 minutes to execute your script. If it takes longer then that, your deployment will fail.

{
        "apiVersion": "2015-06-15",
        "type": "extensions",
        "name": "config-app",
        "location": "[resourceGroup().location]",
        "dependsOn": [
            "[concat('Microsoft.Compute/virtualMachines/', 'FrankDevBox')]"
        ],
        "tags": {
            "displayName": "config-app"
        },
        "properties": {
            "publisher": "Microsoft.Compute",
            "type": "CustomScriptExtension",
            "typeHandlerVersion": "1.9",
            "autoUpgradeMinorVersion": true,
            "settings": {
                "fileUris": [
                    "varaiables('scriptURL')]"
                ]
            },
            "protectedSettings": {
                "commandToExecute": "[concat('powershell -ExecutionPolicy Unrestricted -File ', './SimpleDevBox.ps1')]"
            }
        }
    }
`

The ARM Template


It's finally time to deploy our VM.

# First, we need a Resource Group
    az group create --name frankDemo --location eastus

    # ALWAYS, always validate first... you will save a lot of time
    az group deployment validate --resource-group frankDemo --template-file /home/frank/Dev/DevBox/FrankDevBox.json

    #Finally deploy. This script should take between 5 to 10 minutes
    az group deployment create --name FrankDevBoxDemo --resource-group frankDemo --template-file /home/frank/Dev/DevBox/FrankDevBox.json --verbose

What's Next?!


We created one template; you could make it better.

Deploy from anywhere


By moving the computerName, adminUsername, adminPassword, and the script url in the parameters section, you could then put the template in a public place like GitHub. Then with use the one click deploy!

Directly from the Github page or from anywhere you just need to build a URL from those two parts: https://portal.azure.com/#create/Microsoft.Template/uri/ and the HTML Encoded URL to your template.

If my template is available at https://raw.githubusercontent.com/FBoucher/SimpleDevBox/master/azure-deploy.json then the full url become:
https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FFBoucher%2FSimpleDevBox%2Fmaster%2Fazure-deploy.json

Clicking that URL will bring you to the Azure Portal (portal.azure.com) in a customized form to deploy your template.

DeployForm

It cannot be easier! You can see mine on GitHub.

Auto shutdown


It's very easy to forget to turn off those VM. And whatever you are paying for them or your using the limited MSDN credit it's a really good practice to turn them down. Why not do that automatically!
That can be very simply done by adding a new resource in the template.

{
        "name": "[concat('autoshutdown-', 'FrankDevBox')]",
        "type": "Microsoft.DevTestLab/schedules",
        "apiVersion": "2017-04-26-preview",
        "location": "[resourceGroup().location]",
        "properties": {
            "status": "Enabled",
            "taskType": "ComputeVmShutdownTask",
            "dailyRecurrence": {
                "time": "19:00"
            },
            "timeZoneId": "UTC",
            "targetResourceId": "[resourceId('Microsoft.Compute/virtualMachines', 'FrankDevBox')]",
            "notificationSettings": {
                "status": "Enabled",
                "emailRecipient": "frank@frankysnotes.com",
                "notificationLocale": "en",
                "timeInMinutes": "30"
            }
        },
        "dependsOn": [
            "[concat('Microsoft.Compute/virtualMachines/', 'FrankDevBox')]"
        ]
    }


In Video Please!


If you prefer, I also have a video version of that post.

How to Create an Azure VM with Chocolatey


~Enjoy!


References:



How to save huge money by shutting down your VM automatically

Updated on 2018-03-14

Virtual machines (VM) are used in most solutions nowadays as a [ProcessName] server, temporary machine to run tests or make demos, and sometimes even as a development machine. One of the great benefits of the cloud is that you only pay for what you use. So unlike the old server, that you keep paying for, you won pay virtual machine's CPU for when you turned off! In this post, I explain how to do it with your existing machines and also what to do with all the future one that you will be creating.

(Ce billet en aussi disponible en français.)

Already have a VM up and running, here what to do


From the Azure portal (portal.azure.com), select the Virtual Machine (VM) that you which to edit. Then look at the option panel, on the left, for Auto-Shutdown in the Operations section. You should have something that looks like this:

auto-shutdown

At any time you can enable and disable that functionality, it won’t affect the running VM.

Now, to activate it click on the Enabled. Then Select the time you would like to see the VM shutdown. Be sure to select the good time zone, by default it’s UTC. You can adjust the at for UTC of change the time zone, both options are valid.

Now you could decide to enable the notification. That could be useful if you may want to postpone the shutdown for one or two hours, or integrate the shutdown to another process like backup, cleaning…

To activate the notification option just click on the enabled, and enter the email address. If you want to attach the shutdown to a Logic App or an Azure Functions use the webhook. Here an example of notification email, see the Postpone options link.

emailsample

What if you have many VMs running


Let's say you have already twenty (or more) VMs running, you could have executed a PowerShell script like:


$myMVsName = @("franDev1", "frankBuildserver", "demo_sales2018")

For ($i=0; $i -lt $myMVsName.Length; $i++) {     
    Set-AzureRmDtlAutoShutdownPolicy $myMVsName[$i]
    [...]
}

Update - 2018-03-14
Well, today this is only possible for VM part of a DevTest Labs. Not for "regular" VM. However, I'm sure that day will come pretty quick.Does that mean that you need to go in all your VMs and set it manually? No. You can use an Azure Automation that will stop a list of VM on a regular schedule. A big advantage of this solution is that you can be more creative since it offers a lot more flexibility. You could identify the VM to shutdown base on some TAGS, you could have a different schedule base on the week vs weekend. You could even have a task to start VMs in the morning... More to come on that topic in a future post... If you want to read about how to get started to Azure Automation click here.

Multiple VMs that already exist, no problem

Obviously, if you have multiple virtual machines that already exist it is not very efficient to change their configuration one by one via the portal. Here is a small script to change the configuration of a large amount of VM in one shot.


    '# Login-AzureRmAccount

    $Subscription = Get-AzureRmSubscription -SubscriptionName 'YOUR_SUBSCRIPTION_NAME'
    Select-AzureRmSubscription -Subscription $Subscription.Id

    $selectedVMs = Get-Azurermvm -ResourceGroupName cloud5mins
    foreach($vm in $selectedVMs) 
    { 
        $ResourceGroup = $vm.ResourceGroupName
        $vmName = $vm.Name
        $ScheduledShutdownResourceId = "/subscriptions/$Subscription/resourceGroups/$ResourceGroup/providers/microsoft.devtestlab/schedules/shutdown-computevm-$vmName"
    
        $Properties = @{}
        $Properties.Add('status', 'Enabled')
        $Properties.Add('targetResourceId', $vm.Id)
        $Properties.Add('taskType', 'ComputeVmShutdownTask')
        $Properties.Add('dailyRecurrence', @{'time'= 2100})
        $Properties.Add('timeZoneId', 'Eastern Standard Time')
        $Properties.Add('notificationSettings', @{status='Disabled'; timeInMinutes=60})

        New-AzureRmResource -Location $vm.Location -ResourceId $ScheduledShutdownResourceId -Properties $Properties -Force
    }


The variable $selectedVMs contains all the VMS that we wish to edit. In this sample, I only get VMs contained in the RessourceGroup cloud5mins, but there are no limits to what you can do. You could select all VMs with a specific OS, tags, location, name, etc.

The variable $ScheduledShutdownResourceId will be the identity for the configuration for the auto-shutdown we wish to inject. Note that the provider is microsoft.devtestlab.

Next, we create a collection of properties in $Properties. status the one that active or deactivate the auto-shutdonw. targetResourceId is the resourceID of the VM we target.

The only things left is to specify the time and timezone.

If you prefer, I also have a video version that explains all the steps.

How to shutdown automatically all your existing VMs



End Update

Let's create a VM with the auto-shutdown pre-configured with ARM


Of course, a much more efficient way to set the auto-shutdown is at the creation time by adding a new resource of type Microsoft.DevTestLab/schedules to your template. This option was previously only accessible for DevTestLab, but recently was made available to any VMs.
Here an example of the variables that could be added to your template.

"variables": {

    "ShutdowTime": "21:00",
    "TimeZone": "UTC",
    "emailRecipient": "frank@frankysnotes.com",
    "notificationLocale": "en",
    "timeInMinutes": 30
}

And here an example of Microsoft.DevTestLab/schedules resource. One of these should be added for every VM you wish to auto-shutdown. Because your script is for one server, however, only one instance is required.

{
    "name": "[concat('autoshutdown-', variables('vmName'))]",
    "type": "Microsoft.DevTestLab/schedules",
    "apiVersion": "2017-04-26-preview",
    "location": "[resourceGroup().location]",
    "properties": {
        "status": "Enabled",
        "taskType": "ComputeVmShutdownTask",
        "dailyRecurrence": {
            "time": "[variables('ShutdowTime')]"
        },
        "timeZoneId": "[variables('TimeZone')]",
        "targetResourceId": "[resourceId('Microsoft.Compute/virtualMachines', variables('vmName'))]",
        "notificationSettings": {
            "status": "Enabled",
            "emailRecipient": "[variables('emailRecipient')]",
            "notificationLocale": "[variables('notificationLocale')]",
            "timeInMinutes": "[variables('timeInMinutes')]"
        }
    },
    "dependsOn": [
        "[concat('Microsoft.Compute/virtualMachines/', variables('vmName'))]"
    ]
}

Reading Notes #310

2018

Cloud


Programming


Miscellaneous



Reading Notes #278

azure_functions_featured_imageCloud


Programming


Miscellaneous

  • Designing a Conversation (Alexandre Brisebois) - Interesting post that digs into the paradox where "us", humans have been communicated since our beginnings but still have trouble doing it, now we want to plan ahead and architect communication with machines.
  • Introduction to Microsoft To-Do (Gunnar Peipman) - Interesting app Microsoft finally did his ToDo service.



Reading Notes #264

2017-01-22_21-13-48Cloud


Programming

  • Introducing Docker 1.13 (Docker Core Engineering) - This post summarizes all the great features added in the new release and shows again why Docker is such a fantastic tool in the containers' world.

Databases


Miscellaneous


Reading Notes #243

valutoBusinessSuggestion of the week


Cloud


Programming


Miscellaneous



Reading Notes #202

 

Azure automationSuggestion of the week


Cloud


Programming


Miscellaneous



Reading Notes #188


I was really happy to ear about Microsoft Azure Data Center that will be built in Canada. And it was definitely not only good thing that was published this week...

Suggestion of the week


Cloud


Programming


Database


Miscellaneous



Reading notes #183

IMG_20150425_120539[1]Suggestion of the week


Cloud


Programming


Databases


Miscellaneous



Reading Notes #159

AzureConf2014Suggestion of the week

 

Cloud

 

Programming

 

Database

 

Miscellaneous

~Frank Boucher

Reading Notes #154

 

compare_iaas_paas_saasSuggestion of the week


Cloud


Programming


Miscellaneous


~Frank


Reading Notes #136

 

Suggestion of the week


Cloud


Programming


Miscellaneous



Reading Notes #135

 

What is DevOps_coverSuggestion of the week


Cloud


Programming


Book

  • What is DevOps? (Mike Loukides) - Interesting short and free book that gives you a historical overview of devOps.

Miscellaneous



Reading Notes #126

Cloud Architecture Patterns - CoverSuggestion of the week

Books

  • Cloud Architecture Patterns (Bill Wilder) - Very instructive books that explains many different patterns with clear and practical examples. All the patterns presented are also implemented in an application Page of Photos (or PoP for short). A great book that I strongly recommend.

 

Cloud


Programming

  • Code Kata - I love it. I didn't know those kind of websides exists! I will make my visit.

Miscellaneous

~Frank


Reading Notes #111

Cloud


Programming


Database


Miscellaneous


~Frank




Reading Notes #110

 

CloudyFallSuggestion of the week

Cloud

Programming


Miscellaneous

  • 10 best new features in Excel 2013 (Susan Harkins) - Interesting, go you know your Excel?

  • Writing About Code: Structure - Second part of a very interesting article about how to write. This
    one is all about get our writing in a good shape. Because well structured our post will be easier to read, understand and share.

~Frank



How Windows Azure Simplified my Development

(This post was originaly published on Matricis Blog)

Usually my posts are a lot more technical, but this time I decided to share an experience with Windows Azure that saved me a lot of headaches while saving my boss a bunch of money ;)

The Context

Here at Matricis, we often set up our development environments on virtual machines (VMs) witch we host on our internal infrastructure. We have several different development VM configurations, based on the technologies and versions needed. A big advantage to doing so is that if the required environment changes, we simply choose the corresponding VM template. For the project I'm about to talk about, we needed quite a powerful development system, especially since every developer required Visual Studio 2012, SharePoint Foundation 2013, SQL Server, ADFS, and a handful of other tools (fiddler, notepad++, different browsers, etc.)

The Problem

To be able to develop with SharePoint, it is strongly recommended to have at least 8 gigabytes of RAM (I first tried with just 6 gigs, but it was still a nightmare). My laptop only has 8 gigs of memory, so I couldn't run the VM locally. I asked our IT guys if it was possible to host the VM on a local on-premise server. They answered that they didn't have enough space for the environment. They were very sorry, but I was actually quite happy about it; I now had a perfect use-case to work in Windows Azure!

The Solution

I went to see my boss and explained the situation: instead of buying a brand new server for development and test environment purposes, we should simply use Windows Azure’s IaaS! We could start setting up the VM in less than 10 minutes. In an hour we would be ready to code! The development VM would only be up while it was in use, meaning that it wouldn’t cost a cent while nobody was working on it. On project completion… we would delete all the VMs we were using, and no more fees! My boss loved the idea!

The core team for this project involved four full-time developers, and here is a high level look at our development environment: The Active Directory is shared but every developer has their own SharePoint, Sql Server, and ADFS making them autonomous.

Development environment


As you can see, it's a hybrid environment since the Team Foundation Server (TFS) is on one our local servers. In the morning, I start my VM and within a few minutes, I'm connected remotely and I’m ready to work on a great machine. With a little PowerShell script that I wrote, I don't even need to log in to the Azure Portal to start and stop my VM. Another great joy to this scenario is that I can now work from anywhere and on any kind of machine: from home on the family computer without VPN or from a Hotel on my laptop or my Surface Pro! Happiness often comes from simple things.

In general, I would say that the experience was very positive, but on the road we did encounter some issues that we had to resolve. Since all IPs on Azure are dynamic, we discovered that the domain controller that is hosted in Azure, must be started before the other VMs. This way its IP will always be the first one, therefore the other VMs will find it without any issues. Furthermore, in our architecture, the source-control (TFS) is on-premise. When you check-out or check-in your code, you are passing through the firewall. However, since these actions are intensive, the firewall may interpret the activity as attacks.

Because the job of a firewall is to protect your network, you can imagine what happened... the connection was lost. Once we identified this and created Firewall exceptions for the Azure VMs, everything was good.

In Conclution

I hope this post will encourage you to try Windows Azure as a development and test environment, because it's a really effective and cost-beneficial way to execute on different projects. For more information about the Windows Azure Infrastructure as a Service go to the Windows Azure Web Site.


~Frank


Reading Notes #95

Cloud_hammorkSuggestion of the week

Reading Notes #94

cloud_sheepSuggestion of the week

Cloud

Programming

Databases

Integration

Miscellaneous

~Frank