Skip to main content
Introduction

Uncle Ben to Spiderman: “With great power comes great responsibility.”

We can relate to this sentiment within our field. So for all system administrators and engineers who have the great power of operating within the cloud comes great responsibility too – which often is in terms of COST. The idea of spinning up infrastructure whenever needed looks very attractive and easy because of the cloud. You don’t pay any CAPEX and OPEX cost in the cloud; you only pay for what you use. However, uncontrolled and unmonitored usage of resources results in extra cost.

azure on-duty: automatically shut down vm

Most cloud providers charge compute resources on an hourly basis. Unlike others, in Azure, compute resources, like VMs, are charged per minute. As such, it is very important to stop and deallocate unused VMs. With Azure, we can benefit from this pricing model by auto-shutting down VMs when not in use. The number of ways we can do this is as follows:

  1. Using Azure DevTest Labs
  2. Using the auto-shutdown feature in individual VMs
  3. Using Azure Automation

Azure DevTest Labs is an amazing service in Azure which allows you to quickly provision Windows and Linux VMs for development and testing environments. With DevTest Labs, you can set quotas and limits on the number and size of VMs per lab to minimize waste. The automated shutdown feature of DevTest Labs also helps in saving cost by shutting down VMs when not in use.

The auto-shutdown feature can also be used to shut down individual VMs which are not included in DevTest Labs.

Azure Automation is a service that can be used to automate manual, long-running, and repetitive processes. It saves time and effort and increases the reliability of everyday administrative tasks. It can also be used to schedule them automatically at regular intervals. Runbooks can be used to automate these processes or automate configuration management using Desired State Configuration. One such task is the ability to shut down VMs on schedule. Runbooks can be used to shut down VMs in particular resource groups or specifically-tagged VMs.

Here we will quickly go through the auto-shut down feature for VMs and also check how to use Azure Automation and runbooks to create and destroy an HDInsight cluster on schedule.

To use the auto-shutdown feature for VMs, follow these steps:

1. Navigate to the VM settings blade and click on Auto-shutdown under Schedule.

user_60255_5976d5d3442c6.PNG
2. Enable it, set your scheduled time for shutdown, and select the time zone. If you want a notification 15 minutes before auto-shutdown, provide your webhook URL. Click Save.
user_60255_5976d62b483f3.PNG
Similarly, if you want to use the auto-shutdown feature in DevTest Labs, you have the option to enable or disable it during creation. However, we can change this setting later. By default, it will be enabled for new labs we create.

  1. To create a new DevTest Lab, click on + New > Developer Tools > DevTest Labs.user_60255_5976d8bf02853.PNG
  2. Enter the lab name, then select the region and subscription. Click on Auto-shutdown and set an appropriate time to enable the auto-shutdown of the DevTest Lab VMs on that particular schedule and click OK. Click Create to create lab.user_60255_5976d916dc97e.PNG
If you want to change the auto-shutdown schedule for a DevTest Lab, navigate to the Lab Settings blade and click on Configuration and Policies. Select Auto-shutdown, under Schedules, and change the scheduled shutdown time.
Scenario

Consider the scenario where you work as sysadmin for a finance company. The company deals with humongous amounts of data. They make use of Azure cloud platform for running their application workloads. The company uses HDInsight to generate financial reports of their customers at the end of the month. Their main reason for using an HDInsight cluster is that it uses Azure Blob Storage or Azure Data Lake Store to store files. This separation of storage from Compute services allows you to scale with ease. Also, the files that cluster produced or consumed will remain even when the cluster is deleted.

The company used Linux-based HDInsight clusters during last two days of the month. You only pay for the HDInsight cluster when it’s running, so the idea of deleting the cluster after use helps save a good amount of money. They plan to automate this repetitive task to make life easy. As sysadmin, you plan to use Azure Automation to automate the process of creating and destroying the HDInsight cluster.

We shall make use of a PowerShell script to create and destroy the HDInsight cluster.

Create and Delete an HDInsight Cluster using Azure Automation

To automatically create and destroy the HDInsight cluster on schedule using Azure Automation, follow the steps below:

Creating and Configuring an Azure Automation Account

  1. To create an Automation Account, click on New > Monitoring and Management > Automation. Enter the Automation account name, select subscription, choose the resource group (if any) and region, then click Create.
  2. To manage the HDInsight cluster from Azure Automation, an HDInsight module is required. To add the HDInsight module, navigate to the Automation account blade, and click on Assets, under Resources.user_60255_5976e084085c5.PNG
  3. Click on Modules tile, which will display the list of currently installed modules. Click on Browse Gallery to find and import the AzureRM.Profile and AzureRM.HDInsight modules. Click on the module and click Import. Note that AzureRM.HDInsight has a dependency on AzureRM.Profile, so it should be imported first.
    user_60255_5976e18ac1dd7.PNGuser_60255_5976e1b29b88d.PNG
Once both the modules are imported, we can create and store Credentials and Variables in the Automation service. Storing variables and credentials is an easy and secure way to manage them without hardcoding it in scripts or runbooks. These objects, post creation, can be referenced in one or more scripts.

Creating Credentials and Variables

1. We will create a variable for Azure Subscription which will be used in the script to launch the HDInsight cluster. Under the Assets blade, click on Variables. Click Add a Variable and enter the details as shown below.

user_60255_5976e4b3e42fe.PNG
2. Next, we will create credentials for our script. We will need to create three credentials: Azure credentials to connect to Azure from the script, a cluster username and password to connect to the cluster, and SSH credentials to SSH into the cluster. Click on Credentials and click Add a credential. Create two credentials: cluster-admin and ssh-user.

user_60255_5976e620aeffa.PNG
We now need one more to login to Azure Account from the runbook. AzureRM.Profile doesn’t allow default Microsoft Account to login directly from a runbook, so a new user needs to be created in Azure AD and given the Contributor role on Subscription. Here we will use another possibility i.e. AzureRunAsConnection for authentication.

Azure Run As Account provides a standardized way to authenticate with Azure when you’re managing Azure RM resources using a runbook.

user_60255_5976e7d7c9bd3.PNG

Creating Runbooks to Create and Delete an HDInsight Cluster
Create HDInsight Cluster

Now, to use the variable and credentials created in previous steps, we will use Get-AutomationPSCredential and Get-AutomationVariable cmdlets in our PowerShell script. Let’s first write a PowerShell script to create the HDInsight cluster.

1. We should first authenticate to Azure and select a default subscription to create the cluster. This can be done using Azure Run As Account as shown below.

### Authenticate to Azure 
$Conn = Get-AutomationConnection -Name AzureRunAsConnection

Add-AzureRMAccount -ServicePrincipal -Tenant $Conn.TenantID -ApplicationId $Conn.ApplicationID -CertificateThumbprint

### Selecting subscription
$subscriptionID = Get-AutomationVariable –Name 'subscription-id' # Automation variable for Subscription ID

Select-AzureRmSubscription –SubscriptionId $subscriptionID

2. The next thing is to set cluster variables such as resource group, the number of nodes, node size, etc.

### Set cluster variables
$resourceGroup = "<Existing_Resource_Group>"
$storageAccount = "<Existing_Storage_Account>"
$containerName = "<Existing_Container_Name>"

3. A storage account key is required for this script to create and access resources in the Storage account. Instead of specifying static storage account key, we will get it using PowerShell.

$storageAccountKey = (Get-AzureRmStorageAccountKey –Name $storageAccount –ResourceGroupName $resourceGroup)[0].value

4. We can now set the remaining cluster variables and credentials.

$clusterName = "<Cluster_Name>" 

### Setting cluster credentials
# Automation credential for Cluster Admin
$clusterCreds = Get-AutomationPSCredential –Name 'cluster-admin'

# Automation credential for user to SSH into cluster
$sshCreds = Get-AutomationPSCredential –Name 'ssh-user'

# Use any supported cluster type (Hadoop, HBase, Storm, etc.)
$clusterType = "Hadoop"

$clusterOS = "Linux"
$clusterWorkerNodes = 3
$clusterNodeSize = "Standard_D1_v2"
$location = Get-AzureRmStorageAccount –StorageAccountName $storageAccount –ResourceGroupName $resourceGroup | %{$_.Location}

5. We will use all the variables defined above, along with New-AzureRMHDInsightCluster cmdlet, to provision the HDInsight cluster with declared variables.

### Provision HDInsight cluster

New-AzureRmHDInsightCluster –ClusterName $clusterName –ResourceGroupName $resourceGroup –Location $location –DefaultStorageAccountName "$storageAccount.blob.core.windows.net" –DefaultStorageAccountKey $storageAccountKey -DefaultStorageContainer $containerName –ClusterType $clusterType –OSType $clusterOS –Version "3.5" –HttpCredential $clusterCreds –SshCredential $sshCreds –ClusterSizeInNodes $clusterWorkerNodes –HeadNodeSize $clusterNodeSize –WorkerNodeSize $clusterNodeSize
6. Paste this code, in sequence, in a single file and save it with .ps1 extension. We will import this file to create a runbook out of it to create the HDInsight cluster.

7.Click on Runbook on the Automation Account blade. Click Add a runbook and choose Import an existing runbook. Choose the .ps1 file created above, set Runbook type as PowerShell, and give it an appropriate name and description. Click Create.

user_60255_5976f32570631.PNG

8.Click on the new runbook. Click Edit and verify the runbook. Once done, test it using Test Pane. You should see “Completed” in green without any errors in Test Pane before publishing the runbook. Click Publish.

user_60255_5976f38e21538.PNGuser_60255_5976f3a66ba3e.PNG_800.jpg

Once the runbook is published, you can click Start to run it immediately or click Schedule to run it on schedule. In our case, we can schedule it during the last days of the month; i.e., 29th or 30th of every month.

9. To schedule a runbook, click on Schedule > Add a Schedule. Select Link a schedule to your runbook and set appropriate options as shown below.
user_60255_5976f44d07c79.PNG
Once done, you’ll see the schedule as shown below:
user_60255_5976f47ca4880.PNG
Delete HDInsight cluster

Now that we created a runbook to create the HDInsight cluster, we have to create a runbook to delete the HDInsight cluster. Note that in our scenario, we will just delete the HDInsight cluster and not the other connected resources, such as the Storage Account and resource group.

In the above runbook, we used AzureRunAsAccount to authenticate to Azure. Here we will use the user from Azure AD to authenticate to Azure.

Create the azure-admin credential in your Automation account as described in step 4 in the previous task. We will use Get-AutomationPSCredential to get the value of ‘azure-admin’ in the script.

This script will have two main parts:

  1. Authenticate to Azure and select the subscription.
  2. Delete the HDinsight cluster.
The script will be as given below:
### Authenticate to Azure
# Automation Credential for Azure Login
$azureCreds = Get-AutomationPSCredential –Name 'azure-admin'

Login-AzureRmAccount –Credential $azureCreds

### Selecting subscription
# Automation variable for Subscription ID
$subscriptionID = Get-AutomationVariable –Name 'subscription-id'

Select-AzureRmSubscription –SubscriptionId $subscriptionID

### Deleting the HDInsight cluster
Remove-AzureRmHDInsightCluster –ClusterName "<Cluster_Name>"

Save the above code in a file with the .ps1 extension and import this file to create a new runbook, delete-cluster, as done in the previous task. Test and publish the runbook, then schedule it to run on last day of the month.

Thus, we have created two runbooks; one will create an HDInsight cluster and other will delete the HDInsight cluster. We set it to run on a particular schedule. The following improvements can be made in the above scripts:

  • Automate the creation of the Resource Group and Storage Account (one-time process)
  • Create the cluster-name variable and use the Get-AutomationVariable cmdlet in the create-cluster and delete-cluster runbooks to avoid providing cluster name every time.

Sources / Resources

Feel free to download required PowerShell scripts/runbooks for this guide from GitHub. Explore Azure courses in Linux Academy and get certified. Happy learning! Stay tuned.

Comments are disabled for this guide.