Auto scaling worker roles based on Dynamics CRM workloads

Dinesh Niroshan, 11 August 2014

Recently we needed to develop an application to synchronize data between Dynamics CRM 2013 and a financial system. This application has become complex due to the volume of data as well as the requirement in getting the data synchronized as fast as possible while minimizing the costs of running virtual machines.

So we decided to develop this application as Azure Worker Roles. Following diagram shows the rough solution architecture.

Our target was to increase the number of instances according to the number of sync tasks that were generated by Dynamics CRM.

Keeping fixed number of instances running would work, for example if there are 10 Dynamics CRM 2013 instances to be synched we can create 10 instances of “Sync Task Worker role” which would run and clear the queue instantly, but this is not cost effective.

Configuring Auto Scaling
There are three ways to enable auto scaling. Those are Scale by Schedule, By CPU Usage or Based on Queue items.

For this solution we decided to go for auto scaling based on Queue and the way it is configured in Azure Admin is as follows.

Log in to Azure admin and go to cloud service where the worker roles exist. And then click “SCALE” Link.



Find the worker role you need to configure for auto scaling. Select “QUEUE” for “SCALE MY METRIC”.



Decide how many maximum and minimum Instances you need to create and enter it in “INSTANCE RANGE”

Select the Storage Account which the Queue located for “ACCOUNT OR NAMESPACE”

Select Queue name for “QUEUE NAME”. This is the queue we are going to use for auto scaling based on queue items.

And then enter no of Queue items going to consume by a one worker role instance. For example if there are 10 queue items in the queue and target per machine is 2, up to 5 instances (10 / 2) will be created.



Now we need to set how many instances scale up or down by at a time. In This case we set by one instance at a time for sale up or down, and 5 minutes (this is the minimum) to sale up or down waiting time.

All configurations are done now and finally save the changes.




Time require to start working new instance.
To start a new instance take around 10 minutes. 5 minutes start new instance creation and another 5 minutes to start new instance (VM).