Isolating Azure Storage Accounts for greater Virtual Machine resiliency

In my day to day role as an Azure Solution Architect I get involved in some pretty substantial and very complex deployments for customers which requires a lot of planning and design work. One thing I have found, especially in this new cloud world is that there are about a dozen ways to solve the customer’s problem and each one would be technically right. It typically comes down to what gives the customer the best solution without breaking the bank.

One of the more complex issues I’ve found when working on IaaS deployments with a large number of virtual machines is ensuring that the storage account design is sound. One of the accepted practices is to group common VM tiers into shared storage accounts and to of course place each VM into an availability set to ensure that they fall into Microsoft’s 99.95% SLA. Digging a bit deeper this practice isn’t as resilient as one might think. Sure having VMs in an availability set spreads the VMs across separate fault and update domains, but what about storage? If I have both of my VMs in the same storage account and the underlying storage is unavailable then what happens?

Is this right? Do I need to place each VM into its own storage account for greater resiliency?After doing a bit of research I found this great article: https://azure.microsoft.com/en-us/documentation/articles/resiliency-high-availability-checklist/

In particular this section caught my attention: (I have highlighted the key points)

Are you using premium storage and separate storage accounts for each of your virtual machines?

It is a best practice to use premium storage for your production virtual machines. In addition, you should make sure that you use a separate storage account for each virtual machine (this is true for small-scale deployments. For larger deployments you can re-use storage accounts for multiple machines but there is a balancing that needs to be done to ensure you are balanced across update domains and across tiers of your application).

So it seems premium storage and separate storage accounts are the way to go. Things get even more interesting. Read on…

Not only should you use premium storage and separate storage accounts for your VMs you need to name the storage accounts following a specific naming convention or you run the risk of the storage partitions being potentially co-located on the same partition server. That caught my attention. Luckily I was sent this article: https://azure.microsoft.com/en-us/documentation/articles/storage-performance-checklist/ and the section that really cleared everything up for me was this:

Partition Naming Convention

…naming conventions such as lexical ordering (e.g. msftpayroll, msftperformance, msftemployees, etc) or using time-stamps (log20160101, log20160102, log20160102, etc) will lend itself to the partitions being potentially co-located on the same partition server, until a load balancing operation splits them out into smaller ranges.
You can follow some best practices to reduce the frequency of such operations.

  • Examine the naming convention you use for accounts, containers, blobs, tables and queues, closely. Consider prefixing account names with a 3-digit hash using a hashing function that best suits your needs.
  • If you organize your data using timestamps or numerical identifiers, you have to ensure you are not using an append-only (or prepend-only) traffic patterns. These patterns are not suitable for a range -based partitioning system, and could lead to all the traffic going to a single partition and limiting the system from effectively load balancing. For instance, if you have daily operations that use a blob object with a timestamp such as yyyymmdd, then all the traffic for that daily operation is directed to a single object which is served by a single partition server.

So from the above information it seems that the following holds true:

1) Use Premium storage in conjunction with separate storage accounts. This gets around any IOPS limits per storage account as well but there is a limit of 200 storage accounts per subscription which is a hard limit.

2) Prefix your storage accounts with a random 3 digit hash per storage account to ensure that the storage accounts are properly spread across load balanced partition servers. For example naming your storage accounts storageaccount1, storageaccount2 isn’t sufficient. Go with something like fxwstorage1, bcdstorage2 etc. to ensure that the storage accounts are load balanced correctly. Luckily for us we can use ARM templates to provision storage accounts using the naming convention as mentioned above, but that’s for another post….

Advertisements

About Aaron Saikovski

Aaron Saikovski is a Microsoft certified technology specialist, consultant, developer, trainer and cloud solution architect with over 20+ years commercial information technology experience. He has consulted and worked with an incredibly diverse number of both local and international clients across the banking and financial services, utilities, oil and mining, telecommunications, logistics, construction/property, pharmaceutical, transport/airline and FMCG sectors. He has also consulted on a variety of projects to both Federal and State governments in high-profile portfolios such as Prime Minister and Cabinet, Department of Defence, Taxation Office, Department of Education and Training, Department of Employment & Workplace Relations and the Department of Health. Aaron’s skills revolve around the Microsoft platform stack with a core skillset based on the SharePoint platform. He has been involved in some of the largest SharePoint deployments in the world and was a member of the Global SharePoint Ranger team within Microsoft Corporation and was actively involved with the SharePoint Customer Advisory Team (SPCAT) to provide guidance and best practices to the largest Microsoft enterprise global customers. He was also an active MSDN article reviewer and contributor during his time at Microsoft. He has very broad and deep experience with architecting, developing and deploying complex customer solutions leveraging the Microsoft Office365 and Azure cloud platforms. Some of the key cloud customers he has been involved with are Qantas, Caltex, and Fujitsu/Westpac to name a few. He assists customers with their journey to the cloud and provides guidance and best practices to assist in identifying workloads that are suitable candidates to be migrated or redeveloped. These include IaaS, SaaS and PaaS solutions that solve a particular business requirement while providing the best technical outcome for the customer. Aaron is a regular presenter at such events as SharePoint Saturday (http://www.spsevents.org), Microsoft TechEd and is a regular presenter at user groups and conferences. He presents on a wide and varied range of topics that encompass the complete end to end Microsoft solution stack such as Windows 8 App development through to deploying Azure IaaS solutions with PowerShell. Aaron has an active social media profile. He was the founding member of OzMoss (http://www.codify.com/lists/OzMOSS), a very active and prominent online SharePoint community and he can be followed on Twitter at @RuskyDuck72. He blogs regularly at https://aaronsaikovski.wordpress.com/ Aaron is also an active affiliate member of the Institute of Electrical and Electronics Engineers (www.ieee.org) Outside of work Aaron is a very active volunteer for DoggieRescue (www.doggierescue.com) where he volunteers his time to maintain and improve their IT systems to keep them running as optimally and efficiently as possible. ** OPINIONS EXPRESSED ARE MY OWN VIEWS AND NOT THOSE OF MY EMPLOYER **
This entry was posted in Microsoft Azure, Storage, storage accounts, VMs. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s