Category Archives: azure

Developing an app on Microsoft Azure: a few quick reflections

I have recently completed (if applications are ever completed) an application which runs on Microsoft’s Azure platform. I used lots of Microsoft technology:

  • Visual Studio 2013
  • Visual Studio Online with Team Foundation version control
  • ASP.NET MVC 4.0
  • Entity Framework 4.0
  • Azure SQL
  • Azure Active Directory
  • Azure Web Sites
  • Azure Blob Storage
  • Microsoft .NET 4.5 with C#

The good news: the app works well and performance is good. The application handles the upload and download of large files by authorised users, and replaces a previous solution using a public file sending service. We were pleased to find that the new application is a little faster for upload and download, as well as offering better control over user access and a more professional appearance.

There were some complications though. The requirement was for internal users to log in with their Office 365 (Azure Active Directory) credentials, but for external users (the company’s customers) to log in with credentials stored in a SQL Server database – in other words, hybrid authentication. It turns out you can do this reasonably seamlessly by implementing IPrincipal in a custom class to support the database login. This is largely uncharted territory though in terms of official documentation and took some effort.

Second, Microsoft’s Azure Active Directory support for custom applications is half-baked. You can create an application that supports Azure AD login in a few moments with Visual Studio, but it does not give you any access to metadata like to which security groups the user belongs. I have posted about this in more detail here. There is an API of course, but it is currently a moving target: be prepared for some hassle if you try this.

Third, while Azure Blob Storage itself seems to work well, most of the resources for developers seem to have little idea of what a large file is. Since a primary use case for cloud storage is to cover scenarios where email attachments are not good enough, it seems to me that handling large files (by which I mean multiple GB) should be considered normal rather than exceptional. By way of mitigation, the API itself has been written with large files in mind, so it all works fine once you figure it out. More on this here.

What about Visual Studio? The experience has been good overall. Once you have configured the project correctly, you can update the site on Azure simply by hitting Publish and clicking Next a few times. There is some awkwardness over configuration for local debugging versus deployment. You probably want to connect to a local SQL Server and the Azure storage emulator when debugging, and the Azure hosted versions after publishing. Visual Studio has a Web.Debug.Config and a Web.Release.Config which lets you apply a transformation to your main Web.Config when publishing – though note that these do not have any effect when you simply run your project in Release mode. The correct usage is to set Web.Config to what you want for debugging, and apply the deployment configuration in Web.Release.Config; then it all works.

The piece that caused me most grief was a setting for <wsFederation>. When a user logs in with Azure AD, they get redirected to a Microsoft site to log in, and then back to the application. Applications have to be registered in Azure AD for this to work. There is some uncertainty though about whether the reply attribute, which specifies the redirection back to the app, needs to be set explicitly or not. In practice I found that it does need to be explicit, otherwise you get redirected to the deployed site even when debugging locally – not good.

I have mixed feelings about Team Foundation version control. It works, and I like having a web-based repository for my code. On the other hand, it is slow, and Visual Studio sulks from time to time and requires you to re-enter credentials (Microsoft seems to love making you do that). If you have a less than stellar internet connection (or even a good one), Visual Studio freezes from time to time since the source control stuff is not good at working in the background. It usually unfreezes eventually.

As an experiment, I set the project to require a successful build before check-in. The idea is that you cannot check in a broken build. However, this build has to take place on the server, not locally. So you try to check in, Visual Studio says a build is required, and prompts you to initiate it. You do so, and a build is queued. Some time later (5-10 minutes) the build completes and a dialog appears behind the IDE saying that you need to reconcile changes – even if there are none. Confusing.

What about Entity Framework? I have mixed feelings here too, and have posted separately on the subject. I used code-first: just create your classes and add them to your DbContext and all the data access code is handled for you, kind-of. It makes sense to use EF in an ASP.NET MVC project since the framework expects it, though it is not compulsory. I do miss the control you get from writing your own SQL though; and found myself using the SqlQuery method on occasion to recover some of that control.

Finally, a few notes on ASP.NET MVC. I mostly like it; the separation between Razor views (essentially HTML templates into which you pour your data at runtime) and the code which implements your business logic and data access is excellent. The code can get convoluted though. Have a look at this useful piece on the ASP.NET MVC WebGrid and this remark:

grid.Column("Name",
  format: @<text>@Html.ActionLink((string)item.Name,
  "Details", "Product", new { id = item.ProductId }, null)</text>),

The format parameter is actually a Func, but the Razor view engine hides that from us. But you’re free to pass a Func—for example, you could use a lambda expression.

The code works fine but is it natural and intuitive? Why, for example, do you have to cast the first argument to ActionLink to a string for it to work (I can confirm that it is necessary), and would you have worked this out without help?

I also hit a problem restyling the pages generated by Visual Studio, which use the twitter Bootstrap framework. The problem is that bootstrap.css is a generated file and it does not make sense to edit it directly. Rather, you should edit some variables and use them as input to regenerate it. I came up with a solution which I posted on stackoverflow but no comments yet – perhaps this post will stimulate some, as I am not sure if I found the best approach.

My sense is that what ASP.NET MVC is largely a thing of beauty, it has left behind more casual developers who want a quick and easy way to write business applications. Put another way, the framework is somewhat challenging for newcomers and that in turn affects the breadth of its adoption.

Developing on Azure and using Azure AD makes perfect sense for businesses which are using the Microsoft platform, especially if they use Office 365, and the level of integration on offer, together with the convenience of cloud hosting and anywhere access, is outstanding. There remain some issues with the maturity of the frameworks, ever-changing libraries, and poor or confusing documentation.

Since this area is strategic for Microsoft, I suggest that it would benefit the company to work hard on pulling it all together more effectively.

A note on Azure storage and downloading large files

I have written a simple ASP.NET MVC application for upload and download of files to/from Azure storage.

Getting large file upload to work was the first exercise, described here. That is working well; but what about download?

If your files in Azure storage are public, you can simply serve an URL to the file. If it is not public though, you have a couple of choices:

1. Download the file under application control, by writing to Response.OutputStream or using a FileResult action.

2. Issue a Shared Access Signature (SAS) to the client which enables it to retrieve the file directly from Azure storage. The SAS is sent as an URL argument which tells Azure storage that the request is authorised. The browser downloads the file directly, so it makes no difference to your web application if the file is large.

Note that if you use the first option, it will not work with large files if you simply call DownloadToStream or similar:

container.GetBlockBlobReference(FileName).DownloadToStream(Response.OutputStream);

Why not? Well, the way this code works is that it downloads the large file to the web server, then sends it to the browser. What if your large file is 5GB? The browser will wait a long time for the first byte to be served (giving the user an unresponsive page); but before that happens, the web application will probably throw an exception because it does not like downloading such a large file.

This means the SAS option is a good one, though note that you have to specify an expiry time which could cause problems for users on a slow connection.

Another option is to serve the file in chunks. Use CloudBlockBlob.DownloadRangeToStream to write to Response.OutputStream in a loop until the download is complete. Call Response.Flush() after each chunk to send the chunk to the browser immediately.

This gives the user a nice responsive download experience complete with a cancel option as provided by the browser, and does not crash the application on the server. It seems to me a reasonable approach if the web application is also hosted on Azure and therefore has a fast connection to Azure storage.

What about resuming a failed download? The SAS approach should work as Azure supports it. You could also support this in your app with some additional work since Resume means reading the Range header in a GET request. I have not tried doing this but you might find some clues here.

Developing an ASP.NET MVC app with Azure Active Directory: an ordeal

Regular readers will know that I am working on a simple (I thought) ASP.NET MVC application which is hosted on Azure and uses Azure Blob Storage.

So far so good; but since this business uses Office 365 it seemed to me logical to have users log in using Azure Active Directory (AD). Visual Studio 2013, with the latest update, has a nice wizard to set this up. Just complete the following dialog when starting your new project:

image

This worked fairly well, and users can log in successfully using Azure AD and their normal Office 365 credentials.

I love this level of integration and it seems to me key and strategic for the Microsoft platform. If an employee leaves, or changes role, just update Active Directory and all application access comes into line automatically, whether on premise or in the cloud.

The next stage though was to define some user types; to keep things simple, let us say we have an AppAdmin role for users with full access to the application, and an AppUser role for users with limited access. Other users in the organisation do not need access at all and should not be able to log in.

The obvious way to do this is with AD groups, but I was surprised to discover that there is no easy way to discover to which groups an AD user belongs. The Azure AD integration which the wizard generates is only half done. Users can log in, and you can programmatically retrieve basic information including the firstname, lastname, User Principal Name and object ID, but nothing further.

Fair enough, I thought, there will be some libraries out there that fill the gap; and this is how the nightmare begins. The problem is that this is the cutting edge of .NET cloud development and is an area of rapid change. Yes there are samples out there, but each one (including the official ones on MSDN) seems to be written at a different time, with a different approach, with different .NET assembly dependencies, and varying levels of alpha/beta/experimental status.

The one common thread is that to get the AD group information you need to use the Graph API, a REST API for querying and even writing to Azure Active Directory. In January 2013, Microsoft identity expert Vittorio Bertocci (Principal Program Manager in the Windows Azure Active Directory team at Microsoft) wrote a helpful post about how to restore IsInRole() and [Authorize] in ASP.NET apps using Azure AD – exactly what I wanted to do. He describes essentially a manual approach, though he does make use of a library called Azure Authentication Library (AAL) which you can find on Nuget (the package manager for .NET libraries used by Visual Studio) described as a Beta.

That would probably work, but AAL is last year’s thing and you are meant to use ADAL (Active Directory Authentication Library) instead. ADAL is available in various versions ranging from 1.0.3 which is a finished release, to 2.6.2 which is an alpha release. Of course Bertocci has not updated his post so you can use the obsolete AAL beta if you dare, or use ADAL if you can figure out how to amend the code and which version is the best/safest to employ. Or you can write your own wrapper for the Graph API and bypass all the Nuget packages.

I searched for a better sample, but it gets worse. If you browse around MSDN you will probably come across this article along with this sample which is a Task Tracker application using Azure AD, though note the warnings:

NOTE: This sample is outdated. Its technology, methods, and/or user interface instructions have been replaced by newer features. To see an updated sample that builds a similar application, see WebApp-GraphAPI-DotNet.

Despite the warnings, the older sample is widely referenced in Microsoft posts like this one by Rick Anderson.

OK then, let’s look at at the shiny new sample, even though it is less well documented. It is called WebApp-GraphAPI-DotNet and includes code to get the user profile, roles, contacts and groups from Azure AD using the latest Graph API client: Microsoft.Azure.ActiveDirectory.GraphClient. This replaces an older effort called the GraphHelper which you will find widely used elsewhere.

If you dig into this new sample though, you will find a ton of dependencies on pre-release assemblies. You are not just dealing the Graph API, but also with OWIN (Open Web Interface for .NET), which seems to be Microsoft’s current direction for communication between web applications.

After messing around with Nuget packages and trying to get WebApp-GraphAPI-DotNet working I realised that I was not happy with all this preview code which is likely to break as further updates come along. Further, it does far more than I want. All I need is actually contained in Bertocci’s January 2013 post about getting back IsInRole.

I ended up patching together some code using the older GraphHelper (as found in the obsolete Task Tracker application) and it is working. I can now use IsInRole based on AD groups.

This is a mess. It is a simple requirement and it should not be necessary to plough through all these complicated and conflicting documents and samples to achieve it.

Notes from the field: putting Azure Blob storage into practice

I rashly agreed to create a small web application that uploads files into Azure storage. Azure Blob storage is Microsoft’s equivalent to Amazon’s S3 (Simple Storage Service), a cloud service for storing files of up to 200GB.

File upload performance can be an issue, though if you want to test how fast your application can go, try it from an Azure VM: performance is fantastic, as you would expect from an Azure to Azure connection in the same region.

I am using ASP.NET MVC and thought a sample like this official one, Uploading large files using ASP.NET Web API and Azure Blob Storage, would be all I needed. It is a start, but the method used only works for small files. What it does is:

1. Receive a file via HTTP Post.

2. Once the file has been received by the web server, calls CloudBlob.UploadFile to upload the file to Azure blob storage.

What’s the problem? Leaving aside the fact that CloudBlob is deprecated (you are meant to use CloudBlockBlob), there are obvious problems with files that are more than a few MB in size. The expectation today is that users see some sort of progress bar when uploading, and a well-written application will be resistant to brief connection breaks. Many users have asynchronous internet connections (such as ADSL) with slow upload; large files will take a long time and something can easily go wrong. The sample is not resilient at all.

Another issue is that web servers do not appreciate receiving huge files in one operation. Imagine you are uploading the ISO for a DVD, perhaps a 3GB file. The simple approach of posting the file and having the web server upload it to Azure blob storage introduces obvious strain and probably will not work, even if you do mess around with maxRequestLength and maxAllowedContentLength in ASP.NET and IIS. I would not mind so much if the sample were not called “Uploading large files”; the author perhaps has a different idea of what is a large file.

Worth noting too that one developer hit a bug with blobs greater than 5.5MB when uploaded over HTTPS, which most real-world businesses will require.

What then are you meant to do? The correct approach, as far as I can tell, is to send your large files in small chunks called blocks. These are uploaded to Azure using CloudBlockBlob.PutBlock. You identify each block with an ID string, and when all the blocks are uploaded, called CloudBlockBlob.PutBlockList with a list of IDs in the correct order.

This is the approach taken by Suprotim Agarwal in his example of uploading big files, which works and is a great deal better than the Microsoft sample. It even has a progress bar and some retry logic. I tried this approach, with a few tweaks. Using a 35MB file, I got about 80 KB/s with my ADSL broadband, a bit worse than the performance I usually get with FTP.

Can performance be improved? I wondered what benefit you get from uploading blocks in parallel. Azure Storage does not mind what order the blocks are uploaded. I adapted Agarwal’s sample to use multiple AJAX calls each uploading a block, experimenting with up to 8 simultaneous uploads from the browser.

The initial results were disappointing. Eventually I figured out that I was not actually achieving parallel uploads at all. The reason is that the application uses ASP.NET session state, and IIS will block multiple connections in the same session unless you mark your ASP.NET MVC controller class  with the SessionStateBehavior.ReadOnly attribute.

I fixed that, and now I do get multiple parallel uploads. Performance improved to around 105 KB/s, worthwhile though not dramatic.

What about using a Windows desktop application to upload large files? I was surprised to find little improvement. But can parallel uploading help here too? The answer is that it should happen anyway, handled by the .NET client library, according to this document:

If you are writing a block blob that is no more than 64 MB in size, you can upload it in its entirety with a single write operation. Storage clients default to a 32 MB maximum single block upload, settable using the SingleBlobUploadThresholdInBytes property. When a block blob upload is larger than the value in this property, storage clients break the file into blocks. You can set the number of threads used to upload the blocks in parallel using the ParallelOperationThreadCount property.

It sounds as if there is little advantage in writing your own chunking code, except that if you just call the UploadFromFile or UploadFromStream methods of CloudBlockBlob, you do not get any progress notification event (though you can get a retry notification from an OperationContext object passed to the method). Therefore I looked around for a sample using parallel uploads, and found this one from Microsoft MVP Tyler Doerksen, using C#’s Parallel.For.

Be warned: it does not work! Doerksen’s approach is to upload the entire file into memory (not great, but not as bad as on a web server), send it in chunks using CloudBlockBlob.PutBlock, adding the block ID to a collection at the same time, and then to call CloudBlockBlob.PutBlockList. The reason it does not work is that the order of the loops in Parallel.For is indeterminate, so the block IDs are unlikely to be in the right order.

I fixed this, it tested OK, and then I decided to further improve it by reading each chunk from the file within the loop, rather than loading the entire file into memory. I then puzzled over why my code was broken. The files uploaded, but they were corrupt. I worked it out. In the following code, fs is a FileStream object:

fs.Position = x * blockLength;
bytesread = fs.Read(chunk, 0, currentLength);

Spot the problem? Since fs is a variable declared outside the loop, other threads were setting its position during the read operation, with random results. I fixed it like this:

lock (fs)
{
fs.Position = x * blockLength;
bytesread = fs.Read(chunk, 0, currentLength);
}

and the file corruption disappeared.

I am not sure why, but the manually coded parallel uploads seem to slightly but not dramatically improve performance, to around 100-105 KB/s, almost exactly what my ASP.NET MVC application achieves over my broadband connection.

image

There is another approach worth mentioning. It is possible to bypass the web server and upload directly from the browser to Azure storage. To do this, you need to allow cross-origin resource sharing (CORS) as explained here. You also need to issue a Shared Access Signature, a temporary key that allows read-write access to Azure storage. A guy called Blair Chen seems to have this all figured out, as you can see from his Azure speed test and jazure JavaScript library, which makes it easy to upload a blob from the browser.

I was contemplating going that route, but it seems that performance is no better (judging by the Test Upload Big Files section of Chen’s speed test), so I should probably be content with the parallel JavaScript upload solution, which avoids fiddling with CORS.

Overall, has my experience with the Blob storage API been good? I have not found any issues with the service itself so far, but the documentation and samples could be better. This page should be the jumping off point for all you need to know for a basic application like mine, but I did not find it easy to find good samples or documentation for what I thought would be a common scenario, uploading large files with ASP.NET MVC.

Update: since writing this post I have come across this post by Rob Gillen which addresses the performance issue in detail (and links to working Parallel.For code); however I suspect that since the post is four years old the conclusions are no longer valid, because of improvements to the Azure storage client library.

New features in Windows Azure, including web site backup, .NET mobile services

Microsoft has announced new features in Windows Azure, its cloud platform, described by VP Scott Guthrie on his blog.

Aside: I agree with this comment to his post:

Thank you Scott for update. I wish dozens of MS folks and MS representatives would have a clue about Azure roadmap to help businesses plan their release schedules / migration plans. Till that happens, this blog will remain the main source of updates and a hint of roadmap.

The changes are significant. ExpressRoute offers connectivity to Azure without going through the public internet. Currently you have to use an Equinix datacentre, Level 3 cloud connect, or an AT&T MPLS (Multiprotocol Label Switching) VPN. For enterprises that can meet the requirements and who are wary about data passing through the internet, or who want better connectivity, it is an interesting option.

Next up is backup and restore for Azure web sites. Azure web sites are a way of deploying web applications, ranging from free to multi-instance with automatic scaling. You need at least a Standard site for serious use, as I explained here.

Now you can set up scheduled backup for both the web site and a supporting database. The feature is in preview but you can try it now using the Azure web management portal.

image

I noticed a couple of things. One is that the storage account used must be in the same subscription as the web site. I also spotted this warning:

image

which states that “frequent backups can increase you database costs by up to 100%”. Still, it is a handy feature.

Azure mobile services, designed to supply data to mobile apps, has been extended to support .NET code (previously you had to use Javascript). If you download the code, notes Guthrie, you find that it is  “simply an ASP.NET Web API project with additional Mobile Service NuGet packages included.”

Mobile Services also have new support for notification hubs and for PhoneGap (a way of building mobile apps using HTML and JavaScript).

Another feature that caught my eye is easy linking of third-party apps to Azure Active Directory (which is also used by Office 365). For example, if you are struggling with SharePoint and its poor clients for Windows, iOS and Android, you might consider using Dropbox for business instead. Now you can integrate Dropbox for Business with your Office 365 user directory by selecting  it from the Azure management portal.

image

A closer look at Azure web sites: beware suspension

I am investigating moving this site to Windows Azure. The major benefit would be scalability. Currently it runs on a Linux VM which works very well, but in the event of a major spike in traffic (which is always possible with a news/comment site) it cannot scale.

Windows Azure web sites have nice scalability features. You can add and remove instances, or set up autoscaling based on a schedule or by CPU usage.

Unfortunately this does not come cheap. For autoscaling, you need a Standard web site, which starts at around £35 per month (1 core, 1.75GB RAM) for a Small instance, if you buy a 6 month plan.

Backtracking a little, Azure offers three levels of web site:

  • Free: Shared hosting, 1GB storage, 165MB per day outbound transfer
  • Shared: Shared hosting, 1GB storage, 5GB per day outbound transfer, up to 6 instances. In preview but currently around £7.00 per month per instance
  • Standard: Dedicated hosting, starts at £48 per month pay as you go, £35 per month 6 month plan.

For a quick test, I set up Brandoo WordPress from the Azure app gallery. Brandoo WordPress uses SQL Server rather than MySQL. MySQL on Azure is only available from a third party, ClearDB, which puts me off using it, unless you go a different route and use your own Windows or Linux VM to run it, losing the scalability benefit.

I started with a free web site. I have used free web sites in the past to prototype .NET applications, for which purpose they are excellent. The experience with WordPress was not so good. The site seemed to hang during the WordPress install wizard. My second go was successful, but the site was slow even just navigating the dashboard. Hopeless for any serious use beyond prototyping.

I converted the site to Shared hosting. The price is modest, and I wondered if the ability to scale manually up to 6 instances when needed might be sufficient. The performance improved markedly, compared to the free version. However I noticed these odd metrics in the dashboard: CPU time and Memory Usage, with notes like “Resets in 5 hours” or “Resets in 33 minutes”.

image

In particular, I noted that I had used nearly half of my allocated “Memory usage” just installing an empty WordPress site.

I am not familiar with measuring memory usage per hour and I am not even sure what it means. However, it seems that the consequences of exceeding either the CPU or the memory limit is extreme. The web site is suspended. See for example here:

Something strange happened today. My website on windowsAzure was suspended and inaccessible because my site had exceeded the CPU quotum. I am running the website in shared mode, but I had removed all quota. So why is my website suspended, I thought I would pay for the extra usage, not that my website would be suspended. What is happening here? And what should I do to prevent this from happening again?

See also Jonas Gauffin’s post Azure Failed Me (which has a more positive conclusion than you might expect from the title):

Today I were going to search my blog (http://blog.gauffin.org) about how to do a (almost) a generic type constraint for enums. But instead I got this screen: This site is currently not available. The scary thing is that I’ve got no notification what so ever about my site being down. So I browsed to the manage web site part of Azure and was greeted by the following screen: Suspended. As you see the CPU time has been consumed.

The same problem, I am guessing, hit this user, who has a tutorial on setting up a WordPress blog to Azure, but says at the end not to use it:

Also, even after perfect configuration of the website, Windows Azure was still showing some problems like this “This site is currently not available…” what a joke, the site is in cloud – global cloud – and not available, so when will?

though he apparently did not identify the reason.

It does not seem to me unreasonable that the free option suspends your site; it is free after all. The shared site is low-budget, but still paid for, and it seems to me that the problem of suspension should be spelt out more clearly. If you read:

A web site running in Shared mode benefits from high availability even with a single instance, but you can add up to 6 instances ("scale out") for even greater performance and fault tolerance.

you do not expect that the site will simply stop responding for up to a day if it exceeds CPU or memory limits that are often not easy to predict.

There are thousands of hosting services offering shared hosting or WordPress blogs at low prices, not least WordPress.com, and in general these sites do not get suspended because of exceeding CPU or memory limits.

The conclusion though is that if you want to use Azure for a site whose uptime you care about, you should plan to use no less than a Standard instance.

Billion dollar revenue or not, Microsoft Azure is growing fast

Is Microsoft Azure now a billion dollar business? Maybe, maybe not. The milestone was announced by Curt Anderson, CFO for Server and Tools at Microsoft, in this Bloomberg piece:

Microsoft Corp. (MSFT)’s Windows Azure software and related programs have surpassed $1 billion in annual sales for the first time … Microsoft’s $1 billion sales figure includes Azure, as well as software provided to partners to create related Windows cloud services, Anderson said in an interview.

The remarks have prompted discussion of what exactly makes up this billion dollars of sales. In particular, what is this software sold to partners for “related Windows cloud services” and how much is it worth?

Timothy Prickett Morgan on the Register takes the most sceptical line:

It seems likely, however, that the bulk of that revenue figure comes from peddling Windows Server, Systems Center, SQL Server, and any other wares that service providers, telcos, and hosters have bought to build Windows-based clouds.

It’s hard to imagine it being even a 20-80 split for Azure proper versus Azure-alike, and the ratio is probably something on the order of 10-90 if you put a gun to our head and made us guess. And maybe more like 5-95.

He overstates the case though. Context: Server and Tools earned revenue of over $18bn in the Microsoft’s last financial year ending June 30 2012 and is set to exceed that in 2013. As Mary-Jo Foley reports here, System Center (which forms the basis for Microsoft’s “private cloud” offering) was already over $1bn last year, so it seems unlikely that Anderson would now lump System Center revenue in with Azure and call it Azure revenue.

At the same time, the qualification in Anderson’s statement does imply that Azure on its own, without this “software provided to partners” does not quite make it.

It matters little. It is clear to me that Azure is a rapidly growing business for Microsoft, and that the energy put in by Scott Guthrie and his team is paying off. Check his blog for a stream of strong announcements.

Server and Tools boss Brad Anderson told me that Azure is a “massive public cloud that doubles every six months.”

It makes sense too. If your business runs on Microsoft’s platform and you want to scale into the cloud, Azure is a strong contender now that its usability and features are maturing. Azure Virtual Machines, providing infrastructure as a service, are of key importance; note that while they have been available for a while they only came out of preview officially on April 16th, a couple of weeks ago. Azure Active Directory and the possibility of federation with on-premise AD is another critical feature, and so is virtual networking, which became generally available at the same time as the Virtual Machines.

On the other hand, Prickett Morgan’s response and other exclamations of surprise around the web (Say What? says Gigaom) show the extent to which Microsoft botched the Azure launch back in 2008 and 2009, and how far it has to go before it is perceived as a strong cloud platform contender beyond the circles of Microsoft partners.

Amazon Web Services on the other hand won its cloud reputation years ago and shows no sign of letting go of its lead, with energetic development of its platform that at least matches Microsoft’s efforts as well as commodity pricing.

Still, with both Office 365 and Azure now booming, it seems to me that the time when you could laugh off Microsoft’s cloud efforts has passed. Expect an unqualified $1bn revenue for Azure before too long.

Making sense of Microsoft’s Cloud OS

People have been talking about “the internet operating system” for years. The phrase may have been muttered in Netscape days in the nineties, when the browser was going to be the operating system; then in the 2000s it was the Google OS that people discussed. Most notably though, Tim O’Reilly reflected on the subject, for example here in 2010 (though as he notes, he had been using the phrase way earlier than that):

Ask yourself for a moment, what is the operating system of a Google or Bing search? What is the operating system of a mobile phone call? What is the operating system of maps and directions on your phone? What is the operating system of a tweet?

On a standalone computer, operating systems like Windows, Mac OS X, and Linux manage the machine’s resources, making it possible for applications to focus on the job they do for the user. But many of the activities that are most important to us today take place in a mysterious space between individual machines.

It is still worth reading, as he teases out what OS components look like in the context of an internet operating system, and notes that there are now several (but only a few) competing internet operating systems, platforms which our smart mobile phones or tablets tap into and to some extent lock us in.

But what on earth (or in the heavens) is Microsoft’s “Cloud OS”? I first heard the term in the context of Server 2012, when it was in preview at the end of 2011. Microsoft seems to like how it sounds, because it is getting another push in the context of System Center 2012 Service Pack 1, just announced. In particular, Michael Park from Server and Tools has posted on the subject:

At the highest level, the Cloud OS does what a traditional operating system does – manage applications and hardware – but at the scope and scale of cloud computing. The foundations of the Cloud OS are Windows Server and Windows Azure, complemented by the full breadth of our technology solutions, such as SQL Server, System Center and Visual Studio. Together, these technologies provide one consistent platform for infrastructure, apps and data that can span your datacenter, service provider datacenters, and the Microsoft public cloud.

In one sense, the concept is similar to that discussed by O’Reilly, though in the context of enterprise computing, whereas O’Reilly looks at a bigger picture embracing our personal as well as business lives. Never forget though that this is marketing speak, and Microsoft consciously works to blur together the idealised principles behind cloud computing with its specific set of products: Windows Azure, Window Server, and especially System Center, its server and device management piece.

A nagging voice tells me there is something wrong with this picture. It is this: the cloud is meant to ease the administrative burden by making compute power an abstracted resource, managed by a third party far away in a datacenter in ways that we do not need to know. System Center on the other hand is a complex and not altogether consistent suite of products which imposes a substantial administrative burden on those who install and maintain it. If you have to manage your own cloud, do you get any cloud computing benefit?

The benefit is diluted; but there is plentiful evidence that many businesses are not yet ready or willing to hand over their computer infrastructure to a third-party. While System Center is in one sense the opposite of cloud computing, in another sense it counts because it has the potential to deliver cloud benefits to the rest of the business.

Further confusing matters, there are elements of public cloud in Microsoft’s offering, specifically Windows Azure and Windows Intune. Other bits of Microsoft’s cloud, like Office 365 and Outlook.com, do not count here because that is another department, see. Park does refer to them obliquely:

Running more than 200 cloud services for over 1 billion customers and 20+ million businesses around the world has taught us – and teaches us in real time – what it takes to architect, build and run applications and services at cloud scale.

We take all the learning from those services into the engines of the Cloud OS – our enterprise products and services – which customers and partners can then use to deliver cloud infrastructure and services of their own.

There you have it. The Cloud OS is “our enterprise products and services” which businesses can use to deliver their own cloud services.

What if you want to know in more detail what the Cloud OS is all about? Well, then you have to understand System Center, which is not something that can be explained in a few words. I did have a go at this, in a feature called Inside Microsoft’s private cloud – a glossary of terms, for which the link is currently giving a PHP error, but maybe it will work for you.

image

It will all soon be a little out of date, since System Center 2012 SP1 has significant new features. If you want a summary of what is actually new, I recommend this post by Mike Schutz on System Center 2012 SP1; and this post also by Schutz on Windows Intune and System Center Configuration Manager SP1.

My even shorter summary:

  • All System Center products now updated to run on, and manage, Server 2012
  • Upgraded Virtual Machine Manager supports up to 8000 VMs on clusters of up to 64 hosts
  • Management support for Hyper-V features introduced in Server 2012 including the virtual network switch
  • App Controller integrates with VMs offered by hosting service providers as well as those on Azure and in your own datacenter
  • App Controller can migrate VMs to Windows Azure (and maybe back); a nice feature
  • New Azure service called Global Service Monitor for monitoring web applications
  • Back up servers to Azure with Data Protection Manager

and on the device and client management side, new Intune and Configuration Manager features. It is confusing; Intune is a kind-of cloud based Configuration Manager but has features that are not included in the on-premise Configuration Manager and vice versa. So:

  • Intune can now manage devices running Windows RT, Windows Phone 8, Android and iOS
  • Intune has a self-service portal for installing business apps
  • Configuration Manager integrates with Intune to get supposedly seamless support for additional devices
  • Configuration Manager adds support for Windows 8 and Server 2012
  • PowerShell control of Configuration Manager
  • Ability to manage Mac OS X, Linux and Unix servers in Configuration Manager

What do I think of System Center? On the plus side, all the pieces are in place to manage not only Microsoft servers but a diverse range of servers and a similarly diverse range of clients and devices, presuming the features work as advertised. That is a considerable achievement.

On the negative side, my impression is that Microsoft still has work to do. What would help would be more consistency between the Azure public cloud and the System Center private cloud; a reduction of the number of products in the System Center suite; a consistent user interface across the entire suite; and simplification along the lines of what has been done in the new Azure portal so that these products are easier and more enjoyable to use.

I would add that any business deploying System Center should be thinking carefully about what they still feel they need to manage on-premise, and what can be handed over to public cloud infrastructure, whether Azure or elsewhere. The ability to migrate VMs to Azure could be a key enabler in that respect.

ThoughtWorks bemoans excessive software complexity, advocates small, focused services

ThoughtWorks has released its latest Technology Radar, an opinionated analysis of software development trends.

Things the folk at ThoughtWorks like include automated build and deployment, essential for Continuous delivery; NOSQL database managers especially Neo4j; mobile-first development; the AppCode IDE for Apple’s Objective-C; the Graphite realtime graphing tool for creating dashboards; Clojure and Scala for programming.

I meet some of the ThoughtWorks team at developer conferences from time to time, and generally find them smart and though-provoking to talk to. They must be the despair of the big enterprise software vendors, with a liking for open source and an aversion to heavyweight high-maintenance systems.

This remark particularly caught my eye:

Simple architectures—Simple continues to gain traction, including both techniques for building and composing applications, as well as infrastructure-based techniques to enable simple deployment, failover and recovery. This theme is a recurring one for us, but we have not yet seen the usage shifts we believe are necessary.

I asked consultant James Lewis and practice lead Sam Newman to expand on that. Why do we continue to choose complexity over simplicity?

“A lot of people like to stay inside their big box, and don’t understand the complexities that then creates,” said Newman. “There’s a lack of critical thought given to how services talk to each other. A lot of them are driven by whatever the vendor says you do. Java makes RMI very easy. [Microsoft] .NET makes binding to WSDL [SOAP] schemas very easy. All these tools make bad things very easy to do.

“When you start talking to organisations about smaller services that are focused on doing one thing well, they have the horror associated with now having more than one box to manage and operate. So it’s hard to talk about moving from one big monolithic complicated box that is hard to change, to lots of little boxes, without also having conversations with those clients about how they get better at managing multiple services.

“Netflix has 300 services. Each service runs on at least six machines. They are very good at deploying those services. Yet they are not an overly complicated domain compared to some of our clients.”

“Amazon were talking about this in 2004,” adds Lewis, “the idea that you join up development of these small, simple applications with the operational control, so the same people who build them are also the people who run them. Now that we’re seeing both private and public clouds, and the ability to spin up machines becoming more and more prevalent, its starting to become more attractive.”

This is a consistent theme from ThoughtWorks. Break up complex solutions into many small services, think about how they talk to each other (with REST and HTTP favoured), and think about the infrastructure and how to automate it as well as the software itself.

“In many organisations these conversations are happening,” Newman told me. “I go to clients now, and they talk about the Enterprise Service Bus as being something they’d love to get rid of if they only knew how. Five to ten years ago, to even mention the Enterprise Service Bus as being a problem, they’d look at you with daggers in their eyes.”

“It’s almost like we’re now able to fulfil the promise of service orientation,” says Lewis. “It needed these additional practices, around things like automated deployment, automated rollback, and an understanding that people and process are tied intrinsically with it.”

Another issue, claims Lewis, is that software architects simply get out of touch with best practice.

“Most architects who build big systems are quite a long way from their codebases. They sit in rooms talking to other architects. They might have last written a line of code five or ten years ago. What they do is to design systems as they would have done ten years ago. People do get divorced from the latest trends and perpetuate less effective ways of doing things.”

Finally, here’s something for the Microsoft platform people who read this site. ThoughtWorks is not altogether averse to Microsoft and mentions the Azure cloud platform as something which is becoming interesting. But Windows Phone:

Despite a promising start to Windows Phone, a well thought-out user interface, and probably the best development experience of any mobile platform, we have seen several stumbles in the execution of the platform strategy by Microsoft and its partners. This makes us less optimistic about the future of the platform than we were in the last radar.

Translation: nice mobile platform, but nobody’s buying it. Then again, on Monday next week Windows Phone 8 will be properly unveiled. Still hope?