Category Archives: development

QCon London 2017: IoT insecurity, serverless computing, predicting technical debt, and why .NET Core depends on a 36,000 line C++ file

I’m at the QCon event in London, a multi-vendor conference aimed primarily at enterprise developers and architects.

image
Adam Tornhill speaks at QCon London 2017

A few notes on day one. Alasdair Allan gave a keynote on security and the internet of things; it was an entertaining and disturbing résumé of all that is wrong with the mad rush to connect everything to the internet though short on answers; our culture has to change so that organisations such as hotels, toy manufacturers, appliance vendors and even makers of medical equipment take security seriously but it is not clear how this will come about unless so many bad things happen that customers start to insist on it.

Michael Feathers spoke on strategic code deletion, part of a track on “Dark code: the legacy/tech debt dilemma.” This was an excellent session; code is added to projects more often than it is removed, and lack of hygiene in this regard has risks including security, reliability and performance. But discovering which code is safe to remove is not always trivial, and Feathers explored some of the nuances and suggested some techniques.

Steve Faulkner gave a session on serverless JavaScript, or more specifically, using Amazon Web Services (AWS) Lambda and API Gateway. Faulkner said that the API Gateway was the piece that made Lambda viable for them; he is Director of Platform Engineering at Bustle, a busy content site based in the USA. In a nutshell, moving from EC2 VMs to Lambda has yielded both financial savings and easier management. The only downside is performance; each call to a Lambda function takes a minimum of 100ms whereas the same function on a WM might take 20ms. In the end it is not critical as performance remains satisfactory.

Faulkner said that AWS is ahead of its competitors (Microsoft, Google and IBM were mentioned) but when pressed said that both Microsoft and Google offered strong alternatives. Microsoft’s Azure Functions are spoilt by the need to specify a maximum scale, rather than scaling automatically, but its routing solution is in some ways ahead of AWS, he said. Google’s Functions will be great when out of beta.

Adam Tornhill spoke on A Crystal Ball to prioritise Technical Debt, another session in the dark code track. This was my favourite of the day. Tornhill presented a relatively simple way to discover what code you should refactor now in order to avoid future issues. His method is based on looking for files with many lines of code (a way of measuring complexity) and many commits (suggesting high importance and activity), the “hotspots” in your projects. For more detail and some utilities see Tornhill’s blog.

Why do we end up with bad or risky code in our software? Tornhill said that developers often mistake organisational problems for technical problems and try unsuccessfully to fix them with tools.

He also mentioned an example of high-risk code, the file gc.cpp which performs garbage collection in .NET Core, the next generation of Microsoft’s .NET Framework. This file is over 36,000 lines and should be refactored. There is a discussion on the subject here. It exactly bears out Tornhill’s point. A developer proposes to refactor the file, back in March 2015. Microsoft’s Karel Zikmund defends the status quo:

Why it is this way? … Partly historical reasons (it is this way since the start). Partly because devs working on it didn’t feel the urge to refactor it. Partly because splitting of gc.cpp is non-trivial and risky and because it does not bring too big value (ramp up in the code base can be gained also in the combination of reading BOTR and debugging the code). Why it is staying this way? … Cost/benefit/risk ratio is IMO not in favor of a change here.

Few additional thoughts:
Am I happy that there is only 1 large file? No, but it doesn’t hurt me much either.
Do I see the disadvantages of large file? Yes, but I don’t think they are huge. More like minor annoyances with easy workarounds.
And to turn it around: Do you see the risk of any changes here? Do you see the cost of extra careful code reviews to mitigate the risk?

Strictly technically, we truly believe this is a formatting change. If it was simple to split it up and if it would be low risk and if it would be very easy to review, it might be worth the ‘minor’ improvements mentioned above … but I don’t see that combo happening (not on a noticeable scale in gc.cpp).
On a personal note: I also trust CLR team that if all these three things were true, the refactoring would have happened long time ago.

Note that some of this code goes back beyond .NET Core to the .NET Framework, the “historical reasons” that Zikmund mentions. We can see that the factors preventing change are as much organisational as technical.

Finally I attended a session on Microsoft’s Cognitive Services. Note this was in the “Sponsored solution track”. Microsoft also has a stand here focused on its Cognitive Services.

There is not much Microsoft Platform content at QCon and it seems under-represented, though many of the sessions are applicable to developers on any platform. I am not sure of all the reasons for this; there used to be an Advanced .NET track at QCon. It does reflect some overall development trends as well as the history and evolution of QCon itself. That said, there is a session on SQL Server on Linux so the company is not completely invisible here.

As for the session, it was a reasonable overview of Microsoft’s expanding Cognitive Services APIs, which covers things like image recognition, speech recognition and more. I would have liked more depth and would have preferred to hear from a practitioner, in other words, “we built an application on Cognitive Services and this is what we learned.” I am not altogether clear why the company is pushing this so hard, except that it is a driver for developers to use Azure. I asked about how developers should deal with the problem of uncertainty*, in other words, that Cognitive Services does not deliver absolute results but rather draws conclusions with a confidence score – eg it might be pretty sure that an image contains a human face, fairly sure that it is male, and somewhat confident that the age of the person is mid forties. When the speaker demoed speech recognition it went pretty well except that “Start” was transcribed as “Stop.” This stuff is difficult.

Looking forward now to Day Two: Containers, Machine Learning, and more.

*More concisely expressed as “Systems are moving from the deterministic to the probabilistic” by Stephen Whitworth, who is now speaking on Machine Learning.

Microsoft sets Visual Studio LightSwitch to off

Microsoft has officially announced the end of development of LightSwitch, a rapid application builder for desktop and mobile applications.

LightSwitch was introduced in July 2011 as a tool to build multi-tier applications using a data-first approach. You can design you database using an excellent visual designer, design screens for viewing and editing the data using a non-visual designer, and generate applications with the server-side code hosted either on your own server or on Microsoft Azure. The client application in the original LightSwitch was based on Silverlight, but this was later extended with an option for HTML. You can get a feel for the general approach from my early hands-on here.

As I noted at the time, LightSwitch abstracts a number of difficult tasks. This is a good thing, though as with any application generated you had to take time to learn its quirks. That said, it is more usable than most model-driven development tools, in my experience.

LightSwitch had some bad luck. It was conceived at a time when Silverlight looked like the future of Microsoft’s client development platform, but by the time it launched Silverlight was heading for obsolescence. It also fell victim to ideologies within Microsoft (which persist today) that chase the dream of code-free application development that anyone can do. The documentation for LightSwitch on launch was dreadful, a series of how-tos that neglected to explain how the tool worked. You had to get the software development kit, aimed at those building LightSwitch components, to have any hope of understanding the tool.

image

The abandonment of LightSwitch is not a surprise. Microsoft had stopped talking about it and adoption was poor. There will be no tooling for it in the next Visual Studio, though you can keep using it for a while if you want.

I think it is a shame since it is a promising tool and I cannot help thinking that with more intelligent positioning and a few tweaks to the product and its documentation it could have been a success. Those who did get to grips with it found it very good.

What is unfortunate is that Microsoft has lost the faith of many developers thanks to the many shifts in its development strategy. I know component vendors have also been caught out by the Silverlight and then LightSwitch debacle. Here is one of the comments on the announcement:

Microsoft keeps doing this over and over, we invest months even years to master a technology, just to find out it’s being phased out prematurely. Perfectly good, one-of-a-kind niche tools too. So much investment on both sides (both MS and customers) down the drain. What’s worse, it is done is a non-transparent, dishonest manner, letting things dry up over a couple years so that when the announcement comes, no-one really cares any more, no more noise – just look at this blog.

This makes it hard for the company to convince developers that its new strategies de jour have a longer life ahead of them. I am thinking of the UWP (Universal Windows Platform), which has already changed substantially since its first conception, and of PowerApps, the supposed replacement for LightSwitch, and yet another attempt to promote code-free development.

Developers do not want code-free development. They like tools that do stuff for them, if they are intuitive and transparent, but they also like an easy route to adding and modifying code in order to have the application work the way they want.

On GitHub and GitHub Universe

I’ve been at GitHub Universe in San Francisco for the last few days. Around 1500 developers (not sure if that figure includes staff and exhibitors) in a warehouse at Pier 70. The venue was beautifully converted into an interaction space. Here is the view from outside as we were leaving; Octocat seems to be waving goodbye:

image

The main stage done up like a spaceship:

image

There was a large area for mingling, overseen by Octocat:

image

Plenty of space outside too, with a high standard of food and drink on offer.

image

There was also a “send a postcard” area where you could write a card; with cards, pens, stamps and postbox supplied there was no excuse not to do so:

image

You are probably thinking, when do we get to the techie stuff; but in some ways it is better to look at the space GitHub created and ask what it tells you about the company.

Running an event like this is not cheap, and I think we can conclude that GitHub has a business model that works. Further, there was a generous and inclusive spirit to the event which was good to experience. Kimberly Bryant from Black Girls Code spoke at the opening keynote – the event concert was a Black Girls Code benefit – and while there is often an element of PR in the causes which businesses choose to sponsor, I don’t question the authenticity of GitHub’s efforts to promote both coding and diversity in our sadly imbalanced software industry.

image

In some ways then the actual technical content was not the most important thing about this event. That said, there was some excellent content on themes including how GitHub scales its own service, new project management and code review features in GitHub, and how the product is evolving its add-in or “integrations” platform.

I also learned a bit about Electron, a framework for “creating native Desktop applications with web technologies” based on Chromium and Node.js. Microsoft’s Visual Studio Code uses Electron, as does GitHub’s own Atom editor.

If you are developer, you will be familiar with GitHub; it is the obvious choice for hosting an open source project (free) and a popular option for private repositories. When Google Code closed in 2015, the announcement cited the migration of developers to GitHub as the key reason and acknowledge that it was among “a wide variety of better hosting services” than Google’s own. “To meet developers where they are, we ourselves migrated nearly a thousand of our own open source projects from Google Code to GitHub,” remarked Google’s Chris DiBona. That was a pivotal moment, showing how GitHub has become a core part of the open source ecosystem as well as a strong commercial product for private and enterprise repositories.

GitHub does seem to take its responsibilities seriously and the fact that is has found a successful balance between free and commercial services is something to be thankful for.

Time for another look at “pure .NET”

Back in the Nineties there was a lot of fuss about “pure Java”. This meant Java code without any native code invocations that tie the application to a specific operating system.

It is possible to write cross-platform Java code that invokes native code, but it adds to the complexity. If it is an operating system API you need conditional code so that the write API is called on each platform. If it is a custom library it will have to be compiled separately for each platform.

Over on the Microsoft .NET site, developers have tended to have a more casual approach. After all, in the great majority of cases the code would only ever run on Windows. Further, Microsoft tended to steer developers towards Windows-only dependencies like SQL Server. After all, that is the value of owning a developer platform.

Times change. Microsoft has got the cross-platform bug, with its business strategy based on attracting businesses to its cloud properties (Office 365 and Azure) rather than Windows. The .NET Framework has been forked to create .NET Core, which runs on Mac and Linux as well as Windows. SQL Server is coming to Linux.

Another issue is porting applications from 32-bit to 64-bit, as I was reminded recently when migrating some ASP.NET applications to a new site. If your .NET code avoids P/Invoke (Platform Invoke) then you can compile for “Any CPU” and 64-bit will just work. If you used P-invoke and want to support both 32-bit and 64-bit it requires more care. IntPtr, used frequently in P/Invoke calls, is a different size. If you have custom native libraries, you need to compile them separately for each platform. The lazy solution is always to run as 32-bit but that is a shame.

What this means is that P/Invoke should only be used as a last resort. Arguably this has always been true, but the reasons are stronger today.

This is also an issue for libraries and components intended for general use, whether open source or commercial. It is early days for .NET Core support, but any native code dependencies will be a problem.

Breaking the P/Invoke habit will not be easy but “Pure .NET” is the way to go whenever possible.

Reflections on QCon London 2016 – part one

I attended QCon in London last week. This is a software development conference focused on large-scale projects and with a tradition oriented towards Agile methodology. It is always one of the best events I get to attend, partly because it is vendor-neutral (it is organised by InfoQ), and partly because of the way it is structured. The schedule is divided into tracks, such as “Back to Java” or “Architecting for failure”, each of which has a track leader, and the track leader gets to choose who speaks on their track. This means you get a more diverse range of speakers than is typical; you also tend to hear from practitioners or academics rather than product managers or evangelists.

image

The 2016 event was well up to standard from my perspective – though bear in mind that with 6 tracks on each day I only got to attend a small fraction of the sessions.

This post is just to mention a few highlights, starting with the opening keynote from Adrian Colyer, who specialised in finding interesting IT-related research papers and writing them up on his blog. He seems to enjoy being contrarian and noted, for example, that you might be doing too much software testing – drawing I guess on this post about the art of testing less without sacrificing quality. The takeaway for me is that it is always worth analysing what you do and trying to avoid the point where the cost exceeds the benefit.

Next up was Gavin Stevenson on “love failure” – I wrote this up on the Reg – there is a perhaps obvious point here that until you break something, you don’t know its limitations.

On Monday evening we got a light-hearted (virtual) look at Babbage’s Analytical Engine (1837) which was never built but was interesting as a mechanical computer, and Ada Lovelace’s attempts to write code for it, thanks to John Graham-Cumming and illustrator Sydney Padua (author of The Thrilling Adventures of Lovelace and Babbage).

image

Tuesday and the BBC’s Stephen Godwin spoke on Microservices powering BBC iPlayer. This was a compelling talk for several reasons. The BBC is hooked on AWS (Amazon Web Services) apparently and stores 21TB daily into S3 (Simple Storage Service). This includes safety copies. iPlayer was rebuilt in 2013, Godwin told us, and the team of 25 developers achieves 34 live deployments per week on average; clearly the DevOps stuff is working here. Godwin advocates genuinely “micro” services. “How big should a microservice be? For us, about 600 Java statements,” he said.

Martin Thompson spoke on the characteristics of a good software engineer, though oddly the statement that has stayed with me is that an ORM (Object-Relational Mapping) “is the wrong abstraction for a database”, something that chimes with me even though I get the value of ORMs like Microsoft’s Entity Framework for rapid development where database performance is non-critical.

Then came another highlight: Google’s Micah Lemonik on Architecting Google Docs. This talk sadly was not recorded; a touch of paranoia from Google? This was fascinating both from a historical perspective – Lemonik was involved in a small company called 2Web technologies which developed an Excel-like engine in 2003-4, and joined Google (which acquired 2Web) in 2005 to work on Google Sheets. The big story here was the how Google Sheets became collaborative, so more than one person could work on a spreadsheet simultaneously. “Google didn’t like it initially,” said Lemonik. “They thought it was too weird.” The team persisted though, thinking about the editing process as “messages being transferred between collaborators” rather than as file updates; and it worked.

You can actually use today’s version in your own projects, with Google’s Realtime API, provided that you are happy to have your stuff on Google Drive.

I particularly enjoyed Lemonik’s question to the audience. Two people are working on a sheet, and one types “6” into a cell. Then the same person overtypes this with “7”. Then the collaborator overtypes the same cell with “8”. Next, the first person presses Ctrl-z for undo. What should be the result?

The audience split neatly into “6”, “7”, and just a few “8” (the rationale for “8” is that undo should only undo your own changes and not touch those made by others).

Google, incidentally, settled on “6”, maintaining a separate undo stack for each user. But there is no right answer.

Lemonik also discussed the problem of consistency when there are large numbers of contributors. A hard problem. “There have to be bounds to the system in order for it to perform well,” he said. “The biggest takeaway for me in building the system is that you just can’t have it all. All of engineering is this trade-off.”

image

I have more to say about QCon so look out for part two shortly.

Adapting a native code DLL to be called from a Store or Universal Windows app

I am writing a Bridge game in C# – yes, I have been doing this for some time, it does run now but it is not ready for public unveiling.

It is good fun though and a learning experience, as I am writing it as a Windows 8 Store app. This means it can also be a Universal Windows Platform app but I have kept it compatible with Window 8.1 as I don’t want to lose that large market of Windows 8 users who have not upgraded to 10. Hmm.

Bridge is a card game in which a pack of 52 cards is dealt into 4 hands of 13 cards. Each hand is played as a sequence of 13 4-card “tricks”, and each trick is won one of two opposing pairs of players according to the cards played. Each pair of course tries to win as many tricks as possible, so one of the points of interests is how many tricks can be won if you play perfectly (ie with full knowledge of all four hands). Another point of interest is how each card played affects the potential number of tricks you can win with best play. For example, leading a King might cost you a trick (or more) if your opponents hold both the Ace and the Queen of that suit.

This is called “double dummy” analysis and smart people have written algorithms to calculate the answers. A double dummy analysis is useful in a bridge game for two reasons. One is that users may like to know, after playing a hand, what their best score could have been, or even to analyse the hand and see how if they played this card rather than that card at trick such-and-such the outcome would have varied. The other is that you can use it to assist the software in finding the best play. Of course it is important that the software plays fair by not using knowledge of all four hands beyond what would be known by human players; but it is legitimate to try out various possible hands that match what is currently known and use double dummy analysis on these hands.

One such smart person is Bo Haglund who wrote a C++ Windows library for double dummy analysis, called Double Dummy Solver (DDS) and released it as open source under the Apache 2 license. It works very well and is widely used in the Bridge software community, and has now been ported to Mac and Linux; you can find the latest code on Github.

Modifying a native code DLL to use with a Store app

I wanted to use the library in my own Bridge game but faced a compatibility problem. Windows Store apps can only call into DLLs that meet certain requirements, such as using only a subset of the Windows API, and DDS did not meet those requirements. My choice was either to port the DLL to C#, or to modify the code so that it would work as a Windows Runtime native DLL.

I have no doubt that the code could be ported to C# but it looks like rather a long job that would result in a library with slower performance (please feel free to prove me wrong). I thought it would be more realistic to modify the code, so I created a new Windows 8.1 DLL project in Visual Studio 2013 (I am now using Visual Studio 2015 but it is the same for this) and set about modifying the code so that it would compile.

In no particular order, here are some notes on what I learned.

I was able to get the DLL to compile after disabling the multi-threading support (more on this later), and commenting out some functions that I don’t yet need.

Another issue I hit was that Visual C++ by default performs “Security Development Lifecycle” checks (compile with /sdl). This means that that common functions like strcpy, strcat, sprintf and others will not compile. You have to use “secure” versions of those functions, strcpy_s, strcat_s, sprintf_s and so on. These are specific to Microsoft’s libraries though. Of course you can just not compile with /sdl, or define _CRT_SECURE_NO_WARNINGS, but I chose to fix all of these. Now the library compiled.

But did it work? No. I had introduced a stupid bug which took me a while to fix. Did it then work? Yes, but it took me some time to get it working from C#.

Next, I kept getting DLLNotFound exceptions. OK, so you have to add the DLL as content in your C# project, and make sure it is set to copy to your output. I still got DLLNotFound exceptions. It turns out that you get this exception even when the DLL is present, if there is a dependency in the DLL which is not found. What dependency was not found? I downloaded the Sysinternals Process Monitor utility and set the filter to monitor my C# game. I excluded SUCCESS results. Then I tried to load the DLL. This told me that it was looking for the file msvcr120_app.dll (the Windows Runtime version of the Visual C++ runtime library). My first thought was to add runtime libraries from the appx deployment packages, in:

C:\Program Files (x86)\Microsoft SDKs\Windows\v8.1\ExtensionSDKs\Microsoft.VCLibs\12.0

image

Then I discovered that all you need to do is to add a reference to the Visual C++ runtime packages, much easier. That fixed DLLNotFound.

Next, I had some problems calling the 64-bit DLL with Platform Invoke (PInvoke) from C#. I found it easier to compile both my C# app and the DLL itself as 32-bit code. I may go back to the 64-bit option later.

Concurrency issues

Now I had everything working; except that my DDS port was far inferior to the standard one because it was single-threaded. The original used QueueUserWorkItem which is not available in a Windows Runtime DLL. I searched for what to do, and came across this MSDN article which recommends using RunAsync, WorkItemHandler and IAsyncAction. However my DLL was not currently compiled using /ZW for “Consume Windows Runtime Extension”. I could add that of course; but then my DLL would have a dependency on the Windows Runtime and if I wanted to use the code for, say, Windows 7, it would not work. or not without yet more #ifdef blocks. No big deal perhaps; but my preference was to avoid this dependency.

There may be other solutions, but the one that I found was to use the Concurrency Runtime. Previously, QueueUserWorkItem was called in a for loop. I simply modified this to use a parallel_for loop instead, using the example here for guidance. I also added:

#include <ppltasks.h>

using namespace concurrency;

to the top of the code. It works well, speeding performance by about three times on my quad-core desktop. Of course I was greatly helped by the fact that the code was already written with concurrency in mind.

image

The effect is spoiled by the time it takes to load the DLL but fortunately you can get DDS to solve multiple boards in one call though I have yet to experiment with this.

Running ASP.NET 5.0 on Nano Server preview

I have been trying out Microsoft’s Nano Server Preview and wrote up initial experiences for the Register. One of the things I mentioned is that I could not get an ASP.NET app successfully deployed. After a bit more effort, and help from a member of the team, I am glad to say that I have been successful.

image

What was the problem? First, a bit of background. Nano Server does not run the .NET Framework, presumably because it has too many dependencies on pieces of Windows which Microsoft wanted to omit from this cut-down deployment. Nano Server does support .NET Core, also known as Core CLR, which is the open source fork of the .NET Framework. This enables it to run PowerShell, although with a limited range of cmdlets, and my main two ways of interacting with Nano Server are with PowerShell remoting, and Windows file sharing for copying files across.

On your development machine, you need several pieces in order to code for ASP.NET 5.0. Just installing Visual Studio 2015 RC will do, except that there is currently an incompatibility between the version of the ASP.NET 5.0 .NET Core runtime shipped with Visual Studio, and what works on Nano Server. This meant that my first effort, which was to build an empty ASP.NET 5.0 template app and publish it to the file system, failed on Nano Server with a NativeCommandError.

This meant I had to dig a bit more deeply into ASP.NET 5.0 running on .NET Core. Note that when you deploy one of these apps, you can include all the dependencies in the app directory. In other words, apps are self-hosting. The binary that enables this bit of magic is called DNX (.NET Execution Environment); it was formerly known as the K runtime.

Developers need to install the DNX SDK on their machines (Windows, Mac or Linux). There is currently a getting started guide here, though note that many of the topics in this promising documentation are as yet unwritten.

image

However, after installation you will be able to use several handy commands:

dnvm This is the .NET Version manager. You can have several versions of the DNX runtime installed and this utility lets you list them, set aliases to save typing full paths, and manage defaults.

image

dnu This is the .NET Development Utility (formerly kpm) that builds and publishes .NET Core projects. The two commands I found myself using regularly are dnu restore which downloads Nuget (.NET repository) packages and dnu publish which packages an app for deployment. Once published, you will find .cmd files in the output which you use to start the app.

dnx This is the binary which you call to run an app. On the development machine, you can use dnx . run to run the console app in the current directory and dnx . web to run the web app in the current directory.

Now, back to my deployment issues. The Visual Studio templates are all hooked to DNX beta 4, and I was informed that I needed DNX beta 5 for Nano Server. I played around with trying to get Visual Studio to target the updated DNX but ran into problems so decided to ignore Visual Studio and do everything from the command line. This should mean that it would all work on Mac and Linux as well.

I had a bit of trouble persuading DNX to update itself to the latest unstable builds; the main issue I recall is targeting the correct repository. You NuGet sources must include (currently) https://www.myget.org/F/aspnetvnext/api/v2.

Since I was not using Visual Studio, I based my samples on these, Hello World Console, MVC and Web apps that you can use for testing that everything works. My technique was to test on the development machine using dnx . web, then to use dnu publish and copy the output to Nano Server where I could run ./web.cmd in a remote PowerShell session.

Note that I found it necessary to specify the CoreClr 64-bit runtime in order to get dnu to publish the correct files. I tried to make this the default but for some reason* it reverted itself to x86:

dnu publish –runtime "c:\users\[USERNAME]\.dnx\runtime\dnx-coreclr-win-x64.1.0.0-beta5-11701"

Of course the exact runtime version to use will change soon.

If you run this command and look in the /bin/output folder you will find web.cmd, and running this should start the app. The port on which the app listens is set in project.json in the top level directory of the project source. I set this to 5001, opened that port in the Windows Firewall on the Nano Server, and got a started message on the command line. However I still could not browse to the app running on Nano Server; I got a 400 error. Even on the development machine it did not work; the browser just timed out.

It turned out that there were several issues here. On the development machine, which is running Windows 10 build 10074, I discovered to my annoyance that the web app worked fine with Internet Explorer, but not in Project Spartan, sorry Edge. I do not know why.

Support also gave me some tips to get this working on Nano Server. In order for the app to work across the network, you have to edit project.json so that localhost is replaced either with the IP number of the server, or with a *. I was also advised to add dnx.exe to the allowed apps in the firewall, but I do not think this is necessary if the port is open (it is a nuisance, since the location of dnx.exe changes for every app).

Finally I was successful.

Final observations

It seems to me that ASP.NET vNext running on .NET Core has the characteristic of many open source projects, a few dedicated people who have little time for documentation and are so close to the project that their public communications assume a fair amount of pre-knowledge. The site I referenced above does have helpful documentation though, for the few topics that are complete. Some other posts I found helpful are this series by Steve Perkins, and the troubleshooting suggestions here especially David Fowler’s post.

I like The .NET Core initiative overall since I like C# and ASP.NET MVC and now it is becoming a true cross-platform framework. That said, the code does seem to be in rapid flux and I doubt it will really be ready when Visual Studio 2015 ships. The danger I suppose is that developers will try it in the first release, find lots of problems, and never go back.

I also like the idea of running apps in Nano Server, a low-maintenance environment where you can get the isolation of a dedicated server for your app at low cost in terms of resources.

No doubt though, the lack of pieces that you expect to find on Windows Server will be an issue and I am not sure that the mainstream Microsoft developer ecosystem will take to it. Aidan Finn is not convinced, for example:

Am I really expected to deploy a headless OS onto hardware where the HCL certification has the value of a bucket with a hole in it? If I was to deploy Nano, even in cloud-scale installations, then I would need a super-HCL that stress tests all of the hardware enhancements. And I would want ALL of those hardware offloads turned OFF by default so that I can verify functionality for myself, because clearly, neither Microsoft’s HCL testers nor the OEMs are capable of even the most basic test right now.

Finn’s point is that if your headless server is having networking issues it is hard to troubleshoot, since of course remote tools will not work reliably. That said, I have personally run Hyper-V Server (which is essentially Server Core with just the Hyper-V role) with great success for several years; I started keeping notes on how to troubleshoot from the command line and found solutions to common problems. If networking fails with Nano Server then yes, you have a problem, but there is always something you can do, even if it means mounting the Nano Server VHD or VHDX on another VM. Windows Server admins have become accustomed to a local GUI though and adjusting even to Server Core has not been easy.

*the reason was that I did not use the –p argument with dnvm use which would have made it persistent

Imagination at Mobile World Congress 2015: what is the strategy?

At MWC earlier this month I met with Imagination, best known for its PowerVR video design but also now the owner of the MIPS CPU. Apple is a shareholder and uses Imagination video technology in the iPhone and iPad. This market is highly competitive though, especially since ARM has its own Mali GPU. “You need complete platforms, you need a processor,” Tony King-Smith, executive VP of Technology Marketing, told me. “All the markets that matter to us are integrating towards a single chip. For a single chip you need some mix of central processing, communications, and multimedia.”

MIPS is a supported CPU for Android 2.3 or higher but most Android devices run ARM or Intel CPUs. Why no MIPS devices at MWC?

“There is one and a half to two years between a licensee picking up the IP, and delivering silicon based on it,” an Imagination’s spokesperson said. “We are engaged with customers but until something shows up we cannot disclose any names. Next year we are going to see some progress and potentially something I can show you.” Watch this space then.

What is Imagination’s strategy overall? King-Smith told me that the company is well placed to satisfy the need for optimisation and differentiation in an increasingly mature mobile market. It is also eyeing the IoT (Internet of things) space with interest. “Wearables need completely new architectures,” said King-Smith. “Not just tweaking a mobile chip. That’s where we’re going.”    

I was also interested to see a real demo of Vulkan, the successor to OpenGL, on the Imagination stand, based on the preliminary specification. “It will enable people to make more use of our platform”, said King-Smith, because of the lower level access it offers to the GPU.

image_thumb[10]

For more on Vulkan see this piece on the Reg.

What about the Creator board which Imagination has released, a low-priced starter kit along the lines of Raspberry Pi but of course with MIPS and more powerful graphics? It is an effort to build the ecosystem, said King-Smith. “It is a means for us to deliver our IP and make it easier for developers to engage with us. We also want to enable start-ups and new solutions.” It is primarily for developing and testing ideas, then, but if you want to go into production with it, that is fine too. “That board has been designed to ramp in volume,” King-Smith told me.

So that was 2014: Samsung stumbles, all change for Microsoft, Sony hack, more cloud, more mobile

What happened in 2014? One thing I did not predict is that Samsung lost its momentum. Here are Gartner’s figures for global smartphone sales by vendor, for the third quarter of 2014:

image

Samsung is still huge, of course. But in 2013, Samsung seemed to be in such control of its premium brand that it could shape Android as it wished, rather than being merely an OEM for Google’s operating system. In the enterprise, Samsung KNOX held promise as a way to bring security and manageability to Android, but only in Samsung’s flavour. Today, that seems less likely. Market share is declining, and much of KNOX has been rolled into Android Lollipop. What is going wrong? The difficulty for Samsung is how to differentiate its products sufficiently, to avoid bleeding market share to keenly priced competition from vendors such as Xiaomi and Huawei. This is difficult if you do not control the operating system.

What of the overall mobile OS wars? 2013 brought few surprises: the Apple/Android duopoly continued, Blackberry further diminished its share, and Windows Phone struggles on, though it was not looking good for Microsoft’s OS as 2013 closed; the Nokia acquisition may have been fumbled.

All change at Microsoft

That brings me to Microsoft, a company I watch closely. 2014 saw Satya Nadella appointed as CEO and several strategic changes, though the extent to which Nadella introduced those changes is uncertain. What changes?

Office is going truly cross-platform, with first-class support for iOS and Android. I covered this recently on the Register; the summary is that there will be mobile versions of Office for iOS, Android and Windows (this last a Store app) with similar features, and that more and more of the functionality of desktop Office will turn up in the mobile versions. I learned from my interview with Technical Product Manager Kaberi Chowdhury that ODF (Open Document) support is planned, as is some level of programmability.

The plans for Office are a clue to the company’s wider strategy, which is focused on cloud and server. Key products include Office 365, Windows Azure, Active Directory (and Azure Active Directory), SQL Server, SharePoint, and System Center as a management tool for hybrid cloud.

The Windows client strategy is to bring back users who disliked Windows 8 with a renewed focus on the desktop in the forthcoming Windows 10, while retaining the Store app model for apps that are secure, touch-friendly, and easily deployed. It is still not clear what Windows 10 phones and tablets will look like, but we can expect convergence; no more Windows RT, but perhaps tablets running Windows Phone OS that are in effect the next generation of Windows RT without a desktop personality.

The company will also hedge its bets with full app support for Office and its cloud services on iOS and Android, and in doing so will make its Windows mobile offerings less compelling.

Microsoft’s developer tools are changing in line with this strategy. The next generation of .NET is open source and cross-platform on the server side, for Windows, Mac and Linux. Xamarin plugs the gap for .NET on iOS and Android, while Microsoft is also adding native support (not .NET based) for cross-platform mobile in the next Visual Studio.

These are big changes to the developer stack, and Microsoft is forking .NET between the continuing Windows-only .NET Framework, and the new cross-platform .NET Core. Developers have many questions about this; see this interview on the Register for what I could glean about the current plans. Watch our for the Build conference at the end of April when the company will attempt to put it all together into a coherent whole for developers targeting either Windows 10, or cloud apps, or cloud services with cross-platform mobile clients.

This entire strategy is a logical progression from the company’s failure in mobile. Can it now succeed with client apps running on platforms controlled by its competitors? Alternatively, is there hope that Windows 10 can keep businesses hooked on Windows clients? Maybe 2015 will bring some answers, though with Windows 10 not expected until towards the end of the year there will be a long wait while iOS, Android and even Chrome OS (the operating system of Chromebook) continue to build.

A side effect is that C# now has a better chance of building a cross-platform user base, rather than being a Windows language. This has already happened in game development, thanks to the use of Mono and C# in the popular Unity game engine. Could it also happen with ASP.NET, deployed to Linux servers, now that this will be officially supported? Or is there little room for it alongside Java, PHP, Ruby, Node.js and the rest? 

The puzzle with Microsoft is that there is still too much mediocrity and complacency that damages the company’s offerings. How can it expect to succeed in the crowded wearable market with a band that is uncomfortable to wear? There is still an attitude in some parts of the company that the world will be happy to put up with problems that might be fixed in a future version after some long interval. Then again, the Azure team is doing great things and Windows server continues to impress. Win or lose, there will be plenty of Microsoft news this year.

A theme for 2015: cloud optimization

Late last year I attended Amazon’s re:Invent conference in Las Vegas; I wrote this up here. The key announcement for me was Amazon Aurora, a MySQL clone, not so much because of its merits as a cloud database server, but more because it represents a new breed of applications that are designed for the cloud. If you design database storage with the knowledge that it will only ever run on a huge cloud-scale infrastructure, you can make optimizations that cannot be replicated on smaller systems. I tried to summarize what this means in another Register piece here. The fact that this type of technology can be rented by any of us at commodity prices increases the advantage of public cloud, despite reservations that many still have concerning security and control. It also poses a challenge for companies like Oracle and Microsoft whose technology is designed for on-premises as well as cloud deployment; they cannot achieve the same advantage unless they fork their products, creating cloud variants that use different architecture.

The Sony hack

The cyber invasion of Sony Pictures in late November was not just another hack; it was a comprehensive takedown in which (as far as I can tell) the company’s entire IT systems were entirely compromised and significantly damaged.

According to this report:

Mountains of documents had been stolen, internal data centers had been wiped clean, and 75 percent of the servers had been destroyed.

Most IT admins worry about disaster recovery (what to do after catastrophic system failure such as a fire in your data center) as well as about security (what to do if hackers gain access to sensitive information). In this case, both seemed to happen simultaneously. Further, as producing movies is in effect a digital business, the business suffered loss of some of its actual products, such as the unreleased “Annie”.

The incident is fascinating in itself, especially as we do not know the identity of the hackers or their purpose, but what interests me more are the implications.

Specifically, how many companies are equally at risk? It seems clear that Sony’s security was towards the weak end of the scale, but there is plenty of weak security out there, especially but not exclusively in smaller businesses.

With the outcome of the Sony hack so spectacular, it is likely that there will be similar efforts in 2015, as well as many businesses looking nervously at their own practices and wondering what they can do to protect themselves.

Cloud may be part of the answer though even if the cloud provider does security right, that is no guarantee that their customers do the same.   

Looking back on looking back

Here is what I wrote a year or so ago, Reflecting on 2013- the year of not the PC, no privacy, and the Internet of Things. Most of it still applies. I have not achieved any of the three goals I set for myself though. Maybe this year…

Amazon Reinvent: new products announced including Aurora database with claimed performance 5 times that of MySQL

Amazon is holding its third Reinvent conference in Las Vegas – 13,500 attendees catching up on Amazon’s Web Services platform. In this morning’s keynote, Amazon’s Senior VP of cloud services Andy Jassy evangelised the platform and announced a number of new services which, in typical Amazon style, are now available in preview.

image

Amazon is well ahead of its competitors in cloud services, in terms of market share and mindshare, and Jassy had no problems reeling off impressive statistics and case studies. A slide showing that AWS is not only larger but also growing faster yea-on-year than its competition prompted a small protest. Microsoft claims that Amazon understated its rate of growth:

cloudwars

The refrain from those who spoke on behalf of companies such as Intuit (which intends to move 100% of its applications to AWS) was that no alternative cloud provider could offer a realistic alternative to AWS. With the progress being made by competitors I wonder for how long this will be true – and bear in mind that this is an Amazon conference – but it testifies to the dominance that Amazon has achieved.

Jassy made a key point about security and compliance. The relative security of public cloud versus private datacenters has long been debated, initially on the assumption that computing resources you own and guard yourself must be more secure than those hosted by third-parties. The counter is that few organisations can afford the level of security that big public cloud providers can achieve. Jassy’s point though was that the number of certifications now achieved by AWS is now such that security and compliance is now a driver towards cloud computing.

The main news though was a series of product announcements:

Aurora relational database: a MySQL compatible database as a service for which Jassy claims 5x the performance of MySQL. He says that businesses stick with commercial, proprietary database managers because open source solutions lack the performance, but that Aurora now provides a solution at a commodity price. Unfortunately Aurora is not going to help those with applications locked into Oracle, SQL Server or others. Still, 5x performance is always welcome.

CodeDeploy: apparently based on a service Amazon uses internally, this is a deployment tool for pushing out updated applications to EC2 (Elastic Compute Cloud) VMs without downtime.

CodeCommit: a source code management service for Git repositories.

CodePipeline: automate your software release by defining a workflow of tests and approvals.

Key Management Service: if you manage encrypted data you will be familiar with the hassles of managing and rotating encryption keys. Here is a service to manage that.

AWS Config: A discovery service for the AWS resources you are using.

Service Catalog: a custom portal for users to browse and use AWS resources offered by an organisation.

This was day one; there is another keynote tomorrow and there may be more announcements.

There is no doubting the momentum behind AWS, and according to Jassy, there is still a long way to grow. Towards the end of the keynote he talked about businesses moving entire datacenters to AWS, for example when leases expire, and in the press Q&A session later he expressed the belief that eventually few companies will operate their own datacentres; he does not see much future for private cloud – in the sense of self-managed clouds on your own infrastructure. That is of course what you would expect Amazon to say.

Partnerships are key in this industry and I was interested to note the Reinvent sponsors:

image

The Diamond sponsors (who I presume have paid the most) are Accenture, Cloudnexa (AWS consultants), CSC (also consultants), Intel (I guess Amazon buys a lot of CPUs), Trend Micro and twilio (who must me doing well to be on this list).