The Windows Runtime App Certification Kit: not too good detecting crashes and hangs

Now that I have a lovely ITWriting.com App I thought I should check out whether it is ready to fly.

I therefore ran the App Certification Kit that installs with Visual Studio 2012.

image

The tool asks you to select an installed app and then exercises it. I saw my app open, though I did not see it get beyond the first screen.

image

Eventually – bad news:

image

However, there is only one thing wrong with it:

image

Yes, the version installed is the debug build. I can fix this simply by rebuilding in release configuration.

What does the Kit test? Here is the list:

  • Crashes and hangs test
  • App manifest compliance test
  • Windows security features test
  • Supported API test
  • Performance test
  • App manifest resources test
  • Debug configuration test
  • File encoding
  • Direct3D feature level support
  • App Capabilities test
  • Windows Runtime metadata validation

That sounds most impressive and makes a great list for you to show to your customer.

I am sceptical though. If the app was not exercised beyond the opening screen, might it not be a bit buggy after all?

I inserted the following line of code into the the Click event handler for reading a blog:

int iCrash = 1 / string.Empty.Length;

I then rebuilt the app in release mode and ran the App Certification Test. Great news!

image

and specifically:

image

Thanks though to my umm, bug-unfix, the app crashes whenever I click to read a blog.

I mention this not to poke fun at the App Certification Kit, but to observe that it does not do a good job of automatically detecting crashes and hangs.

The implication is that the human testers are the ones who will do this before an app is admitted to the store. I think they would find my obvious bug; but how much time will they have to test every feature of an app?

3 thoughts on “The Windows Runtime App Certification Kit: not too good detecting crashes and hangs”

  1. I think a proof that an app does not crash or hang is essentially a version of the halting problem, and as such I did not expect this little tool to solve it.
    I submitted an app last week with a rather obvious bug in it, and indeed the testers from Microsoft found it and added a short description to the certification report.
    I found that very helpful.

  2. Shatters your faith in the App Certification Kit when it proclaims to have done something it hasn’t. Reminds of an incident a few years ago when I used to run a Porsche. For sevice I always took it to the authorized Porsche dealer because they provided a lengthy itemized checklist of work completed. This gave me faith in their thoroughness. Until on collecting the car from one service I noticed the ashtray hadn’t been emptied, although it had been checked on the list as completed. Shattered my faith in the Porsche dealer and I sold the car. So app test or car service either way “don’t say it’s done if it isn’t”.

Comments are closed.