Performance testing Vista: misleading reporting of inadequate tests.

Is Vista really half the speed of XP? That is what CNET reports here, in an article which does not bother to link to the source of the tests, as far as I can see.

The tests seem to be based mainly on scripting Office through OLE automation. This is certainly interesting, but it would be more helpful to have a performance breakdown, looking at various aspects of OS performance. Otherwise, how do we know whether this is more to do with, say, OLE performance than Vista performance?

Second, Vista has richer graphics than XP. Think of it like a game which offers difference graphics options to suit your hardware. Typically, you can set varying levels of background detail, shadow effects, etc, in order to find the right compromise between appearance and performance. Vista sets this higher than XP by default, so you would expect screen operations to be slower.

Third, Vista’s UAC security imposes a significant performance overhead. Again you would expect that, since it applies additional checks to numerous operations.

You could conduct a more useful test by configuring Vista to work as much as possible like XP. Turn off UAC, turn off Aero, turn off the indexer, turn off visual effects. I would also suggest checking what is running in the background and matching it as closely as possible on the two machines. I would be surprised if the performance difference is so striking after doing these things.

Perhaps you will argue that this is not the real-world experience. In practice, users take the machine as supplied and start using it; they do not open Control Panel and look at Performance Information and Tools, or get rid of all that third-party foistware. It’s a fair point, though on a corporate network admins can set these things on the user’s behalf.

When it comes to real-world performance, I also have gripes about Vista. However, I have little problem with Vista’s overall performance, say when working in Office, aside from Outlook 2007 which is a disgrace. What bugs me is unexpected delays. For example, I click the Start button, then All Programs, then scroll down to a program group and click. Vista just stops. I tried it just now, and it took 12 seconds to open the group I chose. I’ve had to train myself not to keep clicking. What’s going on there?

There are real issues here, but not helped by misleading reporting of inadequate tests.

4 thoughts on “Performance testing Vista: misleading reporting of inadequate tests.”

  1. Hmm.. You could conduct a more useful test by configuring Vista to work as much as possible like XP. Turn off UAC, turn off Aero, turn off the indexer, turn off visual effects.

    But as you then say, that’s not how Vista is for people. If flashy graphics and indexers don’t get out of the way, people get frustrated – with both Vista *and* Leopard. Don’t have to look far for people who aren’t delighted with Apple’s latest meisterwerk.

  2. I don’t understand *why* these “fancy graphics” require extra CPU load. Graphics cards are so powerful these days, and their processors underused. Their 2D performance has improved very little over the last decade, but their 3D performance is through the roof! Linux based compositing technologies, such as Compiz-Fusion, actually speed up your desktop by offloading the drawing work to the graphics processor, freeing up the CPU to calculate your spreadsheets (for example) faster. Even with “wobbly windows” etc. my computer running Linux (Ubuntu Gutsy) renders massively faster than it does under either Vista or XP. So, do not jump to the conclusion that

    “Second, Vista has richer graphics than XP […] so you would expect screen operations to be slower.”

    – this does not have to be true.

    By the way, I found your post through the Blog Friends facebook application.

  3. I don’t understand *why* these “fancy graphics” require extra CPU load.

    A point well made, particularly since Vista makes use of DirectX. It’s still a problem for performance tests though. When you enable Aero, you also enable the DWM (Desktop Window Manager) which draws windows to an off-screen buffer; it’s a major change to how screens are rendered. In principle it could actually improve performance over XP. But what do we want to compare: Vista vs XP, or DWM vs old-style window painting? The simplistic performance test is just not very interesting.

    Tim

  4. I am a sysadmin who uses Vista Business with the classic interface, UAC off, and Defender and indexing disabled. The OS feels quick, has been stable and I love the new system/resource monitoring tools.

    I find that I need to make less changes to Vista than XP Pro to make it run cleanly and quickly.

Comments are closed.