Vista’s mysterious compatibility settings: what do they do?

I hate this Program compatibility Assistant in Vista. Why?

First, because it applies settings whether you like it or not. There’s no option to say, “I was happy with how it ran, just leave it alone”.

Second, because it does not tell you what it has done. Sure, there is a link that says, What settings are applied? So you click it.

And you get a generic help dialog with six headings. You click the most promising: What changes does it make? It says:

It depends on the problem, but any changes made are related to how Windows runs the program. No changes are made to the program itself. For example, the Program Compatibility Assistant can resolve conflicts with User Account Control, a new security feature in this version of Windows that can help make your computer safer. Or, it can run the program in a mode that simulates earlier versions of Windows. The changes that Program Compatibility Assistant makes are done automatically, so you don’t need to make them.

Vague and uninformative. And that’s it.

So how do you really discover? Well, you could read this article. Then you can fire up RegEdit and look at:

Software\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers

under both HKEY_CURRENT_USER and HKEY_LOCAL_MACHINE. Here I found an entry for FlexBuilder.exe as follows:

OK, so Flex Builder will now have ELEVATECREATEPROCESS applied. What does that mean? Here’s the scoop:

Here the test program was trying to launch an updater which is required to run as administrator and failed. In this case, PCA will apply the ELEVATECREATEPROCESS compatibility mode, which will enable the program to successfully launch the child exe as administrator the next time. Now when the program is run the next time and while trying to launch the updater, it will not fail and will successfully run as administrator. The user will see the UAC consent UI.

More details on what happens under the covers is explained through Q/A below.

  1. What is the detection logic and how does PCA know that the program failed to launch a child exe which needs to run as administrator?The detection for this scenario is accomplished through instrumentation at the CreateProcess API to detect the cases when a child process launch fails due to the requirement to run as administrator.
  2. Why are there no options in this PCA dialog?Due to the high confidence on the issue detection in this scenario, the solution (ELEVATECREATEPROCESS compatibility mode) is automatically applied and the user is not given any options.

In my case I believe there was some problem with the Flex Builder debugger trying to launch FireFox, which was also trying to update itself.

I believe Adobe could avoid this problem by marking Flex Builder as UAC-aware. Then Windows will leave it alone.