Tag Archives: exchange

Crisis for ASP.Net – how serious is the Padding Oracle attack?

Security vulnerabilities are reported constantly, but some have more impact than others. The one that came into prominence last weekend (though it had actually been revealed several months ago) strikes me as potentially high impact. Colourfully named the Padding Oracle attack, it was explained and demonstrated at the ekoparty security conference. In particular, the researchers showed how it can be used to compromise ASP.NET applications:

The most significant new discovery is an universal Padding Oracle affecting every ASP.NET web application. In short, you can decrypt cookies, view states, form authentication tickets, membership password, user data, and anything else encrypted using the framework’s API! … The impact of the attack depends on the applications installed on the server, from information disclosure to total system compromise.

This is alarming simply because of the huge number of ASP.NET applications out there. It is not only a popular framework for custom applications, but is also used by Microsoft for its own applications. If you have a SharePoint site, for example, or use Outlook Web Access, then you are running an ASP.NET application.

The report was taken seriously by Microsoft, keeping VP Scott Guthrie and his team up all night, eventually coming up with a security advisory and a workaround posted to his blog. It does not make comfortable reading, confirming that pretty much every ASP.NET installation is vulnerable. A further post confirms that SharePoint sites are affected.

It does not help that the precise way the attack works is hard to understand. It is a cryptographic attack that lets the attacker decrypt data encrypted by the server. One of the consequences, thanks to what looks like another weakness in ASP.NET, is that the attacker can then download any file on the web server, including web.config, a file which may contain security-critical data such as database connection strings with passwords, or even the credentials of a user in Active Directory. The researchers demonstrate in a YouTube video how to crack a site running the DotNetNuke content management application, gaining full administrative rights to the application and eventually a login to the server itself.

Guthrie acknowledges that the problem can only be fixed by patching ASP.NET itself. Microsoft is working on this; in the meantime his suggested workaround is to configure ASP.NET to return the same error page regardless of what the underlying error really is. The reason for this is that the vulnerability involves inspecting the error returned by ASP.NET when you submit a corrupt cookie or viewstate data.

The most conscientious ASP.NET administrators will have followed Guthrie’s recommendations, and will be hoping that they are sufficient; it is not completely clear to me whether it is. One of the things that makes me think “hmmm” is that a more sophisticated workaround, involving random time delays before an error is returned, is proposed for later versions of ASP.NET that support it. What does that suggest about the efficacy of the simpler workaround, which is a static error page?

The speed with which the ASP.NET team came up with the workaround is impressive; but it is a workaround and not a fix. It leaves me wondering what proportion of ASP.NET sites exposed to the public internet will have implemented the workaround or do so before attacks are widespread?

A characteristic of the attack is that the web server receives thousands of requests which trigger cryptographic errors. Rather than attempting to fix up ASP.NET and every instance of web.config on a server, a more robust approach might be to monitor the requests and block IP numbers that are triggering repeated errors of this kind.

More generally, what should you do if you run a security-critical web application and a flaw of this magnitude is reported? Applying recommended workarounds is one possibility, but frankly I wonder if they should simply be taken offline until more is known about how to protect against it.

One thing about which I have no idea is the extent to which hackers are already trying this attack against likely targets such as ecommerce and banking sites. Of course in principle virtually any site is an attractive target, because of the value of compromised web servers for serving spam and malware.

If you run Windows servers and have not yet investigated, I recommend that you follow the links, read the discussions on Scott Guthrie’s blog, and at least implement the suggested actions.

Exchange 2007: ESEUTIL beats the wizard

Today I was asked to help find missing email in Small Business Server 2008, in other words Exchange 2007. Somehow, thousands of emails had disappeared from a user’s mailbox. They were there a couple of days earlier, so we restored a backup. The procedure is nicely explained by John Bay. You restore the Exchange database to a temporary directory, leaving Exchange up and running. Then you mount the restored backup into a “recovery storage group”, from where you can merge items from the recovered mailbox into the live one.

All went well, until the point where you mount the restored backup. The database would not mount. The event viewer said it was in an inconsistent state. Bay anticipates this, and suggests using the Exchange Troubleshooting Assistant to repair the database. The repair chugged away and completed; but the database still would not mount. I tried again, same result.

I gave up on the Assistant and reverted to the traditional command-line tools. I used the sequence:

ESEUTIL /P

ESEUTIL /D

ISINTEG

running against the recovered database. The first does a repair; the second defragments; and the third runs integrity checks and applies fixes.

Now, you would have thought that the Troubleshooting Assistant would use these very same tools underneath its pretty GUI, but that cannot be the case. Apart from anything else, ESEUTIL /P took much longer than the Assistant. In particular, it appeared to hang while doing something mysterious called Deleting Unicode fixup table.

image

It carried on saying this for around 7 hours. There was evidence that the process was still alive though, so I left it be.

It worked. I ran the other two commands, mounted the database, and merged the mailbox to recover about 4,000 emails.

image

The question remains: where did the emails go? All I know is that the problem coincided with a newly installed Windows and Outlook, which I’m guessing is significant.

The mystery of the slow Exchange 2007: when hard-coded values come back to haunt you

Following a migration from Microsoft Small Business Server 2003 to SBS 2008 users were complaining that Exchange was slower than before in some scenarios. How could this be? The new machine had 64-bit goodness and far more RAM than before.

I checked out the machine’s performance and noticed something odd. Store.exe, the Exchange database, usually grabs vast amounts of RAM, but in this case it was using surprisingly little, around 640MB. Could this be related to the performance issue?

I speculated that Exchange memory usage was limited in some way, so looked up where such a limit is set. I found this article. Ran ADSI Edit and there it was, a 640MB limit (or thereabouts), set in msExchESEParamCacheSizeMax.

I removed the limit, restarted Exchange 2007, and it immediately said “thank you very much” and grabbed 8GB instead.

Why did this setting exist? No doubt because back in the days of SBS 2003 and a much less powerful 32-bit machine, someone set it in order to prevent store.exe from crippling the box. It is another example of why Small Business Server is harder to manage than full server setups when Exchange invariably has a dedicated server (or several).

SBS 2008 cannot be installed as an in-place upgrade; but the official migration process does preserve Active Directory; and since that is where this value lives, and since it is not specific to any version of Exchange, it was dutifully transferred.

Why wasn’t the setting discovered and changed before? Well, you will observe that it is somewhat hidden. The main chances of finding it would be either if you were deeply schooled in the ways of Exchange, or if one of the Best Practices Analyzer (BPA) tools picked it up, or if the users screamed that Exchange was slow (which is what happened) and you figured out what was wrong.

The SBS BPA did not notice it. The Exchange BPA did, kind-of. It was not shown as a critical problem, but listed for information under “Non-Default Settings”, ironically with a tick beside it, as “Maximum ESE cache size changed”. Summoning help on this setting leads to this article which refers to Exchange 2000.

An admin failure, yes, but arguably also a defect in Exchange and SBS. Typical Microsoft: critical setting, hard-coded when it would make more sense to use a percentage value, not checked by setup and persistent across major upgrades of Exchange, deeply buried in Active Directory.

Mentioned here just in case it saves someone time when trying to figure out why their shiny 64-bit Exchange 2007 is running worse than 32-bit Exchange 2003 ever did.

Apple Snow Leopard and Exchange: the real story

Apple’s Snow Leopard (OS 10.6) came out last week, and one of its most hyped features is native support for Microsoft Exchange. Here’s what Apple says:

With Snow Leopard, the Mac is the only computer with built-in support for the latest version of Microsoft Exchange Server. So you can use your Mac — with all the features and applications you love — at home and at work and have all your messages, meetings, and contacts in one place.

What this means is that eager Mac users will be upgrading to Snow Leopard and expecting to be able to connect to Exchange at work with Apple-style “it just works” ease of configuration.

The truth is more complex; and I’m disappointed with both Apple’s publicity and the number of reviews that have simply reported its claims without investigation. That said, it is a tricky subject, and I have some sympathy with Apple, which is doing more or less the right thing at a technical level.

Configuring Snow Leopard Mail to use Exchange

The first thing to understand is that there are myriad ways of connecting to Exchange, including:

  • MAPI, which is Microsoft’s proprietary API
  • IMAP, which is a standard protocol for server-based email
  • ActiveSync, which is a Microsoft protocol used for mobile devices
  • RPC over HTTPS, effectively MAPI over SSL, enabling Outlook to connect from outside the network without VPN
  • Outlook Web Access, a web UI for Outlook
  • WebDAV, now deprecated
  • Exchange Web Services, which communicate using SOAP XML messages

Which of these protocols are actually enabled, and whether they are published beyond the internal network, is a matter for Exchange admins to configure.

The usual generic method to connect to Exchange from a miscellaneous client is IMAP, and this is exactly what Apple supported in Mail before Snow Leopard, and still supports. IMAP works pretty well in my experience, but it is only for email and does not expose any Exchange-specific features.

Snow Leopard adds support for Exchange Web Services (EWS), giving a much richer level of access to Exchange. First snag: EWS is only supported in Exchange 2007, which is why Apple says in its small print:

requires Microsoft Exchange Server 2007 Service Pack 1 Update Rollup 4

Second snag: even EWS does not all the features of MAPI, and some features (notably public folder support) were only added in Exchange 2007 SP2, which has just been released. This probably explains why Mail does not (as far as I can tell) support public folders.

The key thing to understand is that Snow Leopard is not using the same protocol as Outlook and therefore does not have access to the same set of features.

What works and what doesn’t

Let’s assume that you have Snow Leopard and Exchange 2007 SP1. What works and what doesn’t? Based on my experience so far:

  • You will be able to connect on an internal network or VPN, provided that EWS is enabled, which it usually is. You may need to install a digital certificate to avoid warning messages.
  • Mail, Calendar (iCal), tasks and notes in your Exchange mailbox all appear nicely.
  • When outside the network, you will only be able to connect over the Internet if EWS is published externally, which it often is not. You cannot use RPC over HTTPS.
  • There is no access to public folders (note that these are deprecated, but still widely used).
  • It is not possible to send from an email address other than the default.
  • You cannot use Exchange delegation features, such as accessing other mailboxes.
  • Mail will download the entire mailbox; you cannot set it only to download recent items. There is no “online mode” as there is with Outlook.
  • When offline, you can access existing items, but new messages have to be saved as drafts. This is unlike Outlook, which gives you full access to send mail, delete etc, and synchronises on re-connect.

Snow Leopard vs Entourage

You might imagine that Microsoft’s own Entourage product would do a better job than Apple Mail at connecting to Exchange. This is not necessarily the case. The problem is that Entourage 2008 doesn’t use MAPI either. In its first incarnation it uses WebDAV. This proved so problematic that Microsoft quietly released a new Web Services Edition that uses EWS, like Snow Leopard. Even this is a temporary expedient, as the Mac Business Unit has announced Outlook for the Mac. The implication is that it will be closer to feature-parity with Outlook on Windows, though it’s not clear to me whether this means MAPI, or EWS, or who knows what?

My view is that unless you need some specific feature of Entourage, or find that Entourage mysteriously works where Snow Leopard does not, you are likely better off without it. This presumes Exchange 2007, of course. The fundamental reason is that Mail and iCal are nicely integrated with the operating system, whereas Entourage is not so good in this respect; there have also been quality issues with Entourage.

It would be good to see a detailed technical note from Apple and/or Microsoft on Snow Leopard’s Exchange support, how to configure Exchange for it, and any implications for security etc. In the meantime, there is an interesting discussion on Apple’s forums which highlights the issues.

For all its (many) faults, Outlook on Windows remains a better Exchange client than either Snow Leopard or Entourage.