Tag Archives: exchange

Publishing Exchange with pfSense

pfSense is a FreeBSD-based firewall which you can find here.

I wanted to publish Exchange through pfSense. I installed the Squid plugin which includes specific reverse proxy support for Exchange.

If you search for help with publishing Exchange on pfSense you will find this document by Mohammed Hamada.

Unfortunately the steps given seem to be incorrect in some places, certainly for my version which is 2.3.2.

Here’s what I had to do to get it working:

1. Simple one not mentioned in his steps, you have to enable the Squid Proxy Server otherwise Squid will not run

2. Hamada sets a NAT rule to forward HTTPS traffic to his Exchange server:

image

If you do this, it will bypass your reverse proxy. What you should do instead is to create a Firewall rule to accept HTTPS:

image

You should also verify that the pfSense web GUI is not using the same port (443), in System/Advanced/Admin Access. If it is set to HTTP rather than HTTPS that is OK too. Normally access to the web GUI from the WAN is blocked. One other thing: in order to use port 443 in Squid Reverse Proxy General Settings, I set net.inet.ip.portrange.reservedhigh to 0 in System/Advanced/System Tunables

3. I did this, as well as setting up Exchange in Squid Reverse Proxy General Settings, whereupon OWA worked but remote Outlook and mobile clients did not, or at least not reliably. The main problem was this setting in Squid Reverse Proxy / General:

image

This must be set to Intermediate rather than Modern (the default).

Now it works – though if pfSense experts out there have better ways to achieve the above I would be interested.

Update: one other thing to check, make sure that your pfSense box can resolve the internal hostname of your Exchange server. By default it may use external DNS servers even if you put internal DNS servers in General Setup. This is because of the setting Allow DNS server list to be overridden by DHCP/PPP on WAN.

Email hassles with Android and Exchange: couldn’t open connection to server

I have an Asus/Google Nexus 7 which is set up to receive mail from Exchange 2010.

At least, it was. Some time ago it stopped receiving mail. When the mail client tried to sync, I got this message:

image

“Couldn’t open connection to server due to security error”.

The message is not particularly clear. I went back into the account settings and verified everything (including Accept all SSL certificates, since mine is self-signed) and it was all fine – as I knew it would be, since it used to work.

The error, it turns out, is to do with ActiveSync policies. Exchange is detecting that the device is not in conformance, and refusing to sync. Odd, since my ActiveSync policies are relaxed and allow anything.

I removed the account and added it back. Ah, now I have this dialog:

image

I tapped Activate and everything was fine. Mail now syncs again.

I am still not sure how you find this dialog if it does not pop up automatically.

Fixing a Small Business Server 2008 broken by updates

I had a call last night from a small business whose email no longer worked. They had applied updates to the server but Exchange had failed to restart.

Looking at the services it was easy to see why. All the Exchange services and certain others including the IIS web server were set to disabled:

image

The likely culprit was Update Rollup 5 for Exchange Server 2007 Service Pack 3 (KB 2602324) – or rather, the mechanism which applies the patch, since this seems to be an issue that others have run into as far back as 2008 with other Exchange patches, though it is rare:

I installed the Update Rollup 4 and did a reboot of my Exchange Server 2007. But since then, all my services are disabled. Is this a known issue?

My guess is that the patch disables the services in order to update the binaries and then, for some unknown reason not fixed by Microsoft over these last four years, fails to re-enable them.

It seems that no harm was done other than that the services were disabled, but how can you know which services are meant to be running, which should be set to manual, and which should stay disabled?

I contemplated doing a quick test install of SBS 2008 on a VM just  to see how it is set out of the box, but fortunately found this post by Susan Bradley which shows default SBS 2008 running services.

There were a few other things wrong.  SharePoint Services was raising event 5586:

Unknown SQL Exception 33002 occured. Additional error information from SQL Server is included below. Access to table dbo.Versions is blocked because the signature is not valid.

and there was the related event 33002 from the internal SQL Server used by SharePoint. The cause of this was SharePoint Services 3.0 Service Pack 3. When you apply a major update to SharePoint Services, you have to re-run the SharePoint Products and Technologies Configuration Wizard. This is by design, though it seems odd to me that you apply an update and it silently breaks the product it is updating until you run a further manual process. Of course the error itself does not give you much clue about what is really wrong.

The third major issue was a JRNL_WRAP_ERROR from the NTFrs File Replication Service. You have to be careful with this one, since the advised fix in the event log presumes the presence of a good replica elsewhere, which in the case of SBS is unlikely. See this article for details. With SBS which it is the sole domain controller you should set the BurFlags registry key to D4. Further comment on ServerFault here.

The incident reminds me of how prickly SBS can be. It is great value for what it does, but has all the complexity of Microsoft’s server stack plus the further disadvantage of being crammed onto one machine. I prefer a pseudo multi-server approach, even for small businesses, with at least two physical servers and separate VMs for Exchange, SharePoint, domain controller, backup DC on the other physical machine, and so on. Of course this has complexity of its own.

I would guess that when upgrade time comes around, companies like this will be looking carefully at Office 365. Or Google Apps; but the advantage of Office 365 is that you can make the transition from SBS with relatively little impact on users: just migrate the Active Directory, Exchange and SharePoint. You lose flexibility and some local performance, but hand over the maintenance issues to Microsoft.

Windows Phone, Exchange, and self-signed certificates

I am setting up a Nokia Lumia phone, which runs Windows Phone 7.5 “Mango”. I am impressed with the smoothness of the setup experience, though I had one slight hassle which I do not blame on Nokia or Microsoft because it is is not typical.

I run an Exchange Server (and SharePoint) for test purposes with a self-signed certificate. On the iPhone and I think Android, you just get a warning the first time you connect and after that it all works. Windows Phone though gives you a certificate error when Outlook tries to connect and will not let you proceed further.

Which of these approaches is right is a moot point: convenience versus security I guess.

Fortunately it is not too hard to get Windows Phone to trust your certificates. Here is what I did:

1. Email the certificate(s) to myself on Google Mail.

2. Go to Google Mail in the phone’s browser. For this to work I found I had to use the Basic HTML view, as the mobile view did not let me download the attachments.

3. Open the email you sent to yourself. Tap the attachment. Windows Phone offers to install the certificate, tap to install.

4. Now go back to Outlook and synchronise, everything works.

You will also need this for SharePoint if you want to use Office on the device.

Of course if your Exchange and SharePoint use a certificate from a trusted certificate provider then these steps are not necessary.

Microsoft Office 365: the detail and the developer story

I attended the UK launch of Office 365 yesterday and found it a puzzling affair. The company chose to focus on small businesses, and what we got was several examples of customers who had discovered the advantages of storing documents online. We were even shown a live video conference with a jerky, embarrassing webcam stream adding zero business value and reminding me of NetMeeting back in 1995 – which by the way was a rather cool product. Most of what we saw could have been done equally well in Google Apps, except for a demo of the vile SharePoint Workspace for offline editing of a shared document, though if you were paying attention you could see that the presenter was not really offline at all.

There seems to be a large amount of point-missing going on.

There is also a common misconception that Office 365 is “Office in the cloud”, based on Office Web Apps. Although Office Web Apps is an interesting and occasionally useful feature, it is well down the list of what matters in Office 365. It is more accurate to say that Office 365 is for those who do not want to edit documents in the browser.

I am guessing that Microsoft’s focus on small businesses is partly a political matter. Microsoft has to offer an enterprise story and it does, with four enterprise plans, but it is a sensitive matter considering Microsoft’s relationship with partners, who get to sell less hardware and will make less money installing and maintaining complex server applications like Exchange and SharePoint. The, umm, messaging at the Worldwide Partner Conference next month is something I will be watching with interest.

The main point of Office 365 is a simple one: that instead of running Exchange and SharePoint yourself, or with a partner, you use these products on a multi-tenant basis in Microsoft’s cloud. This has been possible for some time with BPOS (Business Productivity Online Suite), but with Office 365 the products are updated to the latest 2010 versions and the marketing has stepped up a gear.

I was glad to attend yesterday’s event though, because I got to talk with Microsoft’s Simon May and Jo Carpenter after the briefing, and they answered some of my questions.

The first was: what is really in Office 365, in terms of detailed features? You can get this information here, in the Service Description documents for the various components. If you are wondering what features of on-premise SharePoint are not available in the Office 365 version, for example, this is where you can find out. There is also a Support Service Description that sets out exactly what support is available, including response time objectives. Reading these documents is also a reminder of how deep these products are, especially SharePoint which is a programmable platform with a wide range of services.

That leads on to my second question: what is the developer story in Office 365? SharePoint is build on ASP.NET, and you can code SharePoint applications in Visual Studio and deploy them to Office 365. Not all the services available in on-premise SharePoint are in the online version, but there is a decent subset. Microsoft has a Sharepoint Online for Office 365 Developer Guide with more details.

Now start joining the dots with technologies like Active Directory Federation Services – single sign-on to Office 365 using on-premise Active Directory – and Windows Azure which offers hosted SQL Server and App Fabric middleware. What about using Office 365 not only for documents and email, but also as a portal for cloud-hosted enterprise applications?

That makes sense to me, though there are still limitations. Here is a thread where someone asks:

Does some know if it is possible to make a database connection with Office365, SharePoint (Designer) and SQL Azure database ?

and the answer from Microsoft’s Mark Kashman on the SharePoint team:

You cannot do this via SharePoint Designer today. What you can do is to create a Silverlight or javaScript client application that calls out to SQL Azure.

In the near future, we are designing a way to make these connections using the base SharePoint technology called BCS (Business Connectivity Services) where then you could develop a service to service to SQL Azure.

If you cannot wait, check out the Cloud Connector for SharePoint 2010 from Layer 2 GmbH.

It seems obvious that Office 365 and Azure together have potential as a developer platform.

What about third-party applications and extensions for Office 365? This is another thing that Microsoft did not talk about yesterday; but it seems to me that there is potential here as well. It is not well integrated, but you can search Microsoft Pinpoint for Office 365 applications and get some results. If Office 365 succeeds, and I think it will, there is an opportunity for developers here.

Notes from the field: migrating a small business to Microsoft BPOS

Today I assisted a (very) small company migrate from Small Business Server 2003 to BPOS (Business Productivity Online Suite), Microsoft’s hosted Exchange and SharePoint.

Why BPOS, when Office 365 launches later this month? Well, BPOS has all the features they need, and when given the choice between a beta-soon-to-be-just-launched online platform, and one that has been around for a few years, they chose the latter, which is reasonable. Long term it will make no difference, because BPOS users will be migrated to Office 365. It was interesting to me, since I am reviewing Office 365 and this migration gave me good insight into the differences.

Aside: the fact that this is a choice says something about Microsoft. One of the advantages of cloud computing is that improvements can be continuous and incremental, since the software is paid for by subscription rather than through a version upgrade cycle. There is only one Salesforce.com platform; there is only one Google Apps platform. Will there be an Office 720 in two to three years time, or will Microsoft have worked this out by then? It will be hard, because no doubt there are teams working on Exchange 2013 and SharePoint 2013 and these will be delivered as on-premise product upgrades. This also implies that the new features in these products will not be considered ready until the on-premise products go gold; which means that cloud customers have to wait a long time for major enhancements. Changing this cycle will require a profound shift in the way the company functions.

Now a few comments about the process. Overall it was pretty good, and took less time than it normally takes to migrate from one version of Small Business Server to the next. There are  annoyances though, beginning with the migration tools. The challenge is that you want to move mailboxes from SBS Exchange to online Exchange without losing any email.

Email coexistence

The basic approach is this:

1. A directory synchronisation tool copies user accounts to BPOS and keeps them in synch with on-premise Active Directory.

2. A mailbox migration tool copies mailboxes to BPOS and sets up forwarding, so email arriving into on-premise Exchange is forwarded to BPOS.

This is known as email co-existence, because users can log on to either on-premise Exchange or BPOS, and still be able to send and receive mail. Clever stuff, and it does make migration nice and smooth.

The first annoyance: the directory synchronization tool must be installed on a 32-bit Windows Server that is joined to the domain but not a domain controller. Many SBS setups do not have such a thing. In this case, I ran up Virtual PC on Windows 7 64-bit, installed 32-bit Server 2003, joined it to the domain (actually over a VPN), and ran the tool from there.

Actual mailbox migration uses a separate tool which fortunately does run on the SBS server itself. One the users are in place and enabled on BPOS, you run this tool to upload the mailboxes. This takes a while, since you are uploading what is probably several Gb of data. I left this running overnight, but it was only partially successful. Two mailboxes did not upload properly and had to be redone, which was a bit untidy because in one case some folders were duplicated. Fortunately it was not hard to clean up.

Once the mailboxes are migrated, you simply install and run Microsoft’s sign-in utility on each client PC. This automatically configures Outlook with a new BPOS profile, leaving the old profile in place in case of mishaps.

The last step is to change the DNS records so that mail is actually delivered to BPOS rather than to on-premise Exchange.

SharePoint migration

This particular company is reliant on SharePoint for document sharing. Although it is SBS 2003, they have SharePoint Services 3.0 installed; it can be done if you are careful.

Major annoyance: the BPOS documentation is silent on the subject of migrating content. There is a heading in the Migration Help for SharePoint Online; but it does not cover migration from on-premise to BPOS SharePoint at all. There are third-party tools that do this though, and some help from the community.

Of course there are multiple ways to move SharePoint content, though in some cases you will lose version information. I found this article helpful. I was able to start from Step 5, since it was already a SharePoint 3.0 site. Look how clear and concise the steps are; a refreshing contrast to Microsoft’s verbose efforts with seemingly endless sections for overviews, planning and deployment, that take ages to get to the point and still manage to omit key information.

I read the post and the comments, then created a blank site in BPOS. I backed up and then exported the existing site to CMP files, and kept it locked so that no new content would be added. Then I installed SharePoint Designer 2007, which is free, logged into the BPOS site and restored the site.

All the important things restored correctly. Unfortunately the permissions do not migrate, because the BPOS domain is different from the SBS domain; the active directory is only synchronized. I had to fix this up by deleting dud users and groups from the new site and adding groups and users for BPOS. I also added a few web parts to the otherwise blank home page. Nevertheless, considering how painful SharePoint migrations can be this one was pretty good.

I understand though that this simple approach does not always work. I would guess that the more SharePoint is customised, the more likely you are to have problems, which is probably why there is no official tool.

Exchange issues

There were a couple of issues with Exchange. The first was public folders, which are not supported in BPOS. The solution is to use SharePoint lists. Here is how it goes:

  1. Open an on-premise Outlook profile with full access to the public folders. Export each public folder to an Outlook .pst file – the best format for preserving all the data.
  2. Go to SharePoint online and create a new list of the appropriate type. For example, for tasks you need a task list, for contacts a contact list.
  3. Open Outlook with a BPOS profile. In SharePoint online, go to the target list and choose Connect to Outlook from the Actions menu.
  4. The SharePoint list now exists in Outlook. Open the .pst and copy the items exported from the public folder to the new list.
  5. Other users need only connect to the SharePoint list. The magic of synchronization copies the content.

Another issue is mailbox permissions. If you have users who want access to another user’s mailbox, you have to set permissions on the target mailbox to allow this. These permissions do not get migrated automatically. To do this, you have to use PowerShell. This article explains. The easiest route to a correctly configured PowerShell is to use the shortcut to the Migration Command Shell which is installed with the Microsoft Online migration tools.

A note on cost

BPOS costs $10 per user per month, with a minimum of 5 users. In the UK this is £6.72, so from £33.60 per month. Along with Exchange and SharePoint, you get Office Live Meeting (Web Conferencing) and Office Communications Online (Enterprise Live Messenger).

A typical SBS server lasts 3 to 5 years before it has to be replaced. Taking the shorter time for example, BPOS will cost £1209.60 in subscription costs over the lifetime of a physical server.

It seems obvious that if Exchange and SharePoint is all you need, and if you are happy with the implications of the cloud approach, BPOS works out cheaper. Of course the pricing will change, but Office 365 is actually coming out at a similar price for the equivalent features. Yes, you could buy a basic server with SBS for £1209, but that is just the start: installation, firewall, backup and maintenance all add to the cost.

That said, most businesses will still need some kind of on-premise server, even if it is no more than a simple NAS (Network Attached Storage) box. Another real-world problem is that there may be server-based applications which cannot easily be abandoned. If you find you have to run an on-premise server anyway, adding BPOS on top looks less attractive.

There is much more to say about cloud vs on-premise, but it is worth noting that it can be cost-effective.

Disappearing items in Outlook and Exchange

I came across what looks to me like an unusual bug, most likely in Microsoft Outlook. Background: I have used the Notes folder in my Exchange mailbox for all sorts of information going back several years. This morning, I looked at the folder and found it empty, except for one solitary item. Normally there are over 1000. The surviving item was the result of my last search in that folder.

image

Now, the Exchange database is robust in my experience; and most often when items disappear it is not a bug but a result of Outlook working as designed but catching the user out in some way. Here are some common reasons:

  1. The items got auto-archived. Archives can be present on any machine on which you run Outlook. The default location for the archive folder is in a hidden location such as C:\Users\[USERNAME]\AppData\Local\Microsoft\Outlook\archive1.pst, where [USERNAME] is your Windows username. It really is hard for users to find this without expert help. How can the popular corporate mail client have usability like this? But I digress. The solution is to open the archive in Outlook and drag items back where they are wanted.
  2. The items are present, but filtered out by the view. Views in Outlook can be filtered to restrict the items on display, for example to unread items only. The user set the filter by clicking something in Outlook’s labyrinthine user interface, but does not realise it is still set. The effect is that items disappear. The solution is to reset the filter.
  3. The user accidentally dragged a folder inside another folder. This is easily done, as Outlook does not prompt you when you do this; it just moves the folder. The symptom is that a little expand symbol appears in the target folder, if it does not already have subfolders. The solution is to drag it back.
  4. The user accidentally deleted the items or folder. Outlook does not prompt when you delete items. In this case, however, the items end up in the Deleted Items folder. The solution is to drag them back where you want them.

Even if the user has subsequently emptied the deleted items folder, there is hope. Outlook has a little-known feature called Recover Deleted Items. Items go into a kind of hidden deleted items folder for a period after they get removed from the visible deleted items folder, or if they are removed with Shift-Delete. Recover Deleted Items, which is on the Folder tab in Outlook 2010 and on the Tools menu in earlier versions, will let you get them back.

My disappeared notes were nowhere to be found. Further, the evidence is that I had not deleted them, since the surviving item was the result of a search. There is no command that I know of to delete all items in a folder other than the result of a search.

Still, I wanted to get them back if possible; and preferably without restoring Exchange to an earlier date, this being a fairly slow and painful operation. I checked my laptop without connecting it to the network, to see if this had an offline copy. My laptop runs Outlook 2007. There was no offline copy, since it had synchronised subsequent to the items disappearing.

Incidentally, this is why synchronisation and redundancy are not the same as backup.

I had one more go at Recover Deleted Items. Curiously, Outlook 2007 does have a Recover Deleted Items option for the Notes folder, whereas Outlook 2010 does not. Note though that the deleted items live not in the local offline store, but in Exchange.

To my surprise, all my old notes were there. I selected them all in the Recover Deleted Items window and clicked to undelete. Now I am back where I was, except that all my old notes now have a “Created” date of today. A nuisance, but a good outcome nonetheless.

But what happened? I have two questions about this. One is how the items got deleted in the first place. The second is how they ended up in Recover Deleted Items. The documentation for Recover Deleted Items will make your head spin. It is an Exchange feature, but apparently controlled by Outlook. This knowledgebase article says it only works on the Deleted items folder, unless you are using Outlook 2007 when it works on all the folders.

Does that suggest that is was Outlook 2007 that deleted my items? But how could Outlook 2007 on one machine delete all the items except the result of a search in Outlook 2010 on another machine? It does not make sense.

My view is that Outlook has become so obscure and intricate in its inner workings that anything is possible. I think Microsoft should build a new Exchange client.

UK business applications stagger towards the cloud

I spent today evaluating several competing vertical applications for a small business working in a particular niche – I am not going to identify it or the vendors involved. The market is formed by a number of companies which have been serving the market for some years, and which have Windows applications born in the desktop era and still being maintained and enhanced, plus some newer companies which have entered the market more recently with web-based solutions.

Several things interested me. The desktop applications seemed to suffer from all the bad habits of application development before design for usability became fashionable, and I saw forms with a myriad of fields and controls, each one no doubt satisfying a feature request, but forming a confusing and ugly user interface when put together. The web applications were not great, but seemed more usable, because a web UI encourages a simpler page-based approach.

Next, I noticed that the companies providing desktop applications talking to on-premise servers had found a significant number of their customers asking for a web-hosted option, but were having difficulty fulfilling the request. Typically they adopted a remote application approach using something like Citrix XenApp, so that they could continue to use their desktop software. In this type of solution, a desktop application runs on a remote machine but its user interface is displayed on the user’s desktop. It is a clever solution, but it is really a desktop/web hybrid and tends to be less convenient than a true web application. I felt that they needed to discard their desktop legacy and start again, but of course that is easier said than done when you have an existing application widely deployed, and limited development resources.

Even so, my instinct is to be wary of vendors who call desktop applications served by XenApp or the like cloud computing.

Finally, there was friction around integrating with Outlook and Exchange. Most users have Microsoft Office and use Outlook and Exchange for email, calendar and tasks. The vendors with web application found their users demanding integration, but it is not easy to do this seamlessly and we saw a number of imperfect attempts at synchronisation. The vendors with desktop applications had an easier task, except when these were repurposed as remote applications on a hosted service. In that scenario the vendors insisted that customers also use their hosted Exchange, so they could make it work. In other words, customers have to build almost their entire IT infrastructure around the requirements of this single application.

It was all rather unsatisfactory. The move towards the cloud is real, but in this particular small industry sector it seems slow and painful.

Crisis for ASP.Net – how serious is the Padding Oracle attack?

Security vulnerabilities are reported constantly, but some have more impact than others. The one that came into prominence last weekend (though it had actually been revealed several months ago) strikes me as potentially high impact. Colourfully named the Padding Oracle attack, it was explained and demonstrated at the ekoparty security conference. In particular, the researchers showed how it can be used to compromise ASP.NET applications:

The most significant new discovery is an universal Padding Oracle affecting every ASP.NET web application. In short, you can decrypt cookies, view states, form authentication tickets, membership password, user data, and anything else encrypted using the framework’s API! … The impact of the attack depends on the applications installed on the server, from information disclosure to total system compromise.

This is alarming simply because of the huge number of ASP.NET applications out there. It is not only a popular framework for custom applications, but is also used by Microsoft for its own applications. If you have a SharePoint site, for example, or use Outlook Web Access, then you are running an ASP.NET application.

The report was taken seriously by Microsoft, keeping VP Scott Guthrie and his team up all night, eventually coming up with a security advisory and a workaround posted to his blog. It does not make comfortable reading, confirming that pretty much every ASP.NET installation is vulnerable. A further post confirms that SharePoint sites are affected.

It does not help that the precise way the attack works is hard to understand. It is a cryptographic attack that lets the attacker decrypt data encrypted by the server. One of the consequences, thanks to what looks like another weakness in ASP.NET, is that the attacker can then download any file on the web server, including web.config, a file which may contain security-critical data such as database connection strings with passwords, or even the credentials of a user in Active Directory. The researchers demonstrate in a YouTube video how to crack a site running the DotNetNuke content management application, gaining full administrative rights to the application and eventually a login to the server itself.

Guthrie acknowledges that the problem can only be fixed by patching ASP.NET itself. Microsoft is working on this; in the meantime his suggested workaround is to configure ASP.NET to return the same error page regardless of what the underlying error really is. The reason for this is that the vulnerability involves inspecting the error returned by ASP.NET when you submit a corrupt cookie or viewstate data.

The most conscientious ASP.NET administrators will have followed Guthrie’s recommendations, and will be hoping that they are sufficient; it is not completely clear to me whether it is. One of the things that makes me think “hmmm” is that a more sophisticated workaround, involving random time delays before an error is returned, is proposed for later versions of ASP.NET that support it. What does that suggest about the efficacy of the simpler workaround, which is a static error page?

The speed with which the ASP.NET team came up with the workaround is impressive; but it is a workaround and not a fix. It leaves me wondering what proportion of ASP.NET sites exposed to the public internet will have implemented the workaround or do so before attacks are widespread?

A characteristic of the attack is that the web server receives thousands of requests which trigger cryptographic errors. Rather than attempting to fix up ASP.NET and every instance of web.config on a server, a more robust approach might be to monitor the requests and block IP numbers that are triggering repeated errors of this kind.

More generally, what should you do if you run a security-critical web application and a flaw of this magnitude is reported? Applying recommended workarounds is one possibility, but frankly I wonder if they should simply be taken offline until more is known about how to protect against it.

One thing about which I have no idea is the extent to which hackers are already trying this attack against likely targets such as ecommerce and banking sites. Of course in principle virtually any site is an attractive target, because of the value of compromised web servers for serving spam and malware.

If you run Windows servers and have not yet investigated, I recommend that you follow the links, read the discussions on Scott Guthrie’s blog, and at least implement the suggested actions.

Exchange 2007: ESEUTIL beats the wizard

Today I was asked to help find missing email in Small Business Server 2008, in other words Exchange 2007. Somehow, thousands of emails had disappeared from a user’s mailbox. They were there a couple of days earlier, so we restored a backup. The procedure is nicely explained by John Bay. You restore the Exchange database to a temporary directory, leaving Exchange up and running. Then you mount the restored backup into a “recovery storage group”, from where you can merge items from the recovered mailbox into the live one.

All went well, until the point where you mount the restored backup. The database would not mount. The event viewer said it was in an inconsistent state. Bay anticipates this, and suggests using the Exchange Troubleshooting Assistant to repair the database. The repair chugged away and completed; but the database still would not mount. I tried again, same result.

I gave up on the Assistant and reverted to the traditional command-line tools. I used the sequence:

ESEUTIL /P

ESEUTIL /D

ISINTEG

running against the recovered database. The first does a repair; the second defragments; and the third runs integrity checks and applies fixes.

Now, you would have thought that the Troubleshooting Assistant would use these very same tools underneath its pretty GUI, but that cannot be the case. Apart from anything else, ESEUTIL /P took much longer than the Assistant. In particular, it appeared to hang while doing something mysterious called Deleting Unicode fixup table.

image

It carried on saying this for around 7 hours. There was evidence that the process was still alive though, so I left it be.

It worked. I ran the other two commands, mounted the database, and merged the mailbox to recover about 4,000 emails.

image

The question remains: where did the emails go? All I know is that the problem coincided with a newly installed Windows and Outlook, which I’m guessing is significant.