The first day of a new year is a great moment to relax and prepare for what is ahead – but spare a thought for Microsoft Exchange administrators who may have woken up to seized up installations of their on-premises email servers. I was among those affected, but only on my tiny system. Messages were stuck in the submission queue, suspiciously since midnight or thereabouts (somehow a message sneaked through timed 12.14 am) and the last error reported by the queue viewer was “Messages deferred by categorizer agent.”
As usual I went down a number of rabbit holes. Restart the Exchange Transport service. Reboot the server. Delete the first message not to be delivered in case it was corrupt and somehow clogging up the queue. Check for certificate issues.
It was none of these. Here is the guilty party in the event viewer:
The FIPS-FS Microsoft Scan Engine failed to load, with the error can’t convert “2201010001” too long.
The impact was that the malware filter could not check the message, hence the error from the categorizer agent.
The solution is to run the Exchange Shell on the server and navigate to the Scripts directory where Exchange is installed, for example C:\Program Files\Microsoft\Exchange Server\V15\Scripts. Here you will find a script called Disable-AntimalwareScanning.ps1.
& $env:ExchangeInstallPath\Scripts\Disable-AntimalwareScanning.ps1
should work. Run it, restart the Exchange Transport service, and email will start to flow.
Once the problem is patched, there is a companion script called Enable-AntimalwareScanning which restores it. Though I am not sure of the value of the Exchange malware filter since Microsoft considers that even on-premises installations should use the Microsoft 365 services for spam and malware scanning, and the on-premises protection features are not kept up to date, meaning that a third-party or open source spam and malware filter is a necessity anyway, unless you go the Office 365 route.
Another reason not to run Exchange on-premises – but Microsoft still says that hybrid systems using Azure Active Directory Connect should do so in order to manage mailboxes.
Note: the maximum value for a 32-bit signed integer is 2,147,483,647. Yesterday which was perhaps represented as 2,112,310,001 would have fitted within that whereas today 2,202,020,001 did not. Dates and times are awkward for programmers.
Update: Microsoft has an official fix here. Thanks to Erik in the comments for the link.
Thank you so much, Tim! I woke up to this exact same problem and started down the rabbit holes as you did. Was a relief to find someone who had encountered and resolved the issue. I appreciate you taking the time to post this for others encountering this!
Happy New Year! :o)
Dates and times are certainly difficult when programmers re-invent the wheel. How did that ever get adopted as a valid storage format, since presumably it would only ever work for the years 2000 to 2021?
How can I buy you a beer? You saved my new year’s.
I 100% share that feeling.
I just woke up to discover this bug in on of my environments. Thanks for this post!
Thank you, Thank you, Thank you!! Now I can watch some football instead of dealing with this all day.
Thank you so much, prevented headaches!!!
Seems Microsoft has still doesn’t know what their doing -.- (Millenium Issue should be gone I think)
Thank you very much and a happy new year! Seen the last error and the event log errors. Disabling the Anti-Malware scanning did the trick for now. Waiting for MS to fix this issue.
Thanks Tim, Saved my bacon.
Thank you! Your post was a life saver, I checked all the common sticking points and couldnt see an issue. This post was spot on.
Thanks Tim – you saved the day. Happy New Year!
THANK YOU for the article! We had it happen to us as well.
Thank you for your post this resolved my email issue.
Absolute life saver, had two production servers hit with this, both fixed by the above. Cheers!!
Another IT admin here with a heart felt Thank you, what a mess, especially when we/you don’t even utilize this service/feature.
Same issue, Same Rabbi holes!
Thanks!
Are we running into this problem, researching it, and posting it all because we’re just so dedicated?
Thanks for saving our collective backsides!
YOU ARE A LIVE SAVIOUR, MY FRIEND!!!
Oh my goodness Tim, a life saver and it is truly appreciated. Thank you so much.
Not all heroes where capes.
You just saved me a night of pulling my hair out.
Many thanks.
The article saved the day for a client of friends who got me out of bed at midnight to look at an exchange server. THANKS.
You sir, are a Gentlemen and a Scholar. Thank You !
Thank you so much, Tim! You are superman! My congratulation Happy New Year from Russia. I wish you strong health and big future!
Thank you very much , it solved the problem
I went through dead ends of troubleshooting but decided to go to the Microsoft Support route. No call from MS Support for three hours. Me thinks they are getting slammed. Thank you my friend. You Rock!
ECHO the above comments. It is 3 AM CST and I just discovered this post. How sh*t like this happens is unfathomable. I scanned EXCH server for malware, uninstalled and reinstalled DUO 2FA. Tested and retested firewall settings. Was getting ready for an unpleasant day. Thank you for this post Tim!