Category Archives: Notes from the field

Microsoft’s Dynamics CRM 2016/365: part brilliant, part perplexing, part downright sloppy

I have just completed a test installation of Microsoft’s Dynamics CRM on-premises; it is now called Dynamics 365 but the name change is cosmetic, and in fact you begin by installing Dynamics CRM 2016 and it becomes Dynamics 365 after applying a downloaded update.

Microsoft’s Dynamics product has several characteristics:

1. It is fantastically useful if you need the features it offers

2. It is fantastically expensive for reasons I have never understood (other than, “because they can”)

3. It is tiresome to install and maintain

I wondered if the third characteristic had improved since I last did a Dynamics CRM installation, but I feel it has not much changed. Actually the installation went pretty much as planned, though it remains fiddly, but I wasted considerable time setting up email synchronization with Exchange (also on-premises). This is a newish feature called Server-Side Synchronization, which replaces the old Email Router (which still exists but is deprecated). I have little love for the Email Router, which when anything goes wrong, fills the event log with huge numbers of identical errors such that you have to disable it before you can discover what is really going wrong.

Email is an important feature as automated emails are essential to most CRM systems. The way the Server-Side Synchronization works is that you configure it, but CRM mailboxes are disabled until you complete a “Test and Enable” step that sends and receives test emails. I kept getting failures. I tried every permutation I could think of:

  • Credentials set per-user
  • Credentials set in the server profile (uses Exchange Impersonation to operate on behalf of each user)
  • Windows authentication (only works with Impersonation)
  • Basic authentication enabled on Exchange Web Services (EWS)

All failed, the most common error being “Http server returned 401 Unauthorized exception.” The troubleshooting steps here say to check that the email address of the user matches that of the mailbox; of course it did.

An annoyance is that on my system the Test and Enable step does not always work (in other words, it is not even tried). If I click Test and Enable in the Mailbox configuration window, I get this dialog:

image

However if I click OK, nothing happens and the dialog stays. If I click Cancel nothing happens and the dialog stays. If I click X the dialog closes but the test is not carried out.

Fortunately, you can also access Test and Enable from the Mailbox list (select a mailbox and it appears in the ribbon). A slightly different dialog appears and it works.

I was about to give up. I set Windows authentication in the server profile, which is probably the best option for most on-premises setups, and tried the test one more time. It worked. I do not know what changed. As this tech note (which is about server-side synchronization using Exchange Online) remarks:

If you get it right, you will hear Microsoft Angels singing

But what’s this about sloppy? There is plenty of evidence. Things like the non-functioning dialog mentioned above. Things like the date which shows for a mailbox that has not been tested:

image

Or leaving aside the email configuration, things like the way you can upload Word templates for use in processes, but cannot easily download them (you can use a tool like the third-party XRMToolbox).

And the script error dialog which has not changed for a decade.

Or the warning you get when viewing a report in Microsoft Edge, that the browser is not supported:

image

so you click the link and it says Edge is supported.

Or even the fact that whenever you log on you get this pesky dialog:

image

So you click Don’t show this again, but it always reappears.

It seems as if Microsoft does not care much about the fit and finish of Dynamics CRM.

So why do people persevere – in fact, the Dynamics business is growing for Microsoft, largely because of Dynamics 365 online and its integration with Office 365. The cloud is one reason, which removes at least some of the admin burden. The other thing though is that it does bring together a set of features that make it invaluable to many businesses. You can use it not only for sales and marketing, but for service case management, quotes, orders and invoices.

It is highly customizable, which is a mixed blessing as your CRM installation becomes increasingly non-standard, but does mean that most things can be done with sufficient effort.

In the end, it is all about automation, and can work like magic with the right carefully designed custom processes.

With all those things to commend it, it would pay Microsoft to work at making the user interface less annoying and the administration less prone to perplexing errors.

Configuring the Android emulator for Hyper-V

Great news that the Android emulator now supports Hyper-V, but how do you enable it?

Pretty simple. First, you have to be running at least Windows 10 1803 (April 2018 update). Then, go into Control Panel – Programs – Turn Windows Features on and off and enabled both Hyper-V and the Windows Hypervisor Platform:

image

Note: this is not the same as just enabling Hyper-V. The Windows Hypervisor Platform, or WHPX, is an API for Hyper-V. Read about it here.

Reboot if necessary and run the emulator.

image

TroubleshootIng? Try running the emulator from the command line.

emulator -list-avds

will list your AVDs.

emulator @avdname -qemu -enable-whpx

will run the AVD called avdname using WHPX (Windows Hypervisor Platform). If it fails, you may get a helpful error message.

Note: If you get a Qt library not found error, use the full path to the emulator executable. This should be the one in the emulator folder, not the one in the tools folder. The full command is:

[path-to-android-sdk]\emulator\emulator @[avdname] -qemu -enable-whpx

You can also use the emulator from Visual Studio, though you need Visual Studio 2017 version 15.8 Preview 1 or higher with the Xamarin tools installed. That said, I had some success with starting the Hyper-V emulator separately (use the command above), then using it with a Xamarin project in Visual Studio 15.7.5.

image

Notes from the Field: dmesg error blocks MySQL install on Windows Subsystem for Linux

I enjoy Windows Subsystem for Linux (WSL) on Windows 10 and use it constantly. It does not patch itself so from time to time I update it using apt-get. The latest update upgraded MySQL to version 5.7.22 but unfortunately the upgrade failed. The issue is that dpkg cannot configure it. I saw messages like:

invoke-rc.d: could not determine current runlevel

2002: Can’t connect to local MySQL server through socket ‘/var/run/mysqld/mysqld.sock

After multiple efforts uninstalling and reinstalling I narrowed the problem down to a dmesg error:

dmesg: read kernel buffer failed: Function not implemented

It is true, dmesg does not work on WSL. However there is a workaround here that says if you write something to /dev/kmsg then at least calling dmesg does not return an error. So I did:

sudo echo foo > /dev/kmsg

Removed and reinstalled MySQL one more time and it worked:

image

Apparently partial dmesg support in WSL is on the way, previewed in Build 17655.

Note: be cautious about fully uninstalling MySQL if you have data you want to preserve. Export/backup the databases first.

Notes from the field: Windows Time Service interrupts email delivery

A business with Exchange Server noticed that email was not flowing. The internet connection was fine, all the servers were up and running including Exchange 2016. Email has been fine just a few hours earlier. What was wrong?

The answer, or the beginning of the answer, was in the Event Viewer on the Exchange Server. Event ID 1035, only a warning:

Inbound authentication failed with error UnexpectedExchangeAuthBlobCheckForClockSkew for Receive connector Default Mailbox Delivery

Hmm. A clock problem, right? It turned out that the PDC for the domain was five minutes fast. This is enough to trigger Kerberos authentication failures. Result: no email. We fixed the time, restarted Exchange, and everything worked.

Why was the PDC running fast? The PDC was configured to get time from an external source, apparently, and all other servers to get their time from the PDC. Foolproof?

Not so. If you typed:

w32tm /query /status

at a command prompt on the PDC (not the Exchange Server, note), it reported:

Source: Free-running System Clock

Oops. Despite efforts to do the right thing in the registry, setting the Type key in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\Parameters to NTP and entering a suitable list of time servers in the NtpServer key, it was actually getting its time from the server clock. This being a Hyper-V VM, that meant the clock on the host server, which – no surprise – was five minutes fast.

You can check for this error by typing:

w32tm /resync

at the command prompt. If it says:

The computer did not resync because no time data was available.

then something is wrong with the configuration. If it succeeds, check the status as above and verify that it is querying an internet time server. If it is not querying a time server, run a command like this:

w32tm /config /update /manualpeerlist:”0.pool.ntp.org,0x8 1.pool.ntp.org,0x8 2.pool.ntp.org,0x8 3.pool.ntp.org,0x8″ /syncfromflags:MANUAL

until you have it right.

Note this is ONLY for the server with the PDC Emulator FSMO role. Other servers should be configured to get time from the PDC.

Time server problems seem to be common on Windows networks, despite the existence of lots of documentation. There are also various opinions on the best way to configure Hyper-V, which has its own time synchronization service. There is a piece by Eric Siron here on the subject, and I reckon his approach is a safe one (Hyper-V Synchronization Service OFF for the PDC Emulator, ON for every other VM).

I love his closing remarks:

The Windows Time service has a track record of occasionally displaying erratic behavior. It is possible that some of my findings are not entirely accurate. It is also possible that my findings are 100% accurate but that not everyone will be able to duplicate them with 100% precision. If working with any time sensitive servers or applications, always take the time to verify that everything is working as expected.

Kaspersky encrypted connection scanning breaks ADFS login, internet-facing Dynamics CRM

I was asked to look at a case where a user could not log in to Dynamics CRM. This is an internet-facing deployment which uses ADFS (Active Directory Federation Services). The user put in valid credentials but received a 401 – unauthorized: Access is denied due to invalid credentials.

The odd thing from the user’s perspective is that everything worked fine on other PCs; but switching web browsers did not fix it.

I noticed that Kaspersky anti-virus was installed.

image

Pausing Kaspersky made no difference to the error. However I came back to this after eliminating some other possible problems. I noticed that if you looked at the certificate on the ADFS site it was not from the site itself, but a Kaspersky certificate.

image

The reason for this is that Kaspersky wants to inspect encrypted traffic for malware.

I understand the rationale but I dislike this behaviour. Your security software should not hide the SSL certificate of the web site you are visiting. Of course it is particularly dislikeable if it breaks stuff, as in this case. I found the setting in Kaspersky and disabled both this feature, and another which injects script into web traffic (though this proved not to be the culprit here), for the sake of Kaspersky’s “URL Advisor”).

Personally I feel that encrypted traffic should only be decrypted in the recipient application. Kaspersky’s feature is an SSL Man-in-the-Middle attack and to my mind reduces rather than increases the security of the PC. However you made the decision to trust your anti-virus vendor when you installed the software.

There are other anti-virus solutions that also do this so Kaspersky is not alone. As to why it breaks ADFS I am not sure, but regard this as a good thing since the user’s SSL connection is compromised.

image

As it turns out, it isn’t essential to disable the feature entirely. You can simply set an exclusion for the ADFS site by clicking Manage exclusions.

Posted here in case others hit this issue.

Unhealthy Identity synchronization Notification: a trivial solution (and Microsoft’s useless troubleshooter)

If you use Microsoft’s AD Connect, also known as DirSync, you may have received an email like this:

image

It’s bad news: your Active Directory is not syncing with Office 365. “Azure Active Directory did not register a synchronization attempt from the Identity synchronization tool in the last 24 hours.”

I got this after upgrading AD Connect to the latest version, currently 1.1.553.

The email recommends you run a troubleshooting tool on the AD Connect server. I did that. Nothing wrong. I rebooted, it synced once, then I got another warning.

This is only a test system but I still wanted to find out what was wrong. I tweaked the sync configuration, again without fixing the issue.

Finally I found this post. Somehow, AD Connect had configured itself not to sync. You can get the current setting in PowerShell, using get-adsyncscheduler:

image

As you can see, SyncCycleEnabled is set to false. The fix is trivial, just type:

set-adsyncscheduler –SyncCycleEnabled $true

Well, I am glad to fix it, but should not Microsoft’s troubleshooting tool find this simple configuration problem?

How to remove the WINS server feature from Windows Server

The WINS service is not needed in most Windows networks but may be running either for legacy reasons, or because someone enabled it in the hope that it might fix a network issue.

It is now apparently a security risk. See here and Reg article here.

Apparently Microsoft says “won’t fix” despite the service still being shipped in Server 2016, the latest version:

In December 2016, FortiGuard Labs discovered and reported a WINS Server remote memory corruption vulnerability in Microsoft Windows Server. In June of 2017, Microsoft replied to FortiGuard Labs, saying, "a fix would require a complete overhaul of the code to be considered comprehensive. The functionality provided by WINS was replaced by DNS and Microsoft has advised customers to migrate away from it." That is, Microsoft will not be patching this vulnerability due to the amount of work that would be required. Instead, Microsoft is recommending that users replace WINS with DNS.

It should be removed then. I noticed it was running on a server in my network, running Server 2012 R2, and that although it was listed as a feature in Server Manager, the option to remove it was greyed out.

I removed it as follows:

1. Stop the WINS service and set it to manual or disabled.

2. Remove the WINS option in DHCP Scope Options if it is present.

3. Run PowerShell as an administrator and execute the following command:

uninstall-windowsfeature wins

This worked first time, though a restart is required.

Incidentally, if Microsoft ships a feature in a Server release, I think it should be kept patched. No doubt the company will change its mind if it proves to be an issue.

Note: you can also use remove-windowsfeature which is an alias for uninstall-windowsfeature. You do need Windows Server 2008 R2 or higher for this to work.

Fixing “couldn’t parse private ssl key” in Dovecot

I run Debian Linux including a mail server, and part of the system is Dovecot, an open source IMAP and POP3 server which has always worked well for me.

Unfortunately it stopped working after an upgrade. With Linux I am in the habit of doing:

apt-get update

apt-get upgrade

to keep the system patched, and normally everything works fine. Occasionally it does not, and then I need to dig in and work out what is wrong and how to fix it. The upgrade to Apache 2.4, for example, was somewhat painful because of changed configuration directives.

This time it was Dovecot that broke. I use Thunderbird to pick up POP3 mail, and nothing was flowing. Eventually I found the problem logged in syslog:

Fatal: Couldn’t parse private ssl_key: error:0906D06C:PEM routines:PEM_read_bio:no start line: Expecting: ANY PRIVATE KEY

I puzzled over this for some time. The path to the private key was correct in dovecot.conf. The permissions were OK. I regenerated the certificate (it’s self-signed) but still the same.

Eventually I found the solution here. The path to the SSL certs used to look like this:

ssl_cert = /etc/ssl/certs/dovecot.pem
ssl_key = /etc/ssl/private/dovecot.pem

Now it must look like this:

ssl_cert = </etc/ssl/certs/dovecot.pem
ssl_key = </etc/ssl/private/dovecot.pem

Yes, you need that angle bracket, otherwise you get the error.

It used to work, so at some point the Dovecot coders took out the compatibility code that allowed the old-style directive.

Mentioned here in case it helps someone find the solution.

Disabling automatic update restarts in Windows Server 2016

Windows Server 2016 is in effect the Windows 10 version of the server OS. If you look in Settings it seems to have the same attitude to updates; in other words, you get them automatically whether you like it or not. Currently my server is even offering me Windows 10 Creators Update:

image

However, I prefer to have servers just download updates and let me decide when to install them. There can be good reasons for this. For example, I run Exchange Server on a machine that is not really up to spec, and the Exchange services have to be manually started every time it reboots. Well, there are ways round this, but it makes the point.

It turns out that you can after all set Windows Server 2016 to download-only. Just run sconfig from the command line and choose option 5:

image

The sconfig menu will be familiar if you have worked with Server Core or other variants of Windows Server without a GUI.

Incidentally, I tried to install Exchange 2016 on Server 2016 without a GUI but it appears not to be supported. A shame.

Returning to the subject of updates, Brendan Power at Microsoft popped up on Reddit to say that this is a bug in in the settings:

The "Available updates will be downloaded…" text in the UI is a bug that doesn’t represent the actual automatic update settings.

To verify the actual server settings, you can open the command prompt and run sconfig.cmd; in the menu, you should see option 5 set to Manual.

A bug? I am not sure. If so, it seems an odd and obvious one. I think Microsoft is keen to have us update automatically. That said, Windows Server 2016 is meant to follow the Long Term Servicing Branch (LTSB) model rather than the “Windows as a service” approach in Windows 10, unless you run Nano Server, according to this post. So compulsory update to retain a supported configuration does not apply here.

Of course you should patch your Windows Server installations in a timely manner, however you choose to do it.

How to get a Bitlocker recovery key on Android

Scenario: you are out and about with your laptop and phone. Laptop protected with Bitlocker encryption. You start up your laptop and it decides for no obvious reason to demand your Bitlocker key before booting up.

If your laptop is domain-joined, this key is normally stored in Active Directory. If is not domain-joined, it is normally stored in the OneDrive linked with your Microsoft account. These are options when the Bitlocker encryption was applied.

This laptop is not domain-joined and the key is in OneDrive. So I pick up my phone and find the link, which is here.

Android helpfully opened this link in the OneDrive app, but not to the location of the keys, just the home page. No idea how to find the recovery keys in there.

The solution I found was to copy the link and paste it directly into the browser. You might need to get a code sent to your phone if you have 2-factor authentication. I found I needed to check the box for “I log on frequently with this device” before it worked.

Then I could see the key in the Android web browser. Phew.