Category Archives: software development

Fixing a .NET P/Invoke issue on Apple Silicon – including a tangle with Xcode

I have a bridge platform (yes the game) written in C# which I am gradually improving. Like many bridge applications it makes use of the open source double dummy solver (DDS) by Bo Haglund and Soren Hein.

My own project started out on Windows and is deployed to Linux (on Azure) but I now develop it mostly on a Mac with Visual Studio Code. The DDS library is cross-platform and I have compiled it for Windows, Linux and Mac – I had some issues with a dependency, described here, which taught me a lot about the Linux app service on Azure, among other things.

Unfortunately though the library has never worked with C# on the Mac – until today that is. I could compile it successfully with Xcode, it worked with its own test application dtest, but not from C#. This weekend I decided to investigate and see if I could fix it.

I have an Xcode project which includes both dtest and the DDS library, which is configured as a dynamic library. What I wanted to do was to debug the C++ code from my .NET application. For this purpose I did not use the ASP.Net bridge platform but a simple command line wrapper for DDS which I wrote some time back as a utility. It uses the same .NET wrapper DLL for DDS as the bridge platform. The problem I had was that when the application called a function from the DDS native library, it printed: Memory::GetPtr 0 vs. 0 and then quit.

The error from my .NET wrapper

I am not all that familiar with Xcode and do not often code in C++ so debugging this was a bit of an adventure. In Xcode, I went to Product – Scheme – Edit Scheme, checked Debug executable under Info, and then selected the .NET application which is called ddscs.

Adding the .NET application as the executable for debugging.

I also had to add an argument under Arguments passed on Launch, so that my application would exercise the library.

Then I could go to Product – Run and success, I could step through the C++ code called by my .NET application. I could see that the marshalling was working fine.

Stepping through the C++ code in Xcode

Now I could see where my error message came from:

The source of my error message.

So how to fix it? The problem was that DDS sets how much memory it allows itself to use and it was set to zero. I looked at the dtest application and noticed this line of code:

SetResources(options.memoryMB, options.numThreads);

This is closely related to another DDS function called SetMaxThreads. I looked at the docs for DDS and found this remark:

The number of threads is automatically configured by DDS on Windows, taking into account the number of processor cores and available memory. The number of threads can be influenced using by calling

SetMaxThreads. This function should probably always be called on Linux/Mac, with a zero argument for auto­ configuration.

“Probably” huh! So I added this to my C# wrapper, using something I have not used before, a static constructor. It just called SetMaxThreads(0) via P/Invoke.

Everything started working. Like so many programming issues, simple when one has figured out the problem!

Microsoft to remove Azure “Basic” IP number and load balancer in favour of pricier options

Microsoft is removing some features from Azure which were called “Basic,” in favour of alternatives which have more features but are also more expensive.

A load balancer is a network component which balances traffic to virtual machines. The Basic load balancer is free but has a few limitations, such as no compatibility with availability zones, support for only 300 instances, no SLA (Service Level Agreement), and no support for NAT Gateway. Microsoft has emailed customers saying:

On 30 September 2025, Azure Basic Load Balancer will be retired. You can continue to use your existing Basic Load Balancers until then, but you’ll no longer be able to deploy new ones after 31 March 2025.

The Standard load balancer routes to availability zones, supports up to 5000 instances, is secure by default, and has a 99.9% SLA, but it costs $0.025 per hour, or around $18 per month, for up to 5 rules.

A Basic public IP number costs  $0.0036 per hour or about $2.60 per month. It is a perfectly good IP number but does not support zone resiliency. A standard public IP number costs $0.005 per hour or about $3.60 per month, and does support zone resiliency. A similar email has been sent to users, with the same dates.

Although these extra charges will not make much of a ripple in enterprise accounts, they can be noticeable, for example if you are an individual developing an application and trying to keep within a strict budget.

Web page memory usage: how much is too much?

I have a web application that loads names into a picklist and the question came up: how many names would be too much for the web page to handle? What about 5,000 names, for example?

Modern web browsers have a memory snapshot built in. Just press F12 and there it is. So I fired up my application with over 5,000 names loaded into an array and displayed in a scrolling div. 2.5 MB.

Then I visited Facebook. Logged in, the page reported over 78 MB.

image

Google’s clean-looking home page? Not logged in, 11.2 MB.

image

Twitter? Logged in, 83 MB.

image

This was enough for me to stop worrying about the memory impact of 5,000 names in my web application. With our casual acceptance of multi-MB web pages it is easy to forget that the complete works of Shakespeare in plain text is 5.5 MB and compresses to less than half of that.

Just because you can do something, does not mean you should. Smaller is better and faster, and software bloat of various kinds is responsible for many performance issues.

There are trade-offs though both in time and in performance. Maybe it is better to load just a few names, and retrieve more from the server when needed. It is easy to test such things. Nevertheless, I found it useful to get some perspective on the memory usage of modern web sites.

Installing Ubuntu 22.04 on Apple M1 with UTM

I  started with Arch Linux for Linux development  on  M1, which works, but succumbed to Ubuntu just because it is so widely used and therefore easier to find help. It is also supported by VS Code for remote development, as I am aiming for something similar to a WSL setup on Windows and using VS Code on the host side. I had problems installing 22.04 though;  the install completed but trying to boot resulted in:

EFI stub: booting Linux Kernel

EFI stub: Using DTB from configuration table

EFI stub: Exiting boot services

and there it would stay.

The fix I found was to update QEMU:

sudo port selfupdate

sudo port install qemu

after which Ubuntu started without issue (no need to reinstall).

Following this I was able to install and use the Remote – SSH extension in VS Code which worked first time.

Small points to note:

  • I accepted the option in Ubuntu to install the OpenSSH server during installation
  • In UTM I changed the networking for the VM to Emulated VLAN rather than Shared Network, in order to use port forwarding. I forwarded both the SSH port 22 and also the HTTP port 80, to different ports on the Mac, so that I can test web applications running on Ubuntu.

Thanks also to Liz Rice for her post here.

Thoughtworks: do not choose to develop Single Page Applications by default

I have a lot of time for Thoughtworks, a global software development company, and always look at its Technology Radar, the latest version of which appeared recently. Plenty to digest, but what caught my eye was this comment regarding SPAs (Single Page Applications):

The sheer prevalence of teams choosing a single-page application (SPA) by default when they need a website has us concerned that people aren’t even recognizing SPAs as an architectural style to begin with, instead immediately jumping into framework selection. SPAs incur complexity that simply doesn’t exist with traditional server-based websites: search engine optimization, browser history management, web analytics, first page load time, etc. That complexity is often warranted for user experience reasons, and tooling continues to evolve to make those concerns easier to address (although the churn in the React community around state management hints at how hard it can be to get a generally applicable solution). Too often, though, we don’t see teams making that tradeoff analysis, blindly accepting the complexity of SPAs by default even when the business needs don’t justify it.

This struck a chord with me because of my adventures creating an online bridge playing platform using ASP.NET Core. I picked the platform because I was in a hurry, like C#, and had some existing code for implementing a bridge game, done for Windows. Any online game though needs lots of JavaScript and I soon became aware that the traditional ASP.NET approach, where each web page is a separate .cshtml file with server-side rendering and C# code-behind, is at odds with trends towards SPAs and JAMstack (JavaScript, API and Markup, where “Markup” is HTML and CSS).

Note that you can of course do SPAs and JAMstack with ASP.NET; ASP.NET is a nice technology for implementing an API and there are Visual Studio templates for things including “ASP.NET Core with React.js and Redux”. A Razor Pages application is still the default though, and gives you a UI for the ASP.NET Core Identity for free which saved me a lot of gruntwork. Still, as I got deeper into JavaScript libraries, including the AWS JavaScript SDK which I am using for audio and video, I found myself being steered towards React.js (resisted so far) and JavaScript bundling with Webpack (tried but was not a good fit). I also found that even switching my JavaScript code to TypeScript was surprisingly awkward, considering that the creator of TypeScript works for Microsoft. I found myself wondering if I should have started with an SPA, or convert my application to an SPA, in order to fit in well with the new world.

Separately, I’ve been involved with another project, in PHP and JavaScript, which is an SPA, and hitting some of the potential issues. For example, the application made a ton of database queries on first load, the data from which was in most cases never used, as users did not visit the parts of the application that required them. Refactoring to load this data on demand has made the application faster and more efficient.

A problem, which Thoughtworks alludes to in a remark about “closing the gap on user experience,” is that staying in JavaScript rather than loading a new page from the server generally makes for a smoother application. The way my bridge application has evolved is that the main play screen is a kind of SPA: everything is done in JavaScript and API calls, and I have written a ton of JavaScript code for things like rendering HTML tables where server-side rendering with Razor would be much easier, but unacceptable for usability. However, different parts of the application still use separate Razor pages, for things like viewing results, configuring a user profile, finding a game, and admin screens for managing members and running sessions.

JavaScript, now TypeScript, has exceeded my expectations in terms of performance and capability. It is annoying at times but a modern web browser is a phenomenal platform. I was glad though to see Thoughtworks noting that going the SPA route is not always the right decision

Drupal 7 is the version that refuses to die as the majority of sites have not upgraded

image

Drupal, which may be the 2nd most popular content management system after WordPress according to these stats, is now at version 9.2. Version 7.0 was released 11 years ago but when 8.0 was being developed (it was released in 2015) the team decided that there there were so many key improvements, including mobile-first design, multi-language support and HTML 5 forms, that in-place upgrade from 7.0 was too hard. In addition, some modules (used to extend Drupal) had no Drupal 8 version. Read all about the migration story here. It is not trivial.

From 8 on, the team promised, compatibility would be preserved so that upgrades would be easier.

What happened? Did every Drupal 7 site migrate to version 8 in order to enjoy the new features and promised future upgrade path?

No. Last month the team confessed that “a majority  of all sites in the Drupal project are still on Drupal 7.” The date for ending support for Drupal 7 keeps getting pushed back and is now November 1 2023, but to be reviewed annually.  “We will announce by July 2023 whether we will extend Drupal 7 community support an additional year,” said the post.

While this is good news in one sense for Drupal 7 site maintainers, it is not good news for the Drupal project. Having more than half Drupal sites on what is now an ancient version is unhealthy and maintaining it is a distraction.

Should the team have compromised the improvements in Drupal 8  for the sake of compatibility? It is imponderable but underlines a general truth in software development: breaking compatibility in major ways is expensive and can only be worth it if the benefits are correspondingly huge.

Another example that come to mind is Visual  Basic .NET which was incompatible with Visual Basic 6.0 and in consequence there are many  VB 6.0  applications still out there, that have never been upgraded.

Python 2 is another example.

What this also means is that time invested in making upgrade easy, or preserving compatibility in a widely-used project, may seem unrewarding but has a big payback.

A UI lesson: do not ask users to choose between Register and Login

I am developing a web site for playing bridge, a project which kicked off in March when lockdown caused bridge clubs everywhere to close. There are lots of sites where you can play bridge online, but not many options (particularly back in March) for clubs that wanted to run their own online sessions.

It’s going OK with a number of clubs now using it every week, though it is still in development. I have learned a painful lesson though. In order to proceed as quickly as possible, I started my project with the Visual Studio template for an ASP.Net Core application with ASP.NET Core Identity – the latter an easy decision since it handles all the complications of registration, password reset and so on. (I did end up having to re-plumb it to use int rather than GUID for the primary key but that is another story).

The default home page the template generates looks like this, with options in the menu to Register or Login.

image

Registration and login are fundamental concepts that have been part of the web forever. It’s simple for a developer to understand: you register to create an account, you login if you already have an account.

The painful discovery is that this is not obvious to everyone, particularly to an older demographic that did not grow up with computers. Another factor is that cookies, browser-saved passwords and single sign-on with Google/Facebook etc means that this whole area is a bit of a mess and there are people who just kinda expect web sites to know who they are (which in one way horrifies me but I do see the massive convenience).

The consequence is that a surprising (to me) number of people had difficulty knowing whether Registration or Login was what they needed. They would Register, then return to the site and hit Register again. Why is this site asking for my details again? Maybe a security thing? Oh no, why does it now say username not available?

This is because asking the user to make this choice is not good design. Registration is rare, login is common. Further, Register is a confusing word. We sometimes use the word register when accounts already exist. Create Account is better. And a better UI is just Login. I need to access this website. Then, underneath the username/password request, an option that says “I need to create an account”. The two options should not be equally prominent; and if you look at how many prominent sites design this, that is what they do:

image image

Lesson learned; but I wish this had occurred to me sooner!

Point-in-time restore: a handy built-in feature in Azure SQL

I am working on a project that is hosted in Azure and I made a mistake, running a SQL script that was dependent on another SQL script that I had forgotten to run. It messed up the foreign keys and I would have to restore a backup … but my most recent backup was from the day before. Annoying.

But wait. Looking the Azure portal I saw this:

image

This is a plain Azure SQL instance with no extras, but look, you can restore the database from 6 minutes ago.

I did it; it restored to a second database. I deleted the bad one, renamed the restored one, ran my scripts in the right order, and all was well.

I recommend you do not run scripts in the wrong order … but if you do, or make some other error, this is a handy feature of Azure SQL which I was not aware of before.

Developing software for playing bridge

I am a duplicate bridge player in my spare time and enjoyed playing in my local club once or twice a week. That was before COVID-19 and then, in March this year, lockdown. Bridge clubs were no longer able to meet. There are more important things in the world; but bridge is both a lot of fun and a welcome distraction from weightier matters, and my thoughts soon turned to what we could do to continue playing in these new circumstances.

The answer was to play online; but while there are plenty of ways to play bridge online, the existing systems were not designed with the idea of being a way for bridge clubs to meet in a new context. If anything, the reverse is true: online bridge site were designed for people who could not easily get to a club or wanted to play at any time with whoever else happened to be available. Clubs like my own, by contrast, wanted to replicate their face-to-face meetings with an online equivalent. A further complication back in March was that the biggest online bridge site, called Bridgebase, was immediately overloaded and declared that it was unwilling to allow new people to qualify as directors, people allowed to run online bridge sessions.

My immediate instinct was to build a new site for playing bridge. I was not quite starting from scratch. Back in the early days of Windows 8, I started work on a bridge game for Microsoft’s new and as it turned out ill-fated platform. I had got some way with it; I had created a bridge engine that understood about cards and hands and tricks and shuffling and scoring and all the various elements that go into playing bridge. It was written in C# and what is now UWP XAML. It is designed of course for a solo player. Here is the bidding screen:

image

and the play screen:

image

This is how it looks on Windows 10; it looked a bit better on Windows 8 though it would not win any prizes for design. My software could play bridge though; the reason I never finished it was that I never cracked getting the AI working. But for human to human play that did not matter. A weekend or two coding, I thought, and I could have a website up and running so our club could play bridge online. I made an immediate start, registering the domain name YourBridgeClubOnline.co.uk.

Well, three months later and here we are.

image

image

It is, I have to say, still under development. But it works and we have been able to play bridge again, as a club.

What took you so long? Ha! Much of my old bridge engine code remains untouched and has proved useful; it all runs fine on .NET Core. Even the (useless) AI has been handy, as I can test the mechanics of play without involving others. But I had, of course, wildly underestimated the problem of converting a game for solo play on Windows, to a multi-player web application. There is much to think about:

The UI. I am not a designer (I am sure you can tell) but spent ages puzzling over how to get a workable user interface in the browser for everything from tablets to desktops. Not smartphones yet but it is coming. I decided early on to take a view on compatibility. No Internet Explorer. JavaScript fetch API is required. When time is against you, it is easier to say, just use another browser, than to waste too much time supporting old browsers.

Messaging – both the API kind, and the chat kind. I am using C#, ASP.NET Core and SignalR. In general it works well. SignalR uses WebSockets as first preference, but falls back to Server Sent Events or long polling where necessary. In my first experiments I did my own polling and switching to SignalR was a great relief.

Registration and login. I am using the stuff that comes in the box, ASP.NET Core Identity. It has saved me a ton of work. It’s a bit annoying and not too well documented. I don’t really like using GUIDs for the primary key, for example, and I believe there is way to avoid it, but it isn’t top priority when you are going for Minimum Viable Product.

JavaScript. I’ve written tons of it and I don’t even like the language. I have a new respect for it though. The thing is, it is very fast and there is nothing you cannot do. The worst thing is the friction of doing some debugging in the browser, and some in Visual Studio. I am thinking of switching to VS Code for development since it works nicely with ASP.NET Core and is better for JavaScript than Visual Studio.

Scoring. My Windows software could score a hand of bridge. But duplicate is different; you have to compare the scores with others who played the same hands and work out the percentages, then export the results to standard formats for display on club websites and submission to the English Bridge Union. It was more work than I had expected and I am not done yet; the system only understands Pairs at the moment, not Teams (a different way of scoring).

Directing. Someone has to manage an online bridge session, settle any arguments, and fix errors like cards played by accident. It all needs coding and there was nothing like it in the Windows version.

Movements. Imagine you have 28 people playing bridge (or 14 pairs). They need to all play the same hands, but never play the same hand twice, and it has to be so arranged that each pair plays against other pairs in a defined sequence so it is balanced and fair. We call this the movement. Online, you have a bit more flexibility because you don’t need to share physical cards: everyone can play the same hand at the same time if you like. It is still quite fiddly though, and I did not do any of this in the old Windows version. I saved some time by writing an import function to enable re-use of movements made for EBUScore, a widely used scoring and bridge session management application. There is more to do though.

Claims. This is where, half way through the hand, a player says, “There’s no point in playing on, I’m obviously going to win all the remaining tricks.” A trick is a sequence of four cards played one from each hand, which is won by one of the pairs. This statement is called a claim, and has to be agreed by the other players. Getting this working was more difficult than I had expected – because built into my bridge engine was the idea that you could score by counting the tricks each side had won. But claimed tricks are never played. With hindsight, I should have allowed for this from the beginning.

Database. Every detail of play has to be stored on the server. I am using Dapper and SQL Server currently, though it is possible that PostgreSQL would work just as well. I started with Entity Framework Core, still there as it is used by ASP.NET Core Identity, but I am happier with Dapper.

Things that worked well

Three months is longer than I had thought it would take to get to a playable system, but I suppose as a spare time project it is not too bad. It would not be possible without the likes of ASP.NET Core and Dapper and SignalR doing so much for you. C# is a delight for coding. I am also using an Azure App Service for all this test and development and that has worked well. I am deploying to a Linux container of course; but the nice thing about App Service is that it will scale to a considerable extent without the hassle of Kubernetes. If the project succeeds and needs to scale up, there is an Azure SignalR service ready and waiting. I was nevertheless interested to see that AWS now offers .NET Core on Elastic Beanstalk, complete with some nice Visual Studio integration. Trying it there would be an interesting experiment, though I’m not sure AWS is so savvy about SignalR.

Open Source?

Could this have been done quicker by making it open source and seeking collaborators early on? Will it become open source? I need help for sure, though I also feel the code needs some cleaning up before it is fit to share more widely. You will recall though that I had started out thinking that it would be a small matter to convert my solo bridge game to an online multiplayer web application. I figured it would be better to get something working and then ask for help. But I am open to offers! Note: this is not a commercial project.

Rewarding

Most of the software projects I have been involved in have been business applications. Bridge is a lot more fun. I do see software development as a creative act. I recall starting work on the bridge game back in 2011 (I think); starting a new blank project in Visual Studio and thinking, hmm, I had better write a class to represent a pack of cards. From that beginning I ended up with an application that could play bridge, after a fashion, and now one that multiple people can play concurrently. It is rewarding and I will not regret the time spent on it, irrespective of how much actual use it gets.

StackOverflow survey 2020

The StackOverflow developer survey 2020 is out – surveys come out constantly but this one is worth more than most because of the huge reach of StackOverflow among developers. This one has 65,000 responses.  Every survey also has reasons why it is unrepresentative; there is no such thing as a definitive survey because you can never precisely compare like with like, humans are unreliable, and every sample has its biases. Every wondered what a survey of “developers who never respond to surveys” would look like, if you could cajole them into answering questions? I have.

image

Imperfect then, but still interesting. What is notable in this one? The questions that interest me are those in the technology section. I am also more interested in trends than in absolute rankings, because it is the trends that might tell us something about the future. So let’s have a look.

In “Most popular programming languages” the interesting stuff is well down the rankings, since the top places are little changed. Dart has gone from 1.9% last year to 4.0% this year; still small, but that is more than 100% growth, thanks no doubt to Flutter. Rust has also grown, from 3.2% to 5.1%. Swift has fallen slightly, from 6.6% to 5.9%. This no longer seems likely to become a top programming language, important though it is for macOS and iOS. Objective-C is down a bit too (4.8% to 4.1%) and I wonder if this suggests greater interest in cross-platform toolkits and/or web technologies for Apple platforms. Such as Flutter, of course.

What about web frameworks? React.js is up a bit, from 31.3% to 35.9%, and to my mind it does look all-conquering at the moment. jQuery is above it, but that is nonsense really, as jQuery is not an alternative to React.js, being more a low-level plumbing thing. The figures for both Angular and ASP.NET are confusing as hell, since last year there was no separate entry for ASP.NET Core, but this year there is; and this year Angular is split between Angular and Angular.js. So we cannot conclude anything about these technologies. There are signs of growth in JAMstack frameworks. Vue.js is up from 15.2% to 17.3% and Gatsby makes an appearance this year at 4.0%

In “Other frameworks” we see .NET down a little but .NET Core up a little, so probably no change there. Flutter up from 3.4% to 7.2% as noted above. Xamarin down a bit, from 6.5% to 5.8%. Xamarin has not been as popular as I had expected it to become when Microsoft acquired it; the reason seems obvious, which is that Microsoft has given developers mixed messages about whether to use it, with the Windows and Office teams seemingly preferring React Native.

In databases, PostreSQL had another good year, rising from 34.3% to 38.5%. More than MySQL, it seems a natural destination for those migrating from Microsoft SQL Server or Oracle. Though note too that SQL Server is up, from 32.8% to 34.8%. Microsoft pushes developers strongly towards SQL Server in its developer tools and frameworks, and the strategy seems to work (plus it is a pretty good database manager).

In platforms, both Linux and Windows have increased use among developers. Not so surprising when you consider that Microsoft now ships the Linux kernel with Windows 10. MacOS is going in the right direction too, from 22.2% to 24.0%.

StackOverflow stuffs cloud platforms into this part of the survey too, as well as things like WordPress (talk about not comparing like with like!). Still, note that AWS is up from 26.6% to 26.7% (well, hardly moved); Azure is up from 11.9% to 14.5%; and Google Cloud (GCP) from 12.4% to 14.1%. Oh yes, and Kubernetes is here too, up from 8.5% to 11.5%. All of this chimes with my perception that GCP is doing pretty well from a developer perspective, now only just behind Azure in this particular community.

Then there is the entertaining most loved, dreaded and wanted. Rust is still move loved by miles (86.1%, up from 83.5% last year). Not much else I want to say about this section, other than to note that Python tops the “most wanted” list by a bigger margin (30.0% up from 25.7%); and poor old VBA continues to be “most dreaded”, again by a bigger margin (80.4% up from 75.2%).

In web frameworks, the good news for .NET developers is that ASP.NET Core now tops the list of “Most loved”, whereas ASP.NET is well down the list (36.9%). Again you cannot really compare with last year. Angular.js (the old version) tops the list of “Most dreaded”.

Similarly, .NET Core tops the list of most loved “other frameworks, libraries and tools” at 71.5%, though it only manages  8.3% in “Most wanted.” Translation: .NET Core developers love it, but it is still not doing that well in terms of appeal to those not currently using it. Chef should be asking itself why it is top of “Most dreaded” for two years running, in fact more dreaded than last year (72.4% up from 66.7%). Puppet is not far behind. Ansible seems both better liked and less dreaded by developers.

The “Most dreaded platforms” list is also notable, with WordPress at the top (PHP spaghetti anyone?) and IBM “Cloud or Watson” second, both positions unchanged from last year.  Android as overtaken Windows as the most dreaded operating system. All the top cloud platforms are more dreaded than last year, and so is Kubernetes.

Make of all that what you will. The survey seems to me valuable as evidence of things we already know, but there are no huge surprises – and why should there be?