Tag Archives: software development

SQLite with .NET: excellent but some oddities

I have been porting a C# application which uses an MDB database (the old Access/JET format) to one that uses SQLite. The process has been relatively smooth, but I encountered a few oddities.

One is puzzling and is described by another user here. If you have a column that normally stores string values, but insert a string that happens to be numeric such as “12345”, then you get an invalid cast exception from the GetString method of the SQLite DataReader. The odd thing is that the GetFieldType method correctly returns String. You can overcome this by using GetValue and casting the result to a string, or calling GetString() on the result as in dr.GetValue().ToString().

Another strange one is date comparisons. In my case the application only stores dates, not times; but SQLite using the .NET provider stores the values as DateTime strings. The SQLite query engine returns false if you test whether “yyyy-mm-dd 00:00:00” is equal to “yyy-mm-dd”. The solution is to use the date function: date(datefield) = date(datevalue) works as you would expect. Alternatively you can test for a value between two dates, such as more than yesterday and less than tomorrow.

Performance is excellent, with SQLite . Unit tests of various parts of the application that make use of the database showed speed-ups of between 2 and 3 times faster then JET on average; one was 8 times faster. Note though that you must use transactions with SQLite (or disable synchronous operation) for bulk updates, otherwise database writes are very slow. The reason is that SQLite wraps every INSERT or UPDATE in a transaction by default. So you get the effect described here:

Actually, SQLite will easily do 50,000 or more INSERT statements per second on an average desktop computer. But it will only do a few dozen transactions per second. Transaction speed is limited by the rotational speed of your disk drive. A transaction normally requires two complete rotations of the disk platter, which on a 7200RPM disk drive limits you to about 60 transactions per second.

Without a transaction, a unit test that does a bulk insert, for example, took 3 minutes, versus 6 seconds for JET. Refactoring into several transactions reduced the SQLite time to 3 seconds, while JET went down to 5 seconds.

Should you convert your Visual Basic .NET project to C#? Why and why not…

When Microsoft first started talking about Roslyn, the .NET compiler platform, one of the features described was the ability to take some Visual Basic code and “paste as C#”, or vice versa.

Some years later, I wondered how easy it is to convert a VB project to C# using Roslyn. The SharpDevelop team has a nice tool for this, CodeConverter, which promises to “Convert code from C# to VB.NET and vice versa using Roslyn”. You can also find this on the Visual Studio marketplace. I installed it to try out.

image

Why would you do this though? There are several reasons, the foremost of which is cross-platform support. The Xamarin framework can use VB to some extent, but it is primarily a C# framework. .NET Core was developed first for C#. Microsoft has stated that “with regard to the cloud and mobile, development beyond Visual Studio on Windows and for non-Windows platforms, and bleeding edge technologies we are leading with C#.”

Note though that Visual Basic is still under active development and history suggests that your Windows VB.NET project will continue running almost forever (in IT terms that is). Even Visual Basic 6.0 applications still run, though you might find it convenient to keep an old version of Windows running for the IDE.

Still, if converting a project is just a right-click in Visual Studio, you might as well do it, right?

image

I tried it, on a moderately-size VB DLL project. Based on my experience, I advise caution – though acknowledging that the converter does an amazing job, and is free and open source. There were thousands of errors which will take several days of effort to fix, and the generated code is not as elegant as code written for C#. In fact, I was surprised at how many things went wrong. Here are some of the issues:

The tool makes use of the Microsoft.VisualBasic namespace to simplify the conversion. This namespace provides handy VB features like DateDiff, which calculates the difference between two dates. The generated project failed to set a reference to this assembly, generating lots of errors about unknown objects called Information, Strings and so on. This is quick to fix. Less good is that statements using this assembly tend to be more convoluted, making maintenance harder. You can often simplify the code and remove the reference; but of course you might introduce a bug with careless typing. It is probably a good idea to remove this dependency, but it is not a problem if you want the quickest possible port.

Moving from a case-insensitive language to a case-sensitive language is a problem. Visual Studio does a good job of making your VB code mostly consistent with regard to case, but that is not a fix. The converter was unable to fix case-sensitivity issues, and introduced some of its own (Imports System.Text became using System.text and threw an error). There were problems with inheritance, and even subtle bugs. Consider the following, admittedly ugly and contrived, code:

image

Here, the VB coder has used different case for a parameter and for referencing the parameter in the body of the method. Unfortunately another variable with the different case is also accessible. The VB code and the converted C# code both compile but return different results. Incidentally, the VB editor will work very hard to prevent you writing this code! However it does illustrate the kind of thing that can go wrong and similar issues can arise in less contrived cases.

C# is more strict than VB which causes errors in conversion. In most cases this is not a bad thing, but can cause headaches. For example, VB will let you pass object members ByRef but C# will not. In fact, VB will let you pass anything ByRef, even literal values, which is a puzzle! So this compiles and runs:

image

Another example is that in VB you can use an existing variable as the iteration variable, but in C# foreach you cannot.

Collections often go wrong. In VB you use an Item property to access the members of a collection like a DataReader. In C# this is omitted, but the converter does not pick this up.

Overloading sometimes goes wrong. The converter does not always successfully convert overloaded methods. Sometimes parameters get stripped away and a spurious new modifier is added.

Bitwise operators are not correctly converted.

VB allows indexed properties and properties with parameters. C# does not. The converter simply strips out the parameters so you need to fix this by hand. See https://stackoverflow.com/questions/2806894/why-c-sharp-doesnt-implement-indexed-properties if the language choices interest you.

There is more, but the above gives some idea about why this kind of conversion may not be straightforward.

It is probably true that the higher the standard of coding in the original project, the more straightforward the conversion is likely to be, the caveat being that more advanced language features are perhaps more likely to go wrong.

Null strings behave differently

Another oddity is that VB treats a String set to null (Nothing) as equivalent to an empty string:

Dim s As String = Nothing

If (s = String.Empty) Then ‘TRUE in VB
     MsgBox(“TRUE!”)
End If

C# does not:

String s = null;

   if (s == String.Empty) //FALSE in C#
    {
        //won’t run
    }

Same code, different result, which can have unfortunate consequences.

Worth it?

So is it worth it? It depends on the rationale. If you do not need cross-platform, it is doubtful. The VB code will continue to work fine, and you can always add C# projects to a VB solution if you want to write most new code in C#.

If you do need to move outside Windows though, conversion is worthwhile, and automated conversion will save you a ton of manual work even if you have to fix up some errors.

There are two things to bear in mind though.

First, have lots of unit tests. Strange things can happen when you port from one language to another. Porting a project well covered by tests is much safer.

Second, be prepared for lots of refactoring after the conversion. Aim to get rid of the Microsoft.VisualBasic dependency, and use the stricter standards of C# as an opportunity to improve the code.

Gartner on Mobile App Development Platforms: Kony, Mendix, Microsoft, Oracle and Outsystems the winners

Gartner has published a paper and Magic Quadrant on Mobile App Development Platforms (MDAPs), which you can read for free thanks to Progress, pleased to be named as a “Visionary”, and probably from other sources.

According to Gartner, an MDAP has three key characteristics:

  • Cross-platform front-end development tools
  • Back-end services that can be used by diverse clients, not just the vendor’s proprietary tools.
  • Flexibility to support public and internal deployments

Five vendors ranked in the sought-after “Leaders” category. These are:

  • Kony, which offers Kony Visualizer for building clients, Kony Fabric for back-end services, and Kony Nitro Engine, a kind of cross-platform runtime based on Apache Cordova .
  • Mendix, which has visual development and modeling tools and multi-cloud, containerised deployment of back-end services
  • Microsoft, which has Xamarin cross-platform development, Azure cloud services, and PowerApps for low-code development
  • Oracle, which has Oracle Mobile Cloud Enterprise including JavaScript Extension Toolkit and deployment via Apache Cordova
  • Outsystems, a low-code platform which has the Silk UI Framework and a visual modeling language, and hybrid deployment via Apache Cordova

Of course there are plenty of other vendors covered in the report. Further, because this is about end-to-end platforms, some strong cross-platform development tools do not feature at all.

A few observations. One is the prominence of Apache Cordova in these platforms. Personally I have lost enthusiasm for Cordova, now that there are several other options (such as Xamarin or Flutter) for building native code apps, which I feel deliver a better user experience, other things being equal (which they never are).

With regard to Microsoft, Gartner notes the disconnect between PowerApps and Xamarin, different approaches to application development which have little in common other than that both can be used with Azure back-end services.

image
Microsoft PowerApps

I found the report helpful for its insight into which MDAP vendors are successfully pitching their platform to enterprise customers. What it lacks is much sense of which platforms offer the best developer experience, or the best technical capability when it comes to solving those unexpected problems that inevitably crop up in the middle of your development effort and take a disproportionate amount of time and effort to solve.

RemObjects Elements: mix and match languages and platforms as you like

The world of software development has changed profoundly in the last decade or so. Once it was a matter of mainly desktop Windows development for the client, mainly Java for server-based applications with web or Windows clients. Then came mobile and cloud – the iPhone SDK was released in March 2008, kicking off a new wave of mobile applications, while Amazon EC2 (Elastic Compute Cloud) came out of beta in October 2008. Microsoft tussled within itself about what to do with Windows Mobile and ended up ceding the entire market to Android and iOS.

The consequence of these changes is that business developers who once happily developed Windows desktop applications have had to diversify, as their customers demand applications for mobile and web as well. The PC market has not gone away, so there has been growing interest in both cross-platform development and in how to port Windows code to other platforms.

Embarcadero took Delphi, a favourite development tool based on an Object Pascal compiler, down a cross-platform path but not to the satisfaction of all Delphi developers, some of whom looked for other ways to transition to the new world.

Founded in 2002, RemObjects had a project called Chrome, which compiled Delphi’s Object Pascal to .NET executables. This product was later rebranded Oxygene. For a while Embarcadero bundled a version of this with Delphi, calling it Prism, after abandoning its own .NET compilation tools.

The partnership with Embarcadero ended, but RemObjects pressed on, adding language features to its flavour of Object Pascal and adding support for Mac OS X, iPhone and Java.

In February 2015 the company was an early adopter of Apple’s Swift language, introducing a Swift compiler called Silver that targets Android, .NET and native Mac OS X executables.

The company now offers a remarkable set of products for developers who want to target new platforms but in a familiar language:

  • Oxygene: Object Pascal
  • Silver: Swift 3 (and most of Swift 4)
  • Hydrogene: C# 7
  • Iodine: Java 8

Each language can import APIs from the others, and compile to all the platforms – well, there are exceptions, but this is the general approach.

More precisely, RemObjects defines four target platforms:

  • Echoes: .NET and .NET Core including ASP.NET and Mono
  • Cooper: Java and Android
  • Toffee: Mac, iOS, tvOS
  • Island: CPU native and WebAssembly

So if you fancy writing a WPF (Windows Presentation Foundation) application in Java, you can:

image

As you may spot from the above screenshot, the RemObjects tools use Visual Studio as the IDE. This is a limitation for Mac developers, so the company also developed a Mac IDE called Fire, and now a Windows IDE called Water (in preview) for those who dislike the Visual Studio dependency.

image

Important to note: RemObjects does not address the problem of cross-platform user interfaces. In this respect it is similar to the approach taken by Xamarin before that company came up with the idea of Xamarin Forms. So this is about sharing non-visual code and libraries, not cross-platform GUI (Graphical User Interface). If you are targeting Cocoa, you can use Apple’s Interface Builder to design your user interface, for example.

Of course WebAssembly and HTML is an interesting option in this respect.

A notable absentee from the list of RemObjects targets is UWP (Universal Windows Platform), a shame given the importance Microsoft still attaches to this.

RemObjects is mainly focused  on languages and compilers rather than libraries and frameworks. The idea is that you use the existing libraries and frameworks that are native to the platform you are targeting. This is a smart approach for a small company that does not wish to reinvent the wheel.

That said, there is a separate product called Data Abstract which is a multi-tier database framework.

These are interesting products, but as a journalists I have struggled to give them much coverage, because of their specialist nature and also the demands on my time as someone who prefers to try things out rather than simply relay news from press releases. I also appreciate that the above information is sketchy and encourage you to check out the website if these tools pique your interest.

Embarcadero launches free Community Edition of Delphi and C++Builder for mainly non-commercial use

A new Community Edition of Delphi and C++Builder, visual development tools for Windows, Mac, Android and iOS, has been released by Embarcadero.

image

The tools are licensed for non-commercial use or for commercial use (for up to 5 developers) where revenue is less than $5000 per year. It is not totally clear to me, but I believe this means the total revenue (or for non-profits, donations) of the individual or organisation, not just the revenue generated by Community Edition applications. From the EULA:

The Community Edition license applies solely if Licensee cumulative annual revenue (of the for-profit organization, the government entity or the individual developer) or any donations (of the non-profit organization) does not exceed USD $5,000.00 (or the equivalent in other currencies) (the “Threshold”). If Licensee is an individual developer, the revenue of all contract work performed by developer in one calendar year may not exceed the Threshold (whether or not the Community Edition is used for all projects).

Otherwise, the Community Editions are broadly similar to the Professional Editions of these tools. Note that even the Professional Edition lacks database drivers other than for local or embedded databases so this is a key differentiator in favour of the Architect or Enterprise editions.

An annoyance is that you cannot install both Delphi and C++ Builder Community Editions on the same PC. For this you need RAD Studio which has no Community Edition.

Delphi and C++ Builder are amazing tools for Windows desktop development, with a compiler that generates fast native code. For cross-platform there is more competition, not least from Microsoft’s Xamarin tools, but the ability to share code across multiple platforms has a powerful attraction.

Get Delphi Community Edition here and C++Builder Community Edition here.

Using the Xamarin WebView for programmatic display of HTML content

Xamarin Forms is a key framework for C# and .NET developers since it lets you target Android, iOS and to some extent Windows (UWP and therefore Windows 10 only) with maximum code reuse. I have a longstanding interest in embedded web browser controls and was glad to see that Xamarin Forms supports a capable WebView control. The WebView wraps Chrome on Android, Safari on iOS, and Edge on UWP.

I did a quick hands-on. In this example (running in the Android emulator on Hyper-V, of course), the HTML is generated programmatically and the CSS loaded from local storage. I also added some script to show the User Agent string that identifies the browser.

image

There are a few things needed to make this work. Some XAML to put the WebView on a page. Then to load content into the WebView you need an HTMLWebViewSource object. If you are loading external files, you must set the BaseUrl property of this object as well as the HTML itself. The BaseUrl tells the control where to look for files that have a relative address. This varies according to the target platform, so you use the Xamarin Forms Dependency Service to set it correctly for each platform.

In Visual Studio, you place the files you want to load in the appropriate folder for each platform. For Android, this is the Assets folder.

That is about all there is to it. As you can see from the above screenshot, I wrote very little code.

The WebView control can also display PDF documents. Finally, there is an EvaluateJavaScriptAsync method that lets you call JavaScript in a WebView and read the results from C#.

This JavaScript bridge is a workaround for the most obvious missing feature, that you cannot directly read the HTML content from the WebView. If this is a full programmatic solution and you generate all the HTML yourself, you can add JavaScript to do what you want. If the user is allowed to navigate anywhere on the web, you cannot easily grab the HTML; but this could be a good thing, in case the user entered a password or is viewing confidential data. You can grab the destination URL from the Navigating event and read it separately if necessary. But the intent of the control is to let you create rich applications that take advantage of the browser’s ability to render content, not to invade a user’s privacy by tracking their web browsing.

Instant applications considered harmful?

Adrian Colyer, formerly of SpringSource, VMWare, and Pivotal, is running an excellent blog where he looks at recent technical papers. A few days ago he covered The Rise of the Citizen Developer – assessing the security impact of online app generators. This was about online app generators for Android, things like Andromo which let you create an app with a few clicks. Of course the scope of such apps is rather limited, but they have appeal as a quick way to get something into the Play Store that will promote your brand, broadcast your blog, convert your website into an app, or help customers find your office.

It turns out that there are a few problems with these app generators. Andromo is one of the better ones. Some of them just download a big generic application with a configuration file that customises it to your requirements. Often this configuration is loaded from the internet, in some cases over HTTP with no encryption. API keys used for interaction with other services such as Twitter and Google can easily leak. They do not conform to Android security best practices and request more permissions that are needed.

Low code or no-code applications are not confined to Android applications. Appian promises “enterprise-grade” apps via its platform.  Microsoft PowerApps claims to “solve business problems with intuitive visual tools that don’t require code.” It is an idea that will not go away: an easy to use visual environment that will enable any business person to build productive applications.

Some are better than others; but there are inherent problems with all these kinds of tools. Three big issues come to mind:

  1. Bloat. You only require a subset of what the application generator can do, but by trying to be universal there is a mass of code that comes along with it, which you do not require but someone else may. This inevitably impacts performance, and not in a good way.
  2. Brick walls. Everything is going well until you require some feature that the platform does not support. What now? Often the only solution is to trash it and start again with a more flexible tool.
  3. Black box. You app mostly works but for some reason in certain cases it gives the wrong result. Lack of visibility into what it happening behind the scenes makes problems like this hard to fix.

It is possible for an ideal tool to overcome these issues. Such a tool generates human-understandable code and lets you go beyond the limitations of the generator by exporting and editing the project in a full programming environment. Most of the tools I have seen do not allow this; and even if they do, it is still hard for the generator to avoid generating a ton of code that you do not really need.

The more I have seen of different kinds of custom applications, the more I appreciate projects with nicely commented textual code that you can trace through and understand.

The possibility of near-instant applications has huge appeal, but beware the hidden costs.

Inside Azure Cosmos DB: Microsoft’s preferred database manager for its own high-scale applications

At Microsoft’s Build event in May this year I interviewed Dharma Shukla, Technical Fellow for the Azure Data group, about Cosmos DB. I enjoyed the interview but have not made use of the material until now, so even though Build was some time back I wanted to share some of his remarks.

Cosmos DB is Microsoft’s cloud-hosted NoSQL database. It began life as DocumentDB, and was re-launched as Cosmos DB at Build 2017. There are several things I did not appreciate at the time. One was how much use Microsoft itself makes of Cosmos DB, including for Azure Active Directory, the identity provider behind Office 365. Another was how low Cosmos DB sits in the overall Azure cloud system. It is a foundational piece, as Shukla explains below.

image

There were several Cosmos DB announcements at Build. What’s new?

“Multi-master is one of the capabilities that we announced yesterday. It allows developers to scale writes all around the world. Until yesterday Cosmos DB allowed you to scale writes in a single region but reads all around the world. Now we allow developers to scale reads and writes homogeneously all round the world. This is a huge deal for apps like IoT, connected cars, sensors, wearables. The amount of writes are far more than the amount of reads.

“The second thing is that now you get single-digit millisecond write latencies at the 99 percentile not just in one region.

“And the third piece is that what falls out of this high availability. The window of failover, the time it takes to failover from one region when a disaster happens, to the other, has shrunk significantly.

“It’s the only system I know of that has married the high consistency models that we have exposed with multi-master capability as well. It had to reach a certain level of maturity, testing it with first-party Microsoft applications at scale and then with a select set of external customers. That’s why it took us a long time.

“We also announced the ability to have your Cosmos Db database in your own VNet (virtual network). It’s a huge deal for enterprises where they want to make sure that no data leaks out of that VNet. To do it for a global distributed database is specially hard because you have to close all the transitive networking dependencies.”

image
Technical Fellow Dharma Shukla

Does Cosmos DB work on Azure Stack?

“We are in the process of going to Azure Stack. Azure Stack is one of the top customer asks. A lot of customers want a hybrid Cosmos DB on Azure Stack as well as in Azure and then have Active – Active. One of the design considerations for multi master is for edge devices. Right now Azure has about 50 regions. Azure’s going to expand to let’s say 200 regions. So a customer’s single Cosmos DB table spanning all these regions is one level of scalability. But the architecture is such that if you directly attach lots of Azure Stack devices, or you have sensors and edge devices, they can also pretend to be replicas. They can also pretend to be an Azure region. So you can attach billions of endpoints to your table. Some of those endpoints could be Azure regions, some of them could be instances of Azure Stack, or IoT hub, or edge devices. This kind of scalability is core to the system.”

Have customers asked for any additional APIs into Cosmos DB?

“There is a list of APIs, HBase, richer SQL, there are a number of such API requests. The good news is that the system has been built in a way that adding new APIs is relatively easy addition. So depending on the demand we continue to add APIs.”

Can you tell me anything about how you’ve implemented Cosmos DB? I know you use Service Fabric. Do you use other Azure services?

“We have dedicated clusters of compute machines. Cosmos DB is a Ring 0 service. So it’s there any time Azure opens a new region, Cosmos DB clusters have provision by default. Just like compute, storage, Cosmos DB is also one of the Ring 0 services which is the bottommost. Azure Active Directory for example depends on Cosmos DB. So Cosmos DB cannot take a dependency on Active Directory.

“The dependency that we have is our own clusters and machines, on which we put Service Fabric. For deployment of Cosmos DB code itself, we use Service Fabric. For some of the load balancing aspects we use Service Fabric. The partition management, global distribution, replication, is our own. So Cosmos DB is layered on top of Service Fabric, it is a Service Fabric application. But then it takes over. Once the Cosmos DB bits are laid out on the machine then its replication and partition management and distribution pieces take over. So that is the layering.

“Other than that there is no dependency on Azure. And that is why one of the salient aspects of this is that you can take the system and host it easily in places like Azure Stack. The dependencies are very small.

“We don’t use Azure Storage because of that dependency. So we store the data locally and then replicate it. And all of that data is also encrypted at rest.”

So when you say it is not currently in Azure Stack, it’s there underneath, but you haven’t surfaced it?

“It is in a defunct mode. We have to do a lot of work to light it up. When we light up it on such on-prem or private cloud devices, we want to enable this active to active pathway. So you are replicating your data and that is getting synchronized with the cloud and Azure Stack is one of the sockets.”

Microsoft itself is using Cosmos DB. How far back does this go? Azure AD is quite old now. Was it always on Cosmos DB / DocumentDB?

“Over the years Office 365, Xbox, Skype, Bing, and more and more of Azure services, have started moving. Now it has almost become ubiquitous. Because it’s at the bottom of the stack, taking a dependency on it is very easy.

“Azure Active Directory consists of a set of microservices. So they progressively have moved to Cosmos DB. Same situation with Dynamics, and our slew of such applications. Skype is by and large on Cosmos DB now. There are still some fragments of the past.  Xbox and the Microsoft Store and others are running on it.”

Do you think your customers are good at making the right choices over which database technology to use? I do pick up some uncertainty about this.

“We are working on making sure that we provide that clarity. Postgres and MySQL and MariaDB and SQL Server, Azure SQL and elastic pools, managed instances, there is a whole slew of relational offerings. Then we have Cosmos DB and then lots of analytical offerings as well.

“If you are a relational app, and if you are using a relational database, and you are migrating from on-prem to Azure, then we recommend the relational family. It comes with this fundamental scale caveat which is that up to 4TB. Most of those customers are settled because they have designed the app around those sorts of scalability limitations.

“A subset of those customers, and a whole bunch of brand new customers, are willing to re-write the app. They know that that they want to come to cloud for scale. So then we pitch Cosmos DB.

“Then there are customers who want to do massive scale offline analytical processing. So there is, Databricks, Spark, HD Insight, and that set of services.

“We realise there are grey lines between these offerings. We’re tightening up the guidance, it’s valid feedback.”

Any numbers to flesh out the idea that this is a fast-growing service for Microsoft?

“I can tell you that the number of new clusters we provision every week is far more than the total number of clusters we had in the first month. The growth is staggering.”

Is Ron Jeffries right about the shortcomings of Agile?

A post from InfoQ alerted me to this post by Agile Manifesto signatory Ron Jeffries with the rather extreme title “Developers should abandon Agile”.

If you read the post, you discover that what Jeffries really objects to is the assimilation of Agile methodology into the old order of enterprise software development, complete with expensive consultancy, expensive software that claims to manage Agile for you, and the usual top-down management.

All this goes to show that it is possible do do Agile badly; or more precisely, to adopt something that you call Agile but in reality is not. Jeffries concludes:

Other than perhaps a self-chosen orientation to the ideas of Extreme Programming — as an idea space rather than a method — I really am coming to think that software developers of all stripes should have no adherence to any “Agile” method of any kind. As those methods manifest on the ground, they are far too commonly the enemy of good software development rather than its friend.

However, the values and principles of the Manifesto for Agile Software Development still offer the best way I know to build software, and based on my long and varied experience, I’d follow those values and principles no matter what method the larger organization used.

I enjoyed a discussion on the subject of Agile with some of the editors and writes at InfoQ during the last London QCon event. Why is it, I asked, that Agile is no longer at the forefront of QCon, when a few years back it was at the heart of these events?

The answer, broadly, was that the key concepts behind Agile are now taken for granted so that there are more interesting things to discuss.

While this makes sense, it is also true (as Jeffries observes) that large organizations will tend to absorb these ideas in name only, and continue with dark methods if that is in their culture.

The core ideas in Extreme Programming are (it seems to be) sound. Working in small chunks, forming a team that includes the customer, releasing frequently and delivering tangible benefits, automated tests and continuous refactoring, planning future releases as you go rather than in one all-encompassing plan at the beginning of a project; these are fantastic principles and revolutionary when you first come across them. See here for Jeffries’ account of what is Extreme Programming.

These ideas have everything to do with how the team works and little to do with specific tools (though it is obvious that things like a test framework, DevOps strategy and so on are needed).

Equally, you can have all the best tools but if the team is not functioning as envisaged, the methodology will fail. This is why software development methodology and the psychology of human relationships are intimately linked.

Real change is hard, and it is easy to slip back into bad practices, which is why we need to rediscover Agile, or something like it, repeatedly. Maybe the Agile word itself is not so helpful now; but the ideas are as strong as ever.

QCon London: the Ethics track and the psychology of software

The most significant thing about the Ethics track at QCon London, a software development conference I attended last week, is that it existed. I can recall ethics being discussed at QCon in previous years (including a memorable appeal by Martin Fowler at Thoughtworks about rectifying the gender imbalance in IT) but not a specific track.

Why does ethics matter more today? Ethics has always mattered, but the power of software over our lives is increasing. It is possible be that algorithms at Facebook, YouTube and Twitter influenced the result of the last US election and the UK’s Brexit referendum. Algorithms play a large role in influencing many of choices, what to buy, where to eat, where to stay, which airline to book, which vendor to use.

Software also consumes more of our time than ever, as we constantly check our phones for notifications, play games or read online content.

The increasing importance of AI (Artificial Intelligence) also raises ethical questions. Last week I attended the Re-work AI Assistant Summit, also in London. One of the sessions concerned “Building an AI Friend”, presented by Artem Rodichev from Replika. The demos were impressive, showing how a bot can be engaging and help users to talk about what matters to them. I asked though if the company had thought about ethical issues, for example if a child became attached to a bot without realising it was non-human. The answer I got was in effect a blank look, followed by the statement “we have a minimum age limit of 7”. The company has no announced business model, but I would encourage it to form an ethical policy early as these things are hard to bolt on in retrospect – as Facebook is discovering today, in the aftermath of the exposure of how its personal data is being misused by third parties.

AI is also poised to take over more jobs previously done by people. This could be a great liberator for humanity, or alternatively divide society even more deeply into haves and have-nots.

We need more ethics discussion then; but is it too late? Well, it is never too late to improve matters, but perhaps much harm could have been avoided if the industry had focused on this earlier.

I attended a talk by Alexander Steinhart (a technologist at ThoughtWorks) on the psychologist’s perspective on ethics in technology.

image

Steinhart talked about addiction. “We all want to unplug, but cannot”, he said.

“Now we are all connected. On average people are nearly three hours online every day. They check phones every 7 to 15 minutes. Many people have difficulties in finding the right balance.”

When is a habit an addiction? When it “gets into the way of your life and you can’t do anything else, and when you try to change behaviour you don’t manage,” said Steinhart, mentioning that “distraction” is identified as a risk by many people today, including teenagers.

Interruptions and distractions are detrimental to our productivity and also a source of stress, he said. Once you are distracted, it takes 20-25 minutes to recover your focus. “Take care that you are not connected all the time.”

Unfortunately we have also developed an “attention economy” where web sites and apps are rewarded for holding our attention and they have evolved to do that effectively.

A great way, apparently, to get us addicted is to have mechanisms that only occasionally reward us. We will try and try again in hope of reward. Lotteries are like this. So are slot machines. So too, says Steinhart, are things like notifications in apps, or the action of pulling down to refresh emails or other feeds. Most of the time we get nothing of value and we know that. But occasionally something really good arrives. The possibility keeps us hooked.

Another difficulty is that humans do not always cope well with abundance. When a previously scarce resource, food for example, becomes abundant, logically what should happen is that we become more discriminating, selecting only the best and discarding the rest. In practice though this is not the case, and we have seen the ascendance of junk food that does us harm.

We now have abundant information. Answering a question that might once have required a trip to the library or several phone calls can now be done in an instance. That is fantastic; but are we coping well? Somehow, instead of becoming more discriminating about the sources and value of available information, humanity is prone to consuming more and more information of low quality, whether that is banal time-wasting or actual falsehoods and information that is intended to deceive or mislead us.

Steinhart argues that we have moved into a new technological era but have not yet learned how to manage it. He draws an analogy with urbanisation; it tool mankind a while to learn how to build cities that were agreeable places in which to live.

Human needs includes some that are will served by today’s technological landscape. We need to experience “all of the different senses, to small, to taste.” We need privacy and solitude. “If you put managers alone for one hour in a room with nothing to do, they make better decisions the rest of the day,” claims Steinhart. We also need conversations, not just connections. “There is so much human interaction that you cannot digitise, like looking someone in the eye” he said.

How does this translate to ethics in technology? We need positive computing and software design that is “aligned with human goals,” he said.

Free and open source software is helpful in this respect, because the goals of the software are aligned with our needs rather then profit.

What can software developers do? “It is not your fault that technology is distracting,” said Steinhart, “but it’s your responsibility to change something.”

image

It is interesting to imagine what software might look like if designed for human needs rather than business interests. Steinhart’s ideas are around making software quieter, designed to get out of our way rather than to interrupt us, smartphones that encourage us to leave them alone, and of course to avoid anti-patterns which feed addiction or deliberately try to trip us up.

I noticed this tweet today about how an Amazon app behaves when you try to cancel. The user clicks Cancel subscription and gets this:

image

The following screen reverses the button colouring so that if you trained yourself to tap the faint button, you actually do the opposite of what you intend:

image

Until the last screen (there’s another one?) where they switch again:

image

This was not in Steinhart’s talk, but it seems a good example of software designed for the business and not for the user.

I have seen a similar pattern in Amazon’s web checkout where you have to click carefully to avoid being signed up for Amazon’s Prime subscription by accident. Not good.

Ethics and technology

This post is long enough; but there is, I hope, much more to say on this subject.

Despite enjoying Steinhart’s talk and others in the Ethics track, I was not encouraged. We need, of course, regulation as well as more principled businesses, and we do not know what such regulation should look like, nor how to implement it.

One thing though is worth repeating: if as a software developer you are asked to do something that is ethically unacceptable, you should refuse. Professional standards include more than quality of coding.