Evans Data has published its 2009 Software Development Platforms survey, to which around 1200 developers contributed, scoring their chosen development tools in eighteen different categories.
The tools covered are Eclipse, Embarcadero’s Delphi, IBM’s Rational Suite, IntelliJ, Microsoft’s Visual Studio, NetBeans, Oracle JDeveloper and Sun Studio.
I was sorry not to see more products covered. Flex Builder Flash Builder, Zend Studio, Aptana and JBuilder would all have been interesting, for example. Each developer only scored the product they actually use (a good thing), so the sample is not as big as it first appears.
I’m also mistrustful of the survey results, particularly when you look at it in detail. For example, one of the categories is “Support for frameworks”. Visual Studio came top, while Eclipse was last. But hang on: in Visual Studio (for example) are we talking MFC, or .NET Framework? The development experience for each is totally different. And were developers primarily judging on the framework tools, or the framework itself? It is hard to attach much meaning to the scores in this category.
Another flaw: the versions of the products is not specified. That means a weakness may have been fixed in a later version, but the survey does not tell you.
A third flaw: some tools are weak in several categories, dragging down their overall score, but that does not matter to developers who do not use them for that purpose. It is hard to compare like with like.
Still, while I’m wary of the survey overall, I though it brought out some interesting points. One is that developers were asked what features matter most to them. So:
The three things developers care about most (highest priority first):
1. Basic tools (editor/compiler/debugger)
2. Documentation
3. Tool integration
The things developers care about least (lowest priority first):
1. Support for remote development
2. Support for parallel programming (sorry Intel!)
3. App Modeling tools
What about the winners and losers in the survey? I almost forgot. IBM’s Rational Suite came top, followed by Microsoft’s Visual Studio. Eclipse came last, though it still got a decent score, well below its rival NetBeans.
The low ranking for Eclipse (which is nevertheless wildly popular) deserves some comment, particularly as the top tool, IBM Rational Suite, is built on Eclipse. I spoke to Eclipse executive director Mike Milinkovich while researching this Register piece recently. One of the points we discussed was the tension (if there is one, which he disputes) between tools vendors sharing resources to build the best possible platform, and holding resources back to retain commercial differentiation. I’ll write this up in more detail shortly; but it shows why certain areas in Eclipse may not receive the attention they deserve – localization was a specific example.
Another problem with Eclipse is that it is all a bit messy, confusing and hard to manage, particularly in a team where you want every member to have an identical setup. It is still worth it though, for the riches it provides for free.
Unfortunately, there is a fallacy lurking under the nice neat rankings of things developers care about most and least. We have become so accustomed to polling groups of people in our democratic age that it’s easy to forget that sometimes the numbers can hide other patterns.
For example, the survey report tells us that remote development is something that “developers care about least”. But surely it would be a more accurate interpretation of the facts to say that remote development is something that “few developers care about”. That’s simply because most groups of developers are in the same building, so they don’t need support for remote development.
I also think the remark “sorry Intel!” misses the point by a wide margin. Shouldn’t that be “Sorry end-users”?
Is it not worrying to think that programmers are not concerned with parallel programming?
If this survey was taken to be the “be all and end all” of what should be concentrated on when developing a language, we would be on a sorry state.
That said, I think it just highlights how many people like me there are out there. If, like me, you only know a very basic level of C# then naturally parallel programming is of no use to me. If you look at the average Windows application for example, you find a lot of them are running on .NET Framework and you are unlikely to be working deep enough to worry about parallel programming.
Of course if you are writing a physics engine or pretty much anything with “rendering” in the name, you probably are going to be more concerned with parallel programming. This survey just highlights the natural flow of things, that there are far more people doing really high-level programming than low-level. Even games developers likely aren’t worrying about parallel programming as they often license the engine from someone else who had the responsibility for that.