Tag Archives: internet

Google’s search monopoly, the decline of organic search and its implications

A piece by Rand Fishkin tells me what I already knew: that Google has a de facto monopoly in search, and that organic search (meaning clicking on a result from a search engine that is not an ad) is in decline, especially on mobile.

According to Fishkin, using data from digital intelligence firm Jumpshot, Google properties deliver 96.1% of all search in the EU and 93.4% of all US searches. “Google properties” include Google, Google Images, Youtube, and Google Maps.

To the extent that this shows high satisfaction with Google’s service, this is a credit to the company. We should also look carefully though at the outcome of those searches. In the latest figures available (Jan-Sept 2018) they break down as follows (EU figures):

  • Mobile: 36.7% organic, 8.8% paid, 54.4% no-click
  • Desktop: 63.6% organic, 6.4% paid, 30% no-click

On mobile, the proportion of paid clicks has more than doubled since 2016. On the desktop, it has gone up by over 40%.

A no-click search is one where the search engine delivers the result without any click-through to another site. Users like this in that it saves a tap, and more important, spares them the ads, login-in pleas, and navigation challenges that a third-party site may present.

There is a benefit to users therefore, but there are also costs. The user never leaves Google, there is no opportunity for a third-party site to build a relationship or even sell a click on one of its own ads. It also puts Google in control of information which has huge political and commercial implications, irrespective of whether it is AI or Google’s own policies that determine what users see.

My guess is that the commercial reality is that organic search has declined even more than the figures suggest. Not all searches signal a buying intent. These searches are less valuable to advertisers and therefore there are fewer paid ads. On the other hand, searches that do indicate a buying intent (“business insurance”, “IT support”, “flight to New York”) are highly valued and attract more paid-for advertising. So you can expect organic search to me more successful on searches that have less commercial value.

In the early days of the internet the idea that sites would have to pay to get visitors was not foreseen. Of course it is still possible to build traffic without paying a Google tax, via social media links or simply by hosting amazing content that users want to see in full detail, but it is increasingly challenging.

There must be some sort of economic law that says entities that can choose whether to give something away or to charge for it, will eventually charge for it. We all end up paying, since whoever actually provides the goods or services that we want has to recoup the cost of winning our business, including a share to Google.

Around six years ago I wrote a piece called Reflecting on Google’s power: a case for regulation? Since then, the case for regulation has grown, but the prospect of it has diminished, since the international influence and lobbying power of the company has also grown.

Let’s Encrypt: a quiet revolution

Any website that supports SSL (an HTTPS connection) requires a  digital certificate. Until relatively recently, obtaining a certificate meant one of two things. You could either generate your own, which works fine in terms of encrypting the traffic, but results in web browser warnings for anyone outside your organisation, because the issuing authority is not trusted. Or you could buy one from a certificate provider such as Symantec (Verisign), Comodo, Geotrust, Digicert or GoDaddy. These certificates vary in price from fairly cheap to very expensive, with the differences being opaque to many users.

Let’s Encrypt is a project of the Internet Security Research Group, a non-profit organisation founded in 2013 and sponsored by firms including Mozilla, Cisco and Google Chrome. Obtaining certificates from Let’s Encrypt is free, and they are trusted by all major web browsers.

image

Last month Let’s Encrypt announced coming support for wildcard certificates as well as giving some stats: 46 million active certificates, and plans to double that in 2018. The post also notes that the latest figures from Firefox telemetry indicate that over 65% of the web is now served using HTTPS.

image
Source: https://letsencrypt.org/stats/

Let’s Encrypt only started issuing certificates in January 2016 so its growth is spectacular.

The reason is simple. Let’s Encrypt is saving the IT industry a huge amount in both money and time. Money, because its certificates are free. Time, because it is all about automation, and once you have the right automated process in place, renewal is automatic.

I have heard it said that Let’s Encrypt certificates are not proper certificates. This is not the case; they are just as trustworthy as those from the other SSL providers, with the caveat that everything is automated. Some types of certificate, such as those for code-signing, have additional verification performed by a human to ensure that they really are being requested by the organisation claimed. No such thing happens with the majority of SSL certificates, for which the process is entirely automated by all the providers and typically requires that the requester can receive email at the domain for which the certificate is issued. Let’s Encrypt uses other techniques, such as proof that you control the DNS for the domain, or are able to write a file to its website. Certificates that require human intervention will likely never be free.

A Let’s Encrypt certificate is only valid for three months, whereas those from commercial providers last at least a year. Despite appearances, this is not a disadvantage. If you automate the process, it is not inconvenient, and a certificate with a shorter life is more secure as it has less time to be compromised.

The ascendance of Let’s Encrypt is probably regretted both by the commercial certificate providers and by IT companies who make a bit of money from selling and administering certificates.

Let’s Encrypt certificates are issued in plain-text PEM (Privacy Enhanced Mail) format. Does that mean you cannot use them in Windows, which typically uses .cer or .pfx certificates?  No, because it is easy to convert between formats. For example, you can use the openssl utility. Here is what I use on Linux to get a .pfx:

openssl pkcs12 -inkey privkey.pem -in fullchain.pem -export -out yourcert.pfx

If you have a website hosted for you by a third-party, can you use Let’s Encrypt? Maybe, but only if the hosting company offers this as a service. They may not be in a hurry to do so, since there is a bit of profit in selling SSL certificates, but on the other hand, a far-sighted ISP might win some business by offering free SSL as part of the service.

Implications of Let’s Encrypt

Let’s Encrypt removes the cost barrier for securing a web site, subject to the caveats mentioned above. At the same time, Google is gradually stepping up warnings in the Chrome browser when you visit unencrypted sites:

Eventually, we plan to show the “Not secure” warning for all HTTP pages, even outside Incognito mode.

Google search is also apparently weighted in favour of encrypted sites, so anyone who cares about their web presence or traffic is or will be using SSL.

Is this a good thing? Given the trivia (or worse) that constitutes most of the web, why bother encrypting it, which is slower and takes more processing power (bad for the planet)? Note also that encrypting the traffic does nothing to protect you from malware, nor does it insulate web developers from security bugs such as SQL injection attacks – which is why I prefer to call SSL sites encrypted rather than secure.

The big benefit though is that it makes it much harder to snoop on web traffic. This is good for privacy, especially if you are browsing the web over public Wi-Fi in cafes, hotels or airports. It would be a mistake though to imagine that if you are browsing the public web using HTTPS that you are really private: the sites you visit are still getting your data, including Facebook, Google and various other advertisers who track your browsing.

In the end it is worth it, if only to counter the number of times passwords are sent over the internet in plain text. Unfortunately people remain willing to send passwords by insecure email so there remains work to do.

Google’s Eric Schmidt looks forward to an Android in every pocket

Google’s Executive Chairman Eric Schmidt addressed the Mobile World Congress in Barcelona in confident mood, boasting of the strong growth in Android adoption and saying that the world would need to increase its population in order to sustain current rates of growth.

image

His keynote was in three parts. He kicked off with a plug for Chrome for Android, handing over to another Googler to show off its unlimited tabs and predictive background downloading which gives you near-instant page rendering if you pick the top hit after a Google search.

Next, he gave a somewhat political address appealing for light regulation of the Internet, on the grounds that any change to the current setup was likely to make it worse – “regulators regulate, that is what they do”. He also expressed his hope that fast internet access would be better extended to the world beyond the wealthy elite nations, noting the role of connectivity in the Arab Spring and in making public the actions of brutal dictators.

What has all this to do with Google’s business? Mainly, I suppose, that more connections means more Google searches which means more advertising income; Schmidt acknowledged that this forms the great majority of the company’s revenue, in the high nineties percentage-wise.

The third part of Schmidt’s session was the most interesting, when he took questions. He was asked for his thoughts on companies (I am sure Amazon was in the questioner’s mind) which take Android and remove all Google’s services. It is open source, he said, and that is entirely permissible; his hope is that customers will demand Google services and the Android market.

There was a revealing moment when an Iranian in the audience challenged Schmidt over his appeal for a free Internet. Chrome for Android was not available in Iran, he said, because Google, “your company”, is blocking it.

Schmidt immediately communicated with his legal team, who turned out to be sitting in the front row. Then he confirmed the block and said it was a requirement of US law, because of sanctions against Iran. “I’m with you”, he said, but Google has to comply with the law.

Another point of note was the number of references Schmidt made to privacy issues. He said that we will see increasing personalization of search and better targeted advertising, such that users will not mind it. You will be able to opt out, but according to Schmidt you will not want to. He added that personal data collection and use will continue to advance as far as society deems it ethical, a fascinating turn of phrase and one that bears further examination.

What was not mentioned? Apple, Microsoft, patents, the impact of the Motorola Mobility acquisition (he did say that that this last is not yet complete and that currently the company is managed independently).

It was an impressive keynote overall, given from a position of strength, at an event which is dominated in many respects by Android devices.

WebKit dominance threatens mobile web standards – but who will care?

Daniel Glazman, co-chairman of the W3C CSS working group, has written a strongly-worded post describing how the “over-dominance” of the WebKit rendering engine threatens web standards.

Everyone loves the open source WebKit, so how is this so? The issue is a complex one. Those who make web browsers do not want to be tied only to those standards already ratified by the W3C as part of HTML or CSS. Therefore, they add features, sometimes in the hope that they will become standards, but use a vendor-specific prefix such as -webkit-,-moz- or -ms-. If you use those features in your markup, you do so in the awareness that they will only work on that specific vendor’s browser. The idea is that the best vendor-specific extensions become standard, in which case the prefix is dropped; or are replaced by an equivalent standard, in which case the prefix is also dropped. This has become an accepted part of the way standards are formed.

The issue now is that WebKit dominates the mobile web to the extent that web authors can assume its use without losing many users. WebKit is used in Apple iOS, Google Android, RIM BlackBerry 6 and higher, as well as on the desktop in Apple Safari and Google Chrome. Amazon also uses WebKit in the Kindle and of course the Android-based Kindle Fire.

The consequence, says Glazman, is that:

technically, the mobile Web is full of works-only-in-WebKit web sites while other browsers and their users are crying.

The further consequence, and this is Glazman’s strongest point, is that other browsers will have to pretend to be WebKit and support its extensions in order to give users a good experience – even if they have their own vendor-specific extensions that support the same features:

All browser vendors let us officially know it WILL happen, and rather sooner than later because they have, I quote, "no other option".

Glazman says “all browser vendors” which suggests that even Microsoft will do this, though that would be a surprising development.

This would mean that the -webkit- vendor-specific extensions were no longer vendor-specific. It would also meant that WebKit is in effect able to create web standards without the bother of going through the W3C:

It will turn a market share into a de facto standard, a single implementation into a world-wide monopoly. Again. It will kill our standardization process. That’s not a question of if, that’s a question of when.

says Glazman, suggesting that there is a risk of a return to the bad days when the dominance Microsoft’s IE6 prevented standards from evolving.

The parallel with IE6 is weak. IE6 was not an open source project, and the damage it did was in part because Microsoft deliberately chose not to invest in advancing HTML, preferring to drive users towards rich internet-connected Windows applications. It is difficult to see how that can happen to WebKit.

Nevertheless, the situation with WebKit is making it difficult for other mobile browsers to compete and does undermine the standards process. This is not really the fault of the WebKit team, though the W3C would like to see support for obsolete vendor-specific extensions dropped more quickly to discourage their use. Rather, it is a consequence of web authors seeing little value in adding support for other browsers that have little actual use on the mobile web.

It is worth observing that Glazman is a Mozilla guy, and his company Disruptive Innovations makes Mozilla extensions.

How can this be resolved? Glazman and others are right to raise awareness of the issue, but I doubt that many outside the standards community or browser vendors themselves will see this as a major problem.

The best fix would be for non-WebKit browsers to become more popular on the mobile web. Growing use of Windows Phone, for example, would give web authors more incentive to fix their markup. Another route to improving standards is via tools which do the right thing. Adobe’s strong support for CSS in Dreamweaver, for example, gave a significant boost to its use and helped to rescue us from font tags and the like.

Finally, it seems to me that the distinction between the “mobile” web and the “full” web is blurring, and rightly so. Users on mobile devices often tap the “full site” link where available since they have big enough screens to benefit. WebKit does not yet dominate the desktop Web. 

Wikipedia goes dark for a day to protest against proposed US legislation

All Wikipedia English requests today redirect to a page protesting against proposed US legislation, specifically the draft SOPA and PIPA legislation.

image

Other sites will also be protesting, including Reddit (a 12 hour protest) and Mozilla, the Firefox people.

Many web searchers will be discovering the value of the cached pages held by search engines. That aside, this is a profound issue that is about more than just SOPA and PIPA. SOPA stands for Stop Online Piracy, and PIPA Protect IP. The problem is that the internet is the powerful means of sharing information that mankind has devised. This brings many benefits, but not so much if it is your proprietary information that is being exchanged for free (music, video, ebooks, software) or if it gives easy access to counterfeit versions of your products, with designer handbags, watches, drugs and the like being particularly vulnerable, because intellectual property forms a large proportion of the value of the purchase.

If you consider this issue at the highest level, there are three broad solutions:

  1. Do nothing, on the grounds that the world has changed and there is nothing you can do that is effective. Technology has made some forms of copyright impossible to enforce. Affected businesses have to adapt to this new world.
  2. Introduce legislation that widens the responsibility for web sites that enable or facilitate copyright infringement beyond the sites themselves, to include search engines, ISPs and payment processors. One of the debates here is how much the owners of the pipes, the infrastructure on which the internet runs, should take legal responsibility for the content that flows through them. Such legislation might be somewhat effective, but at a heavy cost in terms of forcing many sites and services offline even if they have only a slight and tangential relationship to infringing content, and greatly raising the cost of providing services. At worst we might end up with a censored, filtered, limited, expensive internet that represents a step backwards in technology. The further risk is that that such legislation may put too much power in the hands of the already powerful, since winning legal arguments is in practice about financial muscle as well as facts, rights and wrongs.
  3. Find some middle path that successfully restrains the flow of infringing content but without damaging the openness of the internet or its low cost.

There is of course a risk that legislators may think they are implementing the third option, while in fact coming close to the second option. There is also a risk that attempting to implement the third option may in practice turn out to be the first option. It is a hard, complex problem; and while I agree that the proposed legislation is not the right legislation (though note that I am not in the USA), there is no disputing the core fact, that the internet facilitates copyright infringement.

There are also aspects of today’s internet that concern me as I see, for example, children relying on the outcome of Google searches to paste into their homework with little understanding of the distinction between what is authoritative, what is propaganda, and what is anecdotal or simply wrong.

In other words, the “no control” approach to the internet has a downside, even if the upside is so great that it is worth it.

The power of Google: how the Panda update hit Experts Exchange

Searching Google recently it struck me that I rarely see results from Experts Exchange. I used to see a lot of these, because I typically search on things like error messages or programming issues for which the site is a useful source.

The site is controversial, because it (kind-of) charges for access to its knowledgebase but does not pay its experts. I posted about this back in 2009. That said, the quality of its advice is often good, and most answers are available without payment if you scroll far enough down the page. You can also get free access as an expert if you answer a few queries successfully.

Experts Exchange has to some extent been replaced by the StackOverflow group of websites, which are nicer to use and free, but I have found that the chances of getting your obscure query answered can be higher on Experts Exchange, particularly for admin rather than programming queries (of course for admin I am comparing with ServerFault).

Still, I wanted to test my perception that I no longer see Experts Exchange results in Google. I had a look at the Alexa stats for the site.

image

Wow! That vertical line is around April 2011, which is when Google rolled out its "High Quality Sites Algorithm". The site still ranks in the top 3000 in the world according to Alexa – 2787 at the time of writing – but according to the chart it lost around 50% of its visitors then, and has since declined further.

As noted above, the site is controversial, but I personally never minded seeing Experts Exchange results in my searches since the advice there is often good.

The bit that disturbs me though is simply the power Google has over what we read on the Internet. I appreciate the reasons, but it is not healthy for one corporation to have this level of influence, especially bearing in mind the black box nature of its workings.

First impressions of Google TV – get an Apple iPad instead?

I received a Google TV as an attendee at the Adobe MAX conference earlier this year; to be exact, a Logitech Revue. It is not yet available or customised for the UK, but with its universal power supply and standard HDMI connections it works OK, with some caveats.

The main snag with my evaluation is that I use a TV with built-in Freeview (over-the-air digital TV) and do not use a set top box. This is bad for Google TV, since it wants to sit between your set top box and your TV, with an HDMI in for the set top box and an HDMI out to your screen. Features like picture-in-picture, TV search, and the ability to choose a TV channel from within Google TV, depend on this. Without a set-top box you can only use Google TV for the web and apps.

image

I found myself comparing Google TV to Windows Media Center, which I have used extensively both directly attached to a TV, and over the network via Xbox 360. Windows Media Center gets round the set top box problem by having its own TV card. I actually like Windows Media Center a lot, though we had occasional glitches. If you have a PC connected directly, of course this also gives you the web on your TV. Sony’s PlayStation 3 also has a web browser with Adobe Flash support, as does Nintendo Wii though it is more basic.

image

What you get with Google TV is a small set top box – in my case it slipped unobtrusively onto a shelf below the TV, a wireless keyboard, an HDMI connector, and an IR blaster. Installation is straightforward and the box recognised my TV to the extent that it can turn it on and off via the keyboard. The IR blaster lets you position an infra-red transmitter optimally for any IR devices you want to control from Google TV – typically your set-top box.

I connected to the network through wi-fi initially, but for some reason this was glitchy and would lose the connection for no apparent reason. I plugged in an ethernet cable and all was well. This problem may be unique to my set-up, or something that gets a firmware fix, so no big deal.

There is a usability issue with the keyboard. This has a trackpad which operates a mouse pointer, under which are cursor keys and an OK button. You would think that the OK button represents a mouse click, but it does not. The mouse click button is at top left on the keyboard. Once I discovered this, the web browser (Chrome, of course) worked better. You do need the OK button for navigating the Google TV menus.

I also dislike having a keyboard floating around in the living room, though it can be useful especially for things like Gmail, Twitter or web forums on your TV. Another option is to control it from a mobile app on an Android smartphone.

The good news is that Google TV is excellent for playing web video on your TV. YouTube has a special “leanback” mode, optimised for viewing from a distance that works reasonably well, though amateur videos that look tolerable in a small frame in a web browser look terrible played full-screen in the living room. BBC iPlayer works well in on-demand mode; the download player would not install. Overall it was a bit better than the PS3, which is also pretty good for web video, but probably not by enough to justify the cost if you already have a PS3.

The bad news is that the rest of the Web on Google TV is disappointing. Fonts are blurry, and the resolution necessary to make a web page viewable from 12 feet back is often annoying. Flash works well, but Java seems to be absent.

Google also needs to put more thought into personalisation. The box encouraged me to set up a Google account, which will be necessary to purchase apps, giving me access to Gmail and so on; and I also set up the Twitter app. But typically the living room is a shared space: do you want, for example, a babysitter to have access to your Gmail and Twitter accounts? It needs some sort of profile management and log-in.

In general, the web experience you get by bringing your own laptop, netbook or iPad into the room is better than Google TV in most ways apart from web video. An iPad is similar in size to the Google TV keyboard.

Media on Google TV has potential, but is currently limited by the apps on offer. Logitech Media Player is supplied and is a DLNA client, so if you are lucky you will be able to play audio and video from something like a NAS (network attached storage) drive on your network. Codec support is limited.

In a sane, standardised world you would be able to stream music from Apple iTunes or a Squeezebox server to Google TV but we are not there yet.

One key feature of Google TV is for purchasing streamed videos from Netflix, Amazon VOD (Video on Demand) or Dish Network. I did not try this; they do not work yet in the UK. Reports are reasonably positive; but I do not think this is a big selling point since similar services are available by many other routes. 

Google TV is not in itself a DVR (Digital Video Recorder) but can control one.

All about the apps

Not too good so far then; but at some point you will be able to purchase apps from the Android marketplace – which is why attendees at the Adobe conference were given boxes. Nobody really knows what sort of impact apps for TV could have, and it seems to me that as a means of running apps – especially games – on a TV this unobtrusive device is promising.

Note that some TVs will come with Google TV built-in, solving the set top box issue, and if Google can make this a popular option it would have significant impact.

It is too early then to write it off; but it is a shame that Google has not learned the lesson of Apple, which is not to release a product until it is really ready.

Update: for the user’s perspective there is a mammoth thread on avsforum; I liked this post.

Google favours big brands over diversity

Google has made a change to its search algorithm that means most of the results shown for a search may now come from a single domain. Previously, it would only show a couple of results from one domain, on the assumption that users would prefer to select from a diversity of results.

The example chosen by searchengineland is a good one. Search for Apple iPod and you get a page that is mostly links to Apple’s site.

image

If you search for the single word ipod you get more diversity – odd, since only Apple manufacture the ipod so you could argue that the searches are the same. Some people use ipod as a generic name for MP3 player, but that doesn’t seem to be reflected; all the results still relate to Apple’s device.

image

Personally I’d rather see diversity. I don’t see the need for this change, since the site summary with deep links works well when a particular domain closely matches the search term. You can see an example of this in the top result for the ipod search above. Note that it even has a link for “More results from apple.com”. What is the value of suppressing the results from other domains?

The overall impact is that big brands benefit, while smaller businesses and new entrants to markets suffer. It also makes independent comment that bit harder to find.

While to most of us changes like these are only of passing interest, to some they make the difference between a flourishing business and a dead one. Google has too much power.

Incidentally, I generally find Google significantly better than Bing, now its major competitor. However in this case Bing impresses, with categories such as reviews, prices, accessories, manuals and so on; and in the case of the Apple ipod search, a better balance between the official site and independents.

Google flexes page rank muscles, hits Daily Express?

It’s been reported that the Daily Express newspaper is selling paid links, in other words links that look to Google’s web crawler like links from independent editorial, but in fact are paid for by advertisers.

The consequence of doing this, according to Google:

However, some SEOs and webmasters engage in the practice of buying and selling links that pass PageRank, disregarding the quality of the links, the sources, and the long-term impact it will have on their sites. Buying or selling links that pass PageRank is in violation of Google’s webmaster guidelines and can negatively impact a site’s ranking in search results.

I then saw a report saying that Google has indeed penalised the Daily Express. I checked the page rank of the Daily Express home page here and found that it is indeed lower than could be expected.

image

While it is difficult to say what the page rank should be, Alexa shows the Express site as among the top 15,000 worldwide and in the top 1000 in the UK, with over 4,000 incoming links.

Although the Daily Express is not to my taste, I have misgivings about this process. A significant proportion of web traffic comes via Google, and lower search rankings have a direct effect on traffic and therefore business. Exactly how Google determines search rankings is a commercial secret, even though the gist of how it works is well known. There is not much you can do if mistakes are made, other than to complain to Google and hope someone pays attention to you.

There is nothing illegal about selling paid links, and the article highlighted in the Express is marked as “Sponsored”. While it is right for Google not to count these links as genuine recommendations, I am less sure about whether there should be additional punishment for running them. It puts too much power to make and break other companies into the hands of Google.

Of course we don’t know if the low ranking is a result of the paid links or not; that is speculation. Nor do we know how closely the published page rank corresponds which how Google actually determines the order of search results. Looking at Alexa’s report, there’s no conclusive evidence of declining traffic, though Alexa’s figures are based on a relatively small sample.

Still, I get a ton of these paid link requests and this kind of story makes me glad that I always turn them down.

Magazine chief: iPad users – prepare to be retrained

The Guardian has an interview with Future Chief Executive Stevie Spring. Future is a major magazine publisher based in the UK. I was interested to hear how she believes the iPad could change the industry:

We’ve had a whole decade of people paying, believing that if they paid for the pipes they got the poetry free; [they think] ‘I’ve paid my £15 or £20 for broadband so I get access to a library of content’. The iPad gives us an opportunity to retrain them. Content production is not free and good content is worth paying for.

I am all in favour of more people paying for content. However, there are a couple of aspects of this line of argument which concern me. One is pure scepticism – how many print readers will actually be willing to transition to paying for online content just because the iPad is a convenient way to consume it? The problem is that while print has an unique appeal, once you are online it is easy to find equally good content for free, in the case of the consumer magazines in which Future tends to specialise.

The other concern is a deeper one. I get the sense that Spring is talking about content delivered as apps, since this is a proven business – people will more willingly pay for an app, apparently, than subscribe to a web site.

However, content delivered in an app is one step forward, two steps back. The step forward is possibly richer content, with the full power of the local machine. The steps back are that it is not part of the world wide web – not searchable or linkable.

Finally, there is the Apple problem. Is this a Future where we have to be Apple customers in order to enjoy its publications?