This morning’s Twitter feed informed me of the closure of codespaces.com, a company offering a repository and project management service to developers, using Git or subversion.
The reason was a malicious intrusion into its admin console for Amazon Web Services, which the company used as the back end for its services. The intruder demanded money, and when that was not forthcoming, deleted a large amount of data.
An unauthorised person who at this point who is still unknown (All we can say is that we have no reason to think its anyone who is or was employed with Code Spaces) had gained access to our Amazon EC2 control panel and had left a number of messages for us to contact them using a hotmail address
Reaching out to the address started a chain of events that revolved around the person trying to extort a large fee in order to resolve the DDOS.
Upon realisation that somebody had access to our control panel we started to investigate how access had been gained and what access that person had to data in our systems, it became clear that so far no machine access had been achieved due to the intruder not having our Private Keys.
At this point we took action to take control back of our panel by changing passwords, however the intruder had prepared for this and had a already created a number backup logins to the panel and upon seeing us make the attempted recovery of the account he locked us down to a non-admin user and proceeded to randomly delete artefacts from the panel. We finally managed to get our panel access back but not before he has removed all EBS snapshots, S3 buckets, all AMI’s, some EBS instances and several machine instances.
In summary, most of our data, backups, machine configurations and offsite backups were either partially or completely deleted.
According to the statement, the company is no longer viable and will cease trading. Some data has survived and customers are advised to contact support and recover what they can.
It is a horrible situation both for the company and its customers.
How can these kinds of risk be avoided? That is the question, and it is complex. Both backup and security are difficult.
Cloud providers such as Amazon offer excellent resilience and redundancy. That is, if a hard drive or a server fails, other copies are available and there should be no loss of data, or at worst, only a tiny amount.
Resilience is not backup though, and if you delete data, the systems will dutifully delete it on all your copies.
Backup is necessary in order to be able to go back in time. System administrators have all encountered users who demand recovery of documents they themselves deleted.
The piece that puzzles me about the CodeSpaces story is that the intruder deleted off-site backups. I presume therefore that these backups were online and accessible from the same admin console, a single point of failure.
As it happens, I attended Cloud World Forum yesterday in London and noticed a stand from Spanning, which offers cloud backup for Google Apps, Salesforce.com, and coming soon, Office 365. I remarked light-heartedly that surely the cloud never fails; and was told that yes, the cloud never fails, but you can still lose data from human error, sync errors or malicious intruders. Indeed.
Is there a glimmer of hope for CodeSpaces – is it possible that Amazon Web Services can go back in time and restore customer data that was mistakenly or maliciously deleted? I presume from the gloomy statement that it cannot (though I am asking Amazon); but if this is something the public cloud cannot provide, then some other strategy is needed to fill that gap.