Hacked in the Cloud
Amazon's Elastic Compute Cloud or EC2 provides an easy and economical way to run servers "out in the cloud". In just a minute or two of clicking through some web dashboard screens (or running their command-line interface, or making API calls) you can have one or more servers running in their facilities in the eastern US, California, Oregon, São Paulo, Dublin, Frankfurt, Tokyo, Singapore, or Sydney. The price can be as low as US$ 0.02 per hour.
It's easy, it's cheap, but it isn't necessarily safe.
Nor is it especially dangerous. In fact, Alert Logic's monitoring service shows that in-house computing environments have significantly more problems than those hosted either in the cloud or at service providers.
But wherever your server is, you must be careful.
Amazon provides thousands of images of systems.
These are templates that you can have copied onto a virtualized
platform and started.
Your SSH public key is copied into place on the server
~/.ssh/authorized_keys in the home directory
root or the
Amazon calls these ready-to-run templates
Amazon Machine Images or AMIs.
An image or AMI is the standard image waiting to be copied and started as a running instance.
Bitnami is an open-source project that creates installers for web application and development stacks. For example: an Apache web server front end, plus a MySQL database on the back end, plus some programming in PHP or Python to tie them together, plus some automatic monitoring and backup. Bitnami Cloud Hosting provides Bitnami-based AMIs in Amazon's EC2.
<?php include($_SERVER['DOCUMENT_ROOT'].'/ads/responsive-infosec.html'); ?>
When your browser asks for this page, the server first inserts the referenced file into place.
You can do much more with PHP, and among many other tasks, Bitnami images use it to provide a web-based interface to a MySQL database.
Some of these Bitnami AMIs contained a vulnerability in the PHP language. The vulnerability is described here, it was a problem in PHP version 5.3.* before 5.3.12 and in 5.4.* before 5.4.2.
The first thing we noticed was this e-mail from Amazon telling us that one of the Bitnami instance we were running had launched a denial-of-service attack against a host in Sweden (the target IP address 184.108.40.206 was allocated to portlane.com in Stockholm):
This instance had been started around 17:00 UTC on Monday. In less than two and half days an attacker had managed to find it, break in, and start using it as a largely untraceable tool to attack others. The cited "Abuse Time" was 22:10 UTC on Wednesday. Amazon's message went out at 00:06 PDT or 07:06 UTC Thursday.
Once the attackers know about this PHP vulnerability, they
follow a fairly simple path.
First, figure out which AMIs contain the vulnerability.
The AMI name could announce the presence, something
bitnami-with-php-5.3.9 or similar.
Or maybe someone needs to do a little low-cost experimenting
to see which AMIs contain it.
Second, devise an automated way of scanning the Amazon Web Services blocks of IP addresses looking for running instances containing this vulnerability.
Third, start breaking in and abusing those resources.
The very simple fix for this specific instance was simply to terminate it. The instance had already served its purpose.
That might seem a little unusual or drastic, but a lot of times a cloud instance is only needed for a short time. Why pay to run it when you don't really need it? Many cloud servers are ephemeral, running for a very limited time. Of course, this makes it even harder to notice vulnerabilities! At least it's also more difficult for the bad guys...
Nine days later, news came out from Amazon:
I like a number of things about this.
Let's look at the initial report first. Before Amazon had found the underlying cause, they pointed out that our instance was hacked without acting like the Internet Police. They still use the word "please" multiple times in this message, good for them!
They also made some changes, modifying the firewall rules to block all but SSH and RDP, the safe administrative control mechanisms for Linux and Windows, respectively.
They close by suggesting security references at various levels of detail.
The second message is even better. Now Bitrock has told Amazon what is going on. Amazon has told us how to find our instances still at risk from this vulnerability. They then explain how to work around the problem on an instance we want to keep running, and (more usefully) how to migrate an operation to a compatible instance based on a patched image.
Bitrock has removed the vulnerable images and told us how to find compatible replacements.
With Amazon's warning about CloudFormation and Autoscaling, we should be able to avoid an accidental denial of service inflicted by forgetting to modify our cloud architecture design.
Finally, we are given further suggestions on how to get guidance from both Bitnami and Amazon.
I think it's important to notice that the AMI source and the cloud provider communicated with us, spotting a problem and providing guidance, but they did not step in and clean up the mess for us.
Some people will argue that a cloud provider should do all that and more for us, but they are missing the point. The standard definition of "cloud computing" requires on-demand self-service, otherwise we won't have the rapid elasticity and measured service at very low cost.
Yes, you can get remotely hosted managed services run by a third party. It will cost a lot more, be far less flexible, and it won't really be "cloud computing" by the commonly accepted definition.
Back to the Security Page