Hacked in the Cloud
Amazon's Elastic Compute Cloud or EC2 provides an easy and economical way to run servers "out in the cloud". In just a minute or two of clicking through some web dashboard screens (or running their command-line interface, or making API calls) you can have one or more servers running in their facilities in the eastern US, California, Oregon, São Paulo, Dublin, Frankfurt, Tokyo, Singapore, or Sydney. The price can be as low as US$ 0.02 per hour.
It's easy, it's cheap, but it isn't necessarily safe.
Nor is it especially dangerous. In fact, Alert Logic's monitoring service shows that in-house computing environments have significantly more problems than those hosted either in the cloud or at service providers.
But wherever your server is, you must be careful.
Amazon provides thousands of images of systems.
These are templates that you can have copied onto a virtualized
platform and started.
Your SSH public key is copied into place on the server
~/.ssh/authorized_keys in the home directory
root or the
Amazon calls these ready-to-run templates
Amazon Machine Images or AMIs.
An image or AMI is the standard image waiting to be copied and started as a running instance.
Bitnami is an open-source project that creates installers for web application and development stacks. For example: an Apache web server front end, plus a MySQL database on the back end, plus some programming in PHP or Python to tie them together, plus some automatic monitoring and backup. Bitnami Cloud Hosting provides Bitnami-based AMIs in Amazon's EC2.
<?php include($_SERVER['DOCUMENT_ROOT'].'/ads/responsive-infosec.html'); ?>
When your browser asks for this page, the server first inserts the referenced file into place.
You can do much more with PHP, and among many other tasks, Bitnami images use it to provide a web-based interface to a MySQL database.
Some of these Bitnami AMIs contained a vulnerability in the PHP language. The vulnerability is described here, it was a problem in PHP version 5.3.* before 5.3.12 and in 5.4.* before 5.4.2.
The first thing we noticed was this e-mail from Amazon telling us that one of the Bitnami instance we were running had launched a denial-of-service attack against a host in Sweden (the target IP address 220.127.116.11 was allocated to portlane.com in Stockholm):
From: Amazon EC2 Abuse [mailto:email@example.com]
Sent: Thursday, October 31, 2013 12:06 AM
Subject: Your Amazon EC2 Abuse Report [14200733512-1]
Dear Amazon EC2 Customer,
We've received a report that your instance(s):
Instance Id: i-90b0fcf4
IP Address: 18.104.22.168
has been making Denial of Service attacks against remote hosts on the Internet; check the information provided below by the abuse reporter.
This is specifically forbidden in our User Agreement:
Please immediately restrict the flow of traffic from your instances(s) to cease disruption to other networks and reply this email to send your reply of action to the original abuse reporter. This will activate a flag in our ticketing system, letting us know that you have acknowledged receipt of this email.
It's possible that your environment has been compromised by an external attacker. It remains your responsibility to ensure that your instances and all applications are secured. This link provides some suggestions for securing your instances.
Case number: 14200733512-1
Additional abuse report information provided by original abuse reporter:
* Destination IPs:
* Destination Ports:
* Destination URLs:
* Abuse Time: Wed Oct 30 22:10:00 UTC 2013
* Log Extract:
We received a report from a confidential third party source of Denial of Service (DoS) attacks launched from the implicated instance listed above towards 22.214.171.124.
Please investigate the instance in question and respond detailing the corrective measures you'll be taking to address this activity.
In the mean time, we have blocked all ports except incoming TCP port 22 and 3389 from the instance to prevent further abuse.
If you believe that you were compromised by an external attacker, the best recourse is to back up your data, migrate your applications to a new instance, and terminate the old one. Attempting to repair a compromised instance does not guarantee a successful cleanup in most cases. We recommend reviewing the following resources to ensure your EC2 environment is properly secured:
Tips for Securing Your EC2 Instance:
Security Best Practices
AWS Security Center:
This instance had been started around 17:00 UTC on Monday. In less than two and half days an attacker had managed to find it, break in, and start using it as a largely untraceable tool to attack others. The cited "Abuse Time" was 22:10 UTC on Wednesday. Amazon's message went out at 00:06 PDT or 07:06 UTC Thursday.
Once the attackers know about this PHP vulnerability, they
follow a fairly simple path.
First, figure out which AMIs contain the vulnerability.
The AMI name could announce the presence, something
bitnami-with-php-5.3.9 or similar.
Or maybe someone needs to do a little low-cost experimenting
to see which AMIs contain it.
Second, devise an automated way of scanning the Amazon Web Services blocks of IP addresses looking for running instances containing this vulnerability.
Third, start breaking in and abusing those resources.
The very simple fix for this specific instance was simply to terminate it. The instance had already served its purpose.
That might seem a little unusual or drastic, but a lot of times a cloud instance is only needed for a short time. Why pay to run it when you don't really need it? Many cloud servers are ephemeral, running for a very limited time. Of course, this makes it even harder to notice vulnerabilities! At least it's also more difficult for the bad guys...
Nine days later, news came out from Amazon:
From: Amazon Web Services, Inc.
Sent: Saturday, November 09, 2013 12:12 AM
Subject: An Important Message from AWS Security
Dear AWS Customer,
Your security is important to us. Bitrock, the creator of the Bitnami AMIs published in the EC2 Public AMI catalog, has made us aware of a security issue in several of their AMIs. EC2 instances launched from these AMIs are at increased risk of access by unauthorized parties. Specifically, AMIs containing PHP versions 5.3.x before 5.3.12 and 5.4.x before 5.4.2 are vulnerable and susceptible to attacks via remote code execution. It appears you are running instances launched from some of the affected AMIs so we are making you aware of this security issue. This email will help you quickly and easily address this issue.
This security issue is described in detail at the following link, including information on how to correct the issue, how to detect signs of unauthorized access to an instance, and how to remove some types of malicious code:
Instance IDs associated with your account that were launched with the affected AMIs include:
Bitrock has provided updated AMIs to address this security issue which you can use to launch new EC2 instances. These updated AMIs can be found at the following link:
If you do not wish to continue using the affected instances you can terminate them and launch new instances with the updated AMIs.
Note that Bitnami has removed the insecure AMIs and you will no longer be able to launch them, so you must update any CloudFormation templates or Autoscaling groups that refer to the older insecure AMIs to use the updated AMIs instead.
If you wish to continue using the affected instances you should execute the following command on each instance to prevent unauthorized access:
sudo rm -f /opt/bitnami/apache2/cgi-bin/php-cgi /opt/bitnami/apache2/cgi-bin/php-cgi.bin
You can learn more about how to access your instance via SSH on the Bitnami wiki:
In addition, there are other actions you can take that will further increase the security of your instances:
1. Restrict inbound access to your instance to a specific IP range. This is especially important for administrative ports like TCP 22 and 3389:
2. If you are using password-based authentication, be sure to rotate any existing passwords.
Additional assistance and documentation related to AWS security best practices can be found at:
We hope you find this information useful. Please don't hesitate to follow up with us if you have any questions or concerns.
I like a number of things about this.
Let's look at the initial report first. Before Amazon had found the underlying cause, they pointed out that our instance was hacked without acting like the Internet Police. They still use the word "please" multiple times in this message, good for them!
They also made some changes, modifying the firewall rules to block all but SSH and RDP, the safe administrative control mechanisms for Linux and Windows, respectively.
They close by suggesting security references at various levels of detail.
The second message is even better. Now Bitrock has told Amazon what is going on. Amazon has told us how to find our instances still at risk from this vulnerability. They then explain how to work around the problem on an instance we want to keep running, and (more usefully) how to migrate an operation to a compatible instance based on a patched image.
Bitrock has removed the vulnerable images and told us how to find compatible replacements.
With Amazon's warning about CloudFormation and Autoscaling, we should be able to avoid an accidental denial of service inflicted by forgetting to modify our cloud architecture design.
Finally, we are given further suggestions on how to get guidance from both Bitnami and Amazon.
I think it's important to notice that the AMI source and the cloud provider communicated with us, spotting a problem and providing guidance, but they did not step in and clean up the mess for us.
Some people will argue that a cloud provider should do all that and more for us, but they are missing the point. The standard definition of "cloud computing" requires on-demand self-service, otherwise we won't have the rapid elasticity and measured service at very low cost.
Yes, you can get remotely hosted managed services run by a third party. It will cost a lot more, be far less flexible, and it won't really be "cloud computing" by the commonly accepted definition.
Back to the Security Page