Ubuntu on Amazon ec2 from scratch

Update: I’ve now rolled this blog post into the Ubuntu wiki’s: EC2 Starters Guide page. Hopefully this helps out the Ubuntu community!

The informatics team here at MalariaGEN have been working with ec2 since before I joined them. So naturally, it’s one technology with which I’ve had to come to grips in the course of doing my job. For me, EC2 had a fairly steep learning curve, and after spending a while trying to learn it through doing, I decided that I would just have to spend some time getting properly to grips with how things worked. As part of that I decided to document it in a way that I’d not yet seen on the web: logically, comprehensively, explaining all the strange concepts and quirks that were clouding my understanding and stopping me from getting my job done efficiently.

We’re an Ubuntu shop here, or at least we’re moving to becoming one (the work I’m doing to move us off our ageing RHEL4 servers will be a topic for a future post), so this story will be from an Ubuntu point of view.

So first of all, we need to join up to the Amazon EC2 service. You can do this using your existing Amazon account, or create a completely new Amazon account to attach your EC2 subscription to.

So, we go to https://aws.amazon.com/ and sign up, putting in our credit card details and so forth. Eventually, we will end up at the AWS Management Console which is where we will first set up our credentials. But therein lay my first problem: I didn’t understand the various different types of credentials.

EC2 Credentials

In EC2, there are multiple different kinds of credential, Amazon uses slightly non-standard nomenclature, and it’s not always clear which credential is required for a given application.

1. Signon credentials: These are the email address/password pair that you use when you sign up. You use these to sign on to the AWS Management Console, and can be considered the “master” credentials as they allow you to regenerate all other types of credentials.

2. Access Credentials: There are three types: access keys, X.509 certificates and key pairs. The first and second type allow you to connect to the Amazon APIs. Which type of credential depends on which API and tool you are using. Some APIs and tools support both options, whereas others support just one. The third type is SSH public/private key pairs that are used for initial logins to newly created instances.

1. access keys: Symmetric key encryption. These are for making requests to AWS product REST or Query APIs. Can be obtained/regenerated from the Access Keys tab on the AWS Security Credentials page.
2. X.509 certificates: Public key encryption. Use X.509 certificates to make secure SOAP protocol requests to AWS service APIs. These are the credentials you will use when using the command-line ec2 api tools. Can be obtained/regenerated from the X.509 Certificates tab on the AWS Security Credentials page.
3. key pairs: SSH key pairs. When you create an instance, Amazon inserts the public key of your SSH key pair into your new instance so that you can log in using your private key. You can add new SSH key pairs through the AWS Management Console by clicking on Key Pairs under Networking and Security in the Navigation pane and then the Create Key Pair button. After specifying a name you will be prompted to download and save your private key. EC2 stores the public portion of your key pair, and inserts it into /home/ubuntu/.ssh/authorized_keys when you create your instance. If you lose this private key, it cannot be downloaded again; you will need to regenerate a new key pair.

Now that we understand credentials, we can get them all set up. First, lets set up our SSH key pair:
Go to the AWS Management Console, Networking and Security in the Navigation pane and then click the Create Key Pairbutton (and save your SSH private key in e.g. ~/.ec2/ec2.pem). The private key lives on your computer, while the public key lives in your AWS account.

You will also need to set up your Amazon API credentials. Go to Account->Security Credentials

1. click X.509 Certificates tab
2. Create a new certificate
3. Download the private key and the certificate (save them in e.g. ~/.ec2/cert-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem and ~/.ec2/pk-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem).
4. Make your credential files private: chmod go-rwx ~/.ec2/*.pem
5. Scroll to the bottom of the page and note your account ID (a number of the form XXXX-XXXX-XXXX).

Setting up the command line tools

OK, we now have our X.509 certificate and private key for accessing the ec2-api calls (we’re going to use the linux command line tools, but you can use the AWS console to perform a subset of these tasks, and there are GUI applications that can be used too).

Make sure you have multiverse enabled and run the following command:

sudo apt-get install ec2-api-tools

Next, we set up some variables in our shell environment so that we don’t always have to specify them as options to the command line tools:

export EC2_URL=https://ec2.eu-west-1.amazonaws.com
export EC2_KEYPAIR_EU_WEST_1=rwh-ec2 # ec2 SSH private key (name only, not the file)
export EC2_PRIVATE_KEY=$HOME/.ec2/rwh-ec2/pk-UZGAF4E4GLDA7DIDGUVWFIAKJTPK7MTX.pem # ec2 X.509 private key
export EC2_CERT=$HOME/.ec2/rwh-ec2/cert-UZGAF4E4GLDA7DIDGUVWFIAKJTPK7MTX.pem # ec2 X.509 certificate
export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/

test that the environment is set up correctly:

ec2-describe-images -o self -o amazon

EC2 security groups

Security groups allow you to specify firewalling rules for your instances. These firewalling rules are independent of, and in addition to, the software firewalling provided by the instance’s operating system (iptables in the case of modern Ubuntu systems). Security groups must be defined before you create the instances that you would like to be members of those security groups. You specify the security groups to add an instance to at creation time with the -g option to the ec2-run-instances command. You cannot add an existing instance to a security group.

How you set up your security groups is up to you. You may choose to set up security groups that correspond to server functions, or have a separate security group for each instance. An instance may be a member of multiple security groups. If you don’t specify any security groups when you instantiate an instance, it will be added to the default security group. Our examples use the default security group, but keep in mind that this means that this causes an inability to set up firewalling rules in a granular fashion.

If you wish to create a more complex security group configuration, you can do so with these commands:

ec2-add-group  -d
ec2-delete-group
ec2-describe-group [ ...]

See the Using Security Groups section of the User Guide for Amazon EC2for more information.

Instantiating an image

The first thing to do is work out what kind of instance that you want to instantiate. See Amazon EC2 Instance Types or Instance Families and Types for descriptions of the available instance types, and Amazon EC2 Pricing for the current pricing of instances, data transfer and storage. Next, decide what release of Ubuntu you’d like, then look up the AMI code from the links in this table of official Ubuntu EC2 images:

I’ve chosen a t1.micro and the latest and greatest 11.04 for my example. So let’s do it! Let’s finally create an actual, running instance:

ec2-run-instances ami-4290a636 --instance-type t1.micro --region eu-west-1 --key ${EC2_KEYPAIR_EU_WEST_1}

RESERVATION r-e61ead90 153873855856 default
INSTANCE i-3c75064a ami-4290a636 pending rwh-ec2 0 t1.micro 2011-06-16T14:59:24+0000 eu-west-1c aki-4feec43b monitoring-disabled ebs paravirtual

Now we can find out a little bit about our new virtual machine:

ec2-describe-instances

RESERVATION r-e61ead90 153873855856 default
INSTANCE i-3c75064a ami-4290a636 ec2-46-137-45-101.eu-west-1.compute.amazonaws.com ip-10-227-167-14.eu-west-1.compute.internal running rwh-ec2 0 t1.micro 2011-06-16T14:59:24+0000 eu-west-1c aki-4feec43b monitoring-disabled 46.137.45.101 10.227.167.14 ebs paravirtual
BLOCKDEVICE /dev/sda1 vol-1be88672 2011-06-16T14:59:44.000Z

Now we need to open up the SSH port so that we can connect to the machine:

ec2-authorize default -p 22

GROUP default
PERMISSION default ALLOWS tcp 22 22 FROM CIDR 0.0.0.0/0

Now we should be able to see it in the aws web console, and SSH to it:

ssh -i ~/.ec2/rwh-ec2/rwh-ec2.pem ubuntu@ec2-56-237-35-121.eu-west-1.compute.amazonaws.com

The authenticity of host 'ec2-56-237-35-121.eu-west-1.compute.amazonaws.com (46.137.45.101)' can't be established.
RSA key fingerprint is 09:99:28:1c:5f:1a:89:f3:f9:6b:b4:83:88:b4:4c:23.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ec2-56-237-35-121.eu-west-1.compute.amazonaws.com,56.237.35.121' (RSA) to the list of known hosts.

And we’re in! Now we can begin to set up and use the instance just like any other Ubuntu server machine.

You will be billed as long the host is running, so you will probably want to shut it down when you’re done. Note that each partial instance-hour consumed will be billed as a full hour.

ec2-terminate-instances <instance_id>