March 6, 2013 tate.eskew

Automating Dynamic DNS updating with AWS Instances and Route53

UPDATE: I will refactor this post soon. With the release of the awscli toolset in pypi, things have changed a bit. Also, Amazon is forcing you to VPC on new accounts, I’ll update accordingly when time makes itself available.

Recently I’ve been building the underlying system platform for development of our distributed application on AWS. We do a lot of clustering using Storm and Hadoop, which means that we sometimes spin up hundreds of instances that may only live for a few hours during a run. Getting metrics, logs and all of those ‘must haves’ centralized has been part of this build out. When working with large amounts of machines in short lived clusters, it becomes a real pain in the ass to use the built in DNS/naming mechanism/scheme that AWS provides by default. Everything starts to look the same inside of your reporting/metrics/monitoring tools when working with the arbitrary names given to the instances. Hence, this article.

If you are not aware, there are thousands of machine instances running in AWS and Amazon only has a limited number of IPv4 addresses in their block(s), so all of the virtual machines are provided with DHCP addresses via NAT inside Amazon’s network. This means that most of the time when you reboot an instance it will come up with a new IP address/hostname, so the setup described below not only provides us with an easy to remember hostname, it also dynamically updates our Route53 DNS with the new hostname provided by Amazon. One other reason you might use the information here is that Amazon limits the number of Elastic IPs that you receive with your account (5). You can request more, but with this setup and/or utilizing Elastic Load Balancing (ELB) you won’t need to.

We are also going to make sure our script we use to update our DNS information uses the Name tag of the instance for the hostname. This is a great way to set hostnames automatically with configuration management or your preferred tool for provisioning. I use salt-cloud to spin up instances and it automatically sets the Name tag to the name you provide when initiating your instance. Perhaps there will be a post here about salt-cloud in the future.

It should be noted that the info below is specific to CentOS instances. With a little massaging this can easily be adapted to your distribution of choice. Let’s get started.

First things first…Setup Route53 as your DNS Provider

If you are a very large organization, you might only want to delegate a subdomain to Route53. To do that, go here for direction to make this happen.

If you are wanting Route53 to handle everything for your domain, it’s super easy to setup as well. Go here for easy instructions on how to do this.

Setup IAM role and permissions for updating Route53

We will want to setup a new user and group in IAM that will have specific permissions to update our DNS and read our Name tag. Let’s create a group in IAM first. I called my group dns-admin, but feel free to name your group whatever you want as long as you can remember what the group is for. When you are creating the group, select “No Permissions” in the wizard when it asks about setting a policy. Once you have the group created you need to add two policies to it. The first policy described here is to give any user within the dns-admin group permission to read the tags associated with an instance. Remember, we are going to use the Name tag of our instance to set our hostname. The second policy will be used to give permissions to update Route53 with the correct information.

So, select the group you just created and go to the “Permissions” tab. You will see a button to “Attach a Policy”. Click it and select “Custom Policy”. For the policy name, I used ‘describe-tags’ for my first policy. In the policy document area you will want to copy and paste the text from below:

Repeat the above process, but this time name your policy ‘edit-dns’ and copy and paste the text from below in the policy document form field. NOTE: Make sure you change the text where it says YOUR_ZONE_ID with the zone id for the domain you are using in Route53. To get the ID, just go to the Route53 web console, select the domain zone and you will see the Hosted Zone ID number in the right side column.

Now, let’s create the user that we will add to this group. I named my user the same as my group, ‘dns-admin’. You will be asked to download the credentials/keys for this user as soon as you create the user. If you’ve created users before, you know that this is the ONLY time you get this information, so make sure you download it and keep it safe. We will need these keys in a minute.
Once you download your keys, add the user to the group we just created and we are done with IAM.

Fire up an instance

Once you have Route53 and your IAM user/group setup to use with your domain, fire up and instance using your base AMI. I’m sure you already have a base AMI that you configure with an init script or perhaps configuration management tools like Salt, Puppet or Chef. Right?

After your instance comes up we need to log into the instance and grab the tools that we will utilize in our scripts.

Tools we need

Install and configure cli53

cli53 is a great tool that interfaces easily with the Route53 API, making it easy to do updates. The easiest way to install cli53 is to just use ‘pip’. On CentOS machines, pip’s executable is ‘pip-python’. Other distributions just use the name ‘pip’. Run the following command to check if the ‘python-pip’ package is installed. If it’s not, the command will install it for you.

Now, the command to install ‘cli53’ using pip.

Now, let’s configure the cli53 tool with our AWS keys and other settings. We will create the the file ‘/etc/route53/config’ and enter the details there. Enter the following to create the file and set permissions correctly:

Paste the following in the ‘/etc/route52/config’ file. Make sure to replace the text within the quotes for each setting with your own keys, domain/subdomain and TTL. I use a very short TTL because sometimes instances will be initiated and shortly thereafter rebooted. Use the keys that you downloaded earlier when we created the ‘dns-admin’ user in IAM.

Install and configure ec2 API Tools

The ec2 API tools require Java. The application servers I use require Oracle’s Java, so I bake that into my base AMI. Therefore my $JAVA_HOME env variable is set accordingly. Just make sure that if you use the openjdk or jre, that you point $JAVA_HOME to the correct place in the config below. So, install java however you like and then continue on.

Run the following commands to install the API tools to /opt/aws

Make sure to add ‘/opt/aws’ to your PATH and set the following env variables. You can do so by editing or creating the file ‘/etc/profile.d/aws.sh’ and adding the following:

Source the ‘/etc/profile.d/aws.sh’ file to make sure the PATH is added in your current session.

Also, as mentioned above, make sure the $JAVA_HOME is set. I do this by creating the file ‘/etc/profile.d/java.sh’ and adding the following for Oracle Java:

Install and configure ec2-metadata

The only thing we have to do to install ‘ec2-metadata’ is to download it from the link above, place it in ‘/opt/aws’ and make it executable. That’s it!

Create the script to update Route53 and set hostname

Next on our list is to create the script we will use that will update Route53 and set our hostname based on the Name tag of our instance. Create the file /usr/sbin/update-dns-route53 and paste the script printed below into the file. Make sure to replace the YOUR_DOMAIN_HERE text with the domain you are using. Also, make the file executable after saving it.

Now that we have our script, we will want to run it at boot time every time the instance is started. To do that, let’s put the following in ‘/etc/rc.local’

Notice that we are redirecting the output to ‘/tmp/updatedns’. I’ve done this so that if there is a problem with the image not updating it’s name, you can look in this file for errors. Keep that in mind when getting this to work.

Create ‘delete-dns-route53’ script and initscript

So, we’ll want to delete these entries each time the machine shuts down, so we need a delete script, too. Create a new file ‘/usr/sbin/delete-dns-route53’ and paste the following into it. Also, make sure to make the file executable after saving it.

We want to make sure this delete script runs every time the machine is powered down, because every single time it’s powered down, could be it’s last and we don’t want our zone to become full of obsolete entries. Create the file ‘/etc/init.d/removednsfromroute53’ and add paste the following text in it:

As you can see, we call the ‘delete-dns-route53’ script within the init script.
Now, add it to the correct runlevels by issuing the following command:

Well, there you have it…short and sweet, right? You might now want to create a new AMI based on this instance for your new base image. Moving forward if you spin up your instances with a Name tag, the name within the tag will be set as the hostname, the hosts file will be updated and a new CNAME will be created within Route53. Get DNS’ing…or something like that!

Tagged: , , , , , ,

About the Author

tate.eskew Tate is the owner of the site you are looking at. He sometimes makes posts about interesting stuff and other times, not so much.