UPDATE: This information is long out of date. With the release of the awscli toolset in pypi years ago, things have changed a bit. Also, Amazon is forcing you to VPC on new accounts (years ago now). I will not be updating this post, but maybe you can find something useful here.
Recently I’ve been building the underlying system platform for the development of our distributed application on AWS. We do a lot of clustering using Storm and Hadoop, which means that we sometimes spin up hundreds of instances that may only live for a few hours during a run. Getting metrics, logs and all of those ‘must-haves’ centralized has been part of this build-out. When working with large amounts of machines in short-lived clusters, it becomes a real pain in the ass to use the built-in DNS/naming mechanism/scheme that AWS provides by default. Everything starts to look the same inside of your reporting/metrics/monitoring tools when working with the arbitrary names given to the instances. Hence, this article.
If you are not aware, there are thousands of machine instances running in AWS and Amazon only has a limited number of IPv4 addresses in their block(s), so all of the virtual machines are provided with DHCP addresses via NAT inside Amazon’s network. This means that most of the time when you reboot an instance it will come up with a new IP address/hostname, so the setup described below not only provides us with an easy to remember hostname, it also dynamically updates our Route53 DNS with the new hostname provided by Amazon. One other reason you might use the information here is that Amazon limits the number of Elastic IPs that you receive with your account (5). You can request more, but with this setup and/or utilizing Elastic Load Balancing (ELB) you won’t need to.
We are also going to make sure our script we use to update our DNS information uses the Name tag of the instance for the hostname. This is a great way to set hostnames automatically with configuration management or your preferred tool for provisioning. I use salt-cloud to spin up instances and it automatically sets the Name tag to the name you provide when initiating your instance. Perhaps there will be a post here about salt-cloud in the future.
It should be noted that the info below is specific to CentOS instances. With a little massaging this can easily be adapted to your distribution of choice. Let’s get started.
First things first…Setup Route53 as your DNS Provider
If you are a very large organization, you might only want to delegate a subdomain to Route53. To do that, go here for direction to make this happen.
If you are wanting Route53 to handle everything for your domain, it’s super easy to set up as well. Go here for easy instructions on how to do this.
Setup IAM role and permissions for updating Route53
We will want to set up a new user and group in IAM that will have specific permissions to update our DNS and read our Name tag. Let’s create a group in IAM first. I called my group dns-admin, but feel free to name your group whatever you want as long as you can remember what the group is for. When you are creating the group, select “No Permissions” in the wizard when it asks about setting a policy. Once you have the group created you need to add two policies to it. The first policy described here is to give any user within the dns-admin group permission to read the tags associated with an instance. Remember, we are going to use the Name tag of our instance to set our hostname. The second policy will be used to give permissions to update Route53 with the correct information.
So, select the group you just created and go to the “Permissions” tab. You will see a button to “Attach a Policy”. Click it and select “Custom Policy”. For the policy name, I used ‘describe-tags’ for my first policy. In the policy document area you will want to copy and paste the text from below:
{
"Statement":[
{
"Sid":"Stmt1358183399710",
"Action":[
"ec2:DescribeTags"
],
"Effect":"Allow",
"Resource":[
"*"
]
}
]
}
Repeat the above process, but this time name your policy ‘edit-dns’ and copy and paste the text from below in the policy document form field. NOTE: Make sure you change the text where it says YOUR_ZONE_ID with the zone id for the domain you are using in Route53. To get the ID, just go to the Route53 web console, select the domain zone and you will see the Hosted Zone ID number in the right side column.
{
"Statement":[
{
"Action":[
"route53:ChangeResourceRecordSets",
"route53:GetHostedZone",
"route53:ListResourceRecordSets"
],
"Effect":"Allow",
"Resource":[
"arn:aws:route53:::hostedzone/YOUR_ZONE_ID"
]
},
{
"Action":[
"route53:ListHostedZones"
],
"Effect":"Allow",
"Resource":[
"*"
]
}
]
}
Now, let’s create the user that we will add to this group. I named my user the same as my group, ‘dns-admin’. You will be asked to download the credentials/keys for this user as soon as you create the user. If you’ve created users before, you know that this is the ONLY time you get this information, so make sure you download it and keep it safe. We will need these keys in a minute.
Once you download your keys, add the user to the group we just created and we are done with IAM.
Fire up an instance
Once you have Route53 and your IAM user/group set up to use with your domain, fire up and instance using your base AMI. I’m sure you already have a base AMI that you configure with an init script or perhaps configuration management tools like Salt, Puppet, or Chef. Right?
After your instance comes up we need to log into the instance and grab the tools that we will utilize in our scripts.
Tools we need
- cli53 – https://github.com/barnybug/cli53
- ec2 API Tools – http://aws.amazon.com/developertools/351
- ec2-metadata – http://aws.amazon.com/code/1825
Install and configure cli53
cli53 is a great tool that interfaces easily with the Route53 API, making it easy to do updates. The easiest way to install cli53 is to just use ‘pip’. On CentOS machines, pip’s executable is ‘pip-python’. Other distributions just use the name ‘pip’. Run the following command to check if the ‘python-pip’ package is installed. If it’s not, the command will install it for you.
command -v pip-python > /dev/null 2>&1 || { yum install -y python-pip; }
Now, the command to install ‘cli53’ using pip.
pip-python install cli53
Now, let’s configure the cli53 tool with our AWS keys and other settings. We will create the the file ‘/etc/route53/config’ and enter the details there. Enter the following to create the file and set permissions correctly:
mkdir /etc/route53; chmod 700 /etc/route53; touch /etc/route53/config; chmod 600 /etc/route53/config
Paste the following in the ‘/etc/route52/config’ file. Make sure to replace the text within the quotes for each setting with your own keys, domain/subdomain, and TTL. I use a very short TTL because sometimes instances will be initiated and shortly thereafter rebooted. Use the keys that you downloaded earlier when we created the ‘dns-admin’ user in IAM.
# Set access and secret key of a user that #only has access to the following AWS objects/privileges: #"ec2:DescribeTags" #"route53:ChangeResourceRecordSets", #"route53:GetHostedZone", #"route53:ListResourceRecordSets" #"route53:ListHostedZones" AWS_ACCESS_KEY_ID="YOUR_ACCESS_KEY_ID_HERE" AWS_SECRET_ACCESS_KEY="YOUR_SECRET_ACCESS_KEY_HERE" ZONE="THE_NAME_OF_YOUR_DOMAIN_OR_SUBDOMAIN" TTL="30"
Install and configure ec2 API Tools
The ec2 API tools require Java. The application servers I use require Oracle’s Java, so I bake that into my base AMI. Therefore my $JAVA_HOME env variable is set accordingly. Just make sure that if you use the OpenJDK or JRE, that you point $JAVA_HOME to the correct place in the config below. So, install java however you like and then continue on.
Run the following commands to install the API tools to /opt/aws
mkdir -p /opt/aws wget -q http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip unzip -qq ec2-api-tools.zip rsync -a --no-o --no-g ec2-api-tools-*/ /opt/aws/
Make sure to add ‘/opt/aws’ to your PATH and set the following env variables. You can do so by editing or creating the file ‘/etc/profile.d/aws.sh’ and adding the following:
export EC2_HOME=/opt/aws for i in $EC2_HOME do PATH=$i/bin:$PATH done PATH=/opt/aws/:$PATH
Source the ‘/etc/profile.d/aws.sh’ file to make sure the PATH is added in your current session.
source /etc/profile.d/aws.sh
Also, as mentioned above, make sure the $JAVA_HOME is set. I do this by creating the file ‘/etc/profile.d/java.sh’ and adding the following for Oracle Java:
JAVA_HOME=/usr/java/default export JAVA_HOME
Install and configure ec2-metadata
The only thing we have to do to install ‘ec2-metadata’ is to download it from the link above, place it in ‘/opt/aws’, and make it executable. That’s it!
cd /opt/aws wget -q http://s3.amazonaws.com/ec2metadata/ec2-metadata chmod +x ec2-metadata
Create the script to update Route53 and set the hostname
Next on our list is to create the script we will use that will update Route53 and set our hostname based on the Name tag of our instance. Create the file /usr/sbin/update-dns-route53 and paste the script printed below into the file. Make sure to replace the YOUR_DOMAIN_HERE text with the domain you are using. Also, make the file executable after saving it.
#!/bin/sh #This script will get the Name tag of the instance from EC2 and apply it #both as a CNAME record #in Route53 for the specified domain below and update the hostname on the #machine and in the hosts file. # Make sure only root can run our script if [ "$(id -u)" != "0" ]; then echo "This script must be run as root" >& 2 exit 1 fi # Load configuration . /etc/route53/config.sh . /etc/profile.d/java.sh . /etc/profile.d/aws.sh # Export access key ID and secret for our tools export AWS_ACCESS_KEY_ID export AWS_SECRET_ACCESS_KEY # Replace this with your domain DOMAIN=YOUR_DOMAIN_HERE HOSTNAME=$(/opt/aws/bin/ec2-describe-tags -O $AWS_ACCESS_KEY_ID -W $AWS_SECRET_ACCESS_KEY \<br>--filter "resource-type=instance" \<br>--filter "resource-id=$(/opt/aws/ec2-metadata -i | cut -d ' ' -f2)" \<br>--filter "key=Name" | cut -f5) IPV4=/usr/bin/curl -s http://169.254.169.254/latest/meta-data/public-ipv4 # Set the host name<br>/bin/hostname $HOSTNAME.$DOMAIN echo $HOSTNAME.$DOMAIN > /etc/hostname # Set host name on Red Hat variants /bin/sed -i '/HOSTNAME/d' /etc/sysconfig/network echo HOSTNAME=$HOSTNAME.$DOMAIN >> /etc/sysconfig/network # Add fqdn to hosts file /bin/cat < /etc/hosts # This file is automatically genreated by /usr/sbin/update-dns-route53 script 127.0.0.1 localhost $IPV4 $HOSTNAME.$DOMAIN $HOSTNAME # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts EOF # Use command line scripts to get public hostname PUBLIC_HOSTNAME=$(/opt/aws/ec2-metadata | grep 'public-hostname:' | cut -d ' ' -f 2) # Create a new CNAME record on Route 53, replacing the old entry if nessesary /usr/bin/cli53 rrcreate "$ZONE" "$HOSTNAME" CNAME "$PUBLIC_HOSTNAME" --replace --ttl "$TTL"
Now that we have our script, we will want to run it at boot time every time the instance is started. To do that, let’s put the following in ‘/etc/rc.local’
/bin/bash /usr/sbin/update-dns-route53 > /tmp/updatedns 2>&1
Notice that we are redirecting the output to ‘/tmp/updatedns’. I’ve done this so that if there is a problem with the image not updating its name, you can look in this file for errors. Keep that in mind when getting this to work.
Create ‘delete-dns-route53’ script and initscript
So, we’ll want to delete these entries each time the machine shuts down, so we need a delete script, too. Create a new file ‘/usr/sbin/delete-dns-route53’ and paste the following into it. Also, make sure to make the file executable after saving it.
#!/bin/sh # This script will delete the hostname from Route53 on shutdown of the machine # Make sure only root can run our script if [ "$(id -u)" != "0" ]; then echo "This script must be run as root" 1>&2 exit 1 fi # Load configuration . /etc/route53/config . /etc/profile.d/java.sh . /etc/profile.d/aws.sh # Export access key ID and secret for our tools export AWS_ACCESS_KEY_ID export AWS_SECRET_ACCESS_KEY HOSTNAME=$(/opt/aws/bin/ec2-describe-tags -O $AWS_ACCESS_KEY_ID -W $AWS_SECRET_ACCESS_KEY \ --filter "resource-type=instance" \ --filter "resource-id=$(/opt/aws/ec2-metadata -i | cut -d ' ' -f2)" \ --filter "key=Name" | cut -f5) # Delete the hostname from DNS on shutdown /usr/bin/cli53 rrdelete "$ZONE" "$HOSTNAME"
We want to make sure this delete script runs every time the machine is powered down because every single time it’s powered down, could be its last and we don’t want our zone to become full of obsolete entries. Create the file ‘/etc/init.d/removednsfromroute53’ and add paste the following text in it:
#!/bin/bash
# chkconfig: 35 10 10
# description: Removed DNS entries from Route53
#
. /etc/init.d/functions
lockfile=/var/lock/subsys/removednsfromroute53
case "$1" in
start)
touch $lockfile
;;
stop)
/usr/sbin/delete-dns-route53 1> /tmp/deletedns 2>&1
rm -f $lockfile
;;
restart)
$0 stop
$0 start
;;
*)
echo "Usage: $0 {start|stop|restart|status}"
exit 1
;;
esac
exit 0
As you can see, we call the ‘delete-dns-route53’ script within the init script.
Now, add it to the correct runlevels by issuing the following command:
chkconfig --add removednsfromroute53
Well, there you have it…short and sweet, right? You might now want to create a new AMI based on this instance for your new base image. Moving forward if you spin up your instances with a Name tag, the name within the tag will be set as the hostname, the host’s file will be updated and a new CNAME will be created within Route53. Get DNS’ing…or something like that!
Dennis Moul
This was tremendously helpful, thanks very much! It’s a bit crazy that one has to go through so many hoops just to get a consistent hostname & DNS entry across stop/starts of instances, but so be it.
Two small bumps I ran into, I’ll mention in case anyone else runs into them:
My keypair name has spaces in it, which ec2-metadata choked on; I just took out the looping part (“for i in $x”) in the public key section of the ec2-metadta code since I only have one keypair anyway, and then it worked.
I found that cli53 was creating a name of the form host.domain.com.domain.com – looks like it needs the trailing period at the end of the FQDN the way zone files do. So I added this line before the cli53 call:
if [[ “$HOSTNAME” =~ [^.]$ ]]; then HOSTNAME=$HOSTNAME.; fi
in order to add the dot at the end if it’s not already there.
CJ
Would you use this for internal only instances (no EIP)? They all come out with ‘not available’ for the public hostname. I don’t care about using the internal DNS name that the VPC gives out, if it means my users get prettyname.domain.com internally for testing, etc.
Jason O'Connell
I’m wondering, instead of baking the registration / de-registrations into the AMI, can’t this too be done with Salt?
Jonathan
Note that you are likely need to specify –region in the ec2-describe-tags commands since “region us-east-1” is assumed if not indicated otherwise.
Fred
Hi,
It works fine using internal dns (Route 53).
You just have to change l56 like this :
PUBLIC_HOSTNAME=$(/opt/aws/ec2-metadata | grep ‘local-hostname:’ | cut -d ‘ ‘ -f 2)
(Use local-hostname instead of public-hostname.)
To use Route53 as internal DNS I followed this tutorial :
http://sysadminandnetworking.blogspot.fr/2013/04/set-up-private-internal-dns-for-your.html
tateeskew
I plan on updating this sometime soon as AWS released the official command line tools quite a while back and I was going to update the article to reflect that. So many things going on at the moment, though.
Dennis
Hmm, so you are having cooperation between DNS and DHCP happen from hosts/instances as they power up/down? Is this normal for a networking environment, or just because we don’t have control over the DNS/DHCP services for our VPCs? Seems like DHCP should update DNS directly, and when a lease is expired also update DNS automatically. Not sure if my current understanding is up to this yet.
Dennis
I looked around, this seems the only way to update DNS from DHCP in Amazon. So good idea putting this together. It does have one security flaw though. If a machine gets hacked, and the hacker finds the ‘dns-editor’ credentials on it, your network would be completely comprimised.
tateeskew
Hi Dennis…
Thanks for posting. As mentioned at the top of the article, a lot has changed since I wrote this post. This was before the days of the awscli ( http://aws.amazon.com/cli/ ) and also before IAM roles ( http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html ) that hand out temporary credentials to resources based on calls to other AWS services. These days I would write a short python script that used the Python SDK ( http://aws.amazon.com/sdk-for-python/ ) to ask for credentials at run time and make the call to Route53.
With that said, most of the stuff I’m working on has moved to tag based manipulation as they are much more flexible when dealing with instances that are elastic in nature. There are a handful of instances in the infrastructure I manage that still have this type of updating of DNS (mostly core shared services like configuration management server, internal DNS relay, etc), but for the most part I’m not too awfully concerned with DNS on any of the instances in my auto-scaling groups. If auto-scaling, along with other AWS services, aren’t core to your design in AWS, then it’s probably a good idea to use another service, because nearly anything else would be much cheaper.
Dennis Gearon
@tateeskew:disqus so lets say that you had a set of elasticsearch or couchdb instances that had to know how to get to each other. How would you do it using tags? I was planning on a scheme like esearch1.mydomain.com, esearch2.mydomain.com (all of these in a private zone)
galaxy
What I really don’t like with all these HOWTOs is that they demand that you run the scripts as root when there is no real need for that. Communicating with AWS endpoints is performed over the network, so there is no justification for running the update scripts as root. Just switch to a non-privileged account and do your stuff and if for some reason the 3rd party code you imported to your machine to do the task is compromised the damage will be limited to that non-privileged account (+ the corresponding DNS zone) and not the whole system.
galaxy
Well, one may argue that root privileges are needed to update /etc/hosts, but it’s not really so: just do “chgrp aws /etc/hosts && chmod g+w /etc/hosts” and as long as you do in-place modifications (e.g. “echo ‘127.0.0.1 something’ > /etc/hosts”) under the user who is a member of the aws group you will be fine.
So, the whole article is good, but should be prepended with the usage of the non-privileged account (e.g. installing awssdk and cli53 using virtualenv) and using that non-privileged account to do the job. Then it could be also labelled as a security-conscious way of doing this stuff and would make IT security people happy 🙂
Mikael Arhelger
It seems AWS Route 53 changed somewhat since 2013 (i.e. policy management). Where can one find an updated script, please? I have been looking but cannot find. We plan to have various Raspberry Pi in the field that are optionally accessible remotely. However, their public IP changes and we want to use Router 53 to update their IP automatically. Any help appreciated.
Dennis Gearon
I’m going to implement some of this VERY shortly. I’m actually going to reverse this – I will have the instance go to some webserver addresses on a private webserver in the VPC. Just accessing the URL will do the job with the desired hostname will get it assigned to the IP the request comes from with some other checking of course. Shutting down will be the same
Dennis Moul
cli53 is no longer maintained in python, now you download an executable from github: https://github.com/barnybug/cli53/#installation
Also the command execution format has changed, to do a CNAME it’s now:
cli53 rrcreate –replace “$ZONEID” “$HOSTNAME $TTL CNAME $PUBLIC_HOSTNAME”