coreos

How to setup a CoreOS cluster on EC2 with metrics, logging and continuous deployment

View the Project on GitHub danielnbarbosa/coreos

Intro

Docker is a big step forward in bringing sanity to application deployment. The next question is how will you manage all these containers? CoreOS is one answer. The best way to learn something is get your hands dirty and play with it. If you follow along here you will get more familiar with thinking about a world made only of containers and learn a few things along the way, like:

Summary

Here's a high level view of what we'll be doing:

Setup Docker (assuming OSX)

Setup AWS

ssh-keygen -t rsa -f id_rsa

wget https://s3.amazonaws.com/aws-cli/awscli-bundle.zip
unzip awscli-bundle.zip
sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws

# AWS creds
export AWS_ACCESS_KEY_ID=XXXXXXXXXXXXXX
export AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXXXXXXXXXXX
export AWS_DEFAULT_REGION=us-west-1
export AWS_DEFAULT_OUTPUT=text

Launch CloudFormation Stack

Using the template provided from CoreOS, modify the necessary parameters and fire up CloudFormation. If you already had an AWS account make sure it is Default-VPC and not EC2-Classic or the the CloudFormation template may not work.

The following parameters are passed to the template. Don’t forget to generate a new DiscoveryURL:

The template sets the following defaults. If you want something else, other templates are available from CoreOS:

aws cloudformation create-stack --stack-name CoreOS-alpha --template-url https://s3.amazonaws.com/coreos.com/dist/aws/coreos-alpha-hvm.template --parameters ParameterKey=DiscoveryURL,ParameterValue=https://discovery.etcd.io/0ffffffffffffffffffffffffffff,UsePreviousValue=true ParameterKey=InstanceType,ParameterValue=t2.small,UsePreviousValue=true ParameterKey=KeyPair,ParameterValue=daniel,UsePreviousValue=true

Watch the magic happen in your AWS web console. Once created, adjust ASG Max instances to 3 (to prevent ballooning resource use).

Verify connectivity

Now you should have 3 EC2 instances in an ASG, along with a SecurityGroup. Let's login to the hosts and explore the cluster a little.

aws ec2 describe-instances --filter Name=tag:Name,Values=CoreOS-alpha | grep ASSOCIATION | sort | uniq | awk '{print $NF}'
111.111.111.200
111.111.111.201
111.111.111.202

fleetctl communicates over SSH so using ‘-A' allows for forwarding ssh connections, useful for when running fleetctl on the hosts. etcd acts a global configuration database that you can use for all sorts of things. Try setting and getting some keys and watching them instantly appear on other nodes.

ssh -A core@111.111.111.200
fleetctl list-machines
etcdctl ls

ssh-add ~/.ssh/id_rsa
ssh-add -l

# Connect to remote CoreOS on EC2
export FLEETCTL_TUNNEL=111.111.111.200:22

.~/.bash_profile
fleetctl list-machines

Upload and start DataDog monitoring service

Datadog provides a container for setting up host monitoring. This is a container that runs on your CoreOS hosts. You should never need to install additional tools on the host itself. The container monitors resources on the host by inspecting the local /proc filesystem. Normally containers are not allowed to see outside their world, but if run in '--privileged' mode this can be overridden.

This is also a good time to learn about fleet unit files, which are based on systemd.

etcdctl set /ddapikey XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

sed -i -e "s/danielnbarbosa/YOUR_DOCKERHUB_ID/g" *
rm *-e

fleetctl submit dd-agent@.service
fleetctl start dd-agent@{1..3}.service

fleetctl list-units

Upload and start Loggly log service

Loggly also offers a container for centralized logging. This is just a simple rsyslogd service that fowards the logs on to loggly.

The "Docker Way" (TM) says that you should never install more than one service in a container. Containers are not VMs. They wrap processes. This means we won't be adding agents inside of the container for helping out with sending logs to a central server. Instead we will install a logging container that other containers will forward their logs to.

etcdctl set /logapikey XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

sed -i -e "s/danielnbarbosa/YOUR_DOCKERHUB_ID/g" *
rm *-e

fleetctl submit loggly-agent@.service
fleetctl start loggly-agent@{1..3}.service

fleetctl list-units

echo netcat:"Host test log" | ncat -u -w 1 127.0.0.1 49153

Create ELB

aws ec2 describe-security-groups | grep SECURITYGROUPS | grep default | awk '{print $6}'

aws elb create-load-balancer --load-balancer-name=lb-coreos-alpha --listeners Protocol=HTTP,LoadBalancerPort=80,InstanceProtocol=HTTP,InstancePort=80 --availability-zones us-west-1a us-west-1b --security-groups=sg-111aaa5c

aws ec2 describe-instances --filter Name=tag:Name,Values=CoreOS-alpha | grep INSTANCES | awk '{print $8}'
i-8e1111d1
i-8e1111d2
i-8e1111d3

aws elb register-instances-with-load-balancer --load-balancer-name lb-coreos-alpha --instances i-8e1111d1i-8e1111d2i-8e1111d3

Configure CoreOS cluster security groups

Upload and start noke.bit http service

Now for the good part, an actual website. Keep in mind the "Docker Way" (TM) is one service per container and that service runs in the foreground. That's why if you follow the Dockerfile you'll see that nginx is set to 'daemon off'. Also take note of how this container uses '--link' to link to the loggly logging container. Using environment variables passed from the loggly container, nginx configures it's logs to be sent via syslog to the loggly container.

sed -i -e "s/danielnbarbosa/YOUR_DOCKERHUB_ID/g" *
rm *-e

fleetctl submit noke-nginx@.service
fleetctl start noke-nginx@{1..3}.service

fleetctl list-units

Setup DNS

This part I did all through the AWS web console, but you could do it via the command line tools if you wanted. Also you'll have to pick your own domain name. Feel free to forego the whole .bit thing if you don't want to bother with registering a .bit domain.

Register your domain name

.bit domains are registered on the namecoin blockchain. Namecoin is a fork of bitcoin that uses the blockchain as a decentralized store of identity information, including the TLD .bit. This is pretty cool because it basically acts as a decentralized domain registration service. Unfortunately it's still a bit complicated to register these domains. The directions below are for doing it yourself, but there are also services that will register them on your behalf, though this kind of defeats the whole decentralization thing.

namecoind name_update d/noke '{"ns": ["ns-389.awsdns-48.com", "ns-776.awsdns-33.net", "ns-1125.awsdns-12.org", "ns-1695.awsdns-19.co.uk"]}'

Don’t forget to restart chrome in case the plugin has locally cached your domain. Alternatively you can just hack your local hosts file or resolv.conf.

Rebalance btrfs

btrfs is the default storage driver used by docker. You'll need to periodically rebalance btrfs on the CoreOS hosts to prevent them from running out of filesystem space. Note that 'df -h' will lie to you, use 'sudo btrfs fi show' instead.

sudo btrfs fi show; sudo btrfs balance start /; sudo btrfs fi show

Integrate Circle CI

CircleCI is the bees knees. Yes, that's what I said. It is a super slick continuous integration and deployment tool that also supports building and testing docker containers.

ssh-keygen -t rsa -f id_circleci

cat ~/.ssh/id_circleci.pub

on each host:

vi ~/.ssh/authorized_keys

    cat ~/.dockercfg
    DOCKER_EMAIL=XXXXXXXXXXXX
    DOCKER_AUTH=XXXXXXXXXXXXX

    cd .../docker-dd-agent
    vi deploy.sh
    vi Dockerfile

    git add .
    git commit -m “testing circleci"
    git push

    cd .../docker-noke-nginx
    vi rolling_deploy.sh
    vi Dockerfile

    git add .
    git commit -m “testing circleci"
    git push

Conclusion

I had a lot of fun playing around with all these technologies. Things are moving pretty fast right now in both the docker and the bitcoin ecosystems. If you have questions or comments on anything here or anything DevOps or Bitcoin related you can find me on twitter @danielnbarbosa.