Misc

AWS CLI and JQ For Automation of Environments

So I’ve had a problem recently, it might be familiar to others.

I constantly need to provision a brand new environment and I always run into a snag. Basically, vagrant keeps state in a folder under .vagrant and for the vagrant cloud plugins this has the AWS instance id it THINKS it should provision to, if the folder doesn’t exist, then a new instance is created.

The problem comes in when I might want to shutdown an instance in the ec2 console, the ec2 instance itself blows up, or maybe I want to provision a new environment. Any of these actions causes the jenkins state to go out of sync with the real world, because the instance ID in the .vagrant folder no longer matches reality – and when this happens we lose all ability to provision or re-provision.

I’ve been solving this by wiping out my workspace every time this happens to provision a brand new environment, it’s not that bad, except we have clusters of machines that require this wipe-out and re-do. Also we waste time pulling down for git stuff that hasn’t actually changed. And finally, worst of all it’s manual and that involves the whole tribal knowledge thing…. yuck..

Consider more complex environments when we end up running multiple jenkins instances, one jenkins might have the ./vagrant folder and another doesn’t. Or maybe the worst happens, and our jenkins box gets knocked out. Without this state it would cause us to bring up multiple instances (both expensive, and possibly leading to errors). So what to do?

With a little bash voodoo we can scan amazon for instances using amazon cli by looking for instances already in ‘pending’ or ‘active’ state! And then we can then let vagrant know what really is the state of the world.

We can “find” our instances by issuing the below command using the excellent aws cli along with JQ.

aws ec2 describe-tags --filters Name=resource-type,Values=instance | \
         jq '.Tags[] | {Key,Value,ResourceId}' | \
         jq '. | select(.Key=="Name")' | \ 
         jq '. | select(.Value=="YOURNAMEHERE").ResourceId'

This little gem will return either null or the instance ID of the machine you are looking for (our instances are all uniquely named). So now we can use this to conditionally run either a vagrant up YOURNAMEHERE or vagrant provision YOURNAMEHERE depending on the result.

The trick to getting provisioning to work from scratch (lets say you configure jenkins to reset your git every build) is to create the correct file in .vagrant/machines/YOURNAMEHERE/aws/id when the above yields an instance id that is active.

NAME="YOUR_BASE_VAGRANT_NAME_HERE"

cd puppet
chmod +x *.sh
cd "boxes/$NAME"

for ID in $(seq 1 $NUMBER_OF_CAPACITY_NODES);
do
  NODE_ID=$(printf "%02d" $ID)
  VAGRANT_NAME="$NAME-$NODE_ID"
  EC2_TAG_NAME="$NAME-$NODE_ID-$ENV"
  echo "$EC2_TAG_NAME"

  rm -rf ".vagrant/machines/$NAME/aws" || true
  mkdir -p ".vagrant/machines/$NAME/aws"

  INSTANCE_ID=$(aws ec2 describe-instances --filters Name=instance-state-name,Values=running,pending | jq '.Reservations[]' | jq '.Instances[] | { InstanceId, Tags }' | jq 'select(has("Tags")) | select(.Tags[]["Key"] == "Name" and .Tags[]["Value"] == "'"$EC2_TAG_NAME"'") | .InstanceId')
  INSTANCE_ID=$(echo $INSTANCE_ID | sed "s/\"//g")
  echo $INSTANCE_ID

  if [ -n "$INSTANCE_ID" ]; then
    echo "=================== PROVISION ENVIRONMENT ======================="
    echo $INSTANCE_ID > ".vagrant/machines/$VAGRANT_NAME/aws/id"
    DEPLOYMENT_OUTPUT=`ENV=$ENV NODE=$NODE_ID vagrant provision $VAGRANT_NAME`
    test 0 -eq `echo $DEPLOYMENT_OUTPUT | grep "VM not created" | wc -l` -a 0 -eq $?
  else
    echo "=================== BRAND NEW ENVIRONMENT ======================="
    DEPLOYMENT_OUTPUT=`ENV=$ENV NODE=$NODE_ID vagrant up $VAGRANT_NAME --provider=aws`
    test 0 -eq `echo $DEPLOYMENT_OUTPUT | grep "VM not created" | wc -l` -a 0 -eq $?
  fi

done

Now we can provision/re-provision and also use the standard amazon control panel – so we can blow away instances in the ec2 console, and then on next jenkins push, it will detect no instances, and will automatically provision the new environment.

Important to note, that the above allows us to spindle up (N) number of vagrant instances. Useful, for example, when node 09 was terminated and when we run this the script will provision 01-08 and bring up a new 09.

Standard

Leave a Reply

Your email address will not be published. Required fields are marked *