A few months back I wrote about how we switched the Galaxy Zoo HAProxy load balancers to Amazon Web Services (AWS) Elastic Load Balancers (ELB). At that point we had basically just swapped out HAProxy (running on its own EC2 small instance) for an ELB but weren’t making any use of the auto-scaling features also on offer. For the past few days I’ve been playing around with auto-scaling our API layer with the ELB that’s already in place and this morning I pushed the changes into production.
» Getting started
As I mentioned earlier, we already had an ELB in place so we didn’t need to create a new one - instead we’re adding here auto-scaling to an ELB that’s already in place. For completeness however, this is the command used to create the original ELB:
>> elb-create-lb ApiLoadBalancer --zones us-east-1b --listener
"lb-port=80, instance-port=80, protocol=TCP" --listener "lb-port=443,
» De-register existing ELB instances
As we already had a couple of instances registered with the ELB I found the easiest way to get auto-scaling up and running was to remove the existing instances before proceeding:
>> elb-describe-instance-health ApiLoadBalancer
INSTANCE i-abcdefgh InService INSTANCE i-ijklmnop InService
elb-deregister-instances-from-lb -lb ApiLoadBalancer --instances
No Instances currently registered to
» Create a launch configuration
Before you can introduce auto-scaling you need to have a couple of things in place - an Amazon Machine Image (AMI) that upon boot is immediately ready to serve your application and a launch configuration compatible with your currently ELB-scaled nodes (security groups etc.). Depending upon your setup, always having an AMI ready to launch with the latest version of your production codebase is probably the hardest thing to achieve here. Once you have your AMI in place and your security group and key-pair settings to hand you’re ready to create your launch configuration:
>> as-create-launch-config ApiLaunchConfig --image-id ami-myamiid
--instance-type m1.small --key ssh_keypair --group "elb security group
OK-Created launch config
» Create an auto-scaling group
Once you have a launch configuration in place it’s time to create an auto-scaling group. Auto-scaling groups need as a minimum to know what launch configuration, which load-balancer to use, which availability zone and the minimum and maximum to scale to. We never run the Galaxy Zoo API on anything less than 2 nodes and so to create our auto-scaling group I issued a command something like this:
>> as-create-auto-scaling-group ApiScalingGroup --launch-configuration
ApiLaunchConfig --availability-zones us-east-1b --min-size 2
--max-size 6 --load-balancers ApiLoadBalancer
At this point it’s worth noting that although we’d removed all of the instances being load balanced by the ApiLoadBalancer ELB, because the auto-scaling group set a minimum number of instances of 2 checking the status of the auto-scaling group showed that 2 new instances were spinning up:
>> as-describe-scaling-activities ApiScalingGroup
ACTIVITY 78bf4e0d-f72b-4b5b-a044-6b99942088ed 2009-08-24T07:19:28Z
Successful "At 2009-08-24 07:16:12Z a user request created an
AutoScalingGroup changing the desired capacity from 0 to 2.
At 2009-08-24 07:17:17Z an instance was started in response to a
difference between desired and actual capacity, increasing the capacity
from 0 to 2."
I don’t know about you but I think that’s pretty AWESOME!
» Create some launch triggers
To complete the auto-scaling configuration, you need to define the rules that increase and decrease the number of load-balanced instances. Currently we have a very simple rule based upon CPU load - if the average CPU load over the past 120 seconds is greater than 60% we introduce a new instance, if the CPU average drops below 20% then we remove an instance:
>> as-create-or-update-trigger ApiCPUTrigger --auto-scaling-group
ApiScalingGroup --namespace "AWS/EC2" --measure CPUUtilization
--statistic Average --dimensions "AutoScalingGroupName=ApiScalingGroup"
--period 60 --lower-threshold 20 --upper-threshold 60
--lower-breach-increment=-1 --upper-breach-increment 1
These triggers will almost certainly require refinement but helpfully
as-create-or-update-trigger command will create a new trigger if
one doesn’t exist or update an existing trigger based upon the new
» That’s it!
Once again it’s been a breeze to introduce the latest AWS features into our production stack. Moving Galaxy Zoo to AWS has completely changed the way we think about running our web applications - we’ve gone from having a group of ‘pet’ servers we each know the name of to having a fault-tollerant, auto-scaled web-stack ready for the future.