Using an AWS ELB behind Varnish - is it possible?

Question

I'm trying to put a set of EC2 instances behind a couple of Varnish servers. Our Varnish configuration very seldom changes (once or twice a year) but we are always adding/removing/replacing web backends for all kinds of reasons (updates, problems, load spikes). This creates problems because we always have to update our Varnish configuration, which has led to mistakes and heartbreak.

What I would like to do is manage the set of backend servers simply by adding or removing them from an Elastic Load Balancer. I've tried specifying the ELB endpoint as a backend, but I get this error:

Message from VCC-compiler:
Backend host "XXXXXXXXXXX-123456789.us-east-1.elb.amazonaws.com": resolves to multiple IPv4 addresses.
Only one address is allowed.
Please specify which exact address you want to use, we found these:
123.123.123.1
63.123.23.2
31.13.67.3
('input' Line 2 Pos 17)
.host = "XXXXXXXXXXX-123456789.us-east-1.elb.amazonaws.com";

The only consistent public interface ELB provides is its DNS name. The set of IP addresses this DNS name resolves to changes over time and with load.

In this case I would rather NOT specify one exact address - I would like to round-robin between whatever comes back from the DNS. Is this possible? Or could someone suggest another solution that would accomplish the same thing?

Thanks, Sam

Answer 1

You could use a NGINX web server to deal with the CNAME resolution problem:

User-> Varnish -> NGNIX -> ELB -> EC2 Instances
        (Cache Section)        (Application Section)

You have a configuration example in this post: http://blog.domenech.org/2013/09/using-varnish-proxy-cache-with-amazon-web-services-elastic-load-balancer-elb.html

Juan

Answer 2

I wouldn't recommend putting an ELB behind Varnish.

The problem lies on the fact that Varnish is resolving the name assigned to the ELB, and it’s caching the IP addresses until the VCL get’s reloaded. Because of the dynamic nature of the ELB, the IPs linked to the cname can change at any time, resulting in Varnish routing traffic to an IP which is not linked to the correct ELB anymore.

This is an interesting article you might like to read.

Answer 3

Yes, you can.

in your default.vcl put:

include "/etc/varnish/backends.vcl";

and set backend to:

set req.backend = default_director;

so, run this script to create backends.vcl:

#!/bin/bash
FILE_CURRENT_IPS='/tmp/elb_current_ips'
FILE_OLD_IPS='/tmp/elb_old_ips'
TMP_BACKEND_CONFIG='/tmp/tmp_backends.vcl'
BACKEND_CONFIG='/etc/varnish/backends.vcl'

ELB='XXXXXXXXXXXXXX.us-east-1.elb.amazonaws.com'
IPS=($(dig +short $ELB | sort))

if [ ! -f $FILE_OLD_IPS ]; then
    touch $FILE_OLD_IPS
fi

echo ${IPS[@]} > $FILE_CURRENT_IPS

DIFF=`diff $FILE_CURRENT_IPS $FILE_OLD_IPS | wc -l`

cat /dev/null > $TMP_BACKEND_CONFIG

if [ $DIFF -gt 0 ]; then

    COUNT=0

    for i in ${IPS[@]}; do
        let COUNT++
        IP=$i
        cat <<EOF >> $TMP_BACKEND_CONFIG
backend app_$COUNT {
    .host = "$IP";
    .port = "80";
    .connect_timeout = 10s;
    .first_byte_timeout = 35s;
    .between_bytes_timeout = 5s;
}

EOF
    done

    COUNT=0

    echo 'director default_director round-robin {' >> $TMP_BACKEND_CONFIG

    for i in ${IPS[@]}; do
        let COUNT++
        cat <<EOF >> $TMP_BACKEND_CONFIG
    { .backend = app_$COUNT; }
EOF
    done

    echo '}' >> $TMP_BACKEND_CONFIG

    echo 'NEW BACKENDS'
    mv -f $TMP_BACKEND_CONFIG $BACKEND_CONFIG

fi

mv $FILE_CURRENT_IPS $FILE_OLD_IPS

Answer 4

I wrote this script to have a way to auto update the vcl once a new instance comes up or down.

it requires that the .vcl has an include to backend.vcl

This script is just a part of the solution, the tasks should be: 1. get new servername and IP (auto scale) can use AWS API cmds to do that, also via bash 2. update vcl (this script) 3. reload varnish

The script is here http://felipeferreira.net/?p=1358

Other pepole did it in different ways http://blog.cloudreach.co.uk/2013/01/varnish-and-autoscaling-love-story.html

Answer 5

You don get to 10K petitions if had to resolve an ip on each one. Varnish resolve ips on start and do not refresh it unless its restarted o reloaded. Indeed varnish refuses to start if found two ip for a dns name in a backend definition, like the ip returned for multi-az ELBs.

So we solved a simmilar issue placing varnish in front of nginx. Nginx can define an ELB as a backend so Varnish backend is a local nginx an nginx backend is the ELB.

But I don't feel comfy with this solution.

Answer 6

You Could make the ELB in your private VPC so that it would have a local ip. This way you don't have to use any DNS kind of Cnames or anything which Varnish does not support as easily.

Answer 7

Using internal ELB does not help the problem, because it usually have 2 Internal IP's!

Backend host "internal-XXX.us-east-1.elb.amazonaws.com": resolves to multiple IPv4 addresses. Only one address is allowed. Please specify which exact address you want to use, we found these: 10.30.10.134 10.30.10.46 ('input' Line 13 Pos 12)

What I am not sure is if this IPs will remain always the same or they can change? anyone?

Answer 8

I my previous answer (more than three years ago) I hadn't solve this issue, my [nginx - varnish - nxinx ] -> ELB solution worked until ELB changes IPs

But from some time ago we are using the same setup but with nginx compiled with jdomain plugin

So the idea is to place a nginx in the same host that varnish an there configure the upstream like this:

resolver 10.0.0.2;  ## IP for the aws resolver on the subnet

upstream backend {                                                                              
    jdomain internal-elb-dns-name port=80;
}

that upstream will automatically reconfigure the upstream ips the IP if the ELB changes its addresses

It might not be a solution using varnish but it works as expected