Spring Cloud with Zuul and Eureka cannot handle service instances going down

1
votes

I'm currently having an issue when using Netflix Eureka and Zuul in a Spring Cloud environment using Docker.

My current setup is as follows: 3 Docker containers:

  • a Eureka server
  • a Zuul server that uses Eureka to map routes to services and load balancing between multiple containers of the same service.
  • a service (currently just returning a simple string)

This setup works fine but troubles begin when I start playing with the scaling of the service. When I scale up, Zuul picks up the new servers after 30 seconds. That's fine by me. However, when I take down service containers, calls to Zuul can fail (with a HTTP errorcode 200!) because Zuul still thinks the servers are in the alive pool.

I'm using Spring Boot 1.3.6 and Spring Cloud 1.1.2

My questions:

  • Can Zuul be configured to retry calls?
  • Is there a way to force Zuul to do a ping to all the servers using a REST call? As far as I can see, there is no Spring Cloud endpoint for this. Posting to /routes doesn't do what I want.
  • If that's not possible, can Zuul be configured to actually send something else than a 200 when a server is not responding in time?
1
springspring-cloudnetflix-zuulnetflix-eureka
Couple of questions... What exactly do you mean calls to Zuul fail with an error code 200? Can you provide an example response? Also as you bring instances up/down it will take a while for Zuul (or other Eureka clients) to know since the clients will need to refresh data from the Eureka server. Does Zuul eventually figure out the instances are no longer up after it refreshes the data from Eureka? - Ryan Baxter
When I take containers down, calls to Zuul which are routed to server that are now unreachable return an empty page with HTTP response code 200. So there's no way for the client to know that he might need to retry. I would have expected a 504 or 503. Zuul does indeed figure out after the ping interval (which is 30 seconds) that certain servers are not available anymore and removes them from the LB. But during those 30 seconds, calls are returning empty pages which doesn't make sense. - Lieven Doclo
On a side note, Eureka is aware when a server is removed from the registy, so I'm wondering whether there's a way to hook that event to Zuul so that if a server get unregistered, Zuul gets a signal that it needs to refresh its server list. It's not really a problem if new servers aren't immediately available in the LB, but it is a problem if you downscale the underlying services and it takes 30 seconds for the LB to catch up, especially if there's a system that is aware of it (Eureka). - Lieven Doclo
Could you please attach your Eureka server and Eureka client configuration properties? - bart.s

1 Answers

1
votes

you dont have to inform zuul about change in list of servers, as its done underneath. Real issue here is that eureka decides to keep those instances in its registry. There is a rule: if less than 85% services respond to healthcheck, then eureka thinks its due to network failure/partition and keep all services in registry. This is VERY likely to happen if you for example has only 3 instances of your service. As you can imagine, if one switch off(33,3% of all 3 instances) eureka will ignore it and still serve this instance to clients.

It can be nice feature if you have 100 servers :) Its netflix tool tweaked mostly for their needs.

You can try to switch selfPreservation off:

eureka.server.enable-self-preservation=false

or tweak threshold to some lower value:

eureka.server.renewal-percent-threshold=0.85

http://cloud.spring.io/spring-cloud-static/spring-cloud.html#_appendix_compendium_of_configuration_properties