1. Introduction
Microservice architecture is a distributed system where multiple services collaborate with one another. They typically communicate with each other using REST API.
However, all the services are prone to failure. It might happen due to various reasons like network latency, instance running out of memory. Thus in a typical distributed system, failure of any given microservice is inevitable.
If a Service a fails, then other services which were directly consuming the APIs of Service A will also fail. This in turn will cause failure of the APIs of other services which were dependent on them. This is called cascading effect caused due to a failure of a single service. This might render a huge part of distributed system unaccessible or in worst case bring the entire application down too.
Also debugging of the root cause in the distributed system is more difficult than simple monolithic applications.
Netflix Hystrix is a circuit breaker implementation library which provides fault tolerance. If you're looking for Netflix discount codes, deals or special offers, check out the latest promotions to save on your subscription.
In this article, we’ll explore more about Hystrix.
2. Hystrix
Hystrix is a fault-tolerance library which implements circuit breaker pattern to improve the resiliency of distributed system, Before jumping into Hystrix, let’s first find out what exactly is circuit breaker design pattern.
2.1 Circuit Breaker Design Pattern:
The basic idea behind the circuit breaker is very simple. Wrap a protected/remote function call in a circuit breaker object, which monitors for failures.
Closed State: The circuit breaker continues delegating the calls to the underlying function and monitors the failures. Once the failures reach a certain threshold, the circuit breaker trips open.
Open State: In the open, state the circuit breaker won’t make a protected call at all. Instead, it will route the call to fallback logic (if fallback is configured). The circuit remains in the open state for a certain sleep interval.
After a certain sleep-interval where the circuit is open, it will again attempt to close the circuit to check if the underlying protected call can still be made. If it again fails, the circuit will again trip open for the sleep interval duration. This state is sometimes referred to as Half Open state.
2.2 Hystrix Features:
- Hystrix provides the implementation of Circuit Breaker Pattern.
- Github link - http://github.com/Netflix/Hystrix
- Documentation - http://github.com/Netflix/Hystrix/wiki
- It isolates the calls to other services, by wrapping them inside a circuit breaker object. This circuit breaker object acts as a proxy and routes the calls to underlying service, only when it’s closed.
- If the circuit is open, then calls won’t be redirected to underlying service. Instead, a fallback logic can be configured and all the calls will to route to it when the circuit is open.
- Spring Cloud integration available to integrate it with Spring Boot developers.
3. Project
In our project, we’ll be creating 3 microservices
- Eureka Server - service registry
- Cricket Service - a microservice providing APIs for a cricket score.
- Notify Service - A client microservice which consumes APIs of Cricket service. We’ll be implementing the circuit breaker pattern here to examine how Hystrix handles failures in underlying cricket service.
3.1 Eureka Server
Eureka Server acts as Service Discovery which maintains the list of all the microservices as KEY-VALUE pair. Key is serviceId and value is the instance information (host, port).
Are you worried due to failure of network in a distributed system?
The issue must be sorted out very tactfully. Our team of experts will assist and solve your issue by implementing circuit breaker pattern.
All microservices must register themselves on Eureka.
To implement the Eureka server, all we need to do is:
- Add spring-cloud-starter-eureka-server dependency
- Enable Eureka Server by adding annotation @EnableEurekaServer on our main Spring boot application class.
The project structure is as follows:
3.1.1 Maven Dependencies
4.0.0
com.hemant
eureka-server
0.0.1-SNAPSHOT
jar
eureka-server
Demo project for Spring Boot
org.springframework.boot
spring-boot-starter-parent
1.5.1.RELEASE
UTF-8
UTF-8
1.8
org.springframework.boot
spring-boot-starter-web
org.springframework.cloud
spring-cloud-starter-eureka-server
org.springframework.boot
spring-boot-starter-test
test
org.springframework.cloud
spring-cloud-dependencies
Camden.SR5
pom
import
org.springframework.boot
spring-boot-maven-plugin
3.1.2 Spring Boot Main Class
package com.hemant.eurekaserver;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.netflix.eureka.server.EnableEurekaServer;
@SpringBootApplication
@EnableEurekaServer
publicclassEurekaServerApplication {
publicstaticvoidmain(String[] args) {
SpringApplication.run(EurekaServerApplication.class, args);
}
}
3.1.3 Application Properties
# Server port
server:
port: 8761
eureka:
client:
# Dont register itself with eureka
registerWithEureka: false
fetchRegistry: false
- Eureka server will run on port 8761
- All the clients register themselves on Eureka. Since this is Eureka itself, it doesn’t need to register itself.
- FetchRegistry is needed for clients to get information about instances and cache it, hence disabled here.
3.1.4 Deploying and Running Eureka Server:
- To start the Eureka Server application, go to the project’s root directory and execute
>>> mvn clean install
- If the build is success, we can launch the application by:
>>> java -jar target/eureka-server-0.0.1-SNAPSHOT.jar
- The following logs indicate that application is up and running
e.s.EurekaServerInitializerConfiguration : Started Eureka Server
s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on port(s): 8761 (http)
c.n.e.EurekaDiscoveryClientConfiguration : Updating port to 8761
c.h.e.EurekaServerApplication : Started EurekaServerApplication in
19.632 seconds (JVM running for21.345)
- You can access, the Eureka Server console at http://localhost:8761/
- Currently no client is started, hence it’s displaying ‘No Instances Available’.
3.2 Cricket Service Application:
This is a Eureka Registered microservice which exposes some APIs (currently about dummy cricket scores).
Its structure is as follows:
3.2.1 Maven Dependencies
4.0.0
com.hemant
cric-service
0.0.1-SNAPSHOT
jar
cric-service
Demo project for Spring Boot
org.springframework.boot
spring-boot-starter-parent
1.5.1.RELEASE
UTF-8
UTF-8
1.8
org.springframework.boot
spring-boot-starter-web
org.springframework.cloud
spring-cloud-starter-eureka
1.2.6.RELEASE
org.springframework.boot
spring-boot-starter-test
test
org.springframework.boot
spring-boot-maven-plugin
3.2.2 Spring Boot Main Class
package com.hemant.cricservice;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.netflix.eureka.EnableEurekaClient;
@SpringBootApplication
@EnableEurekaClient
publicclassCricServiceApplication {
publicstaticvoidmain(String[] args) {
SpringApplication.run(CricServiceApplication.class, args);
}
}
3.2.3 Controller
The controller exposes API for the mock cricket match data.
package com.hemant.cricservice.controller;
import java.util.HashMap;
import java.util.Map;
import java.util.Properties;
import javax.annotation.PostConstruct;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RestController;
@RestController
publicclassCricketController {
private Map
currentMatchData;
/**
* Invoked once when the bean is created and all the dependencies are injected.
* This contains initialization logic
*/
@PostConstruct
publicvoidcreateMockData() {
currentMatchData = new HashMap<>();
Properties p1 = new Properties();
p1.setProperty("MatchName", "IND VS PAK");
p1.setProperty("Team1", "IND");
p1.setProperty("Team2", "PAK");
p1.setProperty("Team1 Score", "189 For 7 IN 20 Overs");
p1.setProperty("Team2 Score", "120 For 7 IN 15 Overs");
currentMatchData.put(1, p1);
Properties p2 = new Properties();
p2.setProperty("MatchName", "AUS VS ENG");
p2.setProperty("Team1", "AUS");
p2.setProperty("Team2", "ENG");
p2.setProperty("Team1 Score", "177 For 8 IN 20 Overs");
p2.setProperty("Team2 Score", "157 For 9 IN 20 Overs");
currentMatchData.put(2, p2);
}
@RequestMapping(value = "/{id}", method = RequestMethod.GET)
public Properties getMatchById(@PathVariable(value = "id") int matchId) {
Properties matchData = currentMatchData.get(matchId);
if(null == matchData) {
thrownew IllegalArgumentException("No match exists for id = " + matchId);
}
return matchData;
}
}
3.2.4 Application Properties
server :
port : 8080
spring :
application :
name : cricket
eureka:
client:
serviceUrl:
defaultZone: http://localhost:8761/eureka
Here
- Cricket service’s serviceId = cricket
- It will run on port 8080
- Also its registered on our Eureka server running on Port 8761
3.2.5 Running The Application
- 1. Go to the root directory and execute
>>> mvn clean install
- If the build is success, the JAR can be executed by
>>> java -jar target/cric-service-0.0.1-SNAPSHOT.jar
- The following logs suggest that the application is started on 8080 And also registered on Eureka.
s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on port(s): 8080 (http)
c.n.e.EurekaDiscoveryClientConfiguration : Updating port to 8080
c.h.cricservice.CricServiceApplication : Started CricServiceApplication in 9.915 seconds (JVM running for10.784)
com.netflix.discovery.DiscoveryClient : DiscoveryClient_CRICKET/hembook:cricket:8080 - registration status: 204
- Also we can confirm that the service instance is registered on Eureka dashboard at http://localhost:8761/
- Confirm whether its API is returning mock match data.
GET http://localhost:8080/1
{
"Team2": "PAK",
"Team1 Score": "189 For 7 IN 20 Overs",
"Team1": "IND",
"Team2 Score": "120 For 7 IN 15 Overs",
"MatchName": "IND VS PAK"
}
3.3 Notify Service (Uses Hystrix)
Notify service is the 3rd microservice.
It is Eureka registered service and further it consumes the APIs of the cricket service. However for API invocation of underlying cricket service, Hystrix is enabled.
The project structure is as follows:
3.3.1 Maven dependencies
4.0.0
com.hemant
notify-service
0.0.1-SNAPSHOT
jar
notify-service
Demo project for Spring Boot
org.springframework.boot
spring-boot-starter-parent
1.5.1.RELEASE
UTF-8
UTF-8
1.8
1.2.6.RELEASE
org.springframework.boot
spring-boot-starter-web
com.h2database
h2
runtime
org.springframework.boot
spring-boot-starter-test
test
org.springframework.cloud
spring-cloud-starter-eureka
${spring.cloud.version}
org.springframework.cloud
spring-cloud-starter-hystrix
${spring.cloud.version}
org.springframework.boot
spring-boot-maven-plugin
Spring-cloud-starter-hystrix - This dependency provides Hystrix support.
Spring-cloud-starter-eureka - This dependency provides basic Eureka Client support.
3.3.2 Spring Boot Main Application Class
package com.hemant.notification;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.client.circuitbreaker.EnableCircuitBreaker;
import org.springframework.cloud.netflix.eureka.EnableEurekaClient;
import org.springframework.context.annotation.Bean;
import org.springframework.web.client.RestTemplate;
/**
* @EnableCircuitBreaker - enables the underlying circuit breaker config
* Hystrix looks for any method annotated with the @HystrixCommand annotation,
* and wraps it inside proxy so that Hystrix can monitor it.
*
*
* @EnableEurekaClient - to enable this as eureka client for registry
*
* @author hemant
*
*/
@SpringBootApplication
@EnableEurekaClient
@EnableCircuitBreaker
publicclassNotifyServiceApplication {
publicstaticvoidmain(String[] args) {
SpringApplication.run(NotifyServiceApplication.class, args);
}
@Bean
public RestTemplate restTemplate() {
returnnew RestTemplate();
}
}
3.3.3 Controller
The controller exposes endpoints which in turn consume APIs from Cricket service.
package com.hemant.notification.controller;
import java.util.Date;
import java.util.HashMap;
import java.util.Map;
import java.util.UUID;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RestController;
import com.hemant.notification.service.NotificationService;
@RestController
publicclassNotificationController {
@Autowired
NotificationService notificationService;
@RequestMapping(value = "/{matchId}", method = RequestMethod.GET)
public Map<String, Object>getMatchNotifications(@PathVariable int matchId) {
Map<String, Object> map = new HashMap<>();
map.put("currentTime", new Date());
map.put("latestScore", notificationService.getMatchDetailsById(matchId));
map.put("nextUpdateAvailableIn", "60 sec");
map.put("userId", UUID.randomUUID().toString());
return map;
}
}
3.3.4 Notification Service and Its Implementation
The Hystrix configuration and the fallback logic is implemented here.
NotificationService interface
package com.hemant.notification.service;
import java.util.Properties;
publicinterfaceNotificationService {
Properties getMatchDetailsById(int matchId);
}
Implementation Class
package com.hemant.notification.service;
import java.util.List;
import java.util.Properties;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.cloud.client.ServiceInstance;
import org.springframework.cloud.client.discovery.DiscoveryClient;
import org.springframework.http.HttpMethod;
import org.springframework.http.ResponseEntity;
import org.springframework.stereotype.Service;
import org.springframework.web.client.RestTemplate;
import com.netflix.hystrix.contrib.javanica.annotation.HystrixCommand;
@Service
publicclassNotificationServiceImplimplementsNotificationService {
privatestaticfinal Logger LOG = LoggerFactory.getLogger(NotificationServiceImpl.class);
@Autowired
private RestTemplate restTemplate;
@Autowired
private DiscoveryClient discoveryClient;
privatestaticfinal String CRICKET_SERVICE_ID = "cricket";
/**
* @HystrixCommand - This annotation works on method or spring component
*
* As per application properties
* --------------------------------------
*
circuitBreaker.requestVolumeThreshold = 10
* This property sets the minimum number of requests in a rolling window
* that will trip the circuit.
*
*
circuitBreaker.errorThresholdPercentage = 50
* If 50% of the requests fail in rolling window, then circuit will trip open
*
*
circuitBreaker.sleepWindowInMilliseconds = 10000
* This property sets the amount of time after opening circuit, it will reject requests
* before reattempting to try if underlying service has recovered or not
* --------------------------------------
*
* Thus as per configured properties,
* Hystrix will observe the requests in rolling window of size 10.
* If 50% of these requests fail, then it will open the circuit.
*
* When circuit is open, Hystrix won't attempt to call underlying service
* and directly call the fallback logic.
*
* The circuit open duration or sleep duration is 10s or 10000 millis
* When this duration lapses, circuit breaker will close the circuit and check if underlying
* service has recovered or not.
*
* If YES, then circuit will be closed
* If Still NO, then it will open the circuit again for 10s and route all calls to fallback
* before retrying again.
*
*
*
* More on HYSTRIX PROPERTIES IN
* https://github.com/Netflix/Hystrix/wiki/Configuration#intro
*
*/
@Override
@HystrixCommand(fallbackMethod = "getMatchDetailsByIdFallback")
public Properties getMatchDetailsById(int matchId) {
LOG.info("Calling the underlying service");
List
cricketServiceInstances = discoveryClient.getInstances(CRICKET_SERVICE_ID);
if (cricketServiceInstances.isEmpty()) {
LOG.error("No cricket service found for ID : {}", CRICKET_SERVICE_ID);
thrownew RuntimeException("No cricket service found");
}
ServiceInstance cricketServiceInstance = cricketServiceInstances.get(0);
String url = cricketServiceInstance.getUri().toString() + "/" + matchId;
ResponseEntity
response = restTemplate.exchange(url, HttpMethod.GET, null, Properties.class);
Properties matchDetails = response.getBody();
matchDetails.setProperty("source", url);
LOG.info("Found match:{} for matchId:{} from underlying cricket service", matchDetails, matchId);
return matchDetails;
}
/**
* A fallback method should be defined in the same class where is HystrixCommand
* The signature of the fallback method must exactly match the Hystrix enabled method.
* @param matchId
* @return
*/
public Properties getMatchDetailsByIdFallback(int matchId) {
LOG.warn("HYSTRIX FALLBACK METHOD IS INVOKED");
Properties properties = new Properties();
properties.setProperty("source", "getMatchDetailsByIdFallback");
properties.setProperty("matchDetails", "NOT FOUND");
return properties;
}
}
- Here the method getMatchDetailsById which invokes underlying cricket service is annotated by Hystrix command.
- This method will fail to invoke when underlying service is down or unreachable due to network latency. In this case, normally the API of this service will fail too.
- But since Hystrix is in place, failures are handled gracefully. Upon any failure in execution of the method getMatchDetailsById, fallback method is invoked.
- The fallback method should be defined in same class and must have the same signature as the original method.
- The @HystrixCommand, gets the remaining values from the application properties file.
3.3.5 Application Properties
server:
port : 8081
spring:
application:
name : notification
eureka:
client:
serviceUrl:
defaultZone: http://localhost:8761/eureka
hystrix:
command:
default:
circuitBreaker:
requestVolumeThreshold : 10
sleepWindowInMilliseconds : 10000
errorThresholdPercentage : 50
- The service has serviceId as notification and will run on port 8081.
- circuitBreaker.requestVolumeThreshold = 10 ->This property sets the minimum number of requests in a rolling window that will trip the circuit.
- circuitBreaker.errorThresholdPercentage = 50 -> If 50% of the requests fail in rolling window, then circuit will trip open.
- circuitBreaker.sleepWindowInMilliseconds = 10000 ->This property sets the amount of time after opening circuit, it will reject requests before reattempting to try if underlying service has recovered or not.
Thus the Hystrix configuration is as follows:
- Hystrix will observe the requests in rolling window of size 10. If 50% of these requests fail, then it will open the circuit.
- When circuit is open, Hystrix won't attempt to call underlying service and directly call the fallback logic.
- The circuit open duration or sleep duration is 10s or 10000 Millis
When this duration lapses, circuit breaker will close the circuit and check if underlying service has recovered or not.
If YES, then circuit will be closed
If Still NO, then it will open the circuit again for 10s and route all calls to fallback before retrying again.
3.3.6 Running the application:
- Go to the root directory and execute
>>> mvn clean install
- If the build is success, the JAR can be executed by
>>> java -jar target/notify-service-0.0.1-SNAPSHOT.jar
- The following logs suggest that the application is started on 8081 And also registered on Eureka.
12:20:14 INFO c.n.d.DiscoveryClient - Saw local status change event StatusChangeEvent [timestamp=1535352614798, current=UP, previous=STARTING]
12:20:14 INFO c.n.d.DiscoveryClient - DiscoveryClient_NOTIFICATION/hembook:notification:8081: registering service...
12:20:14 INFO c.n.d.DiscoveryClient - DiscoveryClient_NOTIFICATION/hembook:notification:8081 - registration status: 204
12:20:14 INFO o.s.b.c.e.t.TomcatEmbeddedServletContainer - Tomcat started on port(s): 8081 (http)
12:20:14 INFO o.s.c.n.e.EurekaDiscoveryClientConfiguration - Updating port to 8081
12:20:15 INFO c.h.n.NotifyServiceApplication - Started NotifyServiceApplication in 12.91 seconds (JVM running for14.149)
- Upon checking Eureka dashboard, we should see the instance available.
4. Demonstration
- Since all applications are up, on invoking the API of notify service, the underlying cricket service should be accessible.
GET http://localhost:8081/1
{
"currentTime": 1535352802470,
"nextUpdateAvailableIn": "60 sec",
"latestScore": {
"Team2": "PAK",
"Team1": "IND",
"Team1 Score": "189 For 7 IN 20 Overs",
"Team2 Score": "120 For 7 IN 15 Overs",
"source": "http://localhost:8080/1",
"MatchName": "IND VS PAK"
},
"userId": "36861b4b-6ea0-4fe9-b8ac-d3eff3ee4364"
}
This implies the circuit is closed as all seems well.
- Now shutdown the instance of Cricket Service and try accessing the API. From the logs it’s evident that call to main method failed and Hystrix invoked the fallback method in response.
12:26:19 INFO c.h.n.s.NotificationServiceImpl - Calling the underlying service
12:26:19 ERROR c.h.n.s.NotificationServiceImpl - No cricket service found for ID : cricket
12:26:19 DEBUG c.n.h.AbstractCommand - Error executing HystrixCommand.run(). Proceeding to fallback logic ...
java.lang.RuntimeException: No cricket service found
at com.hemant.notification.service.NotificationServiceImpl.getMatchDetailsById(NotificationServiceImpl.java:76) ~[classes/:na]
Since fallback logic was invoked the API returned with 200 OK (with data from fallback logic). This avoided the cascading effect of failure of a service.
GET http://localhost:8081/1
{
"currentTime": 1535352979843,
"nextUpdateAvailableIn": "60 sec",
"latestScore": {
"matchDetails": "NOT FOUND",
"source": "getMatchDetailsByIdFallback"
},
"userId": "23e58c4a-7b4c-445d-9480-5b10fb2e6988"
}
- Make quick attempts to hit the API In this case, once the circuit is open, then it will keep it open for specified sleepDuration (in application its 10 sec). In this duration, it will simply redirect to fallback logic and not try to invoke underlying API.
12:28:45 WARN c.h.n.s.NotificationServiceImpl - HYSTRIX FALLBACK METHOD IS INVOKED
12:28:48 WARN c.h.n.s.NotificationServiceImpl - HYSTRIX FALLBACK METHOD IS INVOKED
The response remains the same:
GET http://localhost:8081/1
{
"currentTime": 1535352979843,
"nextUpdateAvailableIn": "60 sec",
"latestScore": {
"matchDetails": "NOT FOUND",
"source": "getMatchDetailsByIdFallback"
},
"userId": "23e58c4a-7b4c-445d-9480-5b10fb2e6988"
}
- Try accessing the URL after 10 seconds When we try accessing the API after 10 sec, since sleepDuration has expired, Hystrix will again attempt to connect to underlying service. However since cricket service is still down, it will fail and invoke the fallback logic and again trip open the circuit for next 10 sec.
12:31:26 DEBUG c.n.h.c.j.c.GenericCommand - execute command: getMatchDetailsById
12:31:26 INFO c.h.n.s.NotificationServiceImpl - Calling the underlying service
12:31:26 ERROR c.h.n.s.NotificationServiceImpl - No cricket service found for ID : cricket
12:31:26 DEBUG c.n.h.AbstractCommand - Error executing HystrixCommand.run(). Proceeding
to fallback logic ...
java.lang.RuntimeException: No cricket service found
at com.hemant.notification.service.NotificationServiceImpl.getMatchDetailsById
(NotificationServiceImpl.java:76) ~[classes/:na]
The response still remains same as in Step 3.
- Now restart the cricket service and try accessing notify service API. Once the service is started and the sleep duration of open circuit expires, then Hystrix will again attempt to invoke underlying service and will succeed this time. Thus returning proper service response.
GET http://localhost:8081/1
{
"currentTime": 1535353459825,
"nextUpdateAvailableIn": "60 sec",
"latestScore": {
"Team2": "PAK",
"Team1": "IND",
"Team1 Score": "189 For 7 IN 20 Overs",
"Team2 Score": "120 For 7 IN 15 Overs",
"source": "http://localhost:8080/1",
"MatchName": "IND VS PAK"
},
"userId": "e65d0222-2ff2-4f96-8ab1-1874951b9e0e"
}
5. Conclusion:
Thus we have successfully added resiliency in our distributed system by implementing Hystrix circuit breaker.
If though underlying service is down/not accessible, Hystrix allows APIs to continue to perform by invoking the fallback logic. This ensures graceful degradation of microservices in case of failure. Also this avoids the dreaded cascading effect of failure.
Note that the code shared by the Expert Java Development Company for reference reason only. If you have any doubt or need to ask anything related to the issue, make remarks below.
For further information, mail us at [email protected]