|
Note
|
This repository contains the guide documentation source. To view the guide in published form, view it on the Open Liberty website. |
Explore how to manage the impact of failures by using MicroProfile and Istio Fault Tolerance to add retry and fallback behaviours to microservices.
You will learn how to combine MicroProfile Retry and Fallback policies with Istio Retry to make your microservices more resilient to common failures, such as network problems.
Microservices that are created using Eclipse MicroProfile can be freely deployed in a service mesh to reduce the complexity associated with managing microservices. Istio is a service mesh, meaning that it’s a platform for managing how microservices interact with each other and the outside world. Istio consists of a control plane and sidecars that are injected into application pods. The sidecars contain the Envoy proxy. You can think of Envoy as a sidecar that intercepts and controls all the HTTP and TCP traffic to and from your container. If you would like to learn more about Istio, check out the Managing microservice traffic using Istio guide.
MicroProfile and Istio both provide simple and flexible solutions to build fault-tolerant microservices. Fault tolerance provides different strategies for building robust behaviour to cope with unexpected failures. A few fault tolerance policies that MicroProfile can offer include Retry, Timeout, Circuit Breaker, Bulkhead, and Fallback. There is some overlap that exists between MicroProfile and Istio Fault Tolerance, such as the Retry policy. However, Istio does not offer any fallback capabilities. To view the available fault tolerance policies in MicroProfile and Istio, refer to the comparison between MicroProfile and Istio fault handling.
Use retry policies to fail quickly and recover from brief intermittent issues. An application might experience these transient failures when a microservice is undeployed, a database is overloaded by queries, the network connection becomes unstable, or the site host has a brief downtime. In these cases, rather than failing quickly on these transient failures, a retry policy provides another chance for the request to succeed. Simply retrying the request might be all you need to do to make it succeed.
Fallback offers an alternative execution path when an execution does not complete successfully.
You will use the @Fallback annotation from the MicroProfile Fault Tolerance specification to define criteria
for when to provide an alternative solution for a failed execution.
You will develop microservices that demonstrate MicroProfile Fault Tolerance with Istio fault handling. Both MicroProfile and Istio can be used when you want your microservices to have a service mesh architecture with Istio, and use MicroProfile to provide the extra fault tolerance policies that do not exist within Istio.
The application that you will be working with is an inventory service, which collects, stores, and returns the system properties.
It uses the system service to retrieve the system properties for a particular host.
You will add fault tolerance to the inventory service so that it reacts accordingly when the system service is unavailable.
Navigate to the guide-microprofile-istio-retry-fallback/start directory to begin.
The MicroProfile Fault Tolerance API is included in the MicroProfile dependency that is specified in your pom.xml file.
Look for the dependency with the microprofile artifact ID.
This dependency provides a library that allows you to use the fault tolerance policies in your microservices.
The InventoryResource.java file
makes a request to the system service through the MicroProfile Rest Client API.
If you want to learn more about MicroProfile Rest Client,
you can follow the Consuming RESTful services with template interfaces guide.
pom.xml
link:finish/inventory/pom.xml[role=include]InventoryResource.java
link:finish/inventory/src/main/java/io/openliberty/guides/inventory/InventoryResource.java[role=include]To simulate that your system service is temporarily down due to brief intermittent issues, you will pause the pod that is associated
with your system service, then try to send requests to the service. When the system pod is paused, requests to the service return a 503 status code,
and the systemClient.getProperties() in InventoryResource.java throws a WebApplicationException.
To retry the requests to your system service after a WebApplicationException has occurred, add the @Retry annotation.
Update theInventoryResource.javafile.inventory/src/main/java/io/openliberty/guides/inventory/InventoryResource.java
To retry the service request a maximum of 3 times, only when a WebApplicationException occurs,
add the @Retry annotation before the getPropertiesForHost() method.
InventoryResource.java
link:finish/inventory/src/main/java/io/openliberty/guides/inventory/InventoryResource.java[role=include]A request to a service might fail for many different reasons. The default Retry policy initiates a retry for every java.lang.Exception.
However, you can base a Retry policy on a specific exception by using the retryOn parameter. You can identify more than one exception
as an array of values. For example, @Retry(retryOn = {RuntimeException.class, TimeoutException.class}).
You can set limits on the number of retry attempts to avoid overloading a busy service with retry requests.
The @Retry annotation has the maxRetries parameter to limit the number of retry attempts. The default number for maxRetries is 3 requests.
The integer value must be greater than or equal to -1. A value of -1 indicates to continue retrying indefinitely.
Navigate to the guide-microprofile-istio-retry-fallback/start directory and run the following command to build your application and integrate the Retry policy into your microservices:
mvn packageNext, run the docker build commands to build container images for your application:
docker build -t system:1.0-SNAPSHOT system/.
docker build -t inventory:1.0-SNAPSHOT inventory/.The -t flag in the docker build command allows the Docker image to be labeled (tagged) in the name[:tag] format.
The tag for an image describes the specific image version. If the optional [:tag] tag is not specified, the latest tag is created by default.
To verify that the images for your system and inventory microservices are built, run the docker images command to list all local Docker images.
docker imagesYour two images system and inventory should appear in the list of all Docker images:
REPOSITORY TAG
inventory 1.0-SNAPSHOT
system 1.0-SNAPSHOTTo deploy your microservices to the Kubernetes cluster, use the following command:
kubectl apply -f services.yamlYou will see an output similar to the following:
gateway.networking.istio.io/sys-app-gateway created service/system-service created service/inventory-service created deployment.apps/system-deployment created deployment.apps/inventory-deployment created
The traffic.yaml file contains two virtual services. A virtual service defines how requests are routed to your applications.
Deploy the resources defined in the traffic.yaml file:
kubectl apply -f traffic.yamlRun the following command to check the status of your pods:
kubectl get podsIf all the pods are healthy and running, you will see an output similar to the following:
NAME READY STATUS RESTARTS AGE
inventory-deployment-645767664f-nbtd9 2/2 Running 0 30s
system-deployment-6bd97d9bf6-4ccds 2/2 Running 0 30sCheck that all of the deployments are available. You need to wait until all of your deployments are ready and available before making requests to your microservices.
kubectl get deploymentsNAME READY UP-TO-DATE AVAILABLE AGE
inventory-deployment 1/1 1 1 1m
system-deployment 1/1 1 1 1mYou will make a request to the system service from the inventory service to access the
JVM system properties of your running container. The Istio gateway is the entry point for HTTP requests to the cluster.
As defined in the services.yaml file, the gateway is expecting the
Host header of your system service and inventory service to be system.example.com and inventory.example.com, respectively.
However, requests to system.example.com and inventory.example.com won’t be routed to the appropriate IP address.
To ensure that the gateway routes your requests appropriately, ensure that the Host header is set appropriately.
You can set the Host header with the -H option of the curl command.
services.yaml
link:finish/services.yaml[role=include]traffic.yaml
link:finish/traffic.yaml[role=include]Make a request to the service by using curl:
curl -H Host:inventory.example.com http://localhost/inventory/systems/system-service -IIf the curl command is unavailable, then use Postman. Postman enables you
to make requests using a graphical interface. To make a request with Postman, enter http://localhost/inventory/systems/system-service
into the URL bar. Next, switch to the Headers tab and add a header with key of Host and value of inventory.example.com.
Finally, click the blue Send button to make the request.
export INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].nodePort}')
curl -H Host:inventory.example.com http://`minikube ip`:$INGRESS_PORT/inventory/systems/system-service -IYou will see the following output:
HTTP/1.1 200 OK
x-powered-by: Servlet/4.0
content-type: application/json
date: Mon, 19 Aug 2019 19:49:47 GMT
content-language: en-US
x-envoy-upstream-service-time: 4242
server: istio-envoy
transfer-encoding: chunkedBecause the system service is available, the request to the service is successful and returns a 200 response code.
To see the number of times that the system service was called, check the logs of the system pod by using the kubectl logs command.
Replace [system-pod-name] with the pod name associated with
your system service, which you previously saw when running the kubectl get pods command.
kubectl logs [system-pod-name] -c istio-proxy | grep -c system-service:9090kubectl logs [system-pod-name] -c istio-proxy | find /C "system-service:9090"You will see that the kubectl logs command returns a value of 1, meaning that 1 request is made to the system service:
1Now you will make the system service unavailable and observe that MicroProfile’s Retry policy will take effect.
Pause the system service pod to simulate that the service is unavailable.
Remember to replace [system-pod-name] with the pod name that is associated with your system service.
kubectl exec -it [system-pod-name] -- /opt/ol/wlp/bin/server pauseYou will see the following output:
Pausing the defaultServer server. Pausing the defaultServer server completed.
Make a request to the service by using curl:
curl -H Host:inventory.example.com http://localhost/inventory/systems/system-service -IIf the curl command is unavailable, then use Postman.
curl -H Host:inventory.example.com http://`minikube ip`:$INGRESS_PORT/inventory/systems/system-service -IYou will see the following output:
HTTP/1.1 503 Service Unavailable
x-powered-by: Servlet/4.0
content-length: 91
content-type: text/plain
date: Thu, 15 Aug 2019 13:21:57 GMT
server: istio-envoy
x-envoy-upstream-service-time: 2929
content-language: en-USBecause the the system service is unavailable, the request returns a 503 response code.
However, the request retried several times, as specified by the MicroProfile @Retry annotation.
See the number of times that the service was retried:
kubectl logs [system-pod-name] -c istio-proxy | grep -c system-service:9090kubectl logs [system-pod-name] -c istio-proxy | find /C "system-service:9090"You will see the following output:
37The above command returns 37, because there was a total of 37 requests made to the system service.
By default, Istio will retry 2 times to resolve any issues with a 503 response code.
Including the initial requests and the retries, the requests made by Istio are multiplied by the requests made by MicroProfile.
Hence, the 3 requests from the system service, the 3 requests from the inventory service, and the 4 MicroProfile requests are multiplied together,
giving a total of 36 requests. Including the succesful request that you made before you paused the system service, there was a total of 37 requests.
Previously, you implemented the Retry policy to retry requests to your system service by using MicroProfile Fault Tolerance.
This Retry policy can also be implemented with Istio Fault Tolerance.
Update thetraffic.yamlfile.traffic.yaml
traffic.yaml
link:finish/traffic.yaml[role=include]Add the retries field under the route specification in the traffic.yaml file.
This tells Istio to retry requests a maximum of 4 times when the request returns any 5xx response code.
The attempts field is required in the configuration of the
Istio Retry policy. This field specifies the
maximum number of retries that will be attempted for a given request. To retry a request on specific conditions, use the
retryOn field. Because your paused system service responds with a 503 response code,
you set retryOn to be 5xx.
Other retry conditions
can also be specified in retryOn. Optionally, the perTryTimeout field can be added to Istio’s Retry policy
to specify the amount of time that is allocated to each retry attempt.
After you configure the number of retries that Istio performs, deploy your microservices again:
kubectl replace --force -f services.yaml
kubectl replace --force -f traffic.yamlWait until all of your deployments are ready and available.
The [system-pod-name] will be regenerated and is different than the one you used previously.
Run the kubectl get pods command to get the new [system-pod-name].
Pause the system service pod to simulate that the service is unavailable.
kubectl exec -it [system-pod-name] -- /opt/ol/wlp/bin/server pauseMake a request to the service by using curl:
curl -H Host:inventory.example.com http://localhost/inventory/systems/system-service -IIf the curl command is unavailable, then use Postman.
curl -H Host:inventory.example.com http://`minikube ip`:$INGRESS_PORT/inventory/systems/system-service -IBecause the system service is unavailable, the request still returns a 503 response code.
This time, however, Istio retried the request several more times before failing.
See the number of times that the service was retried:
kubectl logs [system-pod-name] -c istio-proxy | grep -c system-service:9090kubectl logs [system-pod-name] -c istio-proxy | find /C "system-service:9090"You will see the following output:
60The above command returns a value of 60, indicating that a total of 60 requests are made to the system service.
The 3 default Istio requests for the system service, the 5 requests for the inventory service
that you enabled in the traffic.yaml file, and the 4 requests sent by MicroProfile are multiplied together.
Next, you will disable some MicroProfile Fault Tolerance capabilities, so that your system service retries with only Istio’s Retry policy.
When both MicroProfile and Istio Fault Tolerance capabilities are enabled, there is a compounding effect that may be unexpected. If both MicroProfile and Istio set their own Retry policies on a service, the maximum number of retries that occur is not equivalent to either of the number of retries specified in MicroProfile or Istio. The number of retries set by MicroProfile and Istio are actually multiplied.
If you want to use Istio as your service mesh and only its fault tolerance capabilities, you can turn off MicroProfile Fault Tolerance by adding a property. This configuration avoids any overlap in behaviours.
MicroProfile Fault Tolerance offers a config property MP_Fault_Tolerance_NonFallback_Enabled that disables all
MicroProfile Fault Tolerance capabilities except fallback. If MP_Fault_Tolerance_NonFallback_Enabled is set to false, only
the @Fallback behaviour is enabled. The other behaviours specified by the MicroProfile Fault Tolerance annotations,
including @Retry, won’t take effect.
You will define the MP_Fault_Tolerance_NonFallback_Enabled config property in a ConfigMap.
ConfigMaps store configuration settings about a Kubernetes pod. This configuration is loaded into the pod as an environment variable that is
used by the pod’s containers. The environment variables are defined in the pod’s specification by using the envFrom field. To learn more about ConfigMaps,
check out the Configuring microservices running in Kubernetes guide.
Use the MP_Fault_Tolerance_NonFallback_Enabled config property to disable the retries performed by MicroProfile,
so that only Istio performs retries.
Update theservices.yamlfile.services.yaml
services.yaml
link:finish/services.yaml[role=include]Add a ConfigMap into the services.yaml file, and
set the MP_Fault_Tolerance_NonFallback_Enabled config property to false.
Add the envFrom field
to inject the ConfigMap with the MP_Fault_Tolerance_NonFallback_Enabled property into your pods.
The name of the ConfigMap, which is inventory-config,
becomes the environment variable name that is specified in the envFrom field.
Deploy your microservices again to turn off all MicroProfile Fault Tolerance capabilities, except fallback:
kubectl replace --force -f services.yamlWait until all of your deployments are ready and available.
Run the kubectl get pods command to get the new [system-pod-name].
Pause the system service pod to simulate that the service is unavailable:
kubectl exec -it [system-pod-name] -- /opt/ol/wlp/bin/server pauseMake a request to the service by using curl:
curl -H Host:inventory.example.com http://localhost/inventory/systems/system-service -IIf the curl command is unavailable, then use Postman.
curl -H Host:inventory.example.com http://`minikube ip`:$INGRESS_PORT/inventory/systems/system-service -IBecause the system service is unavailable, the request still returns a 503 response code.
This time, however, the request was retried several times with Istio, without any retries from MicroProfile.
See the number of times that the service was retried:
kubectl logs [system-pod-name] -c istio-proxy | grep -c system-service:9090kubectl logs [system-pod-name] -c istio-proxy | find /C "system-service:9090"You will see the following output:
15The above command returns 15, indicating that a total of 15 requests are made to the system service.
Because MicroProfile’s Retry policy is disabled, only Istio’s retries are performed.
traffic.yaml
link:finish/traffic.yaml[role=include]services.yaml
link:finish/services.yaml[role=include]Since retrying the requests to the system service still does not succeed, you need a "fall back" plan.
You will create a fallback method as an alternative solution for when retry requests to the system service have failed.
Although you disabled MicroProfile @Retry and other MicroProfile Fault Tolerance policies using the
MP_Fault_Tolerance_NonFallback_Enabled config property, the fallback policy is still available.
As mentioned before, Istio does not offer any fallback capabilities, so the MicroProfile Fallback capability can be used to complement it.
The @Fallback annotation dictates a method to call when the original method encounters a failed execution.
If your microservices have a Retry policy specified, then the fallback occurs after all of the retries have failed.
Update theInventoryResource.javafile.inventory/src/main/java/io/openliberty/guides/inventory/InventoryResource.java
InventoryResource.java
link:finish/inventory/src/main/java/io/openliberty/guides/inventory/InventoryResource.java[role=include]Create the getPropertiesFallback() method.
Add the @Fallback annotation before the getPropertiesForHost() method,
to call the getPropertiesFallback() method when a failure occurs.
The getPropertiesFallback() method, which is the designated fallback method for the original
getPropertiesForHost() method, prints out a warning message in the browser that says
the system service may not be running.
Rebuild your application to add fallback behaviour to your microservices:
mvn packageNext, run the docker build commands to rebuild the container images for your application:
docker build -t inventory:1.0-SNAPSHOT inventory/.
docker build -t system:1.0-SNAPSHOT system/.Deploy your microservices again to turn off all MicroProfile Fault Tolerance capabilities, except fallback:
kubectl replace --force -f services.yaml
kubectl replace --force -f traffic.yamlWait until all of your deployments are ready and available.
Run the kubectl get pods command to get the new [system-pod-name].
Pause the system service pod to simulate that the service is unavailable:
kubectl exec -it [system-pod-name] -- /opt/ol/wlp/bin/server pauseMake a request to the service by using curl:
curl -H Host:inventory.example.com http://localhost/inventory/systems/system-service -IIf the curl command is unavailable, then use Postman.
curl -H Host:inventory.example.com http://`minikube ip`:$INGRESS_PORT/inventory/systems/system-service -IYou will see the following output:
HTTP/1.1 200 OK
x-powered-by: Servlet/4.0
x-from-fallback: yes
content-type: application/json
date: Mon, 19 Aug 2019 19:49:47 GMT
content-language: en-US
x-envoy-upstream-service-time: 4242
server: istio-envoy
transfer-encoding: chunkedYou can see that the request is now successful and returns a 200 response code, with a header called
x-from-fallback, indicating that the fallback method is called when the system service is not available.
See the number of times that the service is retried before the fallback method is called:
kubectl logs [system-pod-name] -c istio-proxy | grep -c system-service:9090kubectl logs [system-pod-name] -c istio-proxy | find /C "system-service:9090"You will see the following output:
3The above command returns 3, indicating that a total of 3 requests are made to the system service.
The Istio retries that you enabled on the inventory service are not performed, because the Fallback policy is enabled.
However, the 3 default requests by Istio on the server end are still performed.
Because all of these requests failed, the getPropertiesFallback() fallback method is called.
When you are done checking out the MicroProfile and Istio Fault Tolerance features, you might want to tear down all the deployed resources as a cleanup step.
Delete your resources from the cluster:
kubectl delete -f services.yaml
kubectl delete -f traffic.yamlDelete the istio-injection label from the default namespace. The hyphen immediately
after the label name indicates that the label should be deleted.
kubectl label namespace default istio-injection-Navigate to the directory where you extracted Istio and delete the Istio resources from the cluster:
kubectl delete -f install/kubernetes/istio-demo.yamlDelete all Istio resources from the cluster:
istioctl uninstall --purgeistioctl uninstall --purgeistioctl uninstall --purgePerform the following steps to return your environment to a clean state.
-
Point the Docker daemon back to your local machine:
eval $(minikube docker-env -u) -
Stop and delete your Minikube cluster:
minikube stop minikube delete
You learned how to build resilient microservices by using Istio Retry and MicroProfile Fallback. You also observed how MicroProfile Fault Tolerance integrates with and complements Istio Fault Tolerance.