I am tring to implement the spring load balancer retry feature in my project. I added RetryableLoadBalancerFilterExchangeFuntion (By Autowiring ) filter to the webclient builder by following this Reference: https://github.com/spring-cloud/spring-cloud-commons/pull/847/commits
Without this change SCLB is working fine, the traffic was distributed to all the available instances, after adding the filter, we are facing the below errors.
{"instant":{"epochSecond":1702552930,"nanoOfSecond":788156223},"thread":"boundedElastic-3","level":"DEBUG","loggerName":"org.springframework.cloud.kubernetes.fabric8.discovery.Fabric8KubernetesDiscoveryClientUtils","message":"discovering endpoints in namespace : sampleTest","endOfBatch":false,"loggerFqcn":"org.apache.commons.logging.LogAdapter$Log4jLog","threadId":91,"threadPriority":5,"messageTimestamp":"2023-12-14T11:22:10.788+0000"}
{"instant":{"epochSecond":1702552930,"nanoOfSecond":796103469},"thread":"boundedElastic-4","level":"DEBUG","loggerName":"org.springframework.cloud.kubernetes.fabric8.discovery.Fabric8KubernetesDiscoveryClientUtils","message":"filter not present","endOfBatch":false,"loggerFqcn":"org.apache.commons.logging.LogAdapter$Log4jLog","threadId":92,"threadPriority":5,"messageTimestamp":"2023-12-14T11:22:10.796+0000"}
{"instant":{"epochSecond":1702552930,"nanoOfSecond":796920350},"thread":"boundedElastic-4","level":"WARN","loggerName":"org.springframework.cloud.loadbalancer.core.RoundRobinLoadBalancer","message":"No servers available for service: 10.233.111.139","endOfBatch":false,"loggerFqcn":"org.apache.commons.logging.LogAdapter$Log4jLog","threadId":92,"threadPriority":5,"messageTimestamp":"2023-12-14T11:22:10.796+0000"}
{"instant":{"epochSecond":1702552930,"nanoOfSecond":798481135},"thread":"boundedElastic-4","level":"WARN","loggerName":"org.springframework.cloud.client.loadbalancer.reactive.RetryableLoadBalancerExchangeFilterFunction","message":"LoadBalancer does not contain an instance for the service IP","endOfBatch":false,"loggerFqcn":"org.apache.commons.logging.LogAdapter$Log4jLog","threadId":92,"threadPriority":5,"messageTimestamp":"2023-12-14T11:22:10.798+0000"}
{"instant":{"epochSecond":1702552930,"nanoOfSecond":811667123},"thread":"boundedElastic-3","level":"DEBUG","loggerName":"org.springframework.cloud.kubernetes.fabric8.discovery.Fabric8KubernetesDiscoveryClientUtils","message":"filter not present","endOfBatch":false,"loggerFqcn":"org.apache.commons.logging.LogAdapter$Log4jLog","threadId":91,"threadPriority":5,"messageTimestamp":"2023-12-14T11:22:10.811+0000"}
{"instant":{"epochSecond":1702552930,"nanoOfSecond":812302880},"thread":"boundedElastic-3","level":"WARN","loggerName":"org.springframework.cloud.loadbalancer.core.RoundRobinLoadBalancer","message":"No servers available for service: IP","endOfBatch":false,"loggerFqcn":"org.apache.commons.logging.LogAdapter$Log4jLog","threadId":91,"threadPriority":5,"messageTimestamp":"2023-12-14T11:22:10.812+0000"}
NOTE: Our application is using Spring Cloud Kubernetes Discovery Server, there is no implementation of ServiceInstanceListSupplier in the code. Reference : https://docs.spring.io/spring-cloud-kubernetes/reference/spring-cloud-kubernetes-discoveryserver.html#:~:text=The%20Spring%20Cloud%20Discovery%20server,the%20Service%20Account%20on%20Kubernetes.
We have the following properties added in the application.properties. spring.cloud.kubernetes.loadbalancer.mode= POD spring.cloud.loadbalancer.ribbon.enabled= false spring.cloud.loadbalancer.retry.enabled= true spring.cloud.loadbalancer.retry.retry-on-all-exceptions= true spring.cloud.loadbalancer.retry.max-retries-on-next-service-instance=2 spring.cloud.loadbalancer.retry.max-retries-on-same-service-instance=0 spring.cloud.loadbalancer.retry.retry-on-all-operations= true
Please let me know if any additional information is needed.
Thank you