Availability: Backoff and Retry Policy

BackOffer

The retry and timeout mechanism for a request is controlled by a BackOffer object, which is created one per RawKVClient method. The BackOffer will decide how much time the next sleep and retry should spend, and whether to timeout the request if not enough time is left for retrying the request. If we need a back off sleep, we call backOffer.doBackOff(funcType, exception), and the current thread will sleep for a decided time. If the current operation will timeout after sleep, the doBackOff simply throw an exception to abort the operation.

callWithRetry

RegionStoreClient.callWithRetry inherits from AbstractGRPCClient.callWithRetry. The concrete logic is in RetryPolicy.callWithRetry, which implements a retry mechanism, but the specific retry strategy is determined by the ErrorHandler. ErrorHandler’s handler{Request, Response}Error function returns a boolean value indicating whether to retry inside callWithRetry. The control flow for callWithRetry is as follows:

The error handler is chosen obeying the following table:

The handleRequestError function copes with the following situations:

The handleResponseError function copes with the following gRPC errors: