Building Resilient RPC Clients: Auto-Reconnect and Retry Policies

Networks fail. Servers restart. Connections drop. In the real world, distributed systems must handle these failures gracefully. A resilient client stays connected, recovers from failures, and retries when appropriate.

WitRPC provides built-in resilience features: automatic reconnection with exponential backoff, configurable retry policies, and health monitoring. Today, we'll explore how to use these features to build robust, self-healing applications.

Why Resilience Matters

Consider a typical production scenario:

Without resilience features, your application:

Crashes or shows cryptic errors
Requires manual intervention to reconnect
Loses in-flight requests
Frustrates users

With proper resilience:

Client automatically detects disconnection
Reconnects with intelligent backoff
Retries failed requests when safe
Application continues working

Let's build this.

Auto-Reconnection

Basic Configuration

Enable automatic reconnection with WithAutoReconnect():

csharp

var client = WitClientBuilder.Build(options =>
{
    options.WithTcp("server.example.com", 5000);
    options.WithMessagePack();
    options.WithAutoReconnect();  // Enable with defaults
});

With defaults, the client will:

Attempt to reconnect up to 10 times
Start with a 1-second delay
Use exponential backoff (2x multiplier)
Cap delays at 2 minutes

Custom Reconnection Settings

Fine-tune the behavior for your needs:

csharp

var client = WitClientBuilder.Build(options =>
{
    options.WithTcp("server.example.com", 5000);
    options.WithMessagePack();
    
    options.WithAutoReconnect(reconnect =>
    {
        reconnect.MaxAttempts = 5;                              // Give up after 5 attempts
        reconnect.InitialDelay = TimeSpan.FromSeconds(2);       // Wait 2s before first retry
        reconnect.MaxDelay = TimeSpan.FromSeconds(30);          // Never wait more than 30s
        reconnect.BackoffMultiplier = 1.5;                      // Gentler backoff curve
        reconnect.ReconnectOnDisconnect = true;                 // Also reconnect if server disconnects us
    });
});

Understanding Exponential Backoff

Exponential backoff prevents thundering herd problems. Here's how delays progress:

Attempt 1: InitialDelay = 2s
Attempt 2: 2s × 1.5 = 3s
Attempt 3: 3s × 1.5 = 4.5s
Attempt 4: 4.5s × 1.5 = 6.75s
Attempt 5: 6.75s × 1.5 = 10.125s (capped at MaxDelay if exceeded)

With default settings (2x multiplier):

Attempt	Delay	Cumulative Wait
1	1s	1s
2	2s	3s
3	4s	7s
4	8s	15s
5	16s	31s
6	32s	63s
7	64s	~2 min
8+	120s (capped)	...

This prevents overwhelming a recovering server while still attempting reconnection quickly at first.

Reconnection Callbacks

Monitor reconnection attempts with callbacks:

csharp

options.WithAutoReconnect(reconnect =>
{
    reconnect.MaxAttempts = 10;
    reconnect.InitialDelay = TimeSpan.FromSeconds(1);
    
    // Called before each reconnection attempt
    reconnect.OnReconnecting = (attempt, delay) =>
    {
        Console.WriteLine($"Reconnecting... Attempt {attempt}, waiting {delay}");
        UpdateUI("Reconnecting...");
    };
    
    // Called when reconnection succeeds
    reconnect.OnReconnected = () =>
    {
        Console.WriteLine("Reconnected successfully!");
        UpdateUI("Connected");
        ResubscribeToEvents();  // Important: re-register event handlers
    };
    
    // Called when all attempts exhausted
    reconnect.OnReconnectionFailed = () =>
    {
        Console.WriteLine("Failed to reconnect after all attempts");
        UpdateUI("Disconnected - Please check your connection");
        ShowReconnectButton();
    };
});

Handling Reconnection in Your Code

When reconnection occurs, you need to handle state properly:

csharp

public class ResilientServiceClient
{
    private readonly IWitClient _client;
    private IOrderService _orderService;
    
    public ResilientServiceClient(IWitClient client)
    {
        _client = client;
        
        // Initial setup
        _orderService = _client.GetService<IOrderService>();
        SubscribeToEvents();
        
        // Handle reconnection
        _client.Reconnected += OnReconnected;
        _client.Disconnected += OnDisconnected;
    }
    
    private void OnDisconnected()
    {
        // Notify UI, pause operations
        IsConnected = false;
        ConnectionStatusChanged?.Invoke(false);
    }
    
    private void OnReconnected()
    {
        // Get fresh service proxy
        _orderService = _client.GetService<IOrderService>();
        
        // Re-subscribe to events (subscriptions don't survive reconnection)
        SubscribeToEvents();
        
        // Resume operations
        IsConnected = true;
        ConnectionStatusChanged?.Invoke(true);
    }
    
    private void SubscribeToEvents()
    {
        _orderService.OrderCreated += HandleOrderCreated;
        _orderService.OrderStatusChanged += HandleOrderStatusChanged;
    }
    
    public bool IsConnected { get; private set; }
    public event Action<bool> ConnectionStatusChanged;
}

Important: Event subscriptions don't survive reconnection. Always re-subscribe in your OnReconnected handler.

Retry Policies

Auto-reconnection handles connection failures. Retry policies handle individual call failures.

When to Retry

Not all failures should be retried:

Failure Type	Retry?	Reason
Network timeout	✅ Yes	Transient, may succeed next time
Server overloaded (503)	✅ Yes	Server may recover
Connection reset	✅ Yes	Transient network issue
Bad request (400)	❌ No	Your request is invalid
Not found (404)	❌ No	Resource doesn't exist
Unauthorized (401)	❌ No	Credentials are wrong
Business logic error	❌ No	Will fail the same way

Basic Retry Configuration

Enable retry with WithRetryPolicy():

csharp

var client = WitClientBuilder.Build(options =>
{
    options.WithTcp("server.example.com", 5000);
    options.WithMessagePack();
    
    options.WithRetryPolicy(retry =>
    {
        retry.MaxRetries = 3;
        retry.InitialDelay = TimeSpan.FromMilliseconds(100);
        retry.BackoffMultiplier = 2.0;
        retry.MaxDelay = TimeSpan.FromSeconds(5);
    });
});

Backoff Strategies

WitRPC supports three backoff strategies:

csharp

// Fixed: Same delay every time
retry.BackoffType = BackoffType.Fixed;
retry.InitialDelay = TimeSpan.FromMilliseconds(500);
// Attempts: 500ms, 500ms, 500ms, ...

// Linear: Delay increases by fixed amount
retry.BackoffType = BackoffType.Linear;
retry.InitialDelay = TimeSpan.FromMilliseconds(100);
retry.BackoffMultiplier = 2.0;  // Add 100ms × 2 each time
// Attempts: 100ms, 300ms, 500ms, 700ms, ...

// Exponential: Delay multiplies each time (default)
retry.BackoffType = BackoffType.Exponential;
retry.InitialDelay = TimeSpan.FromMilliseconds(100);
retry.BackoffMultiplier = 2.0;
// Attempts: 100ms, 200ms, 400ms, 800ms, ...

Configuring Retryable Conditions

Specify which errors should trigger retries:

csharp

options.WithRetryPolicy(retry =>
{
    retry.MaxRetries = 3;
    retry.InitialDelay = TimeSpan.FromMilliseconds(200);
    
    // Retry on specific communication statuses
    retry.RetryOnStatus(CommunicationStatus.InternalServerError);
    retry.RetryOnStatus(CommunicationStatus.ServiceUnavailable);
    retry.RetryOnStatus(CommunicationStatus.Timeout);
    
    // Retry on specific exception types
    retry.RetryOn<TimeoutException>();
    retry.RetryOn<IOException>();
});

Retry Callbacks

Monitor and log retry attempts:

csharp

options.WithRetryPolicy(retry =>
{
    retry.MaxRetries = 3;
    retry.InitialDelay = TimeSpan.FromMilliseconds(100);
    
    retry.OnRetry = (exception, attempt, delay) =>
    {
        _logger.LogWarning(
            exception,
            "Retry attempt {Attempt} after {Delay}ms. Error: {Message}",
            attempt,
            delay.TotalMilliseconds,
            exception?.Message ?? "Unknown");
        
        // Track metrics
        _metrics.IncrementRetryCount();
    };
});

Idempotency: The Critical Consideration

Before enabling retries, understand idempotency:

Idempotent operations can be safely retried:

Reading data: GetOrder(id) returns the same result every time
Updating with full state: UpdateOrder(order) applies the same state
Delete by ID: DeleteOrder(id), since deleting twice is fine

Non-idempotent operations are dangerous to retry:

Creating resources: CreateOrder() may create duplicates
Incrementing counters: IncrementViews() double-counts
Sending notifications: SendEmail() means duplicate emails

Making Operations Idempotent

Strategy 1: Use Idempotency Keys

csharp

public interface IOrderService
{
    // Idempotent: same key = same operation
    Task<Order> CreateOrderAsync(string idempotencyKey, CreateOrderRequest request);
}

// Client usage
var idempotencyKey = Guid.NewGuid().ToString();
var order = await orderService.CreateOrderAsync(idempotencyKey, request);
// If retried with same key, server returns existing order

Strategy 2: Check-Then-Act

csharp

public interface IInventoryService
{
    Task<bool> ReserveStockAsync(string reservationId, string productId, int quantity);
}

// Server implementation
public async Task<bool> ReserveStockAsync(string reservationId, string productId, int quantity)
{
    // Check if already reserved
    if (await _reservations.ExistsAsync(reservationId))
    {
        return true;  // Already done, return success
    }
    
    // Perform reservation
    return await _inventory.ReserveAsync(reservationId, productId, quantity);
}

Strategy 3: Disable Retry for Non-Idempotent Operations

csharp

public class OrderService
{
    private readonly IOrderService _service;
    private readonly IWitClient _client;
    
    public async Task<Order> CreateOrderAsync(CreateOrderRequest request)
    {
        // Temporarily disable retry for this call
        using (_client.DisableRetry())
        {
            return await _service.CreateOrderAsync(request);
        }
    }
    
    public async Task<Order> GetOrderAsync(int orderId)
    {
        // Retry is fine for reads
        return await _service.GetOrderAsync(orderId);
    }
}

Combining Reconnection and Retry

Use both features together for maximum resilience:

csharp

var client = WitClientBuilder.Build(options =>
{
    options.WithTcp("server.example.com", 5000);
    options.WithMessagePack();
    options.WithEncryption();
    
    // Reconnection for connection-level failures
    options.WithAutoReconnect(reconnect =>
    {
        reconnect.MaxAttempts = 10;
        reconnect.InitialDelay = TimeSpan.FromSeconds(1);
        reconnect.MaxDelay = TimeSpan.FromMinutes(2);
        
        reconnect.OnReconnecting = (attempt, delay) =>
            _logger.LogInformation("Reconnecting, attempt {Attempt}", attempt);
        
        reconnect.OnReconnected = () =>
        {
            _logger.LogInformation("Reconnected");
            ResubscribeToEvents();
        };
        
        reconnect.OnReconnectionFailed = () =>
            _logger.LogError("Reconnection failed, giving up");
    });
    
    // Retry for call-level failures
    options.WithRetryPolicy(retry =>
    {
        retry.MaxRetries = 3;
        retry.InitialDelay = TimeSpan.FromMilliseconds(100);
        retry.BackoffType = BackoffType.Exponential;
        retry.BackoffMultiplier = 2.0;
        retry.MaxDelay = TimeSpan.FromSeconds(2);
        
        // Only retry transient failures
        retry.RetryOnStatus(CommunicationStatus.Timeout);
        retry.RetryOnStatus(CommunicationStatus.ServiceUnavailable);
        
        retry.OnRetry = (ex, attempt, delay) =>
            _logger.LogWarning("Retrying call, attempt {Attempt}", attempt);
    });
});

How They Work Together

Health Checks Integration

Monitor connection health in ASP.NET Core applications:

csharp

// Install: OutWit.Communication.Client.HealthChecks

// Registration
builder.Services.AddHealthChecks()
    .AddWitRpcClientCheck("order-service", 
        tags: new[] { "rpc", "ready" },
        timeout: TimeSpan.FromSeconds(5));

// Endpoint
app.MapHealthChecks("/health", new HealthCheckOptions
{
    ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse
});

Custom health check with more control:

csharp

public class WitRpcHealthCheck : IHealthCheck
{
    private readonly IWitClientFactory _factory;
    private readonly string _clientName;
    
    public WitRpcHealthCheck(IWitClientFactory factory, string clientName)
    {
        _factory = factory;
        _clientName = clientName;
    }
    
    public async Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context,
        CancellationToken cancellationToken = default)
    {
        try
        {
            var client = _factory.GetClient(_clientName);
            
            if (!client.IsConnected)
            {
                return HealthCheckResult.Unhealthy(
                    $"Client '{_clientName}' is disconnected");
            }
            
            // Optional: ping the service
            var service = client.GetService<IHealthService>();
            var response = await service.PingAsync();
            
            if (response.IsHealthy)
            {
                return HealthCheckResult.Healthy(
                    $"Client '{_clientName}' is connected and responsive");
            }
            
            return HealthCheckResult.Degraded(
                $"Client '{_clientName}' connected but service degraded: {response.Message}");
        }
        catch (Exception ex)
        {
            return HealthCheckResult.Unhealthy(
                $"Client '{_clientName}' health check failed",
                exception: ex);
        }
    }
}

Complete Resilient Client Example

Here's a production-ready resilient client setup:

csharp

public class ResilientWitRpcClient : IDisposable
{
    private readonly IWitClient _client;
    private readonly ILogger<ResilientWitRpcClient> _logger;
    private readonly SemaphoreSlim _connectionLock = new(1, 1);
    
    private bool _isConnected;
    private int _reconnectAttempts;
    
    public event Action? Connected;
    public event Action? Disconnected;
    public event Action<int>? Reconnecting;
    
    public ResilientWitRpcClient(
        string host, 
        int port,
        ILogger<ResilientWitRpcClient> logger)
    {
        _logger = logger;
        
        _client = WitClientBuilder.Build(options =>
        {
            options.WithTcp(host, port);
            options.WithMessagePack();
            options.WithEncryption();
            
            ConfigureReconnection(options);
            ConfigureRetry(options);
        });
        
        // Surface the connection lifecycle through this wrapper's own events.
        // WitClient exposes Disconnected; connect/reconnect moments come from
        // ConnectAsync results and the auto-reconnect callbacks below.
        _client.Disconnected += sender => Disconnected?.Invoke();
    }
    
    private void ConfigureReconnection(WitClientBuilderOptions options)
    {
        options.WithAutoReconnect(reconnect =>
        {
            reconnect.MaxAttempts = 0;  // Unlimited attempts
            reconnect.InitialDelay = TimeSpan.FromSeconds(1);
            reconnect.MaxDelay = TimeSpan.FromMinutes(5);
            reconnect.BackoffMultiplier = 2.0;
            
            reconnect.OnReconnecting = (attempt, delay) =>
            {
                _reconnectAttempts = attempt;
                _logger.LogWarning(
                    "Connection lost. Reconnecting in {Delay}s (attempt {Attempt})",
                    delay.TotalSeconds, attempt);
                Reconnecting?.Invoke(attempt);
            };
            
            reconnect.OnReconnected = () =>
            {
                _logger.LogInformation(
                    "Reconnected after {Attempts} attempts", 
                    _reconnectAttempts);
                _reconnectAttempts = 0;
                Connected?.Invoke();
            };
        });
    }
    
    private void ConfigureRetry(WitClientBuilderOptions options)
    {
        options.WithRetryPolicy(retry =>
        {
            retry.MaxRetries = 3;
            retry.InitialDelay = TimeSpan.FromMilliseconds(100);
            retry.MaxDelay = TimeSpan.FromSeconds(2);
            retry.BackoffType = BackoffType.Exponential;
            
            // Retry transient failures only
            retry.RetryOnStatus(CommunicationStatus.Timeout);
            retry.RetryOnStatus(CommunicationStatus.ServiceUnavailable);
            retry.RetryOnStatus(CommunicationStatus.InternalServerError);
            retry.RetryOn<TimeoutException>();
            retry.RetryOn<IOException>();
            
            retry.OnRetry = (ex, attempt, delay) =>
            {
                _logger.LogDebug(
                    "Retry {Attempt}/3 in {Delay}ms: {Error}",
                    attempt, delay.TotalMilliseconds, ex?.Message);
            };
        });
    }
    
    public async Task ConnectAsync(CancellationToken cancellationToken = default)
    {
        await _connectionLock.WaitAsync(cancellationToken);
        try
        {
            if (_isConnected) return;
            
            _logger.LogInformation("Connecting to server...");
            await _client.ConnectAsync(TimeSpan.FromSeconds(30), cancellationToken);
            _logger.LogInformation("Connected successfully");
        }
        finally
        {
            _connectionLock.Release();
        }
    }
    
    public T GetService<T>() where T : class
    {
        if (!_isConnected)
        {
            throw new InvalidOperationException("Not connected. Call ConnectAsync first.");
        }
        return _client.GetService<T>();
    }
    
    public bool IsConnected => _isConnected;
    
    private void OnConnected()
    {
        _isConnected = true;
        Connected?.Invoke();
    }
    
    private void OnDisconnected()
    {
        _isConnected = false;
        Disconnected?.Invoke();
    }
    
    public void Dispose()
    {
        _client.Dispose();
        _connectionLock.Dispose();
    }
}

Usage:

csharp

var client = new ResilientWitRpcClient("server.example.com", 5000, logger);

client.Connected += () => Console.WriteLine("✓ Connected");
client.Disconnected += () => Console.WriteLine("✗ Disconnected");
client.Reconnecting += attempt => Console.WriteLine($"↻ Reconnecting (attempt {attempt})");

await client.ConnectAsync();

var orderService = client.GetService<IOrderService>();

// Subscribe to events (will need to resubscribe after reconnection)
orderService.OrderCreated += order => Console.WriteLine($"New order: {order.Id}");

// Make calls (will automatically retry on transient failures)
var orders = await orderService.GetOrdersAsync();

Best Practices

1. Always Handle Reconnection Events

csharp

// Enabling auto-reconnect is half the job; handle the lifecycle too
reconnect.OnReconnected = () =>
{
    // Re-fetch any cached service proxies
    _service = _client.GetService<IMyService>();
    
    // Re-subscribe to events
    _service.SomethingHappened += HandleEvent;
    
    // Refresh any stale data
    await RefreshCachedDataAsync();
};

2. Use Appropriate Timeouts

csharp

// Connection timeout should exceed reconnection delay
options.WithAutoReconnect(reconnect =>
{
    reconnect.InitialDelay = TimeSpan.FromSeconds(1);
    // ...
});

// Call timeout should be reasonable for your operations
await client.ConnectAsync(TimeSpan.FromSeconds(30));

3. Log Resilience Events

csharp

// In production, you need visibility
reconnect.OnReconnecting = (attempt, delay) =>
    _logger.LogWarning("Reconnecting, attempt {Attempt}", attempt);

retry.OnRetry = (ex, attempt, delay) =>
    _logger.LogWarning(ex, "Retrying, attempt {Attempt}", attempt);

4. Set Sensible Limits

csharp

// Don't retry forever for calls
retry.MaxRetries = 3;  // Fail fast

// But consider unlimited reconnection for long-running apps
reconnect.MaxAttempts = 0;  // 0 = unlimited

5. Test Failure Scenarios

csharp

// Deliberately test resilience
[Test]
public async Task Client_ReconnectsAfterServerRestart()
{
    await client.ConnectAsync();
    
    // Simulate server restart
    await server.StopAsync();
    await Task.Delay(2000);
    await server.StartAsync();
    
    // Wait for reconnection
    await WaitForConditionAsync(() => client.IsConnected, TimeSpan.FromSeconds(30));
    
    // Verify functionality restored
    var result = await service.PingAsync();
    Assert.That(result, Is.True);
}

Conclusion

Resilience isn't optional in production systems. WitRPC's built-in features make it straightforward:

Auto-reconnection keeps your client connected despite network issues
Exponential backoff prevents thundering herd problems
Retry policies handle transient call failures
Health checks let you monitor connection status
Callbacks give you visibility and control

Remember:

Configure both reconnection (connection-level) and retry (call-level)
Handle the reconnection lifecycle (especially event re-subscription)
Be careful with idempotency when retrying
Log everything for production visibility
Test your failure scenarios

With these patterns, your WitRPC applications will gracefully handle the inevitable failures of distributed systems.

Next up: Securing WitRPC Communications: Encryption and Authentication, protecting the channel that resilience keeps alive.

This is part 10 of a series on WitRPC. See previous posts: Introducing WitRPC, Getting Started, Real-World Benefits, WitRPC vs. gRPC, Migrating to WitRPC, Under the Hood, Composite Services, ASP.NET Core Integration, and Performance Tuning.