Resilience & Health Checks

Connections drop, servers restart, transient errors happen. WitRPC ships two complementary mechanisms for surviving them, plus health check integration for observing connection state in production.

The two work independently and combine naturally: reconnection restores the channel, retries repeat the call.

Automatic reconnection

With auto-reconnection enabled, the client monitors its connection and re-establishes it after a drop:

var client = WitClientBuilder.Build(options =>
{
    options.WithTcp("server.example.com", 5000);
    options.WithJson();
    options.WithEncryption();
    options.WithAutoReconnect();   // defaults
});

Behavior is configurable:

options.WithAutoReconnect(reconnect =>
{
    reconnect.MaxAttempts = 10;                        // 0 = unlimited
    reconnect.InitialDelay = TimeSpan.FromSeconds(1);
    reconnect.MaxDelay = TimeSpan.FromMinutes(2);
    reconnect.BackoffMultiplier = 2.0;
    reconnect.ReconnectOnDisconnect = true;            // also reconnect when the server closes the connection
});

Delays grow exponentially between attempts (InitialDelay × BackoffMultiplier^n, capped at MaxDelay), so with the defaults the sequence runs 1s, 2s, 4s, 8s and onward up to two minutes. Backoff prevents a fleet of clients from hammering a server that just came back up.

Reconnection callbacks

Three callbacks expose the process, which is where applications refresh their state after an interruption:

options.WithAutoReconnect(reconnect =>
{
    reconnect.OnReconnecting = (attempt, delay) =>
        Console.WriteLine($"Reconnection attempt {attempt}, waiting {delay}...");

    reconnect.OnReconnected = () =>
    {
        Console.WriteLine("Reconnected.");
        // Refresh state the client may have missed while offline
    };

    reconnect.OnReconnectionFailed = lastException =>
    {
        Console.WriteLine($"Reconnection failed: {lastException?.Message}");
        // Alert the user, switch to offline mode
    };
});

Connection state and manual control

The client exposes its state through ConnectionState:

switch (client.ConnectionState)
{
    case ReconnectionState.Connected:     /* normal operation */          break;
    case ReconnectionState.Reconnecting:  /* attempts in progress */      break;
    case ReconnectionState.Disconnected:  /* idle, not reconnecting */    break;
    case ReconnectionState.Failed:        /* attempts exhausted */        break;
}

await client.StopReconnectionAsync() cancels ongoing attempts, which belongs in shutdown paths. WithoutAutoReconnect() disables the feature explicitly.

Retry policies

A retry policy repeats an individual call that failed with a retryable error:

options.WithRetryPolicy();   // defaults

options.WithRetryPolicy(retry =>
{
    retry.MaxRetries = 3;
    retry.InitialDelay = TimeSpan.FromMilliseconds(500);
    retry.MaxDelay = TimeSpan.FromSeconds(5);
    retry.BackoffMultiplier = 2.0;
    retry.BackoffType = BackoffType.Exponential;
});

BackoffType selects how delays grow between attempts: Fixed keeps them constant, Linear grows them arithmetically, Exponential (the default) doubles them under the configured multiplier.

Which failures count as retryable is configurable through RetryableStatuses (communication statuses that trigger a retry) and RetryableExceptionTypes (exception types that do). An OnRetry callback (Action<Exception?, int, TimeSpan>) fires before each attempt, useful for logging.

Retries and idempotency

A retry re-executes the call, so the server may run the method more than once: the first attempt can succeed on the server while its response is lost in transit. Enable retries for operations where repetition is harmless (reads, state-setting updates like SetStatus(x)) and keep them off, or design idempotency keys, for operations that must not run twice (payments, counters, appends). This is a property of your service semantics; the framework cannot decide it for you.

Health checks

The OutWit.Communication.Client.HealthChecks package plugs WitRPC clients into ASP.NET Core health checks. It works with clients registered through the dependency injection integration, referencing them by name:

builder.Services.AddHealthChecks()
    .AddWitRpcClient("MainClient");   // name of the DI-registered client

The check reports the named client's connection health through the standard health endpoint, alongside databases and other dependencies, so existing monitoring picks it up without special handling. Optional parameters set the check name, failure status, and tags. Registering named clients is covered in Dependency Injection →.

Putting it together

A production client typically enables all three:

var client = WitClientBuilder.Build(options =>
{
    options.WithTcp("server.example.com", 5000);
    options.WithMessagePack();
    options.WithEncryption();
    options.WithAccessToken(token);

    options.WithAutoReconnect(r =>
    {
        r.MaxAttempts = 0;                            // keep trying
        r.OnReconnected = () => RefreshApplicationState();
    });

    options.WithRetryPolicy(r =>
    {
        r.MaxRetries = 3;
        r.BackoffType = BackoffType.Exponential;
    });
});

Reconnection keeps the channel alive across outages, retries absorb transient call failures, and the health check makes both visible to monitoring. What remains application work: refreshing missed state in OnReconnected, and deciding which operations are safe to retry.