Deploying WitRPC Services to Production

witrpcdeploymentproductiondockerdotnet

You've built your WitRPC service. It works in development. Now it's time to deploy to production, where reliability, observability, and operability matter.

This post covers the operational aspects of running WitRPC services: containerization, lifecycle management, logging, monitoring, configuration, and troubleshooting production issues.

Server Lifecycle Management

Starting and Stopping

WitRPC servers have a simple lifecycle:

csharp
// Build the server
var server = WitServerBuilder.Build(options =>
{
    options.WithService(new MyService());
    options.WithTcp(5000, maxNumberOfClients: 100);
    options.WithMessagePack();
    options.WithEncryption();
});

// Start listening for connections
server.StartWaitingForConnection();

// ... server is running ...

// Graceful shutdown
server.StopWaitingForConnection();
server.Dispose();

Hosted Service Integration

In ASP.NET Core, use the hosted service pattern for proper lifecycle management:

csharp
// Program.cs
builder.Services.AddWitRpcServer<IOrderService, OrderService>(
    "order-server",
    options =>
    {
        options.WithTcp(5000, maxNumberOfClients: 100);
        options.WithMessagePack();
        options.WithEncryption();
    },
    autoStart: true  // Starts with the application
);

var app = builder.Build();
app.Run();

The autoStart: true parameter registers a hosted service that:

  • Starts the WitRPC server when the application starts
  • Stops gracefully when the application shuts down
  • Integrates with ASP.NET Core's lifetime events

Manual Lifecycle Control

For more control, use IWitServerFactory:

csharp
public class ServerController : ControllerBase
{
    private readonly IWitServerFactory _factory;
    
    public ServerController(IWitServerFactory factory)
    {
        _factory = factory;
    }
    
    [HttpPost("start")]
    public IActionResult StartServer()
    {
        var server = _factory.GetServer("order-server");
        server.StartWaitingForConnection();
        return Ok("Server started");
    }
    
    [HttpPost("stop")]
    public IActionResult StopServer()
    {
        var server = _factory.GetServer("order-server");
        server.StopWaitingForConnection();
        return Ok("Server stopped");
    }
}

Graceful Shutdown

Handle application shutdown properly:

csharp
var app = builder.Build();

// Register shutdown handler
app.Lifetime.ApplicationStopping.Register(() =>
{
    var logger = app.Services.GetRequiredService<ILogger<Program>>();
    logger.LogInformation("Application stopping, cleaning up WitRPC...");
    
    var factory = app.Services.GetRequiredService<IWitServerFactory>();
    var server = factory.GetServer("order-server");
    
    // Stop accepting new connections
    server.StopWaitingForConnection();
    
    // Give existing calls time to complete
    Thread.Sleep(5000);
    
    logger.LogInformation("WitRPC cleanup complete");
});

app.Run();

Configuration Management

Environment-Based Configuration

Use appsettings.json with environment overrides:

json
// appsettings.json
{
  "WitRpc": {
    "Server": {
      "Port": 5000,
      "MaxClients": 100,
      "Serializer": "MessagePack",
      "EnableEncryption": true
    }
  }
}

// appsettings.Production.json
{
  "WitRpc": {
    "Server": {
      "Port": 443,
      "MaxClients": 1000,
      "EnableEncryption": true
    }
  }
}
csharp
var witRpcConfig = builder.Configuration.GetSection("WitRpc:Server");

builder.Services.AddWitRpcServer<IOrderService, OrderService>(
    "order-server",
    options =>
    {
        var port = witRpcConfig.GetValue<int>("Port");
        var maxClients = witRpcConfig.GetValue<int>("MaxClients");
        
        options.WithTcp(port, maxNumberOfClients: maxClients);
        
        var serializer = witRpcConfig.GetValue<string>("Serializer");
        if (serializer == "MessagePack")
            options.WithMessagePack();
        else
            options.WithJson();
            
        if (witRpcConfig.GetValue<bool>("EnableEncryption"))
            options.WithEncryption();
    },
    autoStart: true
);

Secrets Management

Never store secrets in configuration files:

csharp
// Bad: secrets in appsettings.json
var token = configuration["WitRpc:AccessToken"];

// Good: environment variables
var token = Environment.GetEnvironmentVariable("WITRPC_ACCESS_TOKEN");

// Better: secret manager (development)
// dotnet user-secrets set "WitRpc:AccessToken" "your-secret"
var token = configuration["WitRpc:AccessToken"];

// Best: cloud secret manager (production)
// Azure Key Vault, AWS Secrets Manager, HashiCorp Vault
var token = await secretClient.GetSecretAsync("witrpc-access-token");

Containerization with Docker

Basic Dockerfile

dockerfile
# Dockerfile
FROM mcr.microsoft.com/dotnet/aspnet:8.0 AS base
WORKDIR /app
EXPOSE 5000

FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build
WORKDIR /src
COPY ["MyService/MyService.csproj", "MyService/"]
RUN dotnet restore "MyService/MyService.csproj"
COPY . .
WORKDIR "/src/MyService"
RUN dotnet build "MyService.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "MyService.csproj" -c Release -o /app/publish

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .

# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
    CMD curl -f http://localhost:8080/health || exit 1

ENTRYPOINT ["dotnet", "MyService.dll"]

Docker Compose for Development

yaml
# docker-compose.yml
version: '3.8'

services:
  order-service:
    build:
      context: .
      dockerfile: OrderService/Dockerfile
    ports:
      - "5001:5000"
    environment:
      - ASPNETCORE_ENVIRONMENT=Development
      - WITRPC_ACCESS_TOKEN=${WITRPC_TOKEN}
      - ConnectionStrings__Database=Server=db;Database=orders;User=sa;Password=${DB_PASSWORD}
    depends_on:
      - db
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 30s
      timeout: 10s
      retries: 3

  user-service:
    build:
      context: .
      dockerfile: UserService/Dockerfile
    ports:
      - "5002:5000"
    environment:
      - ASPNETCORE_ENVIRONMENT=Development
      - WITRPC_ACCESS_TOKEN=${WITRPC_TOKEN}
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 30s
      timeout: 10s
      retries: 3

  db:
    image: mcr.microsoft.com/mssql/server:2022-latest
    environment:
      - ACCEPT_EULA=Y
      - SA_PASSWORD=${DB_PASSWORD}
    volumes:
      - db-data:/var/opt/mssql

volumes:
  db-data:

Kubernetes Deployment

yaml
# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
  labels:
    app: order-service
spec:
  replicas: 3
  selector:
    matchLabels:
      app: order-service
  template:
    metadata:
      labels:
        app: order-service
    spec:
      containers:
      - name: order-service
        image: myregistry/order-service:latest
        ports:
        - containerPort: 5000
          name: witrpc
        - containerPort: 8080
          name: health
        env:
        - name: WITRPC_ACCESS_TOKEN
          valueFrom:
            secretKeyRef:
              name: witrpc-secrets
              key: access-token
        - name: ASPNETCORE_ENVIRONMENT
          value: "Production"
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "500m"
        livenessProbe:
          httpGet:
            path: /health/live
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 30
        readinessProbe:
          httpGet:
            path: /health/ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
  name: order-service
spec:
  selector:
    app: order-service
  ports:
  - name: witrpc
    port: 5000
    targetPort: 5000
  - name: health
    port: 8080
    targetPort: 8080
  type: ClusterIP

Logging

Structured Logging Setup

csharp
// Program.cs
builder.Logging.ClearProviders();
builder.Logging.AddConsole();
builder.Logging.AddJsonConsole(options =>
{
    options.JsonWriterOptions = new JsonWriterOptions
    {
        Indented = false  // Single-line JSON for log aggregators
    };
});

// Add Serilog for production
builder.Host.UseSerilog((context, config) =>
{
    config
        .ReadFrom.Configuration(context.Configuration)
        .Enrich.FromLogContext()
        .Enrich.WithMachineName()
        .Enrich.WithThreadId()
        .WriteTo.Console()
        .WriteTo.Seq("http://seq:5341");  // Or Elasticsearch, Splunk, etc.
});

WitRPC-Specific Logging

Enable WitRPC internal logging:

csharp
builder.Services.AddWitRpcServer(
    "order-server",
    context =>
    {
        context.WithService(new OrderService());
        context.WithTcp(5000, maxNumberOfClients: 100);
        context.WithMessagePack();
        
        // WithLogger takes Microsoft.Extensions.Logging.ILogger directly:
        // no adapters, no bridge classes
        context.WithLogger(
            context.ServiceProvider.GetRequiredService<ILogger<Program>>());
    },
    autoStart: true
);

Log Key Events

csharp
public class OrderService : IOrderService
{
    private readonly ILogger<OrderService> _logger;
    
    public async Task<Order> CreateOrderAsync(CreateOrderRequest request)
    {
        using var scope = _logger.BeginScope(new Dictionary<string, object>
        {
            ["CustomerId"] = request.CustomerId,
            ["OrderItems"] = request.Items.Count
        });
        
        _logger.LogInformation("Creating order for customer {CustomerId}", request.CustomerId);
        
        try
        {
            var order = await _repository.CreateAsync(request);
            
            _logger.LogInformation(
                "Order {OrderId} created successfully. Total: {Total:C}", 
                order.Id, 
                order.Total);
            
            return order;
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, 
                "Failed to create order for customer {CustomerId}", 
                request.CustomerId);
            throw;
        }
    }
}

Health Checks

ASP.NET Core Health Checks

csharp
// Program.cs
builder.Services.AddHealthChecks()
    // Basic liveness check
    .AddCheck("self", () => HealthCheckResult.Healthy(), tags: new[] { "live" })
    
    // WitRPC connection check from the HealthChecks package:
    // monitors the DI-registered client named "backend"
    .AddWitRpcClient("backend", tags: new[] { "ready" })
    
    // Database check
    .AddSqlServer(connectionString, tags: new[] { "ready" })
    
    // External dependency check
    .AddUrlGroup(new Uri("https://api.external.com/health"), "external-api", tags: new[] { "ready" });

var app = builder.Build();

// Liveness endpoint (is the process alive?)
app.MapHealthChecks("/health/live", new HealthCheckOptions
{
    Predicate = check => check.Tags.Contains("live")
});

// Readiness endpoint (is the service ready to handle requests?)
app.MapHealthChecks("/health/ready", new HealthCheckOptions
{
    Predicate = check => check.Tags.Contains("ready"),
    ResponseWriter = WriteHealthCheckResponse
});

// Detailed health endpoint (for debugging)
app.MapHealthChecks("/health", new HealthCheckOptions
{
    ResponseWriter = WriteHealthCheckResponse
});

Probing the Server Itself

csharp
// Server-side health is best observed the way clients observe it:
// through a connection. Register a loopback probe client against the
// server's own endpoint and let the packaged check monitor it.

builder.Services.AddWitRpcClient(
    "self-probe",
    options =>
    {
        options.WithTcp("127.0.0.1", 5000);   // the server's own endpoint
        options.WithMessagePack();
    },
    autoConnect: true);

builder.Services.AddHealthChecks()
    .AddWitRpcClient("self-probe", tags: new[] { "ready" });

Health Check Response Format

csharp
private static async Task WriteHealthCheckResponse(
    HttpContext context, 
    HealthReport report)
{
    context.Response.ContentType = "application/json";
    
    var response = new
    {
        status = report.Status.ToString(),
        duration = report.TotalDuration.TotalMilliseconds,
        checks = report.Entries.Select(e => new
        {
            name = e.Key,
            status = e.Value.Status.ToString(),
            duration = e.Value.Duration.TotalMilliseconds,
            description = e.Value.Description,
            data = e.Value.Data,
            exception = e.Value.Exception?.Message
        })
    };
    
    var options = new JsonSerializerOptions { WriteIndented = true };
    await context.Response.WriteAsync(JsonSerializer.Serialize(response, options));
}

Monitoring and Metrics

Prometheus Metrics

csharp
// Install: prometheus-net.AspNetCore

builder.Services.AddSingleton<WitRpcMetrics>();

var app = builder.Build();
app.UseHttpMetrics();
app.MapMetrics();  // Exposes /metrics endpoint
csharp
public class WitRpcMetrics
{
    private readonly Counter _requestsTotal;
    private readonly Histogram _requestDuration;
    private readonly Gauge _connectedClients;
    
    public WitRpcMetrics()
    {
        _requestsTotal = Metrics.CreateCounter(
            "witrpc_requests_total",
            "Total WitRPC requests",
            new CounterConfiguration
            {
                LabelNames = new[] { "service", "method", "status" }
            });
        
        _requestDuration = Metrics.CreateHistogram(
            "witrpc_request_duration_seconds",
            "WitRPC request duration",
            new HistogramConfiguration
            {
                LabelNames = new[] { "service", "method" },
                Buckets = new[] { .001, .005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10 }
            });
        
        _connectedClients = Metrics.CreateGauge(
            "witrpc_connected_clients",
            "Number of connected WitRPC clients",
            new GaugeConfiguration
            {
                LabelNames = new[] { "server" }
            });
    }
    
    public void RecordRequest(string service, string method, string status, double duration)
    {
        _requestsTotal.WithLabels(service, method, status).Inc();
        _requestDuration.WithLabels(service, method).Observe(duration);
    }
    
    public void SetConnectedClients(string server, int count)
    {
        _connectedClients.WithLabels(server).Set(count);
    }
}

Instrumented Service Wrapper

csharp
public class InstrumentedOrderService : IOrderService
{
    private readonly IOrderService _inner;
    private readonly WitRpcMetrics _metrics;
    private readonly ILogger<InstrumentedOrderService> _logger;
    
    public InstrumentedOrderService(
        IOrderService inner,
        WitRpcMetrics metrics,
        ILogger<InstrumentedOrderService> logger)
    {
        _inner = inner;
        _metrics = metrics;
        _logger = logger;
    }
    
    public async Task<Order> CreateOrderAsync(CreateOrderRequest request)
    {
        var stopwatch = Stopwatch.StartNew();
        var status = "success";
        
        try
        {
            return await _inner.CreateOrderAsync(request);
        }
        catch (Exception ex)
        {
            status = ex switch
            {
                ValidationException => "validation_error",
                NotFoundException => "not_found",
                _ => "error"
            };
            throw;
        }
        finally
        {
            stopwatch.Stop();
            _metrics.RecordRequest(
                "OrderService", 
                "CreateOrder", 
                status, 
                stopwatch.Elapsed.TotalSeconds);
        }
    }
    
    // ... other methods similarly instrumented
}

Scaling Considerations

Horizontal Scaling

WitRPC servers maintain persistent connections with clients. For horizontal scaling:

Considerations:

  • Use sticky sessions for persistent connections
  • Or use connection-aware load balancing
  • Events are only broadcast to clients connected to that server instance

Connection-Aware Architecture

For events that must reach all clients across servers:

csharp
// Use a message bus for cross-server events
public class DistributedOrderService : IOrderService
{
    private readonly IOrderRepository _repository;
    private readonly IMessageBus _messageBus;  // Redis, RabbitMQ, etc.
    
    public event Action<Order> OrderCreated = delegate { };
    
    public DistributedOrderService(
        IOrderRepository repository,
        IMessageBus messageBus)
    {
        _repository = repository;
        _messageBus = messageBus;
        
        // Subscribe to events from other servers
        _messageBus.Subscribe<OrderCreatedMessage>(msg =>
        {
            OrderCreated(msg.Order);  // Notify local clients
        });
    }
    
    public async Task<Order> CreateOrderAsync(CreateOrderRequest request)
    {
        var order = await _repository.CreateAsync(request);
        
        // Publish to all servers
        await _messageBus.PublishAsync(new OrderCreatedMessage { Order = order });
        
        return order;
    }
}

Troubleshooting Production Issues

Common Issues and Solutions

1. Connection Refused

Symptom: Clients can't connect
Causes:
- Server not started
- Wrong port/address
- Firewall blocking
- Max clients reached

Debug:
- Check server logs for "Started listening"
- Verify port is open: netstat -tlnp | grep 5000
- Check firewall rules
- Track your own connected-session count against the configured maxNumberOfClients

2. Connection Drops

Symptom: Clients disconnect unexpectedly
Causes:
- Network instability
- Server restart
- Memory pressure (OOM kill)
- Idle timeout

Debug:
- Check server logs for disconnection events
- Monitor memory usage
- Enable auto-reconnect on clients
- Check load balancer idle timeout settings

3. Slow Responses

Symptom: RPC calls taking too long
Causes:
- Service implementation slow
- Serialization overhead
- Network latency
- Thread pool exhaustion

Debug:
- Add timing logs to service methods
- Monitor request duration metrics
- Check thread pool stats
- Profile service implementation

4. Memory Leaks

Symptom: Memory grows over time
Causes:
- Event handler leaks
- Large objects not disposed
- Connection not cleaned up

Debug:
- Monitor GC metrics
- Take memory dumps and analyze
- Ensure Dispose() called on connections
- Check event subscription/unsubscription pairs

Diagnostic Endpoints

Add diagnostic endpoints for troubleshooting:

csharp
app.MapGet("/debug/server", (IWitServerFactory factory) =>
{
    var server = factory.GetServer("order-server");
    return new
    {
        server.Id,
        server.Name,
        server.Description
    };
});

// Connection-level state is best tracked in your own service code:
// count clients through your service's session logic, or watch the
// self-probe client's ConnectionState from the health checks section.

app.MapGet("/debug/config", (IConfiguration config) =>
{
    // Return non-sensitive configuration
    return new
    {
        port = config["WitRpc:Server:Port"],
        maxClients = config["WitRpc:Server:MaxClients"],
        serializer = config["WitRpc:Server:Serializer"],
        environment = Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT")
    };
});

Production Checklist

Before going to production, verify:

Configuration

  • Encryption enabled
  • Authentication configured
  • Secrets stored securely (not in code/config files)
  • Environment-specific settings configured
  • Connection limits set appropriately

Resilience

  • Auto-reconnect enabled on clients
  • Retry policies configured
  • Graceful shutdown implemented
  • Health checks configured

Observability

  • Structured logging enabled
  • Log aggregation configured
  • Metrics exposed
  • Alerts configured for key metrics

Operations

  • Docker/container configuration tested
  • Resource limits set (CPU, memory)
  • Liveness and readiness probes configured
  • Scaling strategy defined
  • Backup/recovery plan documented

Security

  • TLS configured for network transport
  • Access tokens rotated regularly
  • Network policies restrict access
  • Security audit completed

Conclusion

Running WitRPC in production requires attention to:

  • Lifecycle management: Proper startup and graceful shutdown
  • Configuration: Environment-based settings, secure secrets
  • Containerization: Docker, Kubernetes deployment
  • Observability: Logging, health checks, metrics
  • Resilience: Auto-reconnect, retry, graceful degradation
  • Scaling: Horizontal scaling considerations

With these operational practices in place, your WitRPC services will run reliably in production.

Next up: Building Real-Time Dashboards with WitRPC Events, putting server push to work.

This is part 12 of a series on WitRPC. See previous posts: Introducing WitRPC, Getting Started, Real-World Benefits, WitRPC vs. gRPC, Migrating to WitRPC, Under the Hood, Composite Services, ASP.NET Core Integration, Performance Tuning, Resilience, and Security.