1 Introduction: The Hidden Dangers of Blind Retries in Distributed Systems
A checkout API calls inventory. Inventory slows down, so checkout retries. The retry joins the same queue as the original request, which may still be executing. More checkout requests arrive, more retries follow, and inventory spends its remaining capacity processing work that customers have already abandoned.
The retry code works exactly as configured. The system fails because that configuration assumes another attempt is cheap.
Building resilient ASP.NET Core APIs with Polly starts with a different question: under what conditions is another attempt worth its cost? Answering that requires deadlines, admission control, failure classification, and business semantics alongside the retry strategy.
1.1 The optimistic fallacy of network reliability and transient faults
A transient fault is temporary, but its duration matters. A connection reset might disappear before the next attempt. A saturated database can remain saturated until callers reduce pressure. Treating both as reasons for immediate retries confuses recoverable transport failure with sustained capacity failure.
An HTTP timeout is especially ambiguous. It means the caller did not observe completion within its budget. The server may have rejected the request, committed the transaction, or continued processing after the client disconnected.
That uncertainty has two consequences. First, retries need evidence that another attempt has a reasonable chance of succeeding. Second, writes need a way to recognize repeated intent. Backoff addresses timing; idempotency addresses correctness. Neither replaces the other.
1.2 How default retry mechanisms mask underlying infrastructure degradation
Suppose first-attempt failures climb while final success remains stable. A dashboard showing only endpoint success can make the deployment look healthy, even though every successful operation now consumes more network traffic and downstream capacity.
The hidden cost eventually appears as higher tail latency, larger connection queues, and more timeouts. At that point, increasing retry counts can make the graph look better briefly while moving the dependency closer to collapse.
Inventory every retry owner: browser, gateway, service mesh, application client, gRPC channel, database execution strategy, and message consumer. Defaults in these components are independent. They do not coordinate an end-to-end attempt budget.
Measure logical requests and physical attempts separately. A successful third attempt should remain visible as a recovered failure, with its additional latency and resource cost.
1.3 The shift from simple availability to fault tolerance and graceful degradation
Availability measured as HTTP 200 responses is incomplete. Returning an empty cart after a pricing failure may satisfy that metric while violating the product contract.
Define acceptable degradation per capability. Recommendations can disappear. Product descriptions might come from a timestamped cache. Payment authorization must preserve an explicit pending or failed state rather than invent success.
A resilient endpoint makes bounded promises: it limits time spent, limits work admitted, and returns a result whose meaning remains trustworthy. Sometimes the right result is an immediate 503. Sometimes it is a documented stale read. Sometimes it is a durable operation identifier that the caller can query later.
1.4 Target architecture overview: ASP.NET Core microservices communicating via HTTP/gRPC
Use .NET 10 LTS, ASP.NET Core 10, and Polly v8 APIs for this implementation. Add Microsoft.Extensions.Http.Resilience for normal HTTP integration, Polly.Extensions for registered pipelines, and Polly.RateLimiting for outbound concurrency controls. OpenTelemetry supplies observation; Aspire helps compose and inspect the distributed application.
The running example is an Orders API reading inventory over HTTP. Payment processing remains a separate write workflow with durable idempotency. Two configured tenants have separate inventory circuits, while one shared admission limit protects the Orders process.
For gRPC, use channel-level retry configuration and deadlines, with method-specific safety decisions. Do not assume an HTTP handler understands gRPC application status: HTTP 200 can carry a failed RPC in its trailers. Avoid adding Polly retries around a gRPC method that already retries internally.
2 The Anatomy of a Resilience Cascade: Why Retries Amplify Outages
Retries create additional demand precisely when a dependency may have less capacity. The dangerous part is the feedback loop: slower processing increases concurrency, higher concurrency increases contention, and contention creates more failures eligible for retry.
2.1 The mechanics of a retry storm and the thundering herd problem
Imagine thousands of callers receive a 503 during a deployment. Each waits exactly 200 milliseconds and retries. The dependency sees a synchronized second wave, often before replacement instances are ready.
Exponential backoff reduces retry frequency but does not remove synchronization. Identical clients starting together can still retry together at increasing intervals. Jitter spreads those attempts across time, reducing the concentration of arrivals.
A retry storm need not produce an obvious traffic spike. Requests can accumulate in client-side queues while outbound throughput stays flat. The storm then appears as memory growth, increasing request age, and long recovery after the dependency becomes healthy.
Control both attempts and outstanding work. Jitter changes arrival timing; a concurrency limit bounds occupancy; a circuit breaker stops repeatedly probing a dependency with enough recent evidence of failure.
2.2 Resource exhaustion: Thread pool starvation, socket depletion, and connection queues
An awaited HTTP operation does not reserve a worker thread while waiting for network I/O. Slow asynchronous requests still consume memory, cancellation registrations, connection or stream capacity, and application bookkeeping.
Thread pool starvation commonly enters through blocking: .Result, .Wait(), synchronous database calls, or lock contention. Retrying those paths creates more blocked work. Adding Task.Run around synchronous waits usually moves the problem into another queue.
Socket exhaustion is a separate failure mode. Creating and disposing a new HttpClient for every attempt can churn connections and consume ephemeral ports. Use IHttpClientFactory or a deliberately managed long-lived client with appropriate connection lifetime settings.
Connection pooling does not eliminate queuing. HTTP/1.1 connection limits and HTTP/2 stream limits can make attempts wait before useful work begins. Monitor time waiting for a connection alongside network duration. A request that times out in a local queue never tested downstream recovery.
2.3 Cascading failures across upstream and downstream service boundaries
Suppose Orders calls Inventory, which calls Warehouse. If Orders retries Inventory and Inventory retries Warehouse, a single customer request can produce many Warehouse requests.
The failure then propagates backward. Orders retains requests longer, its own callers time out, and those callers start their retries. Services that do not call Warehouse can suffer through shared thread pools, database connections, or process memory.
Assign retry ownership deliberately. Prefer the layer with enough context to classify the failure and determine replay safety. Other layers should propagate deadlines, reject excess work, and report clear errors.
Also propagate cancellation. An abandoned parent should stop waiting, retrying, and starting new child work. Cancellation cannot undo committed business effects, but it can prevent additional unnecessary work.
2.4 Mathematical modeling of traffic amplification under compounding retry loads
Let each logical call permit r retries, giving at most a = r + 1 attempts. During a complete failure, one retrying layer can multiply demand by a.
For a chain of d retrying layers, each making one downstream call per attempt, the worst-case leaf amplification is:
A = product(a_i), for i = 1 ... dThree layers allowing two retries each can produce 3 × 3 × 3 = 27 leaf attempts per original request. This is an upper bound before deadlines, circuit breakers, admission limits, and transport behavior intervene. Fan-out adds further demand.
For a simpler probabilistic model, assume each attempt fails independently with probability p. The expected attempts at one layer are:
E[N] = 1 + p + p^2 + ... + p^r
= (1 - p^(r+1)) / (1 - p), when p != 1With two retries and p = 0.2, expected amplification is 1.24. At p = 0.9, it is 2.71. Independence is a poor assumption during a shared outage: retry outcomes are correlated, and added load can increase p. Use this model for reasoning, not capacity guarantees.
Little’s Law adds the occupancy problem: average in-flight work is approximately arrival rate multiplied by average time in the system. At a stable 200 requests per second, raising average duration from 100 milliseconds to two seconds raises average occupancy from 20 to 400, even before accounting for additional attempts.
A retry budget limits aggregate extra work. For example, permit retry attempts up to a configured fraction of original attempts over a moving interval. This is a design target, not a built-in guarantee from setting MaxRetryAttempts. Polly’s per-call ceiling still needs a shared budget mechanism when fleet-wide amplification matters.
3 Modern .NET Resilience Stack: Choosing the Right Tools and Libraries
Choose integration boundaries before choosing configuration values. Most HTTP clients should use Microsoft’s HTTP resilience integration. Direct Polly pipelines are useful when the protected operation or resource lifetime extends beyond ordinary HTTP handler behavior.
3.1 Overview of Polly v8 architecture: ResiliencePipeline, strategies, and functional composition
Polly v8 builds reusable ResiliencePipeline instances from ordered strategies. Use ResiliencePipeline<T> when decisions depend on a returned value, such as HttpResponseMessage.StatusCode.
The first strategy added is outermost. A total timeout outside retry includes retry delays. An attempt timeout inside retry applies separately to each execution. A breaker inside retry observes individual attempts; outside retry, it observes the final logical outcome.
Pipelines can be shared across concurrent executions. That matters because circuits and limiters hold state. Building a pipeline inside every request prevents meaningful shared failure sampling and can multiply resource limits.
The execution callback is equally important. Polly repeats that callback; it does not make arbitrary streams replayable, create database transactions, or infer whether a payment is safe to repeat.
3.2 Integration with Microsoft.Extensions.Resilience and Microsoft Aspire
Microsoft.Extensions.Resilience provides general integration around Polly. Microsoft.Extensions.Http.Resilience adds HTTP-specific strategies and client registration. Prefer these packages over copying older Policy.WrapAsync examples into a new v8 implementation.
Aspire’s C# service defaults can configure HTTP resilience, telemetry, health endpoints, and service discovery. Inspect the generated defaults before customizing individual clients: a global standard handler may already be present.
Avoid stacking a second retry handler on top of that default. For a client requiring a custom handler, remove inherited resilience handlers with RemoveAllResilienceHandlers() before adding the replacement.
Aspire improves composition and visibility. It does not decide retry ownership or supply business idempotency. Those remain application architecture decisions.
3.3 Dependency injection patterns for typed HTTP clients vs. resilience pipelines
A typed client provides a dependency-specific API and centralizes serialization, authentication, and endpoint configuration. It is usually the cleanest boundary for Inventory or Payments.
For a read-oriented client, a starting configuration is:
using Microsoft.Extensions.Http.Resilience;
builder.Services.AddHttpClient<InventoryClient>()
.AddStandardResilienceHandler(options =>
{
options.Retry.MaxRetryAttempts = 1;
options.Retry.DisableForUnsafeHttpMethods();
});This disables retries for methods including POST, PATCH, PUT, and DELETE. Safe HTTP semantics are a starting point; verify the actual endpoint contract too. A badly designed GET that mutates state is still unsafe to replay.
Use a registered Polly pipeline when protecting a custom operation, sharing state across handlers, or controlling the full buffered response lifetime. Resolve it through ResiliencePipelineProvider<TKey> rather than rebuilding it. Avoid capturing a transient typed client in a singleton service.
3.4 Open-source ecosystem companions: YARP, OpenTelemetry metrics, and distributed tracing
YARP provides an ASP.NET Core reverse-proxy boundary for routing and forwarding. Document whether your gateway, ingress, or mesh performs retries, including any custom forwarding logic. Do not assume proxy configuration and application retry configuration are coordinated.
OpenTelemetry should expose both the customer operation and dependency attempts. Use traces to answer which hop consumed the deadline, and metrics to identify amplification and saturation across the fleet.
Keep labels bounded: dependency name, operation name, status category, and configured partition are useful. Order identifiers, arbitrary tenant IDs, and full request URLs can create excessive metric cardinality. Put detailed identifiers in appropriately controlled logs or traces instead.
4 Core Resilience Strategies with Polly v8: Beyond Basic Retries
Each strategy answers a different question. Timeouts bound waiting, retries permit another attempt, breakers react to failure evidence, and fallback defines acceptable behavior when the primary path cannot complete.
4.1 Timeouts: Fast-failing hung requests to release resources promptly
Start with the caller’s end-to-end deadline. Reserve time for authentication, application work, response serialization, and any remaining dependency calls. The outbound budget is what remains, not an independently chosen timeout.
Within that budget, distinguish total operation timeout from per-attempt timeout. A two-second total budget might allow two 700-millisecond attempts and a short retry delay. It cannot guarantee all configured attempts will run.
Polly timeouts use cooperative cancellation. The callback must pass the supplied token to HTTP calls, stream reads, and other cancellable operations. A timeout does not terminate a thread or forcibly stop arbitrary code. If the callback ignores cancellation, observed completion can exceed the configured timeout.
For asynchronous HTTP, there usually is no blocked execution thread to free. The benefit is bounding outstanding work and releasing resources when cancellation is honored. Avoid overlapping, poorly understood HttpClient.Timeout and Polly timers; the example later makes Polly the outbound timeout owner.
4.2 Intelligent backoff algorithms: Fixed, exponential, and jitter
Fixed delay is simple and occasionally appropriate for a known recovery interval. Exponential backoff spaces repeated attempts further apart, which helps when recovery needs time. Neither justifies additional attempts when the remaining deadline is insufficient.
Full jitter typically samples uniformly between zero and a capped exponential delay. Decorrelated approaches vary successive delays to reduce synchronization. Polly v8’s exponential backoff with UseJitter = true uses the Decorrelated Jitter Backoff V2 algorithm; it is not interchangeable with every algorithm called decorrelated jitter.
Avoid implementing custom randomness unless you need specific behavior. Test distributions and bounds rather than asserting that each delay is larger than the previous one. Jittered sequences can decrease.
Respect server-directed waiting. Retry-After can contain a duration or an HTTP date. If the indicated wait exceeds the remaining request budget, return or defer the operation rather than shortening that wait to fit. A custom DelayGenerator must enforce its own bounds; do not assume MaxDelay caps a generated value.
4.3 Circuit breakers: Managing Closed, Open, and Half-Open states with sampling windows
A closed circuit permits calls and records outcomes. When the configured failure ratio and minimum throughput are met within the sampling window, it opens and rejects further executions. After the break duration, an arriving call can probe recovery in the half-open state.
A successful probe closes the circuit; a handled failure reopens it. This is request-driven recovery, not a background health poll. A circuit may remain observationally quiet until another call arrives.
Tune minimum throughput for the partition’s actual traffic. A tenant receiving five calls per minute will not open a circuit requiring dozens of observations within a short window, even if every request fails.
Keep business rejection separate from dependency failure. Invalid input, missing products, and authorization denial generally should not open an availability circuit. Decide explicitly whether throttling is tenant-specific or evidence of shared overload.
Circuit state is local to a pipeline instance. Twenty service replicas can probe independently. Do not assume one open circuit protects the whole fleet or cancels requests already in flight.
4.4 Fallbacks and hedging: Degraded experiences or redundant requests
Fallback belongs near the business contract. Returning cached stock information may be acceptable for browsing, but not for confirming an inventory reservation. Include freshness and source information when clients need it to interpret the result.
Do not hide every exception behind an HTTP 200 fallback. Record that degradation happened, and preserve failure semantics for operations that cannot safely degrade.
Hedging starts another attempt before the first has finished. It can reduce tail latency for safe reads when redundant destinations have genuinely independent failure modes and spare capacity.
Hedging against the same saturated backend adds concurrent work immediately. Keep hedges within attempt and concurrency budgets, cancel losing requests, and remember that cancellation may not stop server execution. Streaming responses and writes need special treatment; a generic hedging strategy does not make either safely replayable.
5 Safeguarding State and Consistency: Idempotency in High-Frequency Retries
The hardest retry failure is a successful write whose response disappears. Transport behavior alone cannot distinguish that outcome from a write that never reached the server.
5.1 The danger of retrying non-idempotent operations
Consider a payment authorization that commits before the connection breaks. A second POST can authorize the payment again. Increasing timeouts merely changes how often this uncertainty appears.
HTTP method semantics help, but business effects determine safety. A PUT may be idempotent for the resource state while a poorly implemented notification hook sends another email on each invocation.
Classify commands separately from queries. Reads can often tolerate bounded retry. Commands require a stable operation identity and an explicit duplicate-handling contract. Some operations should return an uncertain or pending state and require reconciliation instead of immediate replay.
5.2 Architectural patterns for Idempotency Keys in ASP.NET Core
Require a client-generated idempotency key for sensitive commands. The client must reuse it when retrying the same intent, including retries after process restart. Creating a new key for each HTTP attempt defeats the protocol.
Scope the key by authenticated tenant and operation. Store a hash of the meaningful request payload so the same key cannot silently represent different commands.
A durable record commonly contains:
| Field | Purpose |
|---|---|
| Tenant, operation, key | Uniquely identify intent |
| Request hash | Detect incompatible reuse |
| Processing state | Distinguish pending and completed work |
| Result reference or response | Reconstruct the original outcome |
| Creation and expiry times | Enforce the retention contract |
For local database changes, insert the record, perform the business mutation, and save the replayable outcome in one transaction. On duplicate submission, verify the hash and return the recorded result. If processing is still underway, return a documented pending response rather than starting the work again.
Retain records for at least the supported retry and reconciliation horizon. Deleting a key while clients can still retry reintroduces duplicate effects.
5.3 Distributed locking and database unique constraint enforcement
A read-then-insert check races across replicas. Enforce uniqueness in the database:
CREATE UNIQUE INDEX UX_Idempotency_Scope
ON IdempotencyRecords (TenantId, OperationName, IdempotencyKey);The exact column types and bounded key lengths depend on the database. Handle only the expected unique-constraint violation as a duplicate; unrelated persistence failures should remain failures.
A distributed lock can reduce duplicate computation, but leases expire and processes pause. A lock alone does not prove exclusive ownership forever. Durable constraints, transaction boundaries, and, where needed, fencing tokens establish the correctness boundary.
After a duplicate insert failure, some databases require rolling back the failed transaction before reading the existing record. Plan that control flow explicitly rather than continuing on a transaction that is no longer usable.
5.4 Handling partial failures and compensating transactions: Saga pattern overview
A database transaction cannot atomically commit a local order and an unrelated payment provider’s authorization. Model the workflow as durable steps with operation identities and recoverable state.
An outbox can atomically record the order change and the intent to publish a command. A worker delivers that command, potentially more than once. The receiving service needs its own duplicate protection, and an external payment provider should receive a stable provider-supported idempotency key.
If inventory reservation succeeds but payment permanently fails, a saga can issue an idempotent release command. Compensation is another business operation, not a literal rollback of distributed time. It can fail and require retries or reconciliation itself.
Keep brief network retries inside a step. Let durable workflow processing own recovery across minutes or hours, with bounded redelivery and an operator-visible terminal state.
6 Advanced Flow Control: Concurrency Limits, Rate Limiting, and Bulkheads
Retries and circuit breakers do not establish a maximum amount of admitted work. Flow control supplies that missing boundary and protects healthy capabilities while a dependency struggles.
6.1 Bulkhead isolation: Partitioning resource pools
A bulkhead limits how much shared capacity one workload can occupy. Separate inventory lookups from payment calls and background exports, even when they run in the same service.
In Polly v8, use concurrency limiters for this style of isolation rather than copying v7 bulkhead APIs. Combine them with separate database pools or worker queues where the underlying bottleneck requires stronger separation.
Partition carefully. One limiter per tenant helps fairness, but thousands of independently allowed tenants can exceed process capacity. Use a global ceiling above bounded partitions. Likewise, separate circuits only when tenants use meaningfully separate downstream failure domains; otherwise shared failure evidence may be more useful.
6.2 Concurrency limits vs. rate limiting: Protecting downstream services
A concurrency limit bounds simultaneous work. A rate limiter bounds arrivals over time. Neither implies the other: a service can sustain a high request rate when operations are cheap and become overloaded at the same rate when latency rises.
Use concurrency limits to protect finite execution capacity. Use time-based limits for quotas, traffic shaping, and provider request-per-second contracts. Apply both when both constraints exist.
Queueing is a conscious latency trade-off. A short queue can absorb bursts; an unbounded queue converts overload into stale work and memory growth. For latency-sensitive dependency calls, immediate rejection is often easier to operate.
Placement changes meaning. A limiter outside retry counts logical operations, including backoff time. A limiter inside retry counts active attempts and releases capacity between attempts. The implementation below uses both scopes for different purposes.
6.3 ASP.NET Core built-in rate limiting middleware integration
Protect inbound work independently of outbound dependencies. This named policy caps concurrent requests for selected endpoints on each replica:
using Microsoft.AspNetCore.RateLimiting;
builder.Services.AddRateLimiter(options =>
{
options.RejectionStatusCode = StatusCodes.Status429TooManyRequests;
options.AddConcurrencyLimiter("orders", limiter =>
{
limiter.PermitLimit = 100;
limiter.QueueLimit = 0;
});
});
// After app creation, before endpoint execution:
app.UseRateLimiter();
// Apply .RequireRateLimiting("orders") to the selected endpoints.This is a local admission policy, not a distributed tenant quota. A true authenticated tenant policy should derive its partition from verified identity, not an arbitrary header. Authentication must run before that identity-dependent limiter.
Do not retry local limiter rejection inside the same API call. Return overload clearly. An inbound tenant quota often maps to 429; inability to use a required downstream dependency often maps to 503. Neither should become an accidental generic 500.
6.4 Adaptive concurrency control based on server load and telemetry
A fixed limit is a defensible baseline. Adaptive control becomes useful when safe concurrency changes with dependency latency, instance size, or workload mix.
CPU and memory alone are insufficient signals. An I/O-bound dependency can be saturated while application CPU remains low. Include in-flight attempts, connection queue duration, downstream latency, and rejection rates.
Use a slow control loop with minimum and maximum limits, hysteresis, and conservative increases. Reduce admission quickly when queueing becomes harmful, then recover gradually. Avoid having every replica react identically to noisy measurements.
Polly does not automatically turn telemetry into adaptive capacity. Treat the controller as a separately tested component. Replacing an entire pipeline on each adjustment can discard breaker history or leave overlapping old and new limiters, temporarily increasing effective capacity.
7 Production-Grade Implementation: End-to-End ASP.NET Core Architecture Example
The example implements a bounded inventory read path. It deliberately supports only GET and HEAD with no request body, buffers small responses inside the attempt timeout, and gives two configured tenant partitions separate circuit state.
This narrower contract makes replay and ownership explicit. For general-purpose HTTP resilience, prefer the built-in handler. Use this custom approach when you need these response-lifetime and partitioning decisions under application control.
7.1 Setting up a modular ResiliencePipelineBuilder registry
Create a .NET 10 web project and add the required packages:
dotnet new web -n ResilientOrders --framework net10.0
cd ResilientOrders
dotnet add package Polly.Extensions --version '8.*'
dotnet add package Polly.RateLimiting --version '8.*'
dotnet add package Microsoft.Extensions.Http.Resilience --version '10.*'
dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Instrumentation.Http
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocolResolve and lock exact package versions in the repository before deployment. The ranges identify compatible major-version families, not a reproducible production restore.
Put the following imports in GlobalUsings.cs; the remaining C# blocks in this section belong to the same project:
global using Microsoft.AspNetCore.Diagnostics;
global using Microsoft.AspNetCore.Diagnostics.HealthChecks;
global using Microsoft.Extensions.Diagnostics.HealthChecks;
global using Microsoft.Extensions.Http.Resilience;
global using OpenTelemetry.Metrics;
global using OpenTelemetry.Trace;
global using Polly;
global using Polly.CircuitBreaker;
global using Polly.Registry;
global using Polly.Retry;
global using Polly.Timeout;Register a global logical-operation limit and named per-tenant pipelines:
public sealed class InventoryCircuits
{
public IReadOnlyDictionary<string, CircuitBreakerStateProvider> States
{ get; } = new Dictionary<string, CircuitBreakerStateProvider>
{
["north"] = new(),
["south"] = new()
};
}
public static class InventoryResilience
{
public static bool IsUnavailable(HttpResponseMessage response) =>
(int)response.StatusCode is 502 or 503 or 504;
public static bool IsConnectionFailure(HttpRequestException error) =>
error.HttpRequestError == HttpRequestError.ConnectionError;
public static void Register(IServiceCollection services)
{
var circuits = new InventoryCircuits();
services.AddSingleton(circuits);
services.AddResiliencePipeline<string, HttpResponseMessage>(
"inventory-global", pipeline =>
pipeline.AddConcurrencyLimiter(64, 0));
foreach (var entry in circuits.States)
{
var state = entry.Value;
services.AddResiliencePipeline<string, HttpResponseMessage>(
$"inventory-{entry.Key}", pipeline =>
{
pipeline.AddTimeout(TimeSpan.FromSeconds(2));
pipeline.AddRetry(new RetryStrategyOptions<HttpResponseMessage>
{
MaxRetryAttempts = 1,
Delay = TimeSpan.FromMilliseconds(100),
BackoffType = DelayBackoffType.Exponential,
UseJitter = true,
ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
.Handle<HttpRequestException>(IsConnectionFailure)
.Handle<TimeoutRejectedException>()
.HandleResult(response => IsUnavailable(response)
&& !response.Headers.Contains("Retry-After")),
OnRetry = static args =>
{
args.Outcome.Result?.Dispose();
return default;
}
});
pipeline.AddConcurrencyLimiter(32, 0);
pipeline.AddCircuitBreaker(
new CircuitBreakerStrategyOptions<HttpResponseMessage>
{
FailureRatio = 0.5,
MinimumThroughput = 20,
SamplingDuration = TimeSpan.FromSeconds(10),
BreakDuration = TimeSpan.FromSeconds(5),
StateProvider = state,
ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
.Handle<HttpRequestException>(IsConnectionFailure)
.Handle<TimeoutRejectedException>()
.HandleResult(IsUnavailable)
});
pipeline.AddTimeout(TimeSpan.FromMilliseconds(700));
});
}
}
}The breaker observes physical attempts because it is inside retry. Admission rejection is outside the breaker and is not classified for retry. The total timeout includes retry backoff, both attempts, and bounded buffering.
Responses containing Retry-After are intentionally returned without internal retry. That conservative choice avoids sending an early retry when the requested delay exceeds this short interactive budget. The breaker can still count the 503 as dependency failure.
The partition list is finite. Do not construct registry keys from arbitrary caller input: unbounded keys create unbounded state. For tenants sharing one inventory cluster, consider a shared dependency circuit plus separate fairness limits instead.
7.2 Writing a complete custom ASP.NET Core DelegatingHandler
The handler creates a fresh request for every attempt. Reusing the original request through HttpClient.SendAsync is not a safe general replay mechanism, and stream content cannot be assumed repeatable.
public sealed class InventoryReadHandler(
ResiliencePipelineProvider<string> pipelines) : DelegatingHandler
{
public static readonly HttpRequestOptionsKey<string> Partition =
new("InventoryPartition");
protected override async Task<HttpResponseMessage> SendAsync(
HttpRequestMessage request, CancellationToken cancellationToken)
{
if ((request.Method != HttpMethod.Get && request.Method != HttpMethod.Head)
|| request.Content is not null)
throw new InvalidOperationException("Inventory client permits bodyless reads only.");
if (!request.Options.TryGetValue(Partition, out var partition)
|| (partition != "north" && partition != "south"))
throw new InvalidOperationException("Unknown inventory partition.");
var global = pipelines.GetPipeline<HttpResponseMessage>("inventory-global");
var tenant = pipelines.GetPipeline<HttpResponseMessage>($"inventory-{partition}");
return await global.ExecuteAsync(
async outerToken => await tenant.ExecuteAsync(
async attemptToken =>
{
using var attempt = Clone(request);
var response = await base.SendAsync(attempt, attemptToken);
try
{
await BufferAsync(response, attemptToken);
response.RequestMessage = request;
return response;
}
catch
{
response.Dispose();
throw;
}
}, outerToken), cancellationToken);
}
private static HttpRequestMessage Clone(HttpRequestMessage source)
{
var clone = new HttpRequestMessage(source.Method, source.RequestUri)
{
Version = source.Version,
VersionPolicy = source.VersionPolicy
};
foreach (var header in source.Headers)
clone.Headers.TryAddWithoutValidation(header.Key, header.Value);
foreach (var option in source.Options)
clone.Options.Set(new HttpRequestOptionsKey<object?>(option.Key), option.Value);
return clone;
}
private static async Task BufferAsync(
HttpResponseMessage response, CancellationToken token)
{
const int maximumBytes = 1024 * 1024;
var original = response.Content;
await using var stream = await original.ReadAsStreamAsync(token);
using var buffer = new MemoryStream();
var chunk = new byte[8192];
int read;
while ((read = await stream.ReadAsync(chunk.AsMemory(), token)) != 0)
{
if (buffer.Length + read > maximumBytes)
throw new InvalidDataException("Inventory response exceeds 1 MiB.");
await buffer.WriteAsync(chunk.AsMemory(0, read), token);
}
var replacement = new ByteArrayContent(buffer.ToArray());
foreach (var header in original.Headers)
replacement.Headers.TryAddWithoutValidation(header.Key, header.Value);
response.Content = replacement;
original.Dispose();
}
}Discarded responses are disposed before retry; the caller owns the final response. Exceptions during buffering dispose the current response too. The supplied attempt token covers transport and body reading, so a server that sends headers and then stalls cannot escape the attempt timeout.
Buffering trades streaming efficiency for a bounded, replayable read contract. The one-mebibyte limit prevents unbounded payload growth, but buffers and copies still create allocation pressure. For large downloads or streaming RPCs, use a separate client and an explicit streaming deadline instead.
Wire the client and endpoint in Program.cs:
var builder = WebApplication.CreateBuilder(args);
InventoryResilience.Register(builder.Services);
builder.Services.AddTransient<InventoryReadHandler>();
builder.Services.AddExceptionHandler<DependencyExceptionHandler>();
builder.Services.AddProblemDetails();
builder.Services.AddHttpClient("inventory", client =>
{
client.BaseAddress = new Uri(builder.Configuration["Inventory:BaseUrl"]
?? "http://localhost:8081");
client.Timeout = Timeout.InfiniteTimeSpan;
})
.RemoveAllResilienceHandlers()
.ConfigurePrimaryHttpMessageHandler(() => new SocketsHttpHandler
{
MaxConnectionsPerServer = 64,
PooledConnectionLifetime = TimeSpan.FromMinutes(5)
})
.AddHttpMessageHandler<InventoryReadHandler>();
builder.Services.AddHealthChecks()
.AddCheck<InventoryCircuitHealthCheck>("inventory-circuits",
tags: new[] { "dependency" });
builder.Services.AddOpenTelemetry()
.WithMetrics(metrics => metrics.AddMeter("Polly")
.AddAspNetCoreInstrumentation().AddHttpClientInstrumentation()
.AddOtlpExporter())
.WithTracing(traces => traces.AddAspNetCoreInstrumentation()
.AddHttpClientInstrumentation().AddOtlpExporter());
var app = builder.Build();
app.UseExceptionHandler();
app.MapGet("/stock/{sku}", async Task<IResult> (string sku, IHttpClientFactory clients,
CancellationToken token) =>
{
// A real application maps authenticated tenant identity to this bounded key.
using var request = new HttpRequestMessage(HttpMethod.Get,
$"stock/{Uri.EscapeDataString(sku)}");
request.Options.Set(InventoryReadHandler.Partition, "north");
using var response = await clients.CreateClient("inventory")
.SendAsync(request, token);
if (!response.IsSuccessStatusCode)
return Results.StatusCode(StatusCodes.Status503ServiceUnavailable);
return Results.Content(await response.Content.ReadAsStringAsync(token),
"application/json");
});
app.MapHealthChecks("/health/live", new HealthCheckOptions
{
Predicate = _ => false
});
app.MapHealthChecks("/health/dependencies", new HealthCheckOptions
{
Predicate = check => check.Tags.Contains("dependency")
});
app.Run();Configure Inventory:BaseUrl for the real dependency or test proxy. The endpoint assumes successful inventory responses contain JSON. Its broad non-success-to-503 mapping is intentionally simple; preserve documented domain statuses such as product-not-found in a real API.
The inbound limiter from section 6.3 can be added to this program with .RequireRateLimiting("orders"). Replica count multiplies all local limits. Allocate capacity against downstream fleet capacity rather than treating 64 as a universal recommendation.
7.3 Exposing OpenTelemetry metrics and custom health checks for circuit breakers
Registered pipelines enable Polly telemetry. The Polly meter exports strategy events, attempt durations, and pipeline durations through the configured OTLP destination. Verify exporter configuration and collector ingestion in the deployed environment.
Use the same state providers for a passive dependency health check:
public sealed class InventoryCircuitHealthCheck(
InventoryCircuits circuits,
ResiliencePipelineProvider<string> pipelines) : IHealthCheck
{
public Task<HealthCheckResult> CheckHealthAsync(
HealthCheckContext context, CancellationToken cancellationToken = default)
{
foreach (var partition in circuits.States.Keys)
_ = pipelines.GetPipeline<HttpResponseMessage>($"inventory-{partition}");
var impaired = circuits.States.Values.Any(state =>
state.CircuitState != CircuitState.Closed);
return Task.FromResult(impaired
? HealthCheckResult.Degraded("Inventory circuit is not closed.")
: HealthCheckResult.Healthy("Inventory circuits are closed."));
}
}Pipeline resolution ensures the providers are attached before inspection. A closed circuit means no current open-circuit rejection; it does not prove inventory is reachable. This check sends no probe and cannot detect an unused dependency’s outage.
Keep it separate from liveness. Restarting healthy API instances because inventory is down discards circuit history and can synchronize fresh traffic. Whether degradation should affect readiness depends on whether the instance can still serve useful requests.
7.4 Handling edge cases: Custom exception classification and conditional retry predicates
The example retries a narrow connection-failure category, attempt timeouts, and selected gateway/unavailability responses. It does not retry malformed payloads, oversized bodies, caller cancellation, circuit rejection, or limiter rejection.
Complete the registered exception boundary with this implementation. It maps dependency rejection and transport failure to 503, and an expired outbound timeout to 504. Request-abort cancellation is deliberately left to the server rather than translated into a new attempt.
public sealed class DependencyExceptionHandler : IExceptionHandler
{
public async ValueTask<bool> TryHandleAsync(HttpContext context,
Exception exception, CancellationToken cancellationToken)
{
var status = exception switch
{
TimeoutRejectedException => StatusCodes.Status504GatewayTimeout,
BrokenCircuitException => StatusCodes.Status503ServiceUnavailable,
Polly.RateLimiting.RateLimiterRejectedException =>
StatusCodes.Status503ServiceUnavailable,
HttpRequestException => StatusCodes.Status503ServiceUnavailable,
_ => 0
};
if (status == 0 || context.RequestAborted.IsCancellationRequested)
return false;
await Results.Problem(statusCode: status,
title: "Inventory dependency unavailable.").ExecuteAsync(context);
return true;
}
}This handler is appropriate for the sample’s single outbound dependency. In a larger application, wrap dependency failures in typed application exceptions so unrelated HTTP failures are not automatically labeled as inventory failures. Keep sensitive transport details out of public error bodies.
Classify retry and circuit outcomes separately. A deterministic dependency bug may deserve an alert and breaker protection without repeated attempts. A 429 might require tenant-specific cooldown rather than opening a circuit shared by unrelated callers.
Finally, keep authentication refresh outside blind transport retry. Repeated 401s do not become transient because they came from HTTP. Any supported credential refresh should have its own bounded, observable behavior.
8 Chaos Engineering and Resilience Verification: Testing the Unbreakable API
A resilience configuration is a hypothesis about failure behavior. Validate its limits under controlled faults, including recovery, while measuring useful work and additional load.
8.1 Introduction to resilience validation through chaos testing
Begin with deterministic tests around a fake HttpMessageHandler. Verify a transient failure followed by success makes exactly two attempts, while a 400, canceled caller, or open circuit makes no retry.
Test response ownership, fresh request creation, and a server that returns headers immediately but delays the body. Verify concurrent duplicate commands create one durable business effect. Those tests catch errors that a successful happy-path load test misses.
Then introduce network faults in an isolated environment with explicit stop conditions. Establish a baseline first. Change one failure mode at a time, and record the fault start, fault removal, and return to normal behavior.
8.2 Tooling stack: Toxiproxy, Chaos Mesh, and k6 load testing scripts
Toxiproxy is convenient for routing local or containerized TCP traffic through a controllable proxy. Chaos Mesh applies faults within Kubernetes. k6 drives traffic and measures client-visible outcomes.
Use an arrival-rate workload so slower responses do not silently reduce offered traffic:
import http from 'k6/http';
import { Counter } from 'k6/metrics';
const unavailable = new Counter('unavailable_responses');
export const options = {
scenarios: {
steady: {
executor: 'constant-arrival-rate',
rate: 100,
timeUnit: '1s',
duration: '3m',
preAllocatedVUs: 250,
maxVUs: 500,
},
},
thresholds: {
http_req_duration: ['p(99)<2500'],
dropped_iterations: ['count==0'],
},
};
export default function () {
const response = http.get(`${__ENV.BASE_URL}/stock/SKU-42`, {
timeout: '3s',
});
if (response.status === 503) unavailable.add(1);
}The latency threshold is an example experiment criterion, not an availability objective. Count all status classes and compare healthy, fault, and recovery intervals separately. Confirm the load generator has enough virtual users and CPU; dropped iterations invalidate the intended arrival rate.
8.3 Simulating latency spikes, packet loss, and complete blackouts
Assuming an existing Toxiproxy proxy named inventory, inject downstream latency through its management API:
curl -X POST http://localhost:8474/proxies/inventory/toxics \
-H 'Content-Type: application/json' \
-d '{"name":"slow","type":"latency","stream":"downstream","attributes":{"latency":1200,"jitter":100}}'
curl -X DELETE http://localhost:8474/proxies/inventory/toxics/slowPoint the API at the proxy’s listening address, not the management port. In containers, use addresses reachable from the respective container networks rather than assuming localhost refers to the host.
For a silent blackout, use a timeout toxic with timeout: 0; disabling the proxy instead exercises a connection failure. Test both because their resource and latency profiles differ.
Use Chaos Mesh NetworkChaos for packet loss at the network layer. Scope the selector to test inventory pods and add a short experiment duration. Packet loss triggers transport retransmission and congestion behavior; it is not equivalent to returning a synthetic 503.
Also simulate HTTP 503 responses, persistent 429s, cancellation during backoff, and connection recovery after faults are removed. For idempotent commands, inject failure after database commit but before response delivery. That boundary validates the duplicate-prevention design.
8.4 Interpreting metrics: Circuit thresholds and graceful recovery times
Read several signals together: incoming logical requests, outbound attempts, retry events, circuit transitions, limiter rejections, queue duration, and business success. A low error rate with sharply increased attempts is degradation, not free recovery.
For the configured breaker, expect opening only after minimum throughput and failure-ratio conditions are satisfied within the sampling window. Under concurrency, some already-admitted attempts continue after the circuit opens. Do not assert an exact global failure count at the transition.
After removing a fault, observe the first eligible half-open probe and the subsequent traffic ramp. Recovery time includes the remaining break interval, arrival of another request, probe duration, and any downstream warm-up. Local circuits across replicas can recover at different times.
Acceptance criteria should be concrete: bounded latency, bounded admitted work, no retries after caller cancellation, no duplicate business effects, and sustained healthy throughput after recovery. Include a low-traffic partition so an oversized minimum-throughput setting cannot pass unnoticed.
The useful question is whether another attempt improves the customer’s outcome without consuming the capacity needed for recovery. When deadlines, admission controls, circuit state, and business identity all support that decision, retries become a controlled recovery mechanism. Without those boundaries, they are additional traffic arriving at the worst possible moment.
