RELIABILITY · SOFTWARE SYSTEMS · OBSERVABILITY
API resilience under dependency failure
I wanted a smaller way to look at a common service problem: a downstream API is slow or unavailable, and the caller is still waiting. It is easy to say "add retries" or "add a circuit breaker." I wanted to see what each choice actually changes when the conditions are controlled.
The source, experiment notes, raw responses, and verification script live in the R02 folder.
Stack C# / .NET / HTTP / Python
The setup
R02 has two local .NET services. One is a dependency I can switch between healthy, slow, and unavailable. The other is a gateway that calls it. I added one policy at a time so that the runs could be compared without changing the whole system at once.
The working question is: what do timeout, bounded retry, circuit-breaker, and concurrency-limit policies change when a service depends on an API that slows down or fails?
What I ran
I started with direct forwarding. A 750 ms delay in the dependency became a
755 ms wait at the gateway; an unavailable dependency returned 503 straight
through.
The retry case was useful because it was not flattering. With a sustained
503, one gateway request made three dependency calls and still ended in
503. The 250 ms timeout made a different trade-off: it returned 504 after
269 ms instead of waiting for the full 750 ms delay, but it did not make the
dependency available.
For the circuit-breaker run, three consecutive 503 responses opened a
one-second break window. The next request was rejected by the gateway without
a dependency call. After the dependency was healthy and the window had passed,
the next request succeeded.
Finally, I sent six callers to a dependency delayed by 500 ms. With direct
forwarding, all six reached it. With a two-permit, no-queue limit, two reached
the dependency and four received 429 before adding work downstream.
Experiments
Baseline forwarding
Healthy, slow, and unavailable dependency modes passed through the gateway without a resilience policy.
Bounded retry
One sustained-outage request became three dependency attempts and still returned 503.
Timeout boundary
A 250 ms per-attempt timeout returned 504 before the dependency’s 750 ms delay completed.
Circuit breaker
Three consecutive failures opened a one-second break window; the next request was short-circuited.
Concurrency limit
With two permits and no queue, two of six callers reached the slow dependency and four received 429.
What I take from it
None of these policies repair a broken dependency. They decide where the cost lands: extra downstream work, a bounded wait, a temporary refusal, or an upfront rejection. The values here are deliberately local settings, not numbers I would copy into another system.
What I have not tested
Both services ran on one machine. The circuit breaker and concurrency limiter keep state only in the gateway process. I have not tested network partitions, multiple gateway instances, persistent state, production traffic, or a queue with priority rules. Those are separate parts of the problem.