Once a year, on a Saturday morning, the failover exercise runs as planned. Teams switch over to the secondary site, confirm that critical applications restart, record the timings, produce a report. The report concludes that the plan works. It goes to the risk committee, sometimes to the regulator, and is filed away.

Six months later, on a Tuesday afternoon, the online subscription journey becomes unusable for two hours. No server went down, no site was lost. A third-party authentication service was answering in several seconds instead of a few hundred milliseconds; calls piled up, connection pools saturated, and an apparently unrelated back-office became unreachable in turn. The recovery plan was never invoked — there was nothing to recover. The system had not stopped; it had stopped working.

Both scenes tell the same story. The organisation has a plan for the disaster it will probably never see, and no design for the incident it lives through several times a year. That is this article’s argument: resilience is neither a document nor an infrastructure option bought separately. It is a property of the system, arising from the way it is structured — from the dependencies it carries, the delays it tolerates, and what it can still do while part of itself is unavailable. Like security, it is decided in the architecture views. Unlike security, it faces no adversary: failure needs no intent to occur.

The plan answers the wrong question

Continuity and disaster recovery plans reason on a specific model of failure: a clean, localised event that takes out a site, a room, an infrastructure. The response fits that model — a secondary site, restorable backups, a failover procedure, a decision chain. That apparatus is necessary, often mandated, and there is no case for weakening it. The point is simply to see what it does not cover.

The model assumes a binary failure: the component either works or it does not. But the clean outage is the easy case. A component that is plainly dead gets detected, taken out of service, routed around. The incidents that genuinely cost money are of another kind: partial, gradual, ambiguous. Storage answering ten times more slowly. A queue accumulating without ever losing anything. A service returning valid but stale answers. None of that triggers a recovery plan, because nothing has fallen over.

The plan also reasons in infrastructure — sites, machines, backups — whereas failure travels along a business journey. Taking out a contract, executing a transfer, opening a case: each of these crosses a dozen components, several of them outside the organisation’s perimeter — a SaaS service, a partner API, a carrier, an identity provider. No failover plan anticipates a partner slowing down.

The continuity plan is therefore a floor, not an answer. It handles the extreme, rare scenario. Resilience concerns the ordinary behaviour of the system under degraded conditions — and that cannot be declared in a document: it has to be designed.

Stated availability, actual resilience

In most organisations, availability is discussed component by component, in the form of a contractual commitment. Every building block advertises its figure, every supplier its service level, and the whole gives an impression of solidity. That reading has two blind spots.

The first is arithmetic. Rates do not add up, they multiply. A journey that simultaneously needs eight components, each committed to 99.9%, cannot exceed roughly 99.2% — meaning, if failures are independent, on the order of seventy hours of unavailability a year where each block, taken alone, promised fewer than nine. The calculation is trivial, and yet it is rarely done: availability gets negotiated for the parts, never for the journey.

The second blind spot is more serious, because it makes the calculation above optimistic. It assumes failures are independent. But the components of a system share dependencies: a directory, an identity service, name resolution, a network, a payment platform, an orchestration layer. Two redundant instances that both rest on the same dependency are not redundant: they fall together. Redundancy on the diagram is not redundancy in fact.

A word is also due on service commitments themselves: an SLA is not a guarantee of behaviour, it is a commercial commitment backed by a penalty. It restores nothing; it compensates. An organisation that builds its resilience on the figures advertised by its suppliers has transferred a marginal financial risk, not an operational one.

Two levers follow from this reading, and only one is habitually used. The first is to make each component more reliable — useful, expensive, and quick to hit diminishing returns. The second is to reduce the number of components strictly required by a journey, making the others optional. It is by far the more effective, and it is an architectural decision, not an operational one.

RTO and RPO: objectives that must come down into decisions

Two notions structure continuity. RTO (recovery time objective) is the acceptable interruption before the consequence becomes unacceptable. RPO (recovery point objective) is the amount of data one accepts losing — the gap between the last safe state and the moment of failure.

These objectives are set in workshops with the business. That is good practice, and it nearly always fails at the same point: the resulting values never come back down as design constraints. Asked without a counterpart, the business answers “zero” to both questions — which is entirely rational, since the answer is free.

The architectural work consists precisely in making the price of that answer visible, by translating each objective into a technical decision and a cost.

An RPO close to zero requires synchronous replication: no write is confirmed until it has been replicated. That constrains the latency of every write, hence the distance between sites, hence the real protection against a regional disaster. You cannot be both very close for performance and very far for safety: the trade-off is physical.

An RPO of a few minutes allows asynchronous replication, less constraining and far cheaper. But it creates an obligation people forget to design: after recovery, a hole in the data remains. Who fills it? Through what replay, what reconciliation, what re-keying, within what deadline? A non-zero RPO is a commitment made about a business process, not only about a replication technology.

An RTO of a few minutes implies an active or pre-warmed arrangement, with automatic failover. And automatic failover must itself be designed with suspicion: a system that fails over wrongly — on a false positive, or by splitting in two for lack of an arbiter — has just added an outage to its inventory. The recovery mechanism is one more critical component, and often the least exercised of all.

Setting these objectives without translating them creates an illusion of control. Translating them turns continuity into what it should be: a series of cost/risk trade-offs, properly prepared and openly owned.

Four failure modes the plan does not see

The shared dependency. This is the first reflex to acquire: stop asking “which components are redundant?” and start asking “which components fall at the same time?”. An authentication service, a configuration store, an event bus, a payment platform: these cross-cutting blocks turn a local incident into a general one, precisely because everyone leans on them without having decided to.

Grey failure. A slow component is not a dead component. It answers, so health probes declare it alive, so traffic keeps being sent to it, so the queue grows upstream. This is the most destructive failure mode, because the system refuses to acknowledge that it is down. Detection is not a given property: it is designed — on latency thresholds and error rates, not on the single question “is the service responding?”.

The cascade. A partial slowdown becomes a general outage through amplification. Calls fail, clients retry, retries saturate whatever was still holding, and an unbounded retry policy starts attacking the system from within. Recovery traffic is often more violent than nominal traffic: on restart, everyone comes back at once, with caches empty. A system that has not been designed to protect itself from its own clients recovers badly from a minor incident.

The single third party. Outsourcing a capability does not outsource the responsibility. Dependence on a single provider — a host, a software-as-a-service vendor, a payment operator — is a point of failure that the contract frames but the architecture must prepare: what do we do during the outage, not merely what do we recover afterwards. It is now one of the most explicit concerns of regulators, who look at supplier concentration risk as closely as at internal robustness.

Designing for failure: four structuring decisions

Compartmentalise. The question to settle while drawing the target is not “what should be redundant?” but “what must not be able to bring down what?”. Shared resources — a connection pool, a common database, a single queue, one runtime platform — are propagation paths. Compartmentalising means accepting some duplication in order to bound the blast radius. It is decided on the architecture diagram, not in an operational setting.

Decide the degraded mode before the incident. This is the most important decision, and the one most often missing. What does the journey do when a given dependency is unavailable? Refuse cleanly? Queue and replay? Serve slightly stale data, knowingly? Accept the operation with deferred verification? These options are not technical: they are business and risk trade-offs. Accepting a transaction without a real-time check opens a window of accepted risk; refusing carries a commercial cost and sometimes a regulatory one. That choice belongs to the business, and it is documented like any architectural decision. If it is not made in the cold, it will be made at three in the morning by whoever is on call, alone, without a mandate.

Bound the wait. A call with no maximum delay is a dependency with no contract. Timeouts, circuit breakers that isolate a failing component instead of continuing to call it, retries that are spaced out and capped: these are design instruments, not implementation details. It is worth saying bluntly — leaving a library’s default values to determine how the system behaves under incident is delegating an architectural decision to a stranger.

Treat state as the hard part. What is stateless replicates, restarts, moves; the problem is never there. The hard part is data: its consistency, its recovery, and above all the side effects of a retry. An order executed twice because the first call timed out before the answer came back is a far costlier incident than an outage. Idempotence — the guarantee that repeating the same operation produces the same result — must be a requirement of the interface contract, decided at the same time as the interface itself, and not a fix added after the first double debit.

The level of resilience is decided capability by capability

All this engineering is expensive. Applying it uniformly is an allocation error as costly as under-protection: you pay everywhere for what only matters in places, and still miss the critical path.

The right unit of reasoning is neither the server nor even the application: it is the business capability. The question to ask is simple, and it is asked with the business: how long can this capability be unavailable before the consequence becomes irreversible? Irreversible in the regulatory, financial or reputational sense. Some capabilities tolerate a day without lasting damage. Others are counted in minutes. That difference must be legible in the architecture, or it does not exist.

A classic trap waits here: ranking applications by criticality rather than journeys. An application judged secondary but sitting on the path of a critical journey is, in fact, critical — and that is almost always where the incident comes from. Criticality propagates along dependencies; it is not declared application by application.

What is not exercised is not resilient

A degraded mode that has never been exercised is not a degraded mode: it is a hypothesis. And most of the time it is a false one — not through incompetence, but because the behaviour of a system in partial failure cannot be deduced from its documentation.

Testing is built in steps, and there is no need to start at the top. The first step is a design review that takes each dependency and asks what happens if it slows down, then if it disappears. It costs nothing but discipline, and it already exposes blind spots. The second is the failover exercise, provided it covers complete business journeys rather than the restart of machines. The third is controlled fault injection in pre-production — cut a dependency, add latency, observe. The fourth, reserved for organisations that have mastered the previous three, is to run those experiments in production, within a bounded perimeter.

One prerequisite conditions the whole structure: you cannot make resilient what you cannot observe. If the available indicators cannot distinguish “slow” from “dead”, nor tell whether the journey works while every component reports healthy, no protection mechanism will trigger at the right moment. Measuring component availability while ignoring journey availability is like monitoring the organs without ever taking a pulse.

This is also the direction of current regulation. In the European financial sector, the digital operational resilience regulation, applicable since January 2025, shifts the requirement explicitly: having a plan is no longer enough, the ability to absorb a failure must be tested, including the failure of a third-party provider. Regional regulators, Bank Al-Maghrib foremost among them, have long framed business continuity for financial institutions and their use of outsourcing. Whatever the applicable text, the direction is the same: evidence replaces declaration.

What resilience actually asks for

None of the above is an infrastructure programme. Resilience is a set of design decisions — which dependencies we accept to carry, what we keep running when they give way, what we deliberately stop serving, and where we accept paying not to give way at all. Those decisions are taken when the target architecture is drawn. Afterwards, they cost the price of a rebuild.

A resilient organisation is therefore not recognised by the thickness of its recovery plan. It is recognised by the fact that it knows how its system behaves when part of it fails — because it decided that in the cold, wrote it down, and exercised it. On a Tuesday afternoon, that is the whole difference: not avoiding the incident, but knowing in advance what you keep doing, and what you choose to stop.