A useful resilience test can produce an unremarkable-looking result: employees still work after an external dependency disappears, but new requests receive no confirmation and nobody notices. Servers remain responsive. A business workflow is already broken.
Chaos engineering tests such assumptions through controlled faults. Losing an entire region need not be the first exercise. Delaying one dependency and following a specific customer operation is often a more informative place to start.
Ask the question before choosing the tool
Consider a training-registration service that sends confirmations through an external email provider. The team's hypothesis is that losing email does not prevent registration, users can still see their status, and notifications are sent later without creating duplicate registrations.
This is testable. Define observable outcomes: saved registrations remain accessible, unrelated functions retain agreed service quality, and the notification backlog stays within its planned limits. Define recovery too: queued work is accounted for and registrations have not been duplicated.
“Disable email and see what happens” establishes neither expected behaviour nor acceptable harm. The Principles of Chaos Engineering start with a measurable steady state and a hypothesis. Selecting the fault-injection tool comes later.
Inject the relevant failure, not merely an easy one
A refused connection, a slow response and a lost acknowledgement after successful delivery exercise different behaviours. One tells the caller quickly that something failed. Another holds resources while it waits. The third leaves the caller uncertain about the outcome and may lead to a repeated action.
In this example, the point of injection matters. Blocking the wrong endpoint or leaving an unexamined alternate path may fail to affect the intended dependency at all. Confirm the actual impairment and the application's response, rather than relying on a successful status from the experiment runner.
Changing every failure mode at once usually makes the result harder to interpret. Start with one relevant dependency and one condition, keeping the exercise repeatable after a fix.
The boundary has to exist in the system
First choose the environment and a limited set of targets. A test environment is a practical place to verify the mechanism, recovery and observation. Qualify the result: production may differ in traffic, data, permissions and routing.
A production experiment needs an explicit risk decision. Participants should know which customers might be affected, who can stop the exercise and how to reach the on-call team. An existing incident or the start of a major campaign is a poor moment to expand an uncertain fault.
The limit must be enforceable: a test cohort, a selected cell or instance, confirmed resources and minimal permissions. Permission to affect everything with a particular tag is risky when group membership can change after preparation. Inspect the actual target set before starting, not just the selector expression.
For notification tests, also restrict recipients and message content. A test registration must not accidentally reserve a real training place or appear in operational reports as a new sale. Account for those test records and verify that any reserved resources are released as part of cleanup.
Stop is not a time machine
Define a maximum duration, observable stop conditions and someone authorised to halt the test immediately. Use signals tied to the workflow being protected. If email is deliberately delayed, some backlog growth is expected; the stop condition should detect unacceptable harm, such as registration itself failing.
Automatic guardrails also need testing. If the fault takes away monitoring, lack of an alarm cannot be interpreted as safety. Agree in advance how to exit that situation, including stopping when the required visibility is lost.
Stopping the injection and recovering the system are different operations. Stop cannot retract a sent message, restore deleted data or recreate a terminated resource. Verify cleanup for reversible faults and prepare separate recovery for other effects. AWS FIS stop conditions describe halting an experiment; its rollback capabilities depend on the action involved.
The provider returning is not the end of the test
Once email responds again, consumers may drain the backlog too aggressively. Retries add traffic, some notifications become obsolete, and users may already have repeated registration themselves. Rate controls, idempotency and checking current state before delayed actions become important.
Observe that recovery phase as well as the initial fault. Finish when the service is back in its agreed operating state, accepted work is accounted for and temporary settings have been removed. Remaining backlog and manual corrections belong in the result.
For notifications, compare accepted registrations with the intent to notify and the recorded outcome. That is not necessarily perfect proof of arrival in an inbox. It should still distinguish no attempt, an unknown result and provider acceptance. Otherwise the exercise stops exactly where a future customer complaint would begin.
Keep the conclusion as narrow as the evidence
There are at least three outcomes: the observed behaviour supports the hypothesis, contradicts it, or provides insufficient evidence. The third is important. If the selected cohort never used the dependency during the fault, a green dashboard proves very little. Change the experiment instead of counting it as a success.
Record the system version, workload conditions, actual targets, evidence of impairment and observations of the business workflow. “The platform is resilient” is broader than any individual test. A defensible conclusion states which scenario was exercised, under what conditions, and what failed or continued.
A discovered weakness needs an owner and a repeat test after remediation. That is where the return appears: the company stops relying on an unverified assumption. Without credible observation and a safe recovery path, building those capabilities is more useful than adding an impressive server-disabling tool.