What does a controlled integration failure look like?
A SaaS integration should fail inside a controlled boundary: the product preserves the user's intent, records what is known, prevents duplicate side effects, and keeps unrelated workflows available. The external service may be delayed or unavailable, but it should not be able to corrupt the product's core state or make the outcome impossible to understand.
Design the failure before the success screen
The happy path explains what happens when every request returns on time. The dependable design also explains what happens when a response is lost, a webhook arrives twice, credentials expire, a provider rate-limits requests, or an event arrives hours late. Those conditions are ordinary integration states, not exceptional surprises.
Contain the dependency
Keep provider-specific payloads, status names, authentication, and retry behaviour behind an integration boundary. The rest of the product should work with its own stable concepts. This limits how far a provider change can spread and gives the application one place to translate, validate, observe, and recover external interactions.
Who should own the business state during an integration?
The application should own an internal record of the business intent and its current confirmed state. A provider can process a payment, send a message, or create a shipment, but it should not be the only place the product can determine what the user requested, what has been confirmed, and what still needs attention.
Use explicit intermediate states
States such as pending, submitted, confirmed, failed, cancelled, and requires review are more honest than forcing every operation into success or failure. If a request times out after reaching the provider, the result may be unknown rather than failed. Recording that uncertainty prevents an immediate duplicate action and gives reconciliation a state to resolve.
Separate core transactions from secondary work
A completed order should not disappear because an analytics event or notification failed. Decide which external result is required for the core transaction and which work can continue asynchronously. This keeps secondary dependencies from turning a local interruption into a product-wide outage.
How do idempotency and retries prevent duplicate work?
Use idempotency to make repeated requests represent the same intended operation, then retry only failures likely to be temporary. A stable operation identifier lets the receiver recognize a duplicate and return the original outcome instead of charging, booking, sending, or creating the same thing again.
A timeout does not prove that nothing happened
The provider may complete an operation while its response is lost in transit. Retrying without an idempotency rule can repeat the side effect. Generate the operation identifier before the first attempt, store it with the internal record, and reuse it for every retry of that same intent. A genuinely new intent must receive a new identifier.
Retry selectively and with limits
Validation failures, revoked credentials, and permanent business-rule errors usually need correction rather than another attempt. For temporary timeouts, throttling, or service unavailability, use bounded retries with increasing delays and jitter, respect provider retry guidance, and stop when continued attempts are more likely to add load than restore service.
When should an integration use queues and circuit breakers?
Use a queue when work can be accepted now and processed later, especially when traffic is uneven or the provider is temporarily unavailable. Use a circuit breaker when repeated synchronous calls to a failing dependency would consume resources, delay users, or spread the failure through the product. These patterns solve different parts of the problem and may be used together.
Queues preserve work, not correctness by themselves
A queue can buffer requests and retry delivery, but consumers must still tolerate duplicate delivery and partial completion. Record processing attempts, make handlers idempotent, and route repeatedly failing messages to a dead-letter path where they can be inspected, corrected, replayed, or deliberately discarded.
Circuit breakers protect the rest of the product
After a dependency crosses a failure threshold, a circuit breaker stops sending calls for a controlled period and lets the product fail quickly or offer degraded behaviour. It should be paired with meaningful user states and recovery checks; it is not a substitute for handling the business consequence of an unavailable service.
How does reconciliation recover uncertain outcomes?
Reconciliation compares the application's internal records with authoritative provider records and resolves mismatches. It catches operations that succeeded without a usable response, webhooks that were delayed or missed, and local processing that stopped after the provider completed its part. Critical integrations need this repair path as well as real-time delivery.
Design for replay and human recovery
Operators need to see the original intent, provider identifiers, attempts, responses, timestamps, and current ownership of the issue. Recovery tools should support safe replay, confirmation, cancellation, or escalation without direct database edits. The same idempotency rules used in automatic processing must protect manual actions.
Observe business outcomes, not only HTTP errors
Technical metrics should be connected to states the business understands: payments awaiting confirmation, shipments not created, messages delayed, or records requiring review. Alert on sustained failure and growing backlogs, while keeping expected transient retries from becoming noise that hides the incidents requiring action.
What should the user experience show during a failure?
Show only what the system knows. Confirm receipt of the user's request separately from confirming the external outcome, explain when processing is still underway, and provide a safe next step. Never display success merely because a browser request completed, and do not encourage an action that might duplicate an uncertain transaction.
Make pending states useful
A pending state should say what is happening, whether the user can leave, when the system will update, and where the final result will appear. If action is required, explain exactly what can be retried safely. Clear status reduces support demand and prevents users from creating duplicates while trying to recover on their own.
Define the degraded product deliberately
Decide which capabilities remain available when each provider fails. The product might accept work for later processing, offer read-only access, use cached non-sensitive information, or disable one action while preserving the rest. Graceful degradation is a product decision supported by architecture, not an error message added after launch.