The first successful API call proves that two systems can communicate under one set of conditions. A reliable integration must also handle uncertainty: a response is lost, a provider changes a field, a webhook arrives twice, or an operation is still pending when the user asks what happened.
The most useful starting point is the business operation. Define what must happen once, what may happen more than once, and which state the application can show while the external outcome is unknown.
Represent an operation before sending it
Consider an illustrative shipment-creation request. Create a local operation identifier and record the intended shipment before contacting the provider. Track states such as pending, confirmed, retryable failure, and review required. Keep the provider reference when one becomes available.
A timeout belongs to the unknown-outcome category until the contract provides a way to resolve it. It does not prove that the provider rejected the shipment. The provider may have created it and lost the response on the way back.
A provider-supported idempotency key can connect repeated attempts to the same logical operation. Its scope, retention period, and behavior for a different payload must be understood. A local identifier is still useful when a provider lacks that facility, but it cannot by itself prevent an external duplicate; a lookup or reconciliation process may be necessary.
Make retries bounded and specific
Retry a failure only when the provider's contract and the operation's semantics permit it. Invalid credentials and malformed payloads usually need intervention, not repeated traffic. Rate limits may include a retry instruction that the client should honor. A side-effecting request needs an explicit duplicate policy before automatic retries are enabled.
AWS's discussion of timeouts, retries, backoff, and jitter explains why immediate repeated calls can amplify an outage. Backoff spaces attempts; jitter helps clients avoid retrying together. Bound both the number of attempts and the useful time budget for the operation.
Choose the retrying layer deliberately. If an HTTP client, worker, and upstream caller all retry independently, one logical operation can generate many more attempts than expected. Surface retry counts and exhaustion as observable state, rather than letting the operation disappear into logs.
Separate durable acceptance from processing
For an inbound webhook, verify the sender according to the provider's exact specification, then store enough information to process the event durably. A signature scheme can depend on the original request bytes and a timestamp; parsing and reserializing JSON first may change what must be verified.
Recognize duplicate event identifiers within a defined scope. Use a database constraint or another concurrency-safe mechanism for deduplication, and make the handler safe when a worker repeats a partially completed operation. A successful delivery acknowledgment should mean what the provider's contract expects; do not claim durable acceptance before the event is durably recorded.
The Data Integrity Playground demonstrates HMAC using its own documented JSON serialization. It is a learning exercise, not a drop-in verifier for every webhook provider.
Test the uncertainty around the happy path
A useful boundary test suite includes more than a successful sample response. Exercise missing optional fields, unexpected enum values, invalid credentials, rate limiting, repeated delivery, and a timeout after an operation could have been accepted. Keep test data invented and independent of customer records.
When a provider offers a sandbox, use it to verify assumptions that mocks cannot establish. Keep contract fixtures small and versioned so reviewers can see which behavior the application depends on. Structural API changes can be explored in the API Contract Inspector, while behavioral compatibility still needs focused tests and review.
Give an operator a recovery path
Record the local operation identifier, provider reference, attempt count, last outcome category, and the time the operation entered its current state. Avoid recording credentials or complete sensitive payloads merely to make debugging convenient.
A reconciliation task should be able to find operations that have remained uncertain too long and compare them with provider state. An operator needs an action that means “check the external outcome” rather than a button that blindly creates another shipment.
Practice the retry policy
Use the Asynchronous Task & Retry Simulator with a fixed seed. Compare an immediate-retry scenario with bounded backoff and jitter, then inspect completed logical jobs, attempts, waiting work, and dead letters together. The simulation is educational, but the distinction between a logical operation and its delivery attempts is directly useful when reviewing an integration.
Reliability is the ability to explain an operation after the network stops giving clear answers. Build the state, constraints, and recovery tools that make that explanation possible.