I Designed an Integration Layer That Expects Vendors to Fail

Direct API calls create a simple architecture until they become a dependency chain. A slow vendor holds open application requests. A repeated callback creates duplicate records. A changed field breaks parsing. A timeout leaves the business unsure whether the action failed or completed somewhere else.

The system needed to handle uncertainty without producing silent data loss. It also had to support multiple vendors without spreading provider specific logic through the product. Recovery had to be safe enough for operators to use under pressure, with clear evidence of what happened and what the system would do next.

The difficult question was not how to retry a request. It was how to preserve business intent when the external world cannot give a reliable answer.

I placed every vendor behind an adapter contract owned by the platform. The core application sends business commands such as create shipment or capture payment without depending on a provider payload. Each adapter translates that command, validates the response, and stores the external reference separately from the internal record.

Commands enter a durable outbox before delivery. A worker sends them with an idempotency key, records every attempt, and applies bounded retry rules with increasing delay. Circuit protection stops repeated calls when a provider is unhealthy. Failed work moves into a recovery queue with the original intent, current state, and reason for failure intact.

Inbound webhooks pass through signature checks, schema validation, deduplication, and ordering rules before they can change business state. Unknown fields are preserved for diagnosis while invalid events remain isolated from production data. A reconciliation worker compares internal expectations with provider records so ambiguous timeouts can be resolved safely.

The operator view is built around recovery. It shows which vendor is degraded, which workflows are waiting, what has been retried, and which items need a human decision. Traces connect the customer action, internal command, vendor attempt, callback, and final state without exposing secrets.