API Integrations That Don't Break: Webhooks, Retries and Idempotency

By Jezer Niel Blanca, Full Stack Developer · September 15, 2026 · 6 min read

Networks fail, webhooks arrive twice, and events show up out of order. These are the patterns I use to keep API integrations reliable anyway.

A big part of my day-to-day work is API integrations: connecting one system to another so that data flows without anyone copying and pasting. Integrations look simple on a whiteboard. System A calls system B, done. In reality, networks drop connections, providers send the same webhook twice, requests time out after the other side already processed them, and events arrive out of order. The difference between a fragile integration and a reliable one is how it handles those moments. Here are the patterns I rely on.

Assume the Network Will Fail

Every call across the internet can fail in three ways: it fails fast with an error, it hangs, or it succeeds on the other side but the response never reaches you. The third one is the sneaky one, and it's why retries alone aren't enough.

The mindset shift is simple:

Every outbound call needs a timeout.
Every retry must be safe to repeat.
Every incoming event must be safe to receive twice.

If you design with those three rules, most integration bugs never happen.

Outbound Calls: Timeouts and Smart Retries

Laravel's HTTP client makes the right defaults easy. I set explicit timeouts and retry only on errors that are likely temporary:

use Illuminate\Http\Client\ConnectionException;
use Illuminate\Http\Client\RequestException;
use Illuminate\Support\Facades\Http;
use Throwable;

$response = Http::baseUrl(config('services.crm.url'))
    ->withToken(config('services.crm.token'))
    ->connectTimeout(5)
    ->timeout(15)
    ->retry([200, 1000, 3000], when: function (Throwable $exception): bool {
        if ($exception instanceof ConnectionException) {
            return true;
        }

        return $exception instanceof RequestException
            && ($exception->response->status() === 429 || $exception->response->serverError());
    })
    ->post('/contacts', $payload);

Passing an array to retry() sets the wait before each attempt, which gives you a simple backoff. The when callback matters just as much: retrying a 422 Unprocessable Entity will never succeed, because the data itself is wrong. Retry network errors, rate limits, and server errors. Fail fast on everything else.

For anything that isn't needed in the current request, move the call into a queued job. Jobs have their own retries and backoff, and a slow provider no longer slows down your users.

Idempotency: Making Retries Safe

Here's the problem with retries. Your app sends "create invoice", the provider creates it, and then the connection drops before the response arrives. Your code sees a timeout and retries. Now there are two invoices.

The fix is an idempotency key: a unique identifier for the operation that you send with every attempt. Many payment and billing APIs support an Idempotency-Key header, and when they see the same key again they return the original result instead of doing the work twice.

$operation = $invoice->syncOperations()->firstOrCreate(
    ['action' => 'create_remote_invoice'],
    ['idempotency_key' => (string) Str::uuid()],
);

Http::withHeaders(['Idempotency-Key' => $operation->idempotency_key])
    ->post('/invoices', $invoice->toRemotePayload());

The key must be stored and reused, not generated fresh on every attempt. Otherwise it protects nothing.

If the provider doesn't support idempotency keys, you can still get close:

Store the remote ID as soon as you receive it, and check for it before creating anything.
Use a natural unique field, like your own invoice number, and search for it on the remote side before creating.
Prefer "upsert" endpoints when the API offers them.

Retries without idempotency don't make an integration reliable. They just make the duplicates arrive faster.

Receiving Webhooks the Right Way

Webhooks are how other systems tell you something happened. They're also the part of an integration most likely to cause silent bugs. My webhook handlers follow the same four steps every time.

1. Verify the Signature

Most providers sign webhook payloads with a shared secret. Always verify it before trusting the data, and compare with hash_equals to avoid timing attacks:

$expected = hash_hmac('sha256', $request->getContent(), config('services.crm.webhook_secret'));

abort_unless(hash_equals($expected, (string) $request->header('X-Signature')), 401);

Check your provider's documentation for the exact header name and signing scheme. Some include a timestamp in the signed string to prevent replay attacks.

2. Store the Event, Deduplicated

Providers retry webhooks when they don't get a quick success response, so duplicates are normal. A unique index on the provider's event ID turns deduplication into a database guarantee:

Schema::create('webhook_events', function (Blueprint $table) {
    $table->id();
    $table->string('provider');
    $table->string('external_id');
    $table->string('type');
    $table->json('payload');
    $table->timestamp('processed_at')->nullable();
    $table->timestamps();

    $table->unique(['provider', 'external_id']);
});

$event = WebhookEvent::query()->firstOrCreate(
    ['provider' => 'crm', 'external_id' => $request->input('id')],
    ['type' => $request->input('type'), 'payload' => $request->all()],
);

3. Respond Fast, Process Later

Return a 200 as soon as the event is stored, and do the real work in a queued job. If processing takes too long, the provider may time out and send the same event again.

if ($event->wasRecentlyCreated) {
    ProcessWebhookEvent::dispatch($event);
}

return response()->noContent();

4. Process Idempotently

Inside the job, check processed_at first and set it when you finish. If the job is retried after a crash, it won't apply the same change twice.

Ordering, Reconciliation and Visibility

Even with all of the above, two more problems remain.

Events can arrive out of order. An "updated" webhook can arrive before "created", or an old update can arrive after a newer one. Where the payload includes a timestamp or version number, compare it with what you have and ignore anything older. Where it doesn't, treat the webhook as a signal to fetch the latest state from the API instead of trusting the payload.

Some events never arrive. Providers have outages too. I add a scheduled reconciliation job that periodically asks the remote API for recent changes and fixes any drift:

Schedule::job(new ReconcileCrmContacts)->hourly()->withoutOverlapping();

And finally, make the integration visible. Log every outbound request with a correlation ID, keep the webhook events table browsable in your admin, and alert when failed jobs pile up. When something goes wrong, you want to answer "what happened?" in minutes, not hours.

Wrapping up

Reliable integrations come from a short list of habits: timeouts on every call, retries only for temporary errors, idempotency keys that are stored and reused, verified and deduplicated webhooks, fast responses with queued processing, and a reconciliation job as a safety net. None of it is glamorous, but it's what keeps data correct when networks misbehave. If your product needs integrations that just work, I'd love to build them with you and my team.

Tags: APIs, Webhooks, Integrations, Laravel