Reliable Async Processing: Webhooks, Retries, and Idempotent Handlers

A synchronous API call is easy to reason about: send a request, get a response, done. But some work doesn't fit in the lifetime of a single HTTP request — running OCR across 50 documents, screening a batch of signups against a fraud gallery, or processing a folder of images uploaded overnight. For workloads like this, the API accepts the job, hands you a job_id, and does the work in the background. The interesting engineering questions are all about what happens after that: how you find out the job is done, how you handle the case where your server doesn't get the message the first time, and how you make sure processing the same result twice doesn't cause a problem.

This post covers that lifecycle end to end: submitting an async batch job, the two ways to retrieve results (webhooks and polling), how to verify a webhook is genuinely from Quantilence, and how to write a handler that's correct under retries and duplicate deliveries.

The async job lifecycle

Batch endpoints — like POST /v1/ocr/batch, which accepts an array of up to 50 documents — don't process synchronously. The request returns almost immediately with a job identifier and a status:

const job = await client.ocr.batchSubmit({
  documents: files, // up to 50
  callbackUrl: "https://yourapp.com/webhooks/quantilence",
});

console.log(job);
// { job_id: "job_8f3a1c2b", status: "processing" }

From here, the job moves through a small set of states — processing, then completed or failed — and the results become available once it reaches a terminal state. There are two ways to find out when that happens, and most production integrations end up using both.

Two ways to get results: webhooks and polling

Webhooks (recommended). If you provide a callback_url when submitting the job, Quantilence sends a POST request to that URL the moment the job finishes — whether it completes successfully or fails. This is the lower-latency, lower-overhead option: you do no work while the job is processing, and you're notified the instant there's something to act on. Webhooks are available on Growth plans and above; if you're on Starter, polling is your only option.

Polling (fallback). If you don't provide a callback_url — or if your environment can't expose a public endpoint (local development, an internal tool behind a VPN, a CLI script) — you can poll the job status directly:

const status = await client.jobs.get("job_8f3a1c2b");
// { status: "completed", result: { ... } }

Poll on a reasonable interval (a few seconds, with backoff if the job is taking longer than expected) rather than tightly looping — each poll is a request against your rate limit. Polling is simple and requires no inbound infrastructure, but it trades latency and request volume for that simplicity. A common pattern is to use webhooks in production and polling in local development, where exposing a callback URL isn't practical.

The rest of this post focuses on webhooks, because they're where the interesting failure modes live.

Verifying that a webhook is genuinely from Quantilence

Before processing a webhook payload, verify it. Every webhook request includes an X-Quantilence-Signature header — an HMAC-SHA256 signature of the raw request body, computed with a signing secret that's unique to your account (available from your dashboard's webhook settings). This is the same pattern used internally for payment webhooks: compute the expected signature from the raw body and your secret, then compare it to the header using a constant-time comparison.

import crypto from "crypto";

function verifySignature(
  rawBody: string,
  signatureHeader: string,
  secret: string
): boolean {
  const expected = crypto
    .createHmac("sha256", secret)
    .update(rawBody)
    .digest("hex");

  const a = Buffer.from(expected, "utf8");
  const b = Buffer.from(signatureHeader, "utf8");

  return a.length === b.length && crypto.timingSafeEqual(a, b);
}

Two details matter here. First, the signature is computed over the raw request body bytes, not a re-serialized version of the parsed JSON — if your framework parses the body before you can access the raw bytes, you'll need to configure it to preserve the raw payload for this route (most frameworks have a way to do this for webhook endpoints specifically). Re-serializing parsed JSON and signing that will produce a different byte sequence than what was originally signed, even if the data looks identical, because of whitespace and key-ordering differences.

Second, use crypto.timingSafeEqual rather than === for the comparison. A standard string comparison short-circuits on the first mismatched byte, which leaks timing information that could theoretically help an attacker guess a valid signature byte-by-byte. timingSafeEqual always takes the same amount of time regardless of where the strings differ — and it throws if the buffers have different lengths, which is why the length check happens first.

A request with a missing or invalid signature should be rejected with a 401 and never processed:

export async function POST(req: Request) {
  const rawBody = await req.text();
  const signature = req.headers.get("x-quantilence-signature") ?? "";

  if (!verifySignature(rawBody, signature, process.env.QUANTILENCE_WEBHOOK_SECRET!)) {
    return new Response("invalid signature", { status: 401 });
  }

  const event = JSON.parse(rawBody);
  // ... handle event
}

Webhooks are delivered at least once — design for it

Here's the detail that catches most webhook integrations off guard: a webhook can be delivered more than once for the same event. If your endpoint times out, returns a non-2xx status, or simply doesn't respond before Quantilence's delivery timeout, the event is retried — with the same payload and the same X-Quantilence-Event-Id header — on a backoff schedule over roughly 24 hours. From your endpoint's perspective, this is indistinguishable from a brand-new event that happens to look identical. If your handler isn't idempotent, a retried delivery means the same job result gets processed — and any associated side effects (updating a record, sending a notification, charging something) get applied — twice.

This is the same at-least-once delivery problem that the API integration architecture post covers from the client side with Idempotency-Key headers on outbound requests. Webhooks are the mirror image: now you're the receiver, and the deduplication responsibility is on your handler.

The fix is the same shape as the client-side one — track which event IDs you've already processed, and treat a repeat as a no-op:

export async function POST(req: Request) {
  const rawBody = await req.text();
  const signature = req.headers.get("x-quantilence-signature") ?? "";
  const eventId = req.headers.get("x-quantilence-event-id") ?? "";

  if (!verifySignature(rawBody, signature, process.env.QUANTILENCE_WEBHOOK_SECRET!)) {
    return new Response("invalid signature", { status: 401 });
  }

  // Has this event already been processed?
  const alreadyProcessed = await db.webhookEvents.findUnique({
    where: { eventId },
  });
  if (alreadyProcessed) {
    return new Response("ok", { status: 200 }); // duplicate — no-op
  }

  const event = JSON.parse(rawBody);

  // Record the event ID and enqueue the real work.
  await db.webhookEvents.create({ data: { eventId, receivedAt: new Date() } });
  await jobQueue.enqueue("process-batch-result", { jobId: event.job_id });

  return new Response("ok", { status: 200 });
}

The webhookEvents table here is doing one job: turning "have I seen this event ID before?" into a single indexed lookup. It doesn't need to store the payload — just enough to recognize a repeat. A unique constraint on eventId also protects against a race where two retried deliveries arrive concurrently and both pass the "not found" check before either insert completes; the second insert fails the unique constraint, and that failure itself is a signal to treat the request as a duplicate.

Return 200 fast, process slowly

The handler above does two things in sequence: record the event, then enqueue the work — it doesn't perform the OCR result processing, database updates, or notifications inline before responding. This separation matters for reliability in both directions.

If your handler does substantial work before responding — fetching the full result, writing to multiple tables, calling other services — and any of that work is slow or fails, the whole request can time out or return an error. From Quantilence's side, that looks like a failed delivery, and triggers a retry of an event that may have partially succeeded on your end. Untangling "which parts of a partially-applied webhook need to be redone" is much harder than just acknowledging receipt and doing the real work in a job queue, where your own retry and error-handling logic applies.

Acknowledge receipt — verify the signature, dedupe on event ID, enqueue the real work — and return 200 within a few seconds. Let your job queue's retry semantics, which you control and can observe, handle the actual processing.

Common pitfalls

Verifying the signature against the parsed body instead of the raw body. As covered above, this is the most common reason signature verification fails for legitimate webhooks — the parsed-and-re-serialized JSON doesn't byte-for-byte match what was signed, so every request gets rejected as invalid even though it's genuinely from Quantilence.

Doing all the processing inline and returning 200 only at the end. This makes your webhook endpoint as slow and failure-prone as the slowest step in your processing pipeline, and turns transient failures in unrelated systems (a downstream notification service, an analytics call) into webhook delivery failures and retries.

Deduplicating on job ID instead of event ID. A single job can, in principle, generate more than one event over its lifecycle (for example, a processing status update followed by a completed event). Deduplicating on job ID alone could cause you to silently drop a legitimate second event. The X-Quantilence-Event-Id header is unique per event, not per job — dedupe on that.

No monitoring on the webhook endpoint itself. If your endpoint starts returning errors — a bad deploy, an expired database credential, a downstream outage — webhooks will retry for about 24 hours and then stop. If nobody notices during that window, the events are gone for good (beyond what polling the job status API after the fact can recover). Treat your webhook endpoint as a production API with its own uptime and error-rate monitoring, not just a corner of your codebase.

Hardcoding the webhook secret. The signing secret is account-specific and rotatable from your dashboard. Store it as an environment variable (QUANTILENCE_WEBHOOK_SECRET), and if you ever need to rotate it, support verifying against both the old and new secret for a transition window so in-flight retries signed with the old secret aren't rejected.

Frequently asked questions

What's in the webhook payload? The event includes the job_id, a status (completed or failed), and either the result (for completed) or an error description (for failed). The payload shape mirrors what you'd get from polling GET /v1/jobs/{job_id} — webhooks and polling return the same data through different delivery mechanisms.

How long does Quantilence retry a failed webhook delivery? Failed deliveries (non-2xx responses, timeouts, or connection errors) are retried with exponential backoff over approximately 24 hours. After that window, the event is no longer retried, but the result remains available via GET /v1/jobs/{job_id} — polling is always a valid way to recover a result you missed.

Can I use webhooks on the Starter plan? No — webhooks require a Growth plan or above. On Starter, use polling: submit the job without a callback_url, and poll GET /v1/jobs/{job_id} until the status leaves processing. See the pricing page for plan details.

Do I need an idempotency key when calling the batch submit endpoint, in addition to deduplicating webhooks? Yes — these solve different problems. The Idempotency-Key on your submit request (covered in the API integration architecture post) protects against accidentally creating two jobs if your submit request is retried due to a network error. Deduplicating on X-Quantilence-Event-Id in your webhook handler protects against processing the same completion event twice. A robust integration needs both — one at the start of the job's lifecycle, one at the end.

What if my server is down when a webhook is delivered? A connection failure is treated the same as any other failed delivery — it's retried on the same backoff schedule. As long as your server is back up within the ~24-hour retry window, the event will be redelivered. If it's down longer than that, polling GET /v1/jobs/{job_id} for any jobs submitted during the outage will recover the results.

Conclusion

Async processing shifts the hard part of an integration from "did my request succeed" to "what happens between submission and completion, and what happens if a notification about that completion gets lost or duplicated." Submit jobs with an idempotency key, choose webhooks for low-latency notification (with polling as a fallback or recovery path), verify every webhook signature against the raw request body with a constant-time comparison, and write handlers that dedupe on event ID and return quickly. None of these pieces are complicated in isolation — but together, they're the difference between a batch pipeline that quietly handles retries and one that quietly double-processes results until someone notices.

Explore our products →