Understanding Webhooks
Webhooks provide a way for our API to notify your application when certain events occur, such as completion of batch processing jobs. Instead of polling the API repeatedly to check job status, you can register a webhook URL and receive a POST request when the operation completes. This enables efficient asynchronous workflows for large-scale data processing.
Setting Up Webhooks
Configure your webhook endpoint in your dashboard or include it in API requests that support webhook callbacks:
// Include webhook URL in batch request
POST /v1/batch/categorize
{
"domains": ["example1.com", "example2.com", "..."],
"webhook_url": "https://your-app.com/webhooks/categorization",
"webhook_secret": "your_webhook_secret"
}
Webhook Events
We send webhooks for the following event types:
batch.completed
Triggered when a batch categorization job finishes processing all domains. Includes results summary and download link for full results.
batch.failed
Triggered if a batch job fails due to errors. Includes error details and partial results if available.
quota.warning
Triggered when your API usage reaches 80% of your monthly quota. Helps you anticipate and manage usage limits.
quota.exceeded
Triggered when your monthly quota is exhausted. Prompts you to upgrade your plan to continue service.
Webhook Payload Structure
Webhook payloads follow a consistent structure with event metadata and event-specific data:
{
"event": "batch.completed",
"timestamp": "2024-01-15T10:30:00Z",
"webhook_id": "wh_abc123xyz",
"data": {
"batch_id": "batch_def456",
"total_domains": 1000,
"successful": 985,
"failed": 15,
"processing_time_seconds": 45,
"results_url": "https://api.websitecategorizationapi.com/v1/batch/batch_def456/results"
}
}
Verifying Webhook Signatures
All webhooks include a signature header for verification. Always verify signatures to ensure webhooks originated from our servers:
// Webhook headers
X-Webhook-Signature: sha256=abcd1234...
X-Webhook-Timestamp: 1705315800
// Verification in Node.js
const crypto = require('crypto');
function verifyWebhook(payload, signature, secret) {
const timestamp = req.headers['x-webhook-timestamp'];
const signedPayload = `${timestamp}.${JSON.stringify(payload)}`;
const expectedSignature = crypto
.createHmac('sha256', secret)
.update(signedPayload)
.digest('hex');
return `sha256=${expectedSignature}` === signature;
}
Security Best Practice
Always verify webhook signatures before processing. Reject webhooks with invalid signatures to prevent spoofed requests. Store your webhook secret securely using the same practices as your API key.
Handling Webhook Delivery
Your webhook endpoint should respond quickly with a 2xx status code to acknowledge receipt. We retry failed deliveries with exponential backoff:
- Initial delivery attempt immediately after event
- First retry after 30 seconds
- Second retry after 2 minutes
- Third retry after 10 minutes
- Final retry after 1 hour
// Example webhook handler (Express.js)
app.post('/webhooks/categorization', (req, res) => {
// Verify signature first
if (!verifyWebhook(req.body, req.headers['x-webhook-signature'], WEBHOOK_SECRET)) {
return res.status(401).send('Invalid signature');
}
// Acknowledge receipt immediately
res.status(200).send('OK');
// Process webhook asynchronously
processWebhookAsync(req.body);
});