Maximize your classification throughput by running multiple API requests concurrently. Process hundreds of thousands of domains efficiently.
Each API request fetches and analyzes a URL in real time — the response time depends on how quickly the target website loads and how much content it contains. In our benchmark of 100 popular domains, the median response time was ~7 seconds, with some fast sites completing in under 1 second and slower ones taking up to 50 seconds. To classify large batches efficiently, you should send multiple requests in parallel instead of one at a time.
One request at a time. Each waits for the previous one to finish.
100 domains × 10s avg = ~17 minutes
50 requests in flight at the same time. ~50× faster.
100 domains with 50 threads = ~20 seconds
The recommended approach uses Python's built-in concurrent.futures.ThreadPoolExecutor. Each thread sends one request at a time, and the pool manages up to 50 in parallel. The full JSON response from the API is saved for each domain.
#!/usr/bin/env python3
"""
Parallel URL classification using the Website Categorization API.
Saves the full API response for each domain.
"""
import json
import time
import urllib.request
import urllib.parse
from concurrent.futures import ThreadPoolExecutor, as_completed
# ── Configuration ──────────────────────────────────────────────
API_ENDPOINT = "https://www.websitecategorizationapi.com/api/iab/iab_web_content_filtering.php"
API_KEY = "YOUR_API_KEY" # Replace with your API key
MAX_THREADS = 50 # Default concurrency limit
TIMEOUT = 120 # Per-request timeout in seconds
INPUT_FILE = "domains.txt" # One domain per line
OUTPUT_FILE = "results.jsonl" # One JSON object per line
def classify_domain(domain):
"""Classify a single domain. Returns (domain, response_dict, status, elapsed)."""
start = time.time()
try:
data = urllib.parse.urlencode({
"query": domain,
"data_type": "url",
"api_key": API_KEY,
}).encode()
req = urllib.request.Request(API_ENDPOINT, data=data, headers={
"Content-Type": "application/x-www-form-urlencoded",
"User-Agent": "PythonClassifier/1.0",
"Accept": "application/json",
})
with urllib.request.urlopen(req, timeout=TIMEOUT) as resp:
body = json.loads(resp.read().decode("utf-8", "replace"))
return domain, body, resp.status, time.time() - start
except Exception as e:
return domain, {"error": str(e)}, 0, time.time() - start
def main():
# Load domains
with open(INPUT_FILE) as f:
domains = [line.strip() for line in f if line.strip()]
print(f"Classifying {len(domains)} domains with {MAX_THREADS} threads...")
t0 = time.time()
completed = 0
success = 0
# Open output file and write results as they complete
with open(OUTPUT_FILE, "w") as out, \
ThreadPoolExecutor(max_workers=MAX_THREADS) as pool:
futures = {pool.submit(classify_domain, d): d for d in domains}
for future in as_completed(futures):
domain, response, status, elapsed = future.result()
record = {
"domain": domain,
"status": status,
"response_time_s": round(elapsed, 2),
"response": response,
}
out.write(json.dumps(record, ensure_ascii=False) + "\n")
completed += 1
if status == 200:
success += 1
if completed % 100 == 0 or completed == len(domains):
wall = time.time() - t0
rate = completed / wall * 60
print(f" {completed}/{len(domains)} done — "
f"{rate:.0f} domains/min — "
f"{success} successful")
total_time = time.time() - t0
print(f"\nDone in {total_time:.1f}s — {success}/{len(domains)} successful")
print(f"Throughput: {len(domains)/total_time*60:.0f} domains/min")
print(f"Full responses saved to {OUTPUT_FILE}")
if __name__ == "__main__":
main()Save your domains to domains.txt, one per line:
google.com
amazon.com
cnn.com
bbc.co.uk
wikipedia.orgNo external packages needed — the script uses only Python's standard library.
python3 parallel_classify.pyResults are saved to results.jsonl — one JSON object per line containing the domain, HTTP status, response time, and the full API response with all classification data (IAB taxonomy, web filtering categories, confidence scores, and any enrichment fields).
The table below shows estimated processing times based on our live benchmark (100 domains, measured throughput of ~5.4 domains/min per thread). Actual times depend on target website response times.
| Batch Size | 10 Threads | 50 Threads | 100 Threads | 200 Threads | 500 Threads |
|---|---|---|---|---|---|
| 10,000 domains | ~3.1 hours | ~37 min | ~19 min | ~9 min | ~4 min |
| 50,000 domains | ~15.5 hours | ~3.1 hours | ~1.6 hours | ~46 min | ~19 min |
| 100,000 domains | ~31 hours | ~6.2 hours | ~3.1 hours | ~1.6 hours | ~37 min |
| 250,000 domains | ~3.2 days | ~15.5 hours | ~7.8 hours | ~3.9 hours | ~1.6 hours |
| 500,000 domains | ~6.5 days | ~31 hours | ~15.5 hours | ~7.8 hours | ~3.1 hours |
| 1,000,000 domains | ~12.9 days | ~2.6 days | ~31 hours | ~15.5 hours | ~6.2 hours |
For large batches, write results to disk as they complete rather than keeping everything in memory. The example script above does this automatically using JSONL format.
Remove duplicate domains before processing. Each unique domain consumes one API credit, so deduplication saves both time and credits.
Track your remaining credits during processing. The API returns HTTP 429 when credits are exhausted. Plan your batch sizes according to your plan's monthly allocation.
If your project requires faster processing, we can increase your parallel thread limit. Contact us at [email protected] with:
We will configure a higher concurrency limit tailored to your needs.
View Plans API Documentation