Building an in-house URL classifier is a multi-year commitment to training data pipelines, model maintenance, and taxonomy management. URL classification as a service gives your AI agent guardrails production-grade categorization data from day one — through a real-time API, offline database downloads, or both — without the infrastructure overhead of building your own.
Teams that try to build their own URL classification pipeline discover it requires dedicated ML infrastructure, continuous training data, and months of development before reaching production quality.
A URL classification system that works at agent-guardrail scale requires several components that teams underestimate at the outset. You need a web crawler to fetch and analyze page content across millions of domains. You need a training pipeline to label enough domains to train a reliable classifier. You need an ML model that can handle the 700+ category taxonomy without catastrophic accuracy degradation at the long tail. You need an update pipeline that reclassifies domains as their content changes. And you need to do all of this while maintaining sub-millisecond lookup latency so your agent harness does not add perceptible delay to every navigation event.
URL classification as a service eliminates the build phase entirely. Instead of spending months constructing a classification pipeline, you integrate a pre-built service that already classifies 102 million domains across the IAB taxonomy, web filtering categories, and 20+ page types. The service is available in two delivery modes: a real-time API that classifies any URL on demand with sub-200ms latency, and an offline database download that you deploy in your own infrastructure for sub-millisecond lookups.
The hybrid model is the most common deployment pattern: load the offline database for the 99.5% of domains that are pre-classified, and fall back to the real-time API for the 0.5% long tail of newly registered or rarely visited domains. This gives you 100% effective classification coverage with the latency profile of a local lookup and the freshness guarantee of a live API.
Real-time API for flexibility, offline database for performance — or both for complete coverage
Send any URL to the API and receive its classification in under 200 milliseconds. The response includes IAB content categories (all four tiers), web filtering categories, page-type labels, OpenPageRank scores, and global popularity rankings. The API handles any URL — including brand-new domains, deep-linked pages, and dynamically generated paths. Use it as a standalone classification engine or as a fallback for domains not in your local database.
Download the full 102 million domain database as CSV or JSON. Load it into Redis, PostgreSQL, SQLite, DynamoDB, or any data store your agent stack uses. Every lookup completes in under one millisecond with zero external network dependency. The database is a one-time purchase with perpetual license — no per-query fees, no monthly subscriptions, no API rate limits. Optional quarterly updates keep the data current with the latest domain classifications and threat intelligence.
The most robust deployment combines both modes. Load the offline database for primary lookups — it covers 99.5% of domains your agents will encounter. When a URL is not found in the local database (new domain, rare site, or dynamic URL), the middleware automatically falls back to the real-time API. The API response is cached locally, expanding your database organically over time. This hybrid approach delivers sub-millisecond performance for the vast majority of lookups with live classification for the edge cases.
Production-ready snippets for API-first and hybrid deployment patterns
import http.client
import json
import sqlite3
class HybridClassifier:
"""Local DB lookup with API fallback for complete coverage."""
def __init__(self, api_key, db_path="domains.sqlite"):
self.api_key = api_key
self.db = sqlite3.connect(db_path)
self.conn = http.client.HTTPSConnection(
"www.websitecategorizationapi.com"
)
def lookup_local(self, domain):
cursor = self.db.execute(
"SELECT iab_category, page_type, filtering_category, "
"pagerank, global_rank FROM domains WHERE domain = ?",
(domain,)
)
row = cursor.fetchone()
if row:
return {
"source": "local_db",
"iab_category": row[0],
"page_type": row[1],
"filtering_category": row[2],
"pagerank": row[3],
"global_rank": row[4]
}
return None
def classify_api(self, url):
payload = (
f"query={url}&api_key={self.api_key}"
f"&data_type=url&expanded_categories=1"
)
headers = {"Content-Type": "application/x-www-form-urlencoded"}
self.conn.request("POST",
"/api/iab/iab_web_content_filtering.php",
payload, headers)
data = json.loads(
self.conn.getresponse().read().decode("utf-8")
)
return {"source": "api", **data}
def classify(self, url):
from urllib.parse import urlparse
domain = urlparse(url).netloc or url
local = self.lookup_local(domain)
if local:
return local
return self.classify_api(url)
# Usage
classifier = HybridClassifier(api_key="your_api_key")
result = classifier.classify("https://example.com/pricing")
print(f"Source: {result['source']}, Category: {result.get('iab_category')}")
class ClassificationGateway {
constructor(apiKey, localCache = new Map()) {
this.apiKey = apiKey;
this.cache = localCache;
}
async classify(url) {
const domain = new URL(url).hostname;
// Check local cache first
if (this.cache.has(domain)) {
return { source: "cache", ...this.cache.get(domain) };
}
// Fall back to API
const res = await fetch(
"https://www.websitecategorizationapi.com" +
"/api/iab/iab_web_content_filtering.php",
{
method: "POST",
headers: { "Content-Type": "application/x-www-form-urlencoded" },
body: new URLSearchParams({
query: url, api_key: this.apiKey,
data_type: "url", expanded_categories: "1"
})
}
);
const data = await res.json();
// Cache the result for future lookups
const result = {
iabCategory: data.iab_classification?.[0]?.[0] || "Unknown",
pageType: data.page_type || "unknown",
filterCategory: data.filtering_taxonomy?.[0]?.[0] || "",
pageRank: data.open_pagerank || 0
};
this.cache.set(domain, result);
return { source: "api", ...result };
}
}
URL classification as a service — delivered as a downloadable database. IAB categories, page types, filtering labels, and reputation data. One-time purchase with perpetual license.
10 Million Domains — Full SaaS Classification
One-time purchase: Perpetual license | Optional Updates: $1,599/year
20 Million Domains — Complete Intelligence Suite
One-time purchase: Perpetual license | Optional Updates: $2,999/year
50 Million Domains with Complete Intelligence Suite
One-time purchase: Perpetual license | Optional Updates: $4,999/year
Also available: Enterprise URL Database up to 102M domains from $2,499. View all database tiers →
Explore the classification depth of our SaaS database — search any IAB or Web Filtering category to see domain counts and distribution data.
How 102 million domains from our main Enterprise Database are distributed across IAB v3 taxonomy classifications
Spanning Tier 1 through Tier 4 classifications from our 102M Enterprise Database
Charts display domain counts for the top 50 out of 700+ categories in our 102M Enterprise Database. To check the number of domains for the remaining 650+ categories, use the Category Counter tool above .
URL classification is infrastructure, not a feature. When your engineering team builds a URL classifier in-house, they are taking on the maintenance burden of what is effectively a separate product — one that requires its own data pipeline, its own ML models, its own taxonomy governance, and its own operational monitoring. Every hour spent maintaining the classifier is an hour not spent building the agent platform features your customers actually pay for.
Outsourcing classification to a service layer makes the same economic sense as outsourcing authentication to Auth0, payment processing to Stripe, or email delivery to SendGrid. The domain expertise required to maintain a high-quality URL classifier at internet scale — covering 102 million domains, 700+ categories, and continuously evolving web content — is a full-time specialization. Your team's core competency is building agent guardrails, not building URL classifiers.
The total cost of building an in-house URL classification system breaks down into four categories. Infrastructure costs for crawling, storing, and processing 100+ million web pages range from $5,000 to $15,000 per month in cloud compute and storage. ML engineering costs for building, training, and maintaining the classification model require at least one dedicated ML engineer, costing $150,000 to $250,000 per year in fully loaded compensation. Taxonomy management costs for maintaining consistent category definitions, handling edge cases, and updating the taxonomy as new content types emerge require 20-40 hours per month of senior engineering time. And operational costs for monitoring classification accuracy, retraining models, updating the crawl pipeline, and handling data quality issues add another 10-20 hours per month.
Against these costs, a pre-built classification service delivers equivalent or superior coverage for a one-time database purchase of $7,999 to $24,999 (for the AI Agent Database tier), with optional annual updates at a fraction of the build cost. The total cost of ownership over three years is typically 5-10x lower for the service approach, with the additional benefit of immediate deployment rather than a 6-12 month build timeline.
The choice between API-first and database-first integration depends on your agent platform's architecture and performance requirements. API-first integration is the simplest to implement: every URL the agent encounters triggers an API call, and the response drives the guardrail decision. This pattern works well for agent platforms handling fewer than 10,000 URL classifications per day, where the sub-200ms API latency is acceptable and the per-request cost is manageable.
Database-first integration front-loads the classification data into your infrastructure. You download the 102 million domain database, load it into your preferred data store, and serve all lookups locally. This pattern is necessary for agent platforms handling more than 10,000 classifications per day, where the cumulative API latency and cost become significant. With the database deployed locally, every lookup completes in under one millisecond with zero external dependency.
The hybrid pattern combines both: database-first for the 99.5% of domains that are pre-classified, API-first for the 0.5% long tail. This is the recommended pattern for production deployments because it delivers the best combination of performance, coverage, and cost efficiency.
URL classification as a service integrates with every major agent guardrail pattern. For middleware-based guardrails — where classification happens in a proxy layer between the agent and the internet — the service provides the data that the middleware evaluates. For policy-engine guardrails — where classification feeds into a rule evaluation engine — the service provides the inputs that policy rules match against. For observability-based guardrails — where classification is logged for post-hoc analysis — the service provides the metadata that audit reports reference.
Regardless of the guardrail architecture, the integration pattern is the same: intercept the URL, query the classification service (local database or API), receive the structured response (IAB category, web filtering label, page type, PageRank, popularity rank), evaluate the response against the guardrail policy, and execute the allow/block/log decision. The classification service is a data provider; the guardrail platform is a policy enforcer. Keeping these responsibilities separated makes both components easier to maintain, update, and audit independently.
The offline database ships in two formats: CSV for broad compatibility and JSON for structured parsing. The CSV format includes one row per domain with columns for the domain name, IAB v2 categories, IAB v3 categories, web filtering category, page type, OpenPageRank score, global popularity rank, and country-specific ranks. The JSON format nests the same data under a domain key with structured objects for each data dimension.
Both formats are designed for direct ingestion into production data stores. The CSV format loads into PostgreSQL, MySQL, or SQLite with a single COPY or import command. The JSON format loads into MongoDB, DynamoDB, or Elasticsearch with native ingestion tools. For Redis deployments, a simple script parses either format and populates hash keys with domain-level data. The database documentation includes sample ingestion scripts for all major data stores.
Classification accuracy is the single most important quality metric for a URL classification service. An inaccurate classification leads to either a false block (the agent is prevented from accessing a legitimate resource, degrading task completion) or a false allow (the agent accesses a restricted resource, creating a security or compliance incident). Our database maintains a 95%+ accuracy rate across all IAB categories, validated through a continuous sampling and human review process.
For web filtering categories — where false classifications have direct security implications — the accuracy target is 99%+. A domain incorrectly classified as non-malware when it is actually distributing malware is a critical false negative. Our web filtering labels are sourced from multiple threat intelligence feeds and validated against known-bad domain lists maintained by security research organizations.
Agent platform vendors serving multiple customers need URL classification that supports multi-tenant deployment. Each customer may have different guardrail policies — Customer A blocks gambling domains while Customer B allows them for research purposes. The classification data is the same for all tenants; the policy layer differs. Our database supports this pattern natively: deploy one shared classification database across all tenants, and implement per-tenant policy configurations that reference the shared classification data. This architecture minimizes data duplication while allowing unlimited policy customization per tenant.
Deploying URL classification as a service for your agent guardrails takes three steps. First, choose your delivery mode: API-only for development and testing, database-only for air-gapped or high-volume deployments, or hybrid for production environments that need both performance and complete coverage. Second, ingest the classification data into your preferred data store and verify that lookups return expected results for a sample set of domains. Third, wire the classification lookup into your agent harness as a pre-navigation middleware, so every URL is classified before the agent's HTTP request fires. The entire deployment typically takes less than one day for a team familiar with the agent framework.
Skip the build phase. Get production-grade URL classification for your agent guardrails from day one. API access, offline database, or both. One-time purchase, perpetual license.