Autonomous AI agents need hard boundaries. A blocklist API powered by domain categorization data lets you define which website categories, page types, and reputation tiers are off-limits — and enforce those restrictions at the API layer before the agent's HTTP request ever fires. No prompt engineering. No model-based guessing. Deterministic blocking backed by 102 million classified domains.
An AI agent with unrestricted web access is a liability. Without a blocklist, the agent treats every domain on the internet as fair game — including malware sites, adult content, login portals, and internal admin panels.
When you deploy an AI agent to perform web-based tasks — competitive research, lead generation, market analysis, content aggregation — that agent will follow links, chase redirects, and explore domains that no human would visit intentionally. Without a domain-level blocklist, you are relying entirely on the agent's language model to make safety decisions about URLs. This is fundamentally unreliable because language models were not trained to evaluate domain safety — they were trained to generate text.
Our blocklist API wraps a 102 million domain categorization database in a simple REST interface. Before an agent navigates to any URL, your middleware sends the domain to the blocklist API. The API returns the domain's IAB content category, web filtering category, page type, and reputation score. Your policy engine evaluates the response against your blocklist rules and returns an allow or block decision — all in under 200ms for API calls, or under 1ms when using the local database.
The blocklist is fully configurable. Block entire IAB categories (Adult, Illegal Content, Gambling). Block specific page types (login, checkout, admin, settings). Block domains below a reputation threshold. Block domains outside a specific country or language. The rules are declarative — defined in a policy file, not embedded in prompts — which means they are auditable, version-controlled, and consistent across every agent in your fleet.
Three blocking strategies that turn domain categorization into agent-level access control
Define blocklist rules using IAB content taxonomy categories. Block all domains classified as "Adult Content," "Illegal Activities," "Malware/Spyware," or "Gambling." The API evaluates every URL against your category blocklist and returns a block decision with the triggering category. No ambiguity, no false negatives — if the domain is classified in a blocked category, the agent cannot visit it.
Block specific page types regardless of domain category. Login pages, checkout pages, admin panels, and settings pages represent high-risk navigation targets for AI agents. Even on a trusted domain, an agent that reaches a login page can trigger security alerts, lock accounts, or inadvertently submit credentials. Page-type blocking prevents these scenarios at the API layer.
Set minimum reputation thresholds for agent navigation. Domains with low PageRank scores, no global ranking, or minimal web presence are more likely to be spam, parked, or malicious. The API returns reputation signals for every domain, and your policy engine can block any domain that falls below your configured trust threshold.
Production-ready snippets to implement domain blocking in your agent harness
import http.client
import json
from urllib.parse import urlparse
class BlocklistAPI:
"""API-driven blocklist for AI agent domain restriction."""
def __init__(self, api_key, blocklist_config):
self.api_key = api_key
self.config = blocklist_config
self.conn = http.client.HTTPSConnection(
"www.websitecategorizationapi.com"
)
self.decision_log = []
def check_domain(self, target_url):
domain = urlparse(target_url).netloc
payload = (
f"query={domain}"
f"&api_key={self.api_key}"
f"&data_type=domain"
f"&expanded_categories=1"
)
headers = {"Content-Type": "application/x-www-form-urlencoded"}
self.conn.request("POST",
"/api/iab/iab_web_content_filtering.php",
payload, headers)
res = self.conn.getresponse()
return json.loads(res.read().decode("utf-8"))
def evaluate(self, target_url):
data = self.check_domain(target_url)
# Extract categories
iab_cats = [
c[0].split("Category name: ")[1]
for c in data.get("iab_classification", [])
]
filter_cat = (
data.get("filtering_taxonomy", [[""]])[0][0]
.split("Category name: ")[-1]
)
page_type = data.get("page_type", "unknown")
# Check category blocklist
for cat in iab_cats:
if cat in self.config["blocked_categories"]:
return self._block(target_url, f"Blocked category: {cat}")
# Check web filter blocklist
if filter_cat in self.config["blocked_filters"]:
return self._block(target_url, f"Blocked filter: {filter_cat}")
# Check page type blocklist
if page_type in self.config["blocked_page_types"]:
return self._block(target_url, f"Blocked page type: {page_type}")
return self._allow(target_url, iab_cats, page_type)
def _block(self, url, reason):
decision = {"url": url, "action": "BLOCK", "reason": reason}
self.decision_log.append(decision)
return decision
def _allow(self, url, categories, page_type):
decision = {"url": url, "action": "ALLOW",
"categories": categories, "page_type": page_type}
self.decision_log.append(decision)
return decision
# Configuration
blocklist = BlocklistAPI("your_api_key", {
"blocked_categories": ["Adult", "Illegal Content", "Malware"],
"blocked_filters": ["Adult", "Gambling", "Weapons"],
"blocked_page_types": ["login", "checkout", "admin", "settings"]
})
result = blocklist.evaluate("https://example.com/admin/panel")
print(result) # {"action": "BLOCK", "reason": "Blocked page type: admin"}
const express = require("express");
const app = express();
app.use(express.json());
const BLOCKED_CATEGORIES = new Set([
"Adult", "Illegal Content", "Malware", "Phishing", "Gambling"
]);
const BLOCKED_PAGE_TYPES = new Set([
"login", "checkout", "admin", "settings", "signup"
]);
app.post("/api/blocklist/check", async (req, res) => {
const { url, agentId } = req.body;
const classification = await fetch(
"https://www.websitecategorizationapi.com" +
"/api/iab/iab_web_content_filtering.php",
{
method: "POST",
headers: { "Content-Type": "application/x-www-form-urlencoded" },
body: new URLSearchParams({
query: url,
api_key: process.env.CATEGORIZATION_API_KEY,
data_type: "url",
expanded_categories: "1"
})
}
).then(r => r.json());
const filterCat = classification.filtering_taxonomy?.[0]?.[0]
?.replace("Category name: ", "") || "Unknown";
const pageType = classification.page_type || "unknown";
if (BLOCKED_CATEGORIES.has(filterCat)) {
return res.json({ action: "BLOCK", reason: `Category: ${filterCat}` });
}
if (BLOCKED_PAGE_TYPES.has(pageType)) {
return res.json({ action: "BLOCK", reason: `Page type: ${pageType}` });
}
return res.json({
action: "ALLOW",
category: filterCat,
pageType: pageType,
agentId: agentId
});
});
app.listen(3000, () => console.log("Blocklist API on :3000"));
Purpose-built domain databases for AI agent filtering. Includes IAB categories, 20+ page types, reputation scores, and popularity rankings. One-time purchase with perpetual license.
10 Million Domains with Page-Type Intelligence
One-time purchase: Perpetual license | Optional Updates: $1,599/year
20 Million Domains with Full Intelligence Suite
One-time purchase: Perpetual license | Optional Updates: $2,999/year
50 Million Domains with Complete Intelligence Suite
One-time purchase: Perpetual license | Optional Updates: $4,999/year
Also available: Enterprise URL Database up to 102M domains from $2,499. View all database tiers →
Search any IAB or Web Filtering category to see how many domains your blocklist API would cover from our 102M Enterprise Database.
How 102 million domains from our main Enterprise Database are distributed across IAB v3 taxonomy classifications
Spanning Tier 1 through Tier 4 classifications from our 102M Enterprise Database
Charts display domain counts for the top 50 out of 700+ categories in our 102M Enterprise Database. To check the number of domains for the remaining 650+ categories, use the Category Counter tool above .
A blocklist API for AI agents is not a simple keyword filter. It is a policy enforcement layer that must handle millions of domain lookups per day, return decisions in single-digit milliseconds, maintain consistent classifications across time, and produce audit-grade logs for every block and allow decision. Building this system on top of a pre-classified domain database gives you a head start — the classification work is already done, and your engineering effort focuses on the policy logic and the API infrastructure.
The architecture has three components: a domain classification store (the 102M domain database loaded into Redis or PostgreSQL), a policy engine (a rules evaluator that maps classifications to block/allow decisions), and an API layer (a REST or gRPC endpoint that agents call before every navigation event). The database provides the facts. The policy engine provides the rules. The API provides the interface.
Category-level blocking is the broadest and most impactful layer. By blocking entire IAB categories, you eliminate large swaths of the internet that agents should never visit. The standard enterprise blocklist includes Adult Content, Illegal Activities, Malware and Spyware, Phishing, Spam, Gambling, Weapons, Drugs, and Hate Speech. These categories represent domains that are dangerous, inappropriate, or legally problematic regardless of the agent's task.
Beyond the universal blocklist, task-specific category blocks add a second layer. A financial research agent might be blocked from Shopping and Entertainment categories to keep it focused on financial sources. A content marketing agent might be blocked from Competitor sites to prevent accidental content scraping. The blocklist API accepts category rules as configuration — no code changes required to add or remove blocked categories.
Even on trusted domains, certain page types are dangerous for autonomous agents. A login page on your company's own website is a trusted domain — but an agent that navigates to that login page and attempts to interact with it could lock accounts, trigger security alerts, or expose credentials. Page-type blocking operates independently of category blocking: even if the domain category is "allow," the page type can trigger a block.
The critical page types to block for most agent deployments are: login, signup, checkout, payment, settings, admin, dashboard, and account management pages. Our database classifies pages into 20+ distinct types, giving your blocklist API granular control over which interactions are permitted and which are forbidden.
Not every dangerous domain fits neatly into a blocked category. Newly registered domains, parked pages, and domains with no web presence are often used for spam, phishing, or malware distribution — but they may not yet be classified into explicit danger categories. Reputation-based blocking catches these domains by setting minimum thresholds for PageRank scores and global popularity rankings. A domain with a PageRank of 0 and no global ranking is statistically much more likely to be malicious than a domain with established authority and traffic.
The blocklist API can operate in two modes. In local mode, the full 102M domain database is loaded into a local data store (Redis, PostgreSQL, SQLite), and the blocklist API performs lookups against this local store. Local mode delivers sub-millisecond response times and zero external dependencies. In remote mode, the blocklist API calls our classification API for each domain lookup. Remote mode requires no local data storage but adds 100-200ms of network latency per lookup.
The recommended architecture is hybrid: use the local database for the 99.5% of domains it covers, and fall back to the remote API for the 0.5% of domains not in the local store. This hybrid approach delivers the speed of local lookups with the coverage of real-time classification.
Different agent frameworks require different integration patterns for the blocklist API. In LangChain, the blocklist check is implemented as a Tool wrapper that intercepts the browsing tool's URL input, queries the blocklist API, and either passes the URL through or raises a ToolException that the agent handles gracefully. In CrewAI, the blocklist integrates as a pre-task hook that validates all URLs in the task's context before execution begins. In custom agent frameworks, the blocklist API is called as middleware in the HTTP client layer — every outbound request passes through the blocklist check before the TCP connection is established.
A blocklist API generates valuable security telemetry. Every block decision reveals an attempt by an agent to visit a restricted domain. Patterns in block decisions can reveal prompt injection attacks (where malicious instructions direct agents to dangerous URLs), misconfigured agent tasks (where the agent's research scope is too broad), or emerging threats (where previously uncategorized domains start appearing in agent navigation attempts). Feed blocklist logs into your SIEM or observability platform, and set alerts for unusual block rates, new blocked domains, and attempts to access high-severity categories like Malware or Phishing.
Some teams attempt to build blocklist logic directly into their agent's system prompt: "Do not visit adult websites. Do not visit login pages. Do not visit malware domains." This approach fails for three reasons. First, the language model has no reliable mechanism to determine whether a URL is an adult site, a login page, or a malware domain — it can only guess based on the URL string. Second, prompt-based rules are bypassable through prompt injection — a malicious website can include instructions that override the agent's system prompt. Third, prompt-based filtering produces no audit trail — there is no structured log of what was blocked or why.
A database-backed blocklist API solves all three problems. The database provides ground-truth classifications based on actual website content analysis, not URL string patterns. The API operates outside the model's context window, making it immune to prompt injection. And every block decision is logged with full context — domain, category, page type, policy rule, and timestamp — creating an auditable record of every filtering decision.
Deploy a blocklist API backed by 102 million pre-classified domains. Category blocking, page-type blocking, and reputation filtering — all in one database. One-time purchase, perpetual license.