Every AI agent that touches the public internet needs an outer control layer -- a harness -- that decides in real-time which domains to allow, which to block, and which to escalate for review. Our 102 million domain categorization database provides the structured intelligence your harness needs to make deterministic, policy-driven navigation decisions at the domain level before any HTTP request fires.
Popular agent frameworks like LangChain, CrewAI, and AutoGen give agents the ability to browse the web -- but none of them include a domain-level governance layer out of the box.
An agent framework provides the scaffolding for tool use, memory, and reasoning loops. What it does not provide is a runtime boundary around where the agent can go on the internet. Without a dedicated harness layer that enforces domain-level controls, every agent deployment is effectively unbound -- free to visit any URL the model decides is relevant, regardless of whether that domain hosts malware, collects credentials, or violates your organization's acceptable use policy.
An agent harness wraps around the agent runtime and intercepts every outbound navigation request before it reaches the network. The harness extracts the target domain, queries our 102 million domain categorization database, and applies your policy rules to determine whether the request should proceed. This architecture gives you deterministic, sub-millisecond control over every domain your agents interact with -- without modifying the agent's core logic or prompt engineering.
The harness acts as a middleware layer between the agent's intent and the network. It consumes IAB taxonomy categories, page-type labels, web filtering classifications, and domain reputation scores to make allow/block/review decisions. Because the data is pre-computed and stored locally, there is no latency penalty, no external dependency at decision time, and no probabilistic uncertainty in the policy evaluation.
Three architectural components that transform a raw agent into a governed, controllable system
The intercept layer sits between the agent runtime and the HTTP client. Every call to fetch(), requests.get(), or any browser automation command is routed through this layer first. The intercept extracts the target domain from the URL, normalizes it (stripping subdomains, query parameters, and paths as needed), and passes it to the policy evaluator. This layer is framework-agnostic -- it works with LangChain tools, CrewAI actions, AutoGen functions, or custom agent implementations.
The policy evaluator takes the domain's categorization data -- IAB categories, page types, web filtering labels, reputation scores -- and evaluates it against your organization's rule set. Rules can be simple (block all domains in the "Adult" web filtering category) or compound (allow "Business and Finance" domains only if page type is not "login" and reputation score exceeds 5). The evaluator returns a deterministic allow/block/review decision in microseconds.
Every policy decision -- allow, block, or review -- is logged with the full context: timestamp, agent ID, target domain, categorization data returned, rule that matched, and the action taken. This audit trail gives security teams full visibility into agent browsing behavior and provides the evidence trail required for compliance reporting under frameworks like SOC 2, GDPR, and ISO 27001.
Production-ready harness middleware that wraps any agent framework with domain-level controls
import http.client
import json
from datetime import datetime
class DomainLevelHarness:
"""Wraps any agent framework with domain-level allow/block controls."""
POLICY_RULES = {
"blocked_page_types": ["login", "checkout", "settings", "admin"],
"blocked_categories": ["Adult", "Illegal Content", "Malware", "Phishing"],
"review_categories": ["Financial Services", "Healthcare"],
"min_reputation_score": 3
}
def __init__(self, api_key, audit_log_path="harness_audit.jsonl"):
self.api_key = api_key
self.audit_log = audit_log_path
self.conn = http.client.HTTPSConnection(
"www.websitecategorizationapi.com"
)
def classify_domain(self, target_url):
payload = (
f"query={target_url}"
f"&api_key={self.api_key}"
f"&data_type=url"
f"&expanded_categories=1"
)
headers = {
"Content-Type": "application/x-www-form-urlencoded"
}
self.conn.request(
"POST",
"/api/iab/iab_web_content_filtering.php",
payload,
headers
)
res = self.conn.getresponse()
return json.loads(res.read().decode("utf-8"))
def evaluate_policy(self, target_url, agent_id="default"):
data = self.classify_domain(target_url)
categories = [
c[0].split("Category name: ")[1]
for c in data.get("iab_classification", [])
]
page_type = data.get("page_type", "unknown")
reputation = data.get("reputation_score", 0)
decision = {"action": "allow", "reason": "No policy violation"}
if page_type in self.POLICY_RULES["blocked_page_types"]:
decision = {"action": "block",
"reason": f"Blocked page type: {page_type}"}
for cat in categories:
for blocked in self.POLICY_RULES["blocked_categories"]:
if blocked.lower() in cat.lower():
decision = {"action": "block",
"reason": f"Blocked category: {cat}"}
for review in self.POLICY_RULES["review_categories"]:
if review.lower() in cat.lower():
decision = {"action": "review",
"reason": f"Review category: {cat}"}
self._log_decision(agent_id, target_url, decision, data)
return decision
def _log_decision(self, agent_id, url, decision, classification):
entry = {
"timestamp": datetime.utcnow().isoformat(),
"agent_id": agent_id,
"url": url,
"decision": decision,
"classification_summary": {
"page_type": classification.get("page_type"),
"category_count": len(
classification.get("iab_classification", [])
)
}
}
with open(self.audit_log, "a") as f:
f.write(json.dumps(entry) + "\n")
# Wrap your agent's browsing function
harness = DomainLevelHarness(api_key="your_api_key")
result = harness.evaluate_policy(
"https://example.com/admin/settings", agent_id="agent-42"
)
if result["action"] == "block":
print(f"Harness blocked navigation: {result['reason']}")
class DomainHarness {
constructor(apiKey, policyConfig) {
this.apiKey = apiKey;
this.policy = policyConfig;
this.auditLog = [];
}
async interceptNavigation(targetURL, agentId) {
const classification = await this.classifyDomain(targetURL);
const decision = this.applyPolicy(classification);
this.auditLog.push({
timestamp: new Date().toISOString(),
agentId,
url: targetURL,
decision: decision.action,
reason: decision.reason,
pageType: classification.page_type
});
return decision;
}
async classifyDomain(targetURL) {
const response = await fetch(
"https://www.websitecategorizationapi.com" +
"/api/iab/iab_web_content_filtering.php",
{
method: "POST",
headers: {
"Content-Type": "application/x-www-form-urlencoded"
},
body: new URLSearchParams({
query: targetURL,
api_key: this.apiKey,
data_type: "url",
expanded_categories: "1"
})
}
);
return response.json();
}
applyPolicy(classification) {
const pageType = classification.page_type || "unknown";
const filterCat =
classification.filtering_taxonomy?.[0]?.[0]
?.replace("Category name: ", "") || "Unknown";
if (this.policy.blockedPageTypes.includes(pageType)) {
return { action: "block",
reason: `Blocked page type: ${pageType}` };
}
if (this.policy.blockedCategories.includes(filterCat)) {
return { action: "block",
reason: `Blocked category: ${filterCat}` };
}
return { action: "allow", reason: "Policy check passed" };
}
}
Purpose-built domain databases for AI agent filtering. Includes IAB categories, 20+ page types, reputation scores, and popularity rankings. One-time purchase with perpetual license.
10 Million Domains with Page-Type Intelligence
One-time purchase: Perpetual license | Optional Updates: $1,599/year
20 Million Domains with Full Intelligence Suite
One-time purchase: Perpetual license | Optional Updates: $2,999/year
50 Million Domains with Complete Intelligence Suite
One-time purchase: Perpetual license | Optional Updates: $4,999/year
Also available: Enterprise URL Database up to 102M domains from $2,499. View all database tiers →
Search any IAB or Web Filtering category to see how many domains are in our 102M Enterprise Database -- the same data your agent harness policy engine will reference.
How 102 million domains from our main Enterprise Database are distributed across IAB v3 taxonomy classifications
Spanning Tier 1 through Tier 4 classifications from our 102M Enterprise Database
Charts display domain counts for the top 50 out of 700+ categories in our 102M Enterprise Database. To check the number of domains for the remaining 650+ categories, use the Category Counter tool above .
An agent harness is not a feature you bolt on after deployment. It is a foundational architectural component that must be designed into the agent stack from day one. The harness wraps the agent runtime -- whether that runtime is LangChain, CrewAI, AutoGen, Anthropic Computer Use, OpenAI Operator, or a custom framework -- and enforces domain-level controls at the network boundary. Without it, your agent's browsing behavior is ungovernable, unauditable, and uninsurable.
The concept borrows from established patterns in infrastructure security. Just as a service mesh enforces mTLS, rate limiting, and access control between microservices, an agent harness enforces categorization checks, policy rules, and audit logging between an AI agent and the open internet. The difference is that the policy data comes from a domain categorization database rather than a certificate authority or an identity provider.
The decision to enforce controls at the domain level -- rather than at the URL path level or the full page content level -- is deliberate and important. Domain-level control strikes the optimal balance between coverage and performance. A domain classification covers every page on that domain with a single lookup. Path-level classification requires a separate lookup for every distinct URL, which multiplies query volume by 10x to 100x. Full-page content analysis requires actually fetching the page and running NLP inference, which adds seconds of latency and creates a chicken-and-egg problem: you need to visit the page to classify it, but you need to classify it before visiting.
Domain-level categorization is pre-computed and static (until database refresh), which means the lookup is deterministic and sub-millisecond. The 102 million domain database covers 99.5% of the active internet, so the false-negative rate -- domains the agent encounters that have no classification -- is negligible. For the 0.5% long tail, the real-time API provides on-demand classification as a fallback.
Production harness deployments rarely use a flat allow/block list. Instead, they implement a layered policy architecture with multiple tiers of rules that are evaluated in priority order. The first tier is a hard blocklist -- domains and categories that are always blocked regardless of context. This includes web filtering categories like Malware, Phishing, Adult, and Illegal Content. The second tier is a task-scoped allowlist -- categories and page types that are permitted for the agent's specific task. A financial research agent might be allowed to access "Business and Finance" domains but blocked from "Healthcare" or "Entertainment" categories. The third tier is a review queue -- domains that do not match any explicit allow or block rule and are held for human review before the agent proceeds.
This three-tier architecture ensures that security-critical decisions (tier 1) are never overridden, task-relevant decisions (tier 2) are enforced consistently, and ambiguous cases (tier 3) receive human judgment rather than a default allow or default deny that could either halt the agent or expose the organization.
Modern agent workflows are not single-page visits. An agent researching a topic might visit 50 to 200 domains in a single task execution. A stateful harness tracks the sequence of domains visited, the cumulative risk profile, and the total exposure across the session. This enables policy rules that cannot be expressed in stateless per-domain checks. For example, a rule like "block the agent if it has visited more than 5 domains in the Financial Services category within a single task" requires session state. Similarly, "escalate if the agent navigates to 3 different login pages in under 60 seconds" requires temporal awareness.
The harness maintains a session ledger -- a running record of every domain visited, the categorization data returned, the policy decision made, and the timestamp. This ledger feeds into both real-time policy evaluation (for stateful rules) and post-task audit (for compliance reporting). The session ledger is append-only, ensuring that the audit trail is tamper-evident.
The integration pattern varies by framework but follows a consistent middleware approach. In LangChain, the harness is implemented as a custom CallbackHandler that intercepts the on_tool_start event for browsing tools. Before the tool executes, the callback queries the domain database and evaluates the policy. If the result is "block," the callback raises an exception that prevents the tool from executing and returns a structured error message to the agent.
In CrewAI, the harness wraps the agent's browsing action as a pre-execution hook. The hook receives the action parameters (including the target URL), runs the categorization check, and either allows the action to proceed or replaces it with a "navigation blocked" action that the agent can reason about. In AutoGen, the harness is registered as a function tool that the agent must invoke before every browsing call -- the framework's built-in tool calling ensures the check always runs.
Performance is non-negotiable for an agent harness. If the categorization check adds noticeable latency to each navigation decision, it slows the agent's overall workflow and degrades the user experience. A database-driven harness operating against a local data store achieves sub-millisecond lookup times. Redis-backed lookups complete in 0.1ms to 0.5ms. SQLite lookups complete in 0.5ms to 2ms. PostgreSQL lookups with a hash index on the domain column complete in 1ms to 3ms.
Compare this to an API-based or model-based approach. An API call to an external classification service adds 100ms to 500ms per lookup. A secondary LLM evaluation adds 500ms to 3000ms per lookup. Over a 100-domain browsing session, the database approach adds 50ms of total overhead. The API approach adds 10 to 50 seconds. The LLM approach adds 50 to 300 seconds. The performance gap widens with every additional domain the agent visits.
The audit trail generated by the harness serves multiple stakeholders. Security teams use it to investigate incidents -- when an agent exhibits unexpected behavior, the audit trail shows exactly which domains it visited, what categories those domains belonged to, and what policy decisions were made at each step. Compliance teams use it to demonstrate regulatory adherence -- the trail provides evidence that the organization's AI agents are operating within defined boundaries and that policy violations are detected and prevented in real-time.
The audit trail should be stored in an immutable, append-only data store. Common implementations include writing to Amazon S3 with object lock, appending to a Kafka topic with retention policies, or inserting into a PostgreSQL table with row-level security. The key requirement is that audit entries cannot be modified or deleted after creation, ensuring the integrity of the compliance record.
Enterprise deployments run dozens to hundreds of concurrent agent instances. The harness architecture must scale horizontally without introducing contention or single points of failure. The recommended pattern is to deploy the domain database as a shared service (Redis cluster or distributed cache) and the policy evaluator as a stateless function that each agent instance invokes locally. This separates the data layer (shared, replicated) from the compute layer (per-agent, stateless).
Each agent instance maintains its own session ledger in local memory during task execution. At task completion, the ledger is flushed to the centralized audit store. This pattern avoids write contention on the centralized store during task execution while ensuring all audit data is eventually consistent and queryable.
The harness logic -- the intercept layer, policy evaluator, and audit logger -- is typically built in-house because it needs to integrate tightly with your specific agent framework and policy requirements. The domain categorization data, however, is a build-vs-buy decision that almost always favors buying. Building a 102 million domain categorization database from scratch requires web crawling infrastructure, NLP classification models, continuous retraining pipelines, and ongoing data quality management -- a multi-year, multi-million-dollar engineering effort.
Our database provides the categorization data as a one-time purchase with optional annual updates. You get the data in CSV or JSON format, load it into your preferred data store, and your harness queries it locally. No ongoing API dependency, no per-query costs, no external service to rely on during agent runtime. The data is yours to deploy however your architecture requires.
Deploy domain-level controls as the foundation of your agent governance architecture. One-time purchase, perpetual license, 102 million domains classified and ready for harness integration.