Route all AI agent HTTP traffic through a forward proxy that classifies every destination domain and applies category-based filtering rules before the request reaches the internet. Our 102 million domain database provides the classification layer that transforms a generic proxy into a purpose-built agent governance gateway.
Every enterprise filters employee web traffic through a proxy. AI agent traffic bypasses these controls entirely, creating an unmonitored egress path to the public internet.
Enterprise networks route employee web traffic through forward proxies — Zscaler, Broadcom (Symantec), McAfee Web Gateway, or open-source solutions like Squid. These proxies inspect every HTTP request, classify the destination, and apply category-based policies: block adult content, allow business sites, log social media access. But when an AI agent makes HTTP requests, those requests typically bypass the corporate proxy entirely. The agent runtime issues requests from a cloud VM, a container, or a serverless function that has no proxy configuration. The result is an uncontrolled, unmonitored egress channel to the entire internet.
Build a forward proxy specifically designed for AI agent HTTP traffic. Every request from an agent runtime is routed through this proxy. The proxy extracts the destination domain from each request, queries the 102M domain categorization database, and evaluates the result against the agent's category-based policy. Allowed categories pass through. Blocked categories receive an HTTP 403 response. Flagged categories are logged for review. The entire decision happens in the proxy layer — before the request reaches the destination server.
Unlike middleware-based filtering (which requires modifying the agent's code), a proxy-based approach is transparent to the agent. The agent makes standard HTTP requests; the proxy intercepts and filters them. This means you can apply category-based filtering to any agent framework — LangChain, CrewAI, AutoGen, custom implementations — without modifying a single line of agent code. Just point the agent's HTTP client to the proxy, and filtering is active.
Three components that transform a standard proxy into an agent governance gateway
The proxy sits between agent runtimes and the internet. Configure agent containers or VMs to use the proxy via HTTP_PROXY and HTTPS_PROXY environment variables. For HTTPS traffic, the proxy performs TLS interception using a CA certificate installed in the agent runtime's trust store. This gives the proxy visibility into the full URL — not just the domain — enabling path-level filtering for page-type classification.
The proxy's classification engine queries the locally deployed 102M domain database on every intercepted request. The database is loaded into an in-memory store (Redis or an embedded hash map) for sub-millisecond lookups. Each domain resolves to its IAB categories, web filtering classification, page type, and reputation score. For domains not in the local database, the proxy issues a real-time API call as a fallback, caching the result for subsequent requests.
The policy engine evaluates the classification result against the agent's assigned policy. Policies are defined per agent identity or per agent group. Each policy specifies allowed categories, blocked categories, blocked page types, and a default action for uncategorized domains. The decision (allow, block, or flag) is recorded in the proxy access log alongside the domain, category, page type, agent identifier, and timestamp.
Build a category-aware forward proxy for AI agent traffic
import http.client
import json
from urllib.parse import urlparse
class ProxyCategoryFilter:
"""Forward proxy handler that classifies destination
domains and applies category-based filtering rules."""
BLOCKED_CATEGORIES = [
"Adult", "Malware", "Phishing", "Gambling",
"Illegal Content", "Weapons", "Drugs"
]
BLOCKED_PAGE_TYPES = ["login", "checkout", "admin"]
def __init__(self, api_key, local_db=None):
self.api_key = api_key
self.local_db = local_db or {}
self.conn = http.client.HTTPSConnection(
"www.websitecategorizationapi.com"
)
self.request_log = []
def classify_domain(self, domain):
# Check local database first (sub-ms lookup)
if domain in self.local_db:
return self.local_db[domain]
# Fallback to real-time API
payload = (
f"query={domain}"
f"&api_key={self.api_key}"
f"&data_type=url"
f"&expanded_categories=1"
)
headers = {
"Content-Type": "application/x-www-form-urlencoded"
}
self.conn.request(
"POST",
"/api/iab/iab_web_content_filtering.php",
payload,
headers
)
res = self.conn.getresponse()
result = json.loads(res.read().decode("utf-8"))
self.local_db[domain] = result # Cache result
return result
def handle_proxy_request(self, agent_id, request_url):
parsed = urlparse(request_url)
domain = parsed.netloc
data = self.classify_domain(domain)
page_type = data.get("page_type", "unknown")
categories = [
c[0].split("Category name: ")[1]
for c in data.get("filtering_taxonomy", [])
]
decision = "ALLOW"
reason = "Approved category"
# Check page-type blocks
if page_type in self.BLOCKED_PAGE_TYPES:
decision = "BLOCK"
reason = f"Blocked page type: {page_type}"
# Check category blocks
for cat in categories:
if cat in self.BLOCKED_CATEGORIES:
decision = "BLOCK"
reason = f"Blocked category: {cat}"
break
self.request_log.append({
"agent": agent_id,
"domain": domain,
"categories": categories,
"page_type": page_type,
"decision": decision,
"reason": reason
})
return decision, reason
# Proxy integration
proxy = ProxyCategoryFilter(api_key="your_api_key")
decision, reason = proxy.handle_proxy_request(
agent_id="research-agent-01",
request_url="https://example.com/products"
)
print(f"Proxy decision: {decision} — {reason}")
class AgentProxyInterceptor {
constructor(apiKey, policyConfig) {
this.apiKey = apiKey;
this.policy = policyConfig;
this.domainCache = new Map();
this.accessLog = [];
}
async interceptRequest(agentId, requestURL) {
const domain = new URL(requestURL).hostname;
// Check cache first
let classification = this.domainCache.get(domain);
if (!classification) {
const response = await fetch(
"https://www.websitecategorizationapi.com" +
"/api/iab/iab_web_content_filtering.php",
{
method: "POST",
headers: {
"Content-Type":
"application/x-www-form-urlencoded"
},
body: new URLSearchParams({
query: domain,
api_key: this.apiKey,
data_type: "url",
expanded_categories: "1"
})
}
);
classification = await response.json();
this.domainCache.set(domain, classification);
}
const filterCat =
classification.filtering_taxonomy?.[0]?.[0]
?.replace("Category name: ", "") || "Unknown";
const pageType =
classification.page_type || "unknown";
// Evaluate against proxy policy
let decision = "FORWARD"; // proxy forwards request
if (this.policy.blockedCategories.includes(filterCat))
decision = "DROP"; // proxy returns 403
if (this.policy.blockedPageTypes.includes(pageType))
decision = "DROP";
this.accessLog.push({
agentId, domain, filterCat, pageType,
decision, ts: new Date().toISOString()
});
return { decision, domain, filterCat, pageType };
}
}
Purpose-built domain databases for AI agent filtering. Includes IAB categories, 20+ page types, reputation scores, and popularity rankings. One-time purchase with perpetual license.
10 Million Domains with Page-Type Intelligence
One-time purchase: Perpetual license | Optional Updates: $1,599/year
20 Million Domains with Full Intelligence Suite
One-time purchase: Perpetual license | Optional Updates: $2,999/year
50 Million Domains with Complete Intelligence Suite
One-time purchase: Perpetual license | Optional Updates: $4,999/year
Also available: Enterprise URL Database up to 102M domains from $2,499. View all database tiers →
Search any IAB or Web Filtering category to see how many domains are in our 102M Enterprise Database — the same data your proxy filtering rules will reference.
How 102 million domains from our main Enterprise Database are distributed across IAB v3 taxonomy classifications
Spanning Tier 1 through Tier 4 classifications from our 102M Enterprise Database
Charts display domain counts for the top 50 out of 700+ categories in our 102M Enterprise Database. To check the number of domains for the remaining 650+ categories, use the Category Counter tool above .
The proxy model has a thirty-year track record in enterprise security. Every organization with a mature security program routes employee web traffic through a forward proxy. The proxy provides visibility (the security team sees all web traffic), control (category-based policies are enforced at the network layer), and auditability (proxy logs feed into SIEM systems for compliance and incident response). When AI agents began browsing the web, this proven architecture became the natural foundation for agent traffic governance.
The key advantage of a proxy-based approach over middleware-based filtering is transparency. Middleware requires modifying the agent's code — adding a pre-navigation hook, importing a classification library, or wrapping every HTTP call in a policy check. This approach is fragile: agent frameworks update frequently, new tools may bypass the middleware, and developers may forget to apply the filter to new code paths. A proxy, by contrast, operates at the network layer. Every HTTP request from the agent runtime passes through the proxy, regardless of which library, framework, or tool generated the request. There is no code to modify and no hook to maintain.
There are three primary deployment patterns for an agent traffic filtering proxy. The first is a sidecar proxy: deploy the proxy as a container sidecar alongside each agent runtime in Kubernetes. All egress traffic from the agent pod routes through the sidecar. This pattern provides per-agent isolation and allows different agents to have different proxy policies. The second is a centralized gateway proxy: deploy a single proxy instance (or a load-balanced cluster) that all agent runtimes route through. This pattern simplifies management and provides a single point for policy updates. The third is a service mesh integration: if your infrastructure already uses a service mesh like Istio or Linkerd, integrate the category classification logic into the mesh's egress gateway.
Each pattern has trade-offs. Sidecar proxies offer the best isolation but require more resources. Centralized gateways are easier to manage but create a single point of failure. Service mesh integration leverages existing infrastructure but adds complexity to the mesh configuration. The right choice depends on your agent deployment scale, existing infrastructure, and operational preferences.
Modern web traffic is overwhelmingly HTTPS. Without TLS interception, the proxy can only see the destination hostname (via the TLS SNI extension) — not the full URL path. This limits filtering to domain-level decisions: allow or block the entire domain. With TLS interception enabled, the proxy terminates the TLS connection from the agent, decrypts the request, inspects the full URL (including path and query parameters), re-encrypts the request, and forwards it to the destination. This gives the proxy full visibility into the requested path, enabling page-type classification — blocking /admin while allowing /pricing on the same domain.
TLS interception requires installing the proxy's CA certificate in the agent runtime's trust store. For containerized agents, this is typically done at build time by adding the CA certificate to the container image. For VM-based agents, the certificate is installed via configuration management tools.
A well-designed proxy adds minimal latency to agent requests. The domain classification lookup — querying an in-memory database loaded from our 102M domain CSV — takes less than 1 millisecond. The TLS interception overhead adds approximately 2-5 milliseconds per request for the additional handshake. For agents making hundreds of requests per task, this overhead is negligible compared to the network latency of the actual HTTP requests (typically 50-500ms each).
At high request volumes (thousands of agents, millions of requests per day), the proxy's performance depends on the classification database deployment. Loading the full 102M database into Redis on a dedicated instance ensures consistent sub-millisecond lookups regardless of concurrency. For extremely high-throughput deployments, shard the database across multiple Redis instances by domain hash.
Proxy access logs are the definitive record of agent web activity. Each log entry captures the agent's source IP (or container ID), the destination domain, the full URL, the HTTP method, the response status code, the domain's classification (IAB category, web filtering category, page type), and the policy decision (forward or drop). These logs feed directly into existing SIEM systems — Splunk, Elasticsearch, Datadog, or cloud-native logging services — providing the same visibility into agent traffic that security teams already have for employee traffic.
The combination of domain classification and proxy logging enables powerful analytics: which categories do agents visit most frequently? Which agents trigger the most blocks? Are there anomalous access patterns that suggest a compromised agent? These questions are answerable only when agent traffic flows through a classification-aware proxy.
Enterprise Secure Web Gateways (SWGs) like Zscaler and Broadcom already include domain categorization and policy enforcement. Why not just route agent traffic through your existing SWG? You can — but there are practical limitations. First, SWGs are licensed per user or per seat, and agent traffic volumes can exceed employee traffic by orders of magnitude, creating unexpected licensing costs. Second, SWG categorization databases may not include the page-type labels (login, checkout, admin) that are critical for agent governance. Third, SWG policy engines are designed for human browsing patterns, not agent patterns — they lack agent identity integration, task-scoped policies, and agent-specific audit fields.
A purpose-built agent proxy using our domain database addresses all of these gaps. The database is a one-time purchase with no per-seat licensing. It includes 20+ page-type labels specifically designed for agent use cases. And the proxy policy engine can be customized to support agent-specific features like task-scoped permissions, delegation chains, and real-time policy updates.
The most performant proxy deployment loads the entire 102M domain database into memory at startup. The database ships as a CSV file that can be parsed into a hash map keyed by domain name. At 102 million entries, the in-memory footprint is approximately 8-12 GB depending on the data format and hash map implementation. For proxies running on cloud instances with 16+ GB RAM, this is a comfortable fit. For resource-constrained environments, load only the 10M or 20M AI Agent Database tier, which covers the most-visited domains and fits in 1-2 GB of RAM.
Alternatively, deploy the database in Redis alongside the proxy. Redis provides sub-millisecond lookups with minimal configuration. Load the CSV into Redis using a bulk import script, then query Redis from the proxy on each intercepted request. This approach decouples the database from the proxy process, enabling independent scaling and updates.
Route AI agent traffic through a forward proxy powered by 102 million classified domains. Category-based filtering, page-type blocking, and full audit logging in one deployment.