WebsiteCategorizationAPI
Try Categorization
Home
Demo Tools - Categorization
Website Categorization Text Classification URL Database Taxonomy Mapper
Demo Tools - Website Intel
Technology Detector Quality Score Competitor Finder
Demo Tools - Brand Safety
Brand Safety Checker Brand Suitability Quality Checker
Demo Tools - Content
Sentiment Analyzer Context Aware Ads
Resources
API Documentation Pricing Login
Try Categorization

Map of the Internet for AI Agents

Empower your AI agents with a map of the internet — our in-depth 20 million domains database with comprehensive website intelligence. Every domain is categorized with IAB Tier 1-4 categories and Web Filtering classifications. Access pricing pages, login portals, contact information, leadership teams, careers, and more.

See Databases & Pricing

AI Agent Domain Database — Page-Level Intelligence

The only pre-indexed page-level intelligence database for AI agents. Domains sourced from the Google Chrome UX Report (CrUX) — the definitive record of real-world internet usage. Coverage available up to 30M domains.

AI Agent Database

AI Agent Database — 10M

Google CrUX Top 10M

$7,999

One-time purchase • perpetual license

Optional Annual Updates: $1,599/year for monthly refreshes

  • 10M domains (CrUX Top 10M)
  • IAB Tier 1-4 categories
  • Web filtering classifications
  • 20 page-type URLs per domain
  • 2000+ user personas
  • OpenPageRank + Domain Ages
  • CrUX Global & Country Rank
  • Country classification
  • Priority enterprise support
Popular

AI Agent Database — 15M

Extended active web • 99.85% traffic coverage

$14,999

One-time purchase • perpetual license

Optional Annual Updates: $2,999/year for monthly refreshes

  • 15M domains (CrUX + extended)
  • Everything in 10M tier, plus:
  • Long-tail publisher coverage
  • Regional & niche domains
  • 20 page-type URLs per domain
  • Full metadata suite
  • Ideal for AdTech & cybersecurity
  • Priority enterprise support
Maximum Coverage

AI Agent Database — 30M

Maximum coverage • 30M domains

$24,999

One-time purchase • perpetual license

Optional Annual Updates: $4,999/year for monthly refreshes

  • 30M domains — maximum coverage
  • Everything in 15M tier, plus:
  • Long-tail & regional domains
  • Complete active web coverage
  • 20 page-type URLs per domain
  • Full metadata suite
  • Ideal for enterprise-scale filtering
  • Dedicated account manager

Why this pricing? To build this database, we traversed the link structure of every domain and individually analyzed over 10 billion pages through a multi-step AI classification pipeline to identify up to 20 page types per domain. The database returns verified, actual URLs for each page type — not pattern guesses. Building this internally would require crawling millions of domains, traversing billions of links, running multi-stage AI classification, and maintaining freshness — costing hundreds of thousands in infrastructure annually. Our database delivers all of this instantly with a one-time purchase. Page-level intelligence is available across all tiers up to 30M domains.

Do you need a custom database?

Tell us your specific requirements for the AI agents database. We can build a custom slice from our 102 million domain repository tailored exactly to your needs — by industry, geography, page type, or any combination.

[email protected]

Agent Governance & Guardrails

Learn how to build safe, compliant AI agent systems with URL categorization as the policy backbone. Explore our knowledgebase articles.

Enterprise Guardrails for Agentic AI Web Access

How enterprises can control and monitor AI agent browsing behavior using URL classification policies.

The Policy Engine Every AI Agent Browser Needs

Why every autonomous browser agent needs a URL-level policy engine to enforce allow/block rules in real time.

What a CASB for AI Agents Looks Like

Applying Cloud Access Security Broker concepts to AI agent traffic — visibility, control, and compliance.

Zero Trust for Autonomous Web Agents

Implementing zero-trust architecture for AI agents that browse the web — verify every URL before allowing access.

AI Agent Governance Platform with URL Categories

Building a governance platform that uses URL categorization to enforce policies across all AI agent web interactions.
Deep Link Analysis Pipeline

Over 10 Billion Links Individually Analyzed

To identify up to 20 key page types for each of our 102 million domains, we traversed and individually analyzed over 10 billion links using our sophisticated multi-step AI page-type identification pipeline. Each link was visited, its content examined, and its page type classified through multiple AI model stages.

10B+
Links Traversed
102M
Domains Covered
20
Page Types Identified
Multi-Step
AI Classification
Link Structure Traversal
For each domain, we crawl the complete link structure — navigation menus, footer links, sitemaps, and internal references — to discover every reachable page. This is not a simple URL pattern check; we follow the actual link graph of each website.
Multi-Step AI Page-Type Identification
Each discovered page is analyzed through a multi-stage AI pipeline that examines page content, structure, and context to determine its type. A login page might live at /welcome, /portal, or /access — our models identify it regardless of URL format.
Actual URLs Returned Per Domain
The database returns the real, verified URLs for each identified page type per domain. When you query for "login page of example.com," you get the actual URL — not a guess, not a pattern match, but the confirmed link discovered and classified by our pipeline.
Pre-Classified Page-Type URLs

Why Pre-Classified URLs for 102M Domains
Changes Everything for AI Agents

Having pre-classified URLs for 20 page types across 102 million domains at the start of any agent task means your agents skip the discovery phase entirely. The result: orders of magnitude faster task completion.

Orders of Magnitude Faster

Without pre-classified data, an agent must crawl each domain, follow links, load pages, and analyze content to find a login or pricing page. That takes seconds to minutes per domain. With our database, the agent gets the exact URL in under 1ms — a local lookup instead of a live crawl.

From minutes per domain to microseconds

Dramatically Lower Cost

Live crawling and AI classification at runtime burns tokens, compute, and API calls. Every page an agent visits to discover structure costs $0.01–$0.05 in LLM inference. Multiply by thousands of domains and the bill explodes. A one-time database purchase eliminates all per-query classification costs.

One-time cost vs. per-query billing

Zero Hallucination Risk

When agents guess URLs, they hallucinate. An LLM asked to find a company's pricing page might fabricate /pricing, /plans, or /packages — none of which exist. Our database provides verified, real URLs that were actually discovered and classified, eliminating hallucinated navigation entirely.

Verified URLs, not AI guesses
1000x faster lookups
Zero per-query cost
100% verified URLs

What Data You Get for Every Domain

For each domain, we traverse its complete link structure and analyze each discovered page through our multi-step AI classification pipeline to identify up to 20 page types. The database returns actual, verified URLs for each page type — not guessed paths. Not all 20 types will be present on every domain, as availability depends on each site's structure.

Domain-Level Intelligence

Every domain in our database is enriched with comprehensive categorization and metadata. Easily find 99% of AI-themed domains worldwide, filter by popularity, country, and user personas.

99.9%

Google CruX Dataset Coverage

Our database domains are sourced from the Google Chrome User Experience (CruX) dataset, which covers 99.9% of internet usage worldwide. This means our database covers virtually all websites that internet users actually visit.

IAB Categories (Tier 1-4)
Deep categorization using IAB Content Taxonomy up to Tier 4 granularity for precise targeting
Web Filtering Categories
Security-focused categorization for content filtering, compliance, and brand safety
2000+ User Personas
Detailed audience personas to understand who visits each domain
OpenPageRank
Domain authority scores to identify high-quality, trusted websites
Country Detection
Geographic targeting with accurate country identification for each domain
Global Popularity Rank Group
Worldwide popularity ranking based on Google Chrome user experience data (CrUX)
Country Popularity Rank Group
Country-specific popularity ranking showing how domains rank within each country (CrUX)

Page-Level Intelligence

We traverse each domain's link structure and analyze over 10 billion pages through our multi-step AI pipeline to identify 20 page types. The database returns actual, verified URLs — not guessed paths — so your agents access the exact pages they need without runtime crawling.

Pricing Pages
Links to plans and pricing for competitive analysis.
Contact Pages
Support channels and direct contact information.
About Us
Company mission, history, and background details.
Leadership
Executive team profiles and organizational charts.
Careers
Job portals for hiring trends and growth signals.
Partners
Affiliate programs and integration partner directories.
Blog/News
Content hubs for company updates and articles.
Documentation
Developer portals, API docs, and technical guides.
Status Pages
System uptime monitoring and incident history.
Legal/Privacy
Terms of Service, Privacy Policy, and GDPR pages.
Case Studies
Customer success stories and testimonials.
Security
Trust centers, compliance certs (SOC2), and whitepapers.
Product
Feature breakdowns and solution landing pages.
Events
Webinars, conferences, and event registration pages.
Community
Forums, Discord invites, and user groups.
Help Center
Knowledge bases, FAQs, and support portals.
Integrations
App marketplaces and connection directories.
Sitemap
XML or HTML sitemaps for structure crawling.
Press
Newsrooms, media kits, and PR contacts.
DISCOVER 100+ USE CASES

See What Your AI Agents Can Do with Our Database

From competitive pricing intelligence to executive tracking, talent acquisition to security audits — explore 100 powerful applications across 20 page types that transform domain data into actionable business intelligence.

Explore 100 Use Cases
100
Use Cases
20
Page Types
20M+
Domains
700+
Categories
AGENTIC WORKFLOWS

47 Industries • 100 AI Agent Workflows

Discover how AI agents powered by our domain database transform operations across every major industry.

Finance & Professional

Technology & Media

Healthcare & Commerce

Industrial & Logistics

Give Your Agents a Map of the Internet

The most comprehensive, accurate, and up-to-date domain intelligence available for AI agents.

99.9% Internet Coverage
Our database covers virtually all active websites that matter. If users visit it, we have it categorized and indexed with all available page URLs.
300K+ New Domains Daily
We track and categorize over 300,000 newly registered domains every single day. Nothing gets missed - your agents always have access to the latest websites.
Refreshed Twice Monthly
Our database is refreshed twice per month, removing expired, parked, and error domains. Your agents work with fresh, valid, and accurate data.

The Semantic Layer for the Agentic Web

Transforming 20M categorized domains into actionable intelligence for your AI.

Instant Agent Awareness

Stop your agents from flying blind. Instantly inject semantic context for 20M+ domains, distinguishing reputable sources from spam in milliseconds.

RAG-Ready Intelligence

Slash token costs. We provide pre-packaged descriptive summaries—not just IDs—so your LLM understands the source without wasting tokens figuring it out.

Functional Navigation

A sitemap for agents. We identify critical functional paths like /pricing, /docs, or /login across the web, telling your agent exactly where to act.

Curated Training Sets

Train smarter. Access "Clean Web" data packs (e.g., Verified B2B, Medical Safe) to fine-tune SLMs on high-quality content, filtering out the noise.

Complex Multi-Filter AI Agents

Build sophisticated agents that combine domain-level data with page-level intelligence. Filter by category, popularity, country, and specific page types all in one query.

Example Query

Powerful Combined Filtering + Page URLs

With a single API call, you can:

Find all AI-themed domains from the US, among top 500K most popular websites globally, that have pricing pages - and get the direct pricing page URL for each domain returned!
Enterprise Market Intel

Identify high-authority competitors in specific markets with transparent pricing.

IAB: Software Rank > 5 US, UK, DE Pricing Page
Precision Lead Gen

Find ideal prospects matching your buyer persona with verified contact points.

B2B Services Top 1M Traffic Decision Maker Contact Page
Investment Research

Spot high-growth AI startups actively scaling their teams and leadership.

IAB: AI Rising Rank Careers Page Team Page
Strategic Partnership

Discover complementary businesses in target regions for partnership outreach.

Finance EU Region Trusted Site About Page
Security Vendor Scout

Evaluate cybersecurity vendors with enterprise-ready authentication portals.

Cybersecurity High Authority Login Page Legal Page
Developer Tool Finder

Locate developer-first platforms with accessible documentation and API resources.

Dev Tools Global Docs Page Integration Page

Pricing Page Agents

Access direct links to pricing pages. Enable AI agents to extract pricing data, compare plans, and analyze competitive positioning.

Competitive Pricing Analyzer

Monitors competitor pricing in real-time, tracking price changes, plan modifications, and new tier introductions.

  • Real-time price monitoring
  • Historical price tracking
  • Automated alerts on changes

SaaS Pricing Discovery Bot

Discovers and catalogs SaaS pricing models, identifying freemium offerings, trial periods, and enterprise pricing structures.

  • Freemium detection
  • Trial period identification
  • Feature-price mapping

Procurement Assistant Agent

Enterprise procurement AI that compares vendor pricing, identifies bulk discounts, and recommends optimal purchasing strategies.

  • Vendor price comparison
  • Bulk discount detection
  • ROI calculations

Login Page Agents

Direct access to login page URLs. Enable AI agents to perform authentication research, SSO analysis, and security assessments.

Security Audit Agent

Analyzes login page implementations, identifying authentication methods, MFA options, and security configurations.

  • Auth method detection
  • MFA availability check
  • SSL/TLS verification

SSO Integration Mapper

Discovers and maps Single Sign-On implementations, identifying supported identity providers and integration capabilities.

  • IdP detection (Google, Okta, etc.)
  • OAuth/SAML identification
  • Social login mapping

Auth Tech Stack Detector

Identifies authentication technologies, frameworks, and libraries used in login implementations across websites.

  • Framework detection
  • Auth library identification
  • Password policy analysis

Contact Page Agents

Access contact information pages. Enable AI agents to extract contact details, support channels, and communication preferences.

Lead Enrichment Agent

Extracts and enriches lead data from contact pages, building comprehensive company profiles for outreach campaigns.

  • Email extraction
  • Phone number parsing
  • Social profile linking

Support Channel Analyzer

Maps support channels, identifying available contact methods, response expectations, and support infrastructure.

  • Channel availability mapping
  • Live chat detection
  • Support hours identification

Office Location Mapper

Extracts and maps company office locations, headquarters addresses, and regional presence from contact pages.

  • HQ identification
  • Branch office mapping
  • Regional coverage analysis

About Us Page Agents

Access company background information. Enable AI agents to extract company history, mission statements, values, and organizational details.

Company Profiler Agent

Builds comprehensive company profiles by extracting founding year, mission, values, and company story from about pages.

  • Founding date extraction
  • Mission statement capture
  • Company values mapping

Investor Research Agent

Extracts company background, funding history, and growth indicators from about pages for due diligence processes.

  • Funding history extraction
  • Investor identification
  • Growth milestone tracking

Competitive Intel Agent

Analyzes competitor positioning, unique value propositions, and market differentiation from company about pages.

  • Positioning analysis
  • UVP extraction
  • Differentiation mapping

Leadership Page Agents

Access leadership and team pages. Enable AI agents to extract executive profiles, organizational structure, and key decision-maker information.

Executive Tracker Agent

Tracks executive movements, new appointments, and role changes across companies by monitoring team pages.

  • C-suite identification
  • Role change detection
  • Executive background extraction

Sales Navigator Agent

Identifies key decision-makers, stakeholders, and buying committee members from company leadership pages.

  • Decision-maker identification
  • Stakeholder mapping
  • Org chart building

Recruiter Intel Agent

Analyzes team compositions, identifies hiring patterns, and maps organizational structures for executive recruiting.

  • Team structure analysis
  • Hiring pattern detection
  • Talent gap identification

Careers Page Agents

Access career and job pages. Enable AI agents to track hiring trends, analyze job requirements, and monitor company growth signals.

Hiring Signals Agent

Tracks hiring velocity and job posting patterns as leading indicators of company expansion and market activity.

  • Hiring velocity tracking
  • Growth signal detection
  • Department expansion alerts

Tech Stack Discovery Agent

Extracts technology stacks, tools, and skill requirements from job postings to understand company tech landscapes.

  • Tech requirement extraction
  • Skill demand analysis
  • Tool adoption tracking

Culture Analyzer Agent

Extracts company culture indicators, values, perks, and work environment details from careers pages.

  • Culture value extraction
  • Perks cataloging
  • Remote policy detection

Partners Page Agents

Access partnership and integration pages. Enable AI agents to discover partnership opportunities, integration ecosystems, and affiliate programs.

Integration Ecosystem Mapper

Maps partner ecosystems and integration networks, identifying which tools and platforms companies integrate with.

  • Integration discovery
  • API partner mapping
  • Ecosystem analysis

Affiliate Program Scout

Discovers affiliate and referral programs, extracting commission structures, requirements, and partnership benefits.

  • Affiliate program detection
  • Commission rate extraction
  • Referral program analysis

Business Development Agent

Identifies partnership opportunities by analyzing partner programs, requirements, and strategic fit indicators.

  • Partner requirements extraction
  • Strategic fit scoring
  • Partnership tier analysis

Blog & News Page Agents

Access blog and news sections. Enable AI agents to monitor content strategies, track announcements, and analyze thought leadership.

Content Strategy Analyzer

Analyzes competitor content strategies, identifying topics, posting frequency, and content formats across company blogs.

  • Topic analysis
  • Content gap identification
  • Publishing pattern tracking

Announcement Monitor Agent

Monitors company news and announcements, tracking product launches, feature updates, and corporate developments.

  • Product launch tracking
  • Feature announcement alerts
  • Corporate news monitoring

Thought Leadership Scout

Identifies thought leadership content and industry insights, mapping expert voices and trending topics in your market.

  • Expert content discovery
  • Industry trend tracking
  • Influencer identification

IAB Category Intelligence

Leverage IAB Content Taxonomy up to Tier 4 for precise domain targeting. From broad industry segments to ultra-specific niches.

Vertical Market Discovery Agent

Uses IAB Tier 1-4 categories to map entire industry verticals, identifying all players from enterprise to niche specialists.

  • Tier 1-4 hierarchy navigation
  • Industry vertical mapping
  • Niche segment discovery

Ad Targeting Intelligence Agent

Builds precise audience segments using IAB categories for programmatic advertising and contextual targeting campaigns.

  • Contextual targeting lists
  • Brand-safe inventory
  • Audience segment building

Competitor Landscape Mapper

Maps competitive landscapes using precise IAB Tier 4 categories, revealing direct and adjacent competitors in any market.

  • Direct competitor identification
  • Adjacent market discovery
  • Market share estimation

Web Filtering Intelligence

Security-focused categorization for compliance, content filtering, and brand safety applications.

Brand Safety Monitor Agent

Evaluates websites against brand safety guidelines using Web Filtering categories to protect advertising investments.

  • Content risk assessment
  • Brand safety scoring
  • Blocklist generation

Compliance Verification Agent

Uses Web Filtering categories to verify regulatory compliance, identifying restricted or regulated website categories.

  • Regulatory category mapping
  • Restricted content detection
  • Compliance reporting

Threat Category Agent

Identifies potentially risky categories for security operations, supporting threat intelligence and content filtering.

  • Risk category identification
  • Security policy enforcement
  • Content filtering support

User Persona Intelligence

Access over 2000 distinct user personas. Understand exactly who visits each domain for precision targeting and audience research.

Audience Builder Agent

Builds targeted audience lists by finding domains that attract specific user personas matching your ideal customer profile.

  • Persona-based targeting
  • Lookalike audience building
  • ICP domain discovery

Market Research Agent

Analyzes persona distributions across industries to understand market demographics and audience composition.

  • Demographic analysis
  • Industry audience profiling
  • Market sizing by persona

Content Strategy Agent

Maps personas to content types, helping create targeted content strategies based on audience preferences.

  • Content-persona matching
  • Topic opportunity finding
  • Channel strategy insights

OpenPageRank Intelligence

Filter domains by authority and trust. Identify high-quality websites and separate leaders from newcomers in any market.

Authority Domain Finder

Identifies high-authority domains in any category, perfect for finding industry leaders and trusted sources.

  • High-authority filtering
  • Industry leader identification
  • Trust score ranking

Link Prospect Agent

Finds high-PageRank domains for link building and digital PR, filtering by authority thresholds and relevance.

  • Link prospect discovery
  • Authority threshold filtering
  • Outreach list building

Emerging Player Detector

Identifies rising domains with growing authority, spotting emerging competitors and market entrants early.

  • Rising authority detection
  • New entrant monitoring
  • Growth trajectory analysis

Geographic & Popularity Intelligence

Filter by country and real-world popularity using Global and Country Rank Groups from Google Chrome usage data (CrUX). Target specific markets with precision.

Market Entry Research Agent

Analyzes domains by country to understand local market landscapes, identifying key players in target regions.

  • Country-specific filtering
  • Local competitor mapping
  • Regional market analysis

Popularity Ranking Agent

Uses Global and Country Rank Groups (CrUX) to filter domains by real user traffic, identifying the most-visited sites globally and per country.

  • Global Rank Group filtering
  • Country Rank Group filtering
  • Popularity tier analysis

Global Expansion Agent

Combines country data with Global and Country Rank Groups to identify expansion opportunities in new geographic markets.

  • Multi-region analysis
  • Market opportunity scoring
  • Localization priority

Ready to Power Your AI Agents?

Get instant access to our 20 million domain database. Start building smarter AI agents with comprehensive website intelligence today.

View Pricing Plans API Documentation
Stay in the loop

You are on the list!

We will send you updates that matter — no spam.