Emerging

AI Bots & Agentic Security

Automated traffic is no longer just crawlers and attackers. AI agents now browse, research, compare and transact on behalf of users — which breaks the assumption that “bot” means “block”.

The shift: from blocking bots to governing agents

For twenty years bot management was a binary question with a mostly binary answer: search crawlers in, everything else out. Agentic AI dissolves that model, because a large and growing class of automated traffic is acting for a legitimate customer. An AI assistant that researches a product, compares prices and completes a purchase is revenue — if you can recognise it and trust it. Blocked indiscriminately, it is lost revenue and an invisible customer.

Three distinct populations now need distinct policy:

Population	Intent	Sensible default
AI crawlers / training data collectors	Ingest content at scale for model training or index building	Policy decision: allow, restrict, or monetise — but always identify
Retrieval / answer-engine agents	Fetch a page in real time to answer a user's question, often with attribution	Usually allow — this is referral traffic in a new shape
Transacting user agents	Act on behalf of an identified end user: browse, add to cart, purchase	Allow under verified identity, with fraud and rate governance

Malicious automation dressed as an AI agent is the fourth population, and it is precisely why identity — not User-Agent strings — must be the basis of policy.

Agentic Security Framework

Akamai's Agentic Security Framework is aimed at powering trusted AI-driven interactions and commerce. The problem it addresses is trust establishment between three parties who previously had no protocol for it: the user on whose behalf an agent acts, the agent itself, and the site the agent transacts with.

The functional requirements this creates for a defender:

Identify the agent cryptographically, not by self-declaration. A User-Agent header is a claim, not evidence.
Establish delegated authority — is this agent genuinely acting for a real, consenting user, and with what scope?
Differentiate policy by agent and by action — reading a product page, adding to cart and completing a payment warrant different levels of assurance.
Apply commerce-grade controls — inventory protection, rate governance, and fraud checks that are designed for agent behaviour rather than tuned for human browsing patterns.
Keep an audit trail of agentic transactions for dispute resolution and regulatory obligations.

Reference: Akamai unveils Agentic Security Framework to power trusted AI-driven interactions and commerce.

Web Bot Auth — cryptographic bot identity

Web Bot Auth replaces reputation-by-IP-range with reputation-by-signature. The bot operator holds a private key and signs its HTTP requests; the site verifies the signature against the operator's published public key. The practical consequences are significant:

Spoofing a well-known crawler becomes cryptographically infeasible — today anyone can send User-Agent: Googlebot, and defenders must fall back to reverse DNS and IP-range verification that operators must publish and maintain.
Operators can change infrastructure freely without breaking every allow-list on the internet, because identity is no longer tied to an address.
Sites can express fine-grained policy per verified identity — allow this crawler at this rate on these paths, monetise that one, deny the third — with confidence that the identity is real.
It scales to long-tail agents, which matters because the number of distinct AI agents is growing far faster than any manual allow-list can track.

Reference: Redefining trust with web bot authentication.

Visibility into AI bots and agents

Before any policy is possible you need to answer, with data: which AI bots and agents are hitting my properties, what are they fetching, how much origin cost are they generating, and is that traffic producing any value? Improved visibility into AI bots and agents means classification that distinguishes training crawlers from retrieval agents from user-delegated agents, reported per property and per path.

Practical reporting questions to answer before deciding policy:

What share of requests to high-value content is AI automation?
Which specific operators, and are they honouring robots.txt?
Is AI retrieval traffic correlated with referral sessions and conversions, or purely extractive?
What is the incremental origin and egress cost?

Monetization for publishers

For publishers, blanket blocking of AI crawlers converts an asset into nothing; blanket allowing gives it away. Monetization integrations create a third option: identify the AI crawler, then require a commercial arrangement for access to content — per-crawl licensing, negotiated agreements, or tiered access — enforced technically at the edge for operators who have not agreed to terms.

This only works if identification is reliable, which is why Web Bot Auth and monetization are two halves of one strategy. See the Akamai sales-digest posts on monetization integrations and improved visibility into AI bots and agents.

Building an AI bot policy

Measure first. Two weeks of classified AI traffic reporting, by operator and path.
Decide by content class, not site-wide. Marketing pages, documentation, paywalled editorial and transactional flows deserve different answers.
State it machine-readably — robots.txt directives and any emerging preference signals — then enforce technically, because well-behaved operators honour the file and the rest do not.
Prefer verified identity to heuristics. Support signed-request verification and validate self-declared crawlers.
Treat user-delegated agents as customers, with fraud and inventory controls appropriate to automated purchasing rather than human pacing.
Revisit quarterly. This area is moving faster than any other in bot management.