Google on November 17, 2026 published a technical guide explaining how website operators can allow traffic from Google-Agent, the name used by agents hosted on Google infrastructure that browse sites and perform actions when a user requests them. The document is aimed at content delivery networks, web application firewalls and bot-detection services, which can stop automated visitors before they reach a page.
According to the guide, site owners can identify Google-Agent requests through three signals: a User-Agent header that includes "Google-Agent," a source IP address listed in Google's user-triggered-agents.json file and, for some requests, a cryptographic signature under an experimental Web Bot Auth program. Google recommends that verified requests skip bot challenges such as CAPTCHAs and JavaScript tests.
The guide also lists six reasons legitimate requests can still fail after an allow rule is added. They include allow rules that run after broad data center or AI agent block policies, per-IP rate limits that ignore the agent's shared IP pool, outdated IP lists and proxies that strip client IP, User-Agent or signature headers. Google also says permission should cover multistep actions such as search, booking and add-to-cart, along with the APIs behind them.
PPC Land reported that the IP file generated November 16 listed 20 prefixes, 12 IPv4 and 8 IPv6, and that Google-Agent first appeared on Google's fetcher list on March 20, 2026. Google's fetcher documentation says user-triggered fetchers "generally ignore robots.txt rules." The new guide does not name which Google products send these requests.
The publication cited separate findings from Blend, which reported that bot protection blocked its AI shopping agent at 29 of 200 retailers.
Source: PPC Land -- https://ppc.land/google-lists-six-ways-sites-still-block-its-ai-agent-after-allowing-it/
