LLMSearchConsoleBot

LLMSearchConsoleBot is the crawler of LLM Search Console, a service that shows brands which websites AI assistants such as ChatGPT, Gemini and Perplexity cite in their answers. If you found it in your server logs, this page says what it fetches and why.

How it identifies itself

Every request it sends carries this user agent:

LLMSearchConsoleBot/1.0 (+https://llmsearchconsole.com/bot)

What it fetches

• Your robots.txt, on its own at most once every 7 days, when an AI answer that one of our customers tracks has cited your site, or when a customer has added it as one of their own websites, and before each of the checks below. We read it to tell our customers whether AI crawlers, such as OAI-SearchBot, PerplexityBot or Googlebot, may read the site.
• Your sitemaps, at most once every 7 days, and only when a customer has added your site as one of their own websites: up to 10 sitemap files and 5,000 page addresses.
• The headers of up to 3 of your pages, the ones AI answers cite most, at most once every 7 days, to see when they last changed. These are HEAD requests: the pages themselves are not downloaded.

It downloads no other page, and it does not log in, submit forms, run scripts or keep cookies.

robots.txt

Before it fetches a sitemap or checks a page, LLMSearchConsoleBot reads your site's robots.txt and follows the rules it gives for User-agent: LLMSearchConsoleBot, or, when there are none, the rules for all crawlers (User-agent: *). robots.txt itself is always read. When your robots.txt cannot be reached, LLMSearchConsoleBot fetches nothing else from your site that time.

It reads a line without its colon (such as Disallow /) or with a misspelt name (such as Dissallow: or User agent:) as Google does, although some crawlers skip such lines. Write each rule with its colon and the field's exact name.

Contact

Questions about LLMSearchConsoleBot: [email protected]. LLM Search Console is run by Codedesign.