SitepagesBot

Stop it, or slow it down

SitepagesBot follows your robots.txt. It reads it again at least once an hour, so a change takes effect within the hour.

User-agent: SitepagesBot
Disallow: /

User-agent: SitepagesBot
Disallow: /members/

User-agent: SitepagesBot
Crawl-delay: 5

With no group for SitepagesBot, it follows your rules for every crawler (User-agent: *).

What it reads

• Links a course’s owner adds to their course: a page, a document, a recording.
• If the owner chooses, the pages that rank in a web search for the course’s subject.
• Wikipedia, Wikibooks, Wikiversity and Wikidata, through their public APIs, in the course’s language.

It reads only the pages it is given. It doesn’t follow links and it doesn’t crawl whole sites.

How it behaves

• It reads robots.txt before anything else on a site, and checks every redirect against the robots.txt of the site it leads to.
• If your robots.txt is missing (a 4xx answer other than 429), it treats the site as open, as the robots standard (RFC 9309) says.
• If your robots.txt answers with a server error or 429, or can’t be reached, it reads nothing from your site for 10 minutes.
• If a page answers 429 Too Many Requests, it waits as long as your Retry-After asks, up to 30 seconds, tries once more, then leaves the page.
• Only http and https, on ports 80 and 443. Up to 5 redirects. It stops at 20 MB, 20 seconds a request and 2 minutes a page.
• It doesn’t sign in, fill in forms, keep cookies or run scripts. It reads what any visitor without an account would see.

What happens to what it reads

The text it reads is kept with the course it was read for, along with the address it came from, and used to write that course: its notes, the terms it teaches and its lessons.