ShadowAtlasBot
ShadowAtlasBot reads public historical sources (encyclopedias, archives of government records, open research
datasets) for a history and news-analysis project. It identifies itself in every request as
ShadowAtlasBot.
- It makes one request at a time, with pauses between requests to the same site.
- It follows robots.txt, and Wikimedia's maxlag and Retry-After signals.
- When a site asks it to slow down or refuses it, it stops contacting that site for the rest of the run.
- It never disguises itself, uses proxies, or tries to get around access controls.
If it's causing you any trouble, write to bot@shadowatlas.net and we'll
fix it right away.
Shadow Atlas