Codanya — web crawler
About CodanyaBot
CodanyaBot is the crawler that builds Codanya's search index. It reads public web pages, extracts their text and links, and stores what it finds so the search box on this site can return results. It is not a scraper, it runs no analytics, and it does not track people.
What it does
CodanyaBot fetches HTML pages following the links it discovers, the same way any search engine builds its index. It respects a per-host crawl delay, keeps its requests small, and skips binaries, images, and other non-text resources that are no use to a text index.
It identifies itself honestly on every request with the User-Agent string above, which includes a link back to this page and a contact address. There is no attempt to disguise the crawler as a browser or a person.
How to control or block it
You have three standard ways to keep CodanyaBot out of your site, or out of parts of it. All of them are honoured.
-
robots.txt
Add a group for our bot. To block the whole site:
User-agent: CodanyaBotthenDisallow: /on the next line. To block a section, point the Disallow at that path. ACrawl-delaydirective is honoured too, up to 60 seconds. -
Per-page meta tag
Put
<meta name="codanyabot" content="noindex">in a page's<head>to keep that page out of the index. Usenofollowto stop CodanyaBot following links from that page, ornonefor both. -
Contact
If you need the crawler to stop, slow down, or skip something urgently, email [email protected] and it will be handled.
How it behaves
Polite by default. Requests to the same host are spaced out, never parallelised against a single domain. Different hosts are crawled independently.
Verifies robots.txt first. A page is never fetched if the site's robots.txt disallows it for CodanyaBot. A missing robots.txt is treated as "no rules".
No private networks. The crawler refuses to follow any URL that resolves to a private, loopback, or reserved address, so it cannot be turned into a server-side request forgery tool.
Questions?
If CodanyaBot is putting load on your site that you would rather it did not, or you want a range of URLs removed from the index, write to [email protected].
Back to search