Our crawler

A company's home page and at most three public pages it links to: contact, about and services. It reads robots.txt first and follows it, including Crawl-delay. It waits at least 10 seconds between two requests to the same site. It never logs in, never fills in a form, and never reads a page behind a paywall.

Business facts the site itself publishes: the company name, its industry when the site names it, its city and country when its address states them, its contact page, and the phone and email it publishes for business (such as info@ or sales@). Each fact is kept with the page it came from and a short quote of it.

It never keeps people: no names from team pages, no personal email addresses, no social profiles. A site that is parked, gone or blocks us is removed.

Add this to your robots.txt and it stops reading your site:

User-agent: MondailyIndexBot
Disallow: /

A group for MondailyBot blocks it as well. To have your company removed and never read again, write to privacy@mondaily.com with your domain. How we handle data about people found this way: Privacy.