The HatlessLRM crawler

Who we are and what this is

Hatless is an SEO agency in the Netherlands. The bot is called HatlessLRM and belongs to our link monitor: it checks whether links to our clients, placed on pages in agreement with the publisher, are still present.

It identifies itself with the user agent:

HatlessLRM/0.1 (+https://hatless.com/bot)

What the bot does

• It fetches only individual, known-in-advance URLs. It does not spider a whole site, does not follow internal links and does not read your sitemap.
• It makes GET requests to HTML pages only. It follows redirects. It times out after 30 seconds.
• It reads the page to find the outgoing links. It does not run JavaScript, does not fill in forms, does not log in anywhere and sets no cookies.

How often it visits

• By default, once a week per URL.
• At most one request at a time per website, with at least 1.5 seconds between two requests to the same site.
• If a site returns an error, the wait grows: first an hour, then two, four, eight, up to a maximum of a week. A site that is down is not hammered.

What we store

• The HTTP status code, a checksum (hash) of the page, and the outgoing links we find: the URL, the anchor text and the rel attribute.
• We do not keep a copy of the page text and no personal data about visitors.

robots.txt

The bot reads robots.txt before every request (cached for six hours) and obeys Disallow rules. To keep the bot out, add this to your robots.txt:

User-agent: HatlessLRM
Disallow: /

That takes effect within six hours. In fairness: a block at IP level or through a bot firewall is seen by us as "page could not be fetched"; only robots.txt is treated by us as an instruction.

Where it runs from

It runs on infrastructure in the EU (Frankfurt). We do not have fixed IP addresses, so we cannot provide an IP list, filter on the user agent instead.

Contact

Questions or a removal request? Email hi@hatless.com and we will respond as soon as we can.