Monitor AI crawler policy signals via HTTP, Data Tables, email and Slack

Quick overview

Track published AI crawler rules on selected websites and optionally monitor llms.txt presence and content hashes. Receive one email for baselines or changes, with persistent snapshots in a Data Table.

How it works

1. A manual trigger or hourly tick gated by policyCheckHour in Config loads per-site snapshots once daily from a Data Table.
2. Fetch public files using a descriptive User-Agent, a 20-second timeout, paced requests and up to three transient attempts. Preserve successful responses during retries.
3. Parse case-insensitive user-agent groups, select the most specific match, merge equally specific groups and fall back to *. Strip comments, handle BOM and CRLF, ignore empty restrictions, and use the longest matching root rule with Allow winning ties.
4. Compare each bot's status and canonical path lists, plus optional llms.txt presence and hash. A robots.txt 404 means allow all. Other fetch errors are reported without changing that site's saved snapshot.
5. Send one SMTP email if required, then upsert successful snapshots by siteKey. No change sends no email. New baselines can be silent through Config.

Setup

1. Create a Data Table with siteKey, robotsHash, policyJson and llmsTxt as String columns and checkedAt as Date. Enter its ID, site origins, bot tokens and email addresses in Config, then select an SMTP credential. Run twice manually before publishing the daily schedule. See SETUP.md for details.
2. Requires n8n 2.8.4, Data Tables, an SMTP credential and outbound public HTTP access.

Requirements

• n8n 2.8.4, Data Tables, an SMTP credential and outbound public HTTP access. Public GET requests require no API credentials. Keep workflow executions sequential. The included test harness requires Node 20 and cached n8n 2.8.4.

Customization

• Edit Config for origins, crawler tokens, llms.txt checks, baseline emails, request pacing, addresses and policyCheckHour (0 to 23, default 9). The fixed hourly trigger checks the selected hour in the workflow or instance timezone; manual checks run immediately. A fuller version could also audit meta robots tags and X-Robots-Tag headers across many pages.

Additional info

Who is this for

Website owners, marketing teams and IT teams tracking published AI crawler preferences on a small list of public websites.

What it does

Fetches robots.txt and evaluates configured crawler tokens as blocked, allowed or partial, with their allow and disallow paths. Optionally checks llms.txt presence and SHA-256 hash without storing its content. Sends one escaped HTML summary for new baselines or meaningful changes, including short added and removed robots.txt lines. robots.txt is a voluntary convention; this template reports what sites publish and does not assess legal compliance.

Tested: parent-run mock and live suites passed on 2026-10-23 with n8n 2.8.4, including real import, Data Tables, SMTP rejection recovery, unchanged-run silence and three public sites. The review subsequently added a Config schedule gate and disabled duplicate native HTTP retries; its offline checks pass, but the final export needs a parent end-to-end rerun. See TESTING.md.

Third-party product names are used only to describe compatibility. This template is not affiliated with or endorsed by those companies.