Quick Overview
This workflow manually crawls a documentation website with Apify’s hydrocore/website-to-markdown-text actor, converts each discovered page into clean Markdown and text, and outputs a simplified, RAG-ready dataset containing the page URL, title, content, and word count.
How it works
- Runs when you start the workflow with a manual trigger.
- Reads crawl inputs like start URL, page limit, spend cap, crawl scope, crawler type, and output format.
- Uses Apify to run the hydrocore/website-to-markdown-text actor to crawl the site (respecting robots.txt) and return a dataset of pages with extracted markdown/text and metadata.
- Reshapes each dataset item to keep only RAG-friendly fields (url, title, markdown, text, and wordCount) for downstream indexing or storage.
Setup
- Install the n8n Apify community node package (@apify/n8n-nodes-apify) and add an Apify API credential.
- Update the crawl inputs in the Configure crawl step (startUrl, maxPages, maxSpendUsd, crawlScope, crawlerType, outputFormat, and respectRobotsTxt) to match your target site.
- Execute the workflow once and validate the markdown/text output before connecting it to a vector database or storage destination.