See llms.txt for all machine-readable content.

Back to Templates

Convert documentation URLs to Markdown for RAG with Apify

Quick Overview

This workflow manually crawls a documentation website with Apify’s hydrocore/website-to-markdown-text actor, converts each discovered page into clean Markdown and text, and outputs a simplified, RAG-ready dataset containing the page URL, title, content, and word count.

How it works

  1. Runs when you start the workflow with a manual trigger.
  2. Reads crawl inputs like start URL, page limit, spend cap, crawl scope, crawler type, and output format.
  3. Uses Apify to run the hydrocore/website-to-markdown-text actor to crawl the site (respecting robots.txt) and return a dataset of pages with extracted markdown/text and metadata.
  4. Reshapes each dataset item to keep only RAG-friendly fields (url, title, markdown, text, and wordCount) for downstream indexing or storage.

Setup

  1. Install the n8n Apify community node package (@apify/n8n-nodes-apify) and add an Apify API credential.
  2. Update the crawl inputs in the Configure crawl step (startUrl, maxPages, maxSpendUsd, crawlScope, crawlerType, outputFormat, and respectRobotsTxt) to match your target site.
  3. Execute the workflow once and validate the markdown/text output before connecting it to a vector database or storage destination.