Scrape and Analyze Websites with Custom Prompts using Gemini, Apify, and LangChain

Created by

Msaid Mohamed el hadi

Last update

Last update 2 days ago

🔍 AI-Powered Website Prompt Executor (Apify + OpenRouter)

This workflow combines the power of Apify and OpenRouter to scrape website content and execute any custom prompt using AI. You define what you want — whether it’s extracting contact details, summarizing content, collecting job offers, or anything else — and the system intelligently processes the site to give you results.

🚀 Overview

This workflow allows you to:

Input a URL and define a prompt.
Scrape the specified number of pages from the website.
Process each page’s metadata and Markdown content.
Use AI to interpret and respond to the prompt on each page.
Aggregate and return structured output.

🧠 How It Works

Input Example

{
  "enqueue": true,
  "maxPages": 5,
  "url": "https://apify.com",
  "method": "GET",
  "prompt": "collect all contact informations available on this website"
}

Workflow Steps

Step	Action
1	Triggered by another workflow with JSON input.
2	Calls the Apify actor `firescraper-ai-website-content-markdown-scraper` to scrape content.
3	Loops through the scraped pages.
4	AI analyzes each page based on the input prompt.
5	Aggregates AI outputs across all pages.
6	Final AI processing step to return a clean structured result.

🛠 Technologies Used

Apify – Scrapes structured content and Markdown from websites.
OpenRouter – Provides access to advanced AI models like Gemini.
LangChain – Handles AI agent orchestration and prompt interpretation.

🔧 Customization

Customize the workflow via the following input fields:

url: Starting point for scraping
maxPages: Limit the number of pages to crawl
prompt: Define any instruction (e.g., “summarize this website,” “extract product data,” “list all emails,” etc.)

This allows dynamic, flexible use across various use cases.

📦 Output

The workflow returns a JSON result that includes:

Processed prompt responses from each page
Aggregated AI insights
Structured and machine-readable format

🧪 Example Use Cases

🔍 Extracting contact information from websites
📄 Summarizing articles or company profiles
🛍️ Collecting product information
📋 Extracting job listings or news
📬 Generating outreach lists from public data
🤖 Used as a tool within other AI agents for real-time web analysis
🧩 Integrated as an external tool in MCP (Multi-Component Prompt) servers to enhance AI capabilities

🔐 API Credentials Required

You will need:

Apify API token – For running the scraper actor
OpenRouter API key – For AI-powered prompt processing

Set these credentials in your environment or n8n credential manager before running.