+8
π‘π Essential Multipage Website Scraper with Jina.ai
Published 6 days ago
Created by
Joseph LePage
Categories
Template description
π‘π Essential Multipage Website Scraper with Jina.ai
Use responsibly and follow local rules and regulations
This N8N workflow enables automated multi-page website scraping using Jina.ai's powerful web scraping capabilities, with seamless integration to Google Drive for content storage. Here's how it works:
Main Features
The workflow automatically scrapes multiple pages from a website's sitemap and saves each page's content as a separate Google Drive document.
Key Components
Input Configuration
- Starts with a sitemap URL (default: https://ai.pydantic.dev/sitemap.xml)
- Processes the sitemap to extract individual page URLs
- Includes filtering options to target specific topics or pages
Scraping Process
- Uses Jina.ai's web scraper to extract content from each URL
- Converts webpage content into clean markdown format
- Extracts page titles automatically for document naming
Storage Integration
- Creates individual Google Drive documents for each scraped page
- Names documents using the format "URL - Page Title"
- Saves content in markdown format for better readability
Usage Instructions
- Set your target website's sitemap URL in the "Set Website URL" node
- Configure the "Filter By Topics or Pages" node to select specific content
- Adjust the "Limit" node (default: 20 pages) to control batch size
- Connect your Google Drive account
- Run the workflow to begin automated scraping
Additional Features
- Built-in rate limiting through the Wait node to prevent overloading servers
- Batch processing capability for handling large sitemaps
The workflow requires no API key for Jina.ai, making it accessible for immediate use while maintaining responsible scraping practices.
Share Template
More AI workflow templates
AI agent chat
This workflow employs OpenAI's language models and SerpAPI to create a responsive, intelligent conversational agent. It comes equipped with manual chat triggers and memory buffer capabilities to ensure seamless interactions.
To use this template, you need to be on n8n version 1.50.0 or later.
n8n Team
+7
Scrape and summarize webpages with AI
This workflow integrates both web scraping and NLP functionalities. It uses HTML parsing to extract links, HTTP requests to fetch essay content, and AI-based summarization using GPT-4o. It's an excellent example of an end-to-end automated task that is not only efficient but also provides real value by summarizing valuable content.
Note that to use this template, you need to be on n8n version 1.50.0 or later.
n8n Team
+5
AI agent that can scrape webpages
βοΈπ οΈππ€π¦Ύ
This template is a PoC of a ReAct AI Agent capable of fetching random pages (not only Wikipedia or Google search results).
On the top part there's a manual chat node connected to a LangChain ReAct Agent. The agent has access to a workflow tool for getting page content.
The page content extraction starts with converting query parameters into a JSON object. There are 3 pre-defined parameters:
url** β an address of the page to fetch
method** = full / simplified
maxlimit** - maximum length for the final page. For longer pages an error message is returned back to the agent
Page content fetching is a multistep process:
An HTTP Request mode tries to get the page content.
If the page content was successfuly retrieved, a series of post-processing begin:
Extract HTML BODY; content
Remove all unnecessary tags to recude the page size
Further eliminate external URLs and IMG scr values (based on the method query parameter)
Remaining HTML is converted to Markdown, thus recuding the page lengh even more while preserving the basic page structure
The remaining content is sent back to an Agent if it's not too long (maxlimit = 70000 by default, see CONFIG node).
NB:
You can isolate the HTTP Request part into a separate workflow.
Check the Workflow Tool description, it guides the agent to provide a query string with several parameters instead of a JSON object.
Please reach out to Eduard is you need further assistance with you n8n workflows and automations!
Note that to use this template, you need to be on n8n version 1.19.4 or later.
Eduard
+10
Building Your First WhatsApp Chatbot
This n8n template builds a simple WhatsApp chabot acting as a Sales Agent. The Agent is backed by a product catalog vector store to better answer user's questions.
This template is intended to help introduce n8n users interested in building with WhatsApp.
How it works
This template is in 2 parts: creating the product catalog vector store and building the WhatsApp AI chatbot.
A product brochure is imported via HTTP request node and its text contents extracted.
The text contents are then uploaded to the in-memory vector store to build a knowledgebase for the chatbot.
A WhatsApp trigger is used to capture messages from customers where non-text messages are filtered out.
The customer's message is sent to the AI Agent which queries the product catalogue using the vector store tool.
The Agent's response is sent back to the user via the WhatsApp node.
How to use
Once you've setup and configured your WhatsApp account and credentials
First, populate the vector store by clicking the "Test Workflow" button.
Next, activate the workflow to enable the WhatsApp chatbot.
Message your designated WhatsApp number and you should receive a message from the AI sales agent.
Tweak datasource and behaviour as required.
Requirements
WhatsApp Business Account
OpenAI for LLM
Customising this workflow
Upgrade the vector store to Qdrant for persistance and production use-cases.
Handle different WhatsApp message types for a more rich and engaging experience for customers.
Jimleuk
+2
Telegram AI Chatbot
The workflow starts by listening for messages from Telegram users. The message is then processed, and based on its content, different actions are taken. If it's a regular chat message, the workflow generates a response using the OpenAI API and sends it back to the user. If it's a command to create an image, the workflow generates an image using the OpenAI API and sends the image to the user. If the command is unsupported, an error message is sent. Throughout the workflow, there are additional nodes for displaying notes and simulating typing actions.
Eduard
+5
Ask questions about a PDF using AI
The workflow first populates a Pinecone index with vectors from a Bitcoin whitepaper. Then, it waits for a manual chat message. When received, the chat message is turned into a vector and compared to the vectors in Pinecone. The most similar vectors are retrieved and passed to OpenAI for generating a chat response.
Note that to use this template, you need to be on n8n version 1.19.4 or later.
David Roberts
