+12

Autonomous AI crawler

Published 3 months ago

Created by

Oskar

Template description

This workflow with AI agent is designed to navigate through the page to retrieve specific type of information (in this example: social media profile links).

The agent is equipped with 2 tools:

text tool: to retrieve all the text from the page,
URLs tool: to extract all possible links from the page.

💡 You can edit prompt and JSON schema connected to the agent in order to return other data then social media profile links.

👉 This workflow uses Supabase as storage (input/output). Feel free to change it to any other database of your choice.

🎬 See this workflow in action in my YouTube video.

How it works?

The workflow uses the input URL (website) as a starting point to retrieve the data (e.g. example.com). Using the "URLs tool", the agent is able to retrieve all links from the page and navigate to them.

For example, if you want to retrieve contact information, agent will try to find a subpage that might contain this information (e.g. example.com/contact) and extract the information using the text tool.

Set up steps

Connect database with input data (website addresses) or pin sample data to trigger node.
Configure the crawling agent to retrieve the desired data (e.g. modify prompt and/or parsing schema).
Set credentials for OpenAI.
Optionally: split agent tools to separate workflows.

If you like this workflow, please subscribe to my YouTube channel and/or my newsletter.

Share Template

More AI workflow templates

AI agent chat

This workflow employs OpenAI's language models and SerpAPI to create a responsive, intelligent conversational agent. It comes equipped with manual chat triggers and memory buffer capabilities to ensure seamless interactions. To use this template, you need to be on n8n version 1.50.0 or later.

n8n Team

Scrape and summarize webpages with AI

This workflow integrates both web scraping and NLP functionalities. It uses HTML parsing to extract links, HTTP requests to fetch essay content, and AI-based summarization using GPT-4o. It's an excellent example of an end-to-end automated task that is not only efficient but also provides real value by summarizing valuable content. Note that to use this template, you need to be on n8n version 1.50.0 or later.

n8n Team

AI agent that can scrape webpages

⚙️🛠️🚀🤖🦾 This template is a PoC of a ReAct AI Agent capable of fetching random pages (not only Wikipedia or Google search results). On the top part there's a manual chat node connected to a LangChain ReAct Agent. The agent has access to a workflow tool for getting page content. The page content extraction starts with converting query parameters into a JSON object. There are 3 pre-defined parameters: url** – an address of the page to fetch method** = full / simplified maxlimit** - maximum length for the final page. For longer pages an error message is returned back to the agent Page content fetching is a multistep process: An HTTP Request mode tries to get the page content. If the page content was successfuly retrieved, a series of post-processing begin: Extract HTML BODY; content Remove all unnecessary tags to recude the page size Further eliminate external URLs and IMG scr values (based on the method query parameter) Remaining HTML is converted to Markdown, thus recuding the page lengh even more while preserving the basic page structure The remaining content is sent back to an Agent if it's not too long (maxlimit = 70000 by default, see CONFIG node). NB: You can isolate the HTTP Request part into a separate workflow. Check the Workflow Tool description, it guides the agent to provide a query string with several parameters instead of a JSON object. Please reach out to Eduard is you need further assistance with you n8n workflows and automations! Note that to use this template, you need to be on n8n version 1.19.4 or later.

Eduard

Implement complex processes faster with n8n

Get started