Back to templates
HTTP Request node
+2

Search and download torrents using transmission-daemon

Published 2 years ago

Created by

dangerblack
Danger

Template description

Ok google download "movie name"

I develop this automation to improve my quality of life in handling torrents in my media-center.

Goal

Automate the search operations of a movie based on its name and trigger a download using your transmission-daemon.

Setup

Prerequisite

  • Transmission daemon up and running and its authentication method
  • N8N configured self-hosted or with the possibility to add npm package better with docker-compose.yaml
  • Telegram bot credential [optional]

Configuration

Create a folder where your docker-compose.yaml belongs n8n_dir and proceed in installing the node package.

cd ~/n8n_dir
npm i torrent-search-api

Configuring your docker-compose.yaml file this way. You must include all the dependencies of torrent-search-api. This will let you run the new torrent search node presented in this workflow.

version: '3.3'
services:
    n8n:
        container_name: n8n
        ports:
            - '5678:5678'
        restart: always
        volumes:
            - '~/n8n_dir/.n8n:/home/node/.n8n'
            - '~/n8n_dir/node_modules/@tootallnate:/usr/local/lib/node_modules/@tootallnate'
            - '~/n8n_dir/node_modules/accepts:/usr/local/lib/node_modules/accepts'
            - '~/n8n_dir/node_modules/agent-base:/usr/local/lib/node_modules/agent-base'
            - '~/n8n_dir/node_modules/ajv:/usr/local/lib/node_modules/ajv'
            - '~/n8n_dir/node_modules/ansi-styles:/usr/local/lib/node_modules/ansi-styles'
            - '~/n8n_dir/node_modules/asn1:/usr/local/lib/node_modules/asn1'
            - '~/n8n_dir/node_modules/assert:/usr/local/lib/node_modules/assert'
            - '~/n8n_dir/node_modules/assert-plus:/usr/local/lib/node_modules/assert-plus'
            - '~/n8n_dir/node_modules/ast-types:/usr/local/lib/node_modules/ast-types'
            - '~/n8n_dir/node_modules/asynckit:/usr/local/lib/node_modules/asynckit'
            - '~/n8n_dir/node_modules/aws-sign2:/usr/local/lib/node_modules/aws-sign2'
            - '~/n8n_dir/node_modules/aws4:/usr/local/lib/node_modules/aws4'
            - '~/n8n_dir/node_modules/base64-js:/usr/local/lib/node_modules/base64-js'
            - '~/n8n_dir/node_modules/batch:/usr/local/lib/node_modules/batch'
            - '~/n8n_dir/node_modules/bcrypt-pbkdf:/usr/local/lib/node_modules/bcrypt-pbkdf'
            - '~/n8n_dir/node_modules/bluebird:/usr/local/lib/node_modules/bluebird'
            - '~/n8n_dir/node_modules/boolbase:/usr/local/lib/node_modules/boolbase'
            - '~/n8n_dir/node_modules/brotli:/usr/local/lib/node_modules/brotli'
            - '~/n8n_dir/node_modules/bytes:/usr/local/lib/node_modules/bytes'
            - '~/n8n_dir/node_modules/caseless:/usr/local/lib/node_modules/caseless'
            - '~/n8n_dir/node_modules/chalk:/usr/local/lib/node_modules/chalk'
            - '~/n8n_dir/node_modules/cheerio:/usr/local/lib/node_modules/cheerio'
            - '~/n8n_dir/node_modules/cloudscraper:/usr/local/lib/node_modules/cloudscraper'
            - '~/n8n_dir/node_modules/co:/usr/local/lib/node_modules/co'
            - '~/n8n_dir/node_modules/color-convert:/usr/local/lib/node_modules/color-convert'
            - '~/n8n_dir/node_modules/color-name:/usr/local/lib/node_modules/color-name'
            - '~/n8n_dir/node_modules/combined-stream:/usr/local/lib/node_modules/combined-stream'
            - '~/n8n_dir/node_modules/component-emitter:/usr/local/lib/node_modules/component-emitter'
            - '~/n8n_dir/node_modules/content-disposition:/usr/local/lib/node_modules/content-disposition'
            - '~/n8n_dir/node_modules/content-type:/usr/local/lib/node_modules/content-type'
            - '~/n8n_dir/node_modules/cookiejar:/usr/local/lib/node_modules/cookiejar'
            - '~/n8n_dir/node_modules/core-util-is:/usr/local/lib/node_modules/core-util-is'
            - '~/n8n_dir/node_modules/css-select:/usr/local/lib/node_modules/css-select'
            - '~/n8n_dir/node_modules/css-what:/usr/local/lib/node_modules/css-what'
            - '~/n8n_dir/node_modules/dashdash:/usr/local/lib/node_modules/dashdash'
            - '~/n8n_dir/node_modules/data-uri-to-buffer:/usr/local/lib/node_modules/data-uri-to-buffer'
            - '~/n8n_dir/node_modules/debug:/usr/local/lib/node_modules/debug'
            - '~/n8n_dir/node_modules/deep-is:/usr/local/lib/node_modules/deep-is'
            - '~/n8n_dir/node_modules/degenerator:/usr/local/lib/node_modules/degenerator'
            - '~/n8n_dir/node_modules/delayed-stream:/usr/local/lib/node_modules/delayed-stream'
            - '~/n8n_dir/node_modules/delegates:/usr/local/lib/node_modules/delegates'
            - '~/n8n_dir/node_modules/depd:/usr/local/lib/node_modules/depd'
            - '~/n8n_dir/node_modules/destroy:/usr/local/lib/node_modules/destroy'
            - '~/n8n_dir/node_modules/dom-serializer:/usr/local/lib/node_modules/dom-serializer'
            - '~/n8n_dir/node_modules/domelementtype:/usr/local/lib/node_modules/domelementtype'
            - '~/n8n_dir/node_modules/domhandler:/usr/local/lib/node_modules/domhandler'
            - '~/n8n_dir/node_modules/domutils:/usr/local/lib/node_modules/domutils'
            - '~/n8n_dir/node_modules/ecc-jsbn:/usr/local/lib/node_modules/ecc-jsbn'
            - '~/n8n_dir/node_modules/ee-first:/usr/local/lib/node_modules/ee-first'
            - '~/n8n_dir/node_modules/emitter-component:/usr/local/lib/node_modules/emitter-component'
            - '~/n8n_dir/node_modules/enqueue:/usr/local/lib/node_modules/enqueue'
            - '~/n8n_dir/node_modules/enstore:/usr/local/lib/node_modules/enstore'
            - '~/n8n_dir/node_modules/entities:/usr/local/lib/node_modules/entities'
            - '~/n8n_dir/node_modules/error-inject:/usr/local/lib/node_modules/error-inject'
            - '~/n8n_dir/node_modules/escape-html:/usr/local/lib/node_modules/escape-html'
            - '~/n8n_dir/node_modules/escape-string-regexp:/usr/local/lib/node_modules/escape-string-regexp'
            - '~/n8n_dir/node_modules/escodegen:/usr/local/lib/node_modules/escodegen'
            - '~/n8n_dir/node_modules/esprima:/usr/local/lib/node_modules/esprima'
            - '~/n8n_dir/node_modules/estraverse:/usr/local/lib/node_modules/estraverse'
            - '~/n8n_dir/node_modules/esutils:/usr/local/lib/node_modules/esutils'
            - '~/n8n_dir/node_modules/extend:/usr/local/lib/node_modules/extend'
            - '~/n8n_dir/node_modules/extsprintf:/usr/local/lib/node_modules/extsprintf'
            - '~/n8n_dir/node_modules/fast-deep-equal:/usr/local/lib/node_modules/fast-deep-equal'
            - '~/n8n_dir/node_modules/fast-json-stable-stringify:/usr/local/lib/node_modules/fast-json-stable-stringify'
            - '~/n8n_dir/node_modules/fast-levenshtein:/usr/local/lib/node_modules/fast-levenshtein'
            - '~/n8n_dir/node_modules/file-uri-to-path:/usr/local/lib/node_modules/file-uri-to-path'
            - '~/n8n_dir/node_modules/forever-agent:/usr/local/lib/node_modules/forever-agent'
            - '~/n8n_dir/node_modules/form-data:/usr/local/lib/node_modules/form-data'
            - '~/n8n_dir/node_modules/format-parser:/usr/local/lib/node_modules/format-parser'
            - '~/n8n_dir/node_modules/formidable:/usr/local/lib/node_modules/formidable'
            - '~/n8n_dir/node_modules/fs-extra:/usr/local/lib/node_modules/fs-extra'
            - '~/n8n_dir/node_modules/ftp:/usr/local/lib/node_modules/ftp'
            - '~/n8n_dir/node_modules/get-uri:/usr/local/lib/node_modules/get-uri'
            - '~/n8n_dir/node_modules/getpass:/usr/local/lib/node_modules/getpass'
            - '~/n8n_dir/node_modules/graceful-fs:/usr/local/lib/node_modules/graceful-fs'
            - '~/n8n_dir/node_modules/har-schema:/usr/local/lib/node_modules/har-schema'
            - '~/n8n_dir/node_modules/har-validator:/usr/local/lib/node_modules/har-validator'
            - '~/n8n_dir/node_modules/has-flag:/usr/local/lib/node_modules/has-flag'
            - '~/n8n_dir/node_modules/htmlparser2:/usr/local/lib/node_modules/htmlparser2'
            - '~/n8n_dir/node_modules/http-context:/usr/local/lib/node_modules/http-context'
            - '~/n8n_dir/node_modules/http-errors:/usr/local/lib/node_modules/http-errors'
            - '~/n8n_dir/node_modules/http-incoming:/usr/local/lib/node_modules/http-incoming'
            - '~/n8n_dir/node_modules/http-outgoing:/usr/local/lib/node_modules/http-outgoing'
            - '~/n8n_dir/node_modules/http-proxy-agent:/usr/local/lib/node_modules/http-proxy-agent'
            - '~/n8n_dir/node_modules/http-signature:/usr/local/lib/node_modules/http-signature'
            - '~/n8n_dir/node_modules/https-proxy-agent:/usr/local/lib/node_modules/https-proxy-agent'
            - '~/n8n_dir/node_modules/iconv-lite:/usr/local/lib/node_modules/iconv-lite'
            - '~/n8n_dir/node_modules/inherits:/usr/local/lib/node_modules/inherits'
            - '~/n8n_dir/node_modules/ip:/usr/local/lib/node_modules/ip'
            - '~/n8n_dir/node_modules/is-browser:/usr/local/lib/node_modules/is-browser'
            - '~/n8n_dir/node_modules/is-typedarray:/usr/local/lib/node_modules/is-typedarray'
            - '~/n8n_dir/node_modules/is-url:/usr/local/lib/node_modules/is-url'
            - '~/n8n_dir/node_modules/isarray:/usr/local/lib/node_modules/isarray'
            - '~/n8n_dir/node_modules/isobject:/usr/local/lib/node_modules/isobject'
            - '~/n8n_dir/node_modules/isstream:/usr/local/lib/node_modules/isstream'
            - '~/n8n_dir/node_modules/jsbn:/usr/local/lib/node_modules/jsbn'
            - '~/n8n_dir/node_modules/json-schema:/usr/local/lib/node_modules/json-schema'
            - '~/n8n_dir/node_modules/json-schema-traverse:/usr/local/lib/node_modules/json-schema-traverse'
            - '~/n8n_dir/node_modules/json-stringify-safe:/usr/local/lib/node_modules/json-stringify-safe'
            - '~/n8n_dir/node_modules/jsonfile:/usr/local/lib/node_modules/jsonfile'
            - '~/n8n_dir/node_modules/jsprim:/usr/local/lib/node_modules/jsprim'
            - '~/n8n_dir/node_modules/koa-is-json:/usr/local/lib/node_modules/koa-is-json'
            - '~/n8n_dir/node_modules/levn:/usr/local/lib/node_modules/levn'
            - '~/n8n_dir/node_modules/lodash:/usr/local/lib/node_modules/lodash'
            - '~/n8n_dir/node_modules/lodash.assignin:/usr/local/lib/node_modules/lodash.assignin'
            - '~/n8n_dir/node_modules/lodash.bind:/usr/local/lib/node_modules/lodash.bind'
            - '~/n8n_dir/node_modules/lodash.defaults:/usr/local/lib/node_modules/lodash.defaults'
            - '~/n8n_dir/node_modules/lodash.filter:/usr/local/lib/node_modules/lodash.filter'
            - '~/n8n_dir/node_modules/lodash.flatten:/usr/local/lib/node_modules/lodash.flatten'
            - '~/n8n_dir/node_modules/lodash.foreach:/usr/local/lib/node_modules/lodash.foreach'
            - '~/n8n_dir/node_modules/lodash.map:/usr/local/lib/node_modules/lodash.map'
            - '~/n8n_dir/node_modules/lodash.merge:/usr/local/lib/node_modules/lodash.merge'
            - '~/n8n_dir/node_modules/lodash.pick:/usr/local/lib/node_modules/lodash.pick'
            - '~/n8n_dir/node_modules/lodash.reduce:/usr/local/lib/node_modules/lodash.reduce'
            - '~/n8n_dir/node_modules/lodash.reject:/usr/local/lib/node_modules/lodash.reject'
            - '~/n8n_dir/node_modules/lodash.some:/usr/local/lib/node_modules/lodash.some'
            - '~/n8n_dir/node_modules/lru-cache:/usr/local/lib/node_modules/lru-cache'
            - '~/n8n_dir/node_modules/media-typer:/usr/local/lib/node_modules/media-typer'
            - '~/n8n_dir/node_modules/methods:/usr/local/lib/node_modules/methods'
            - '~/n8n_dir/node_modules/mime:/usr/local/lib/node_modules/mime'
            - '~/n8n_dir/node_modules/mime-db:/usr/local/lib/node_modules/mime-db'
            - '~/n8n_dir/node_modules/mime-types:/usr/local/lib/node_modules/mime-types'
            - '~/n8n_dir/node_modules/monotonic-timestamp:/usr/local/lib/node_modules/monotonic-timestamp'
            - '~/n8n_dir/node_modules/ms:/usr/local/lib/node_modules/ms'
            - '~/n8n_dir/node_modules/negotiator:/usr/local/lib/node_modules/negotiator'
            - '~/n8n_dir/node_modules/netmask:/usr/local/lib/node_modules/netmask'
            - '~/n8n_dir/node_modules/nth-check:/usr/local/lib/node_modules/nth-check'
            - '~/n8n_dir/node_modules/oauth-sign:/usr/local/lib/node_modules/oauth-sign'
            - '~/n8n_dir/node_modules/object-assign:/usr/local/lib/node_modules/object-assign'
            - '~/n8n_dir/node_modules/on-finished:/usr/local/lib/node_modules/on-finished'
            - '~/n8n_dir/node_modules/optionator:/usr/local/lib/node_modules/optionator'
            - '~/n8n_dir/node_modules/pac-proxy-agent:/usr/local/lib/node_modules/pac-proxy-agent'
            - '~/n8n_dir/node_modules/pac-resolver:/usr/local/lib/node_modules/pac-resolver'
            - '~/n8n_dir/node_modules/parseurl:/usr/local/lib/node_modules/parseurl'
            - '~/n8n_dir/node_modules/performance-now:/usr/local/lib/node_modules/performance-now'
            - '~/n8n_dir/node_modules/prelude-ls:/usr/local/lib/node_modules/prelude-ls'
            - '~/n8n_dir/node_modules/process-nextick-args:/usr/local/lib/node_modules/process-nextick-args'
            - '~/n8n_dir/node_modules/promise-polyfill:/usr/local/lib/node_modules/promise-polyfill'
            - '~/n8n_dir/node_modules/proxy-agent:/usr/local/lib/node_modules/proxy-agent'
            - '~/n8n_dir/node_modules/proxy-from-env:/usr/local/lib/node_modules/proxy-from-env'
            - '~/n8n_dir/node_modules/psl:/usr/local/lib/node_modules/psl'
            - '~/n8n_dir/node_modules/punycode:/usr/local/lib/node_modules/punycode'
            - '~/n8n_dir/node_modules/qs:/usr/local/lib/node_modules/qs'
            - '~/n8n_dir/node_modules/querystring:/usr/local/lib/node_modules/querystring'
            - '~/n8n_dir/node_modules/raw-body:/usr/local/lib/node_modules/raw-body'
            - '~/n8n_dir/node_modules/readable-stream:/usr/local/lib/node_modules/readable-stream'
            - '~/n8n_dir/node_modules/request:/usr/local/lib/node_modules/request'
            - '~/n8n_dir/node_modules/request-promise:/usr/local/lib/node_modules/request-promise'
            - '~/n8n_dir/node_modules/request-promise-core:/usr/local/lib/node_modules/request-promise-core'
            - '~/n8n_dir/node_modules/request-x-ray:/usr/local/lib/node_modules/request-x-ray'
            - '~/n8n_dir/node_modules/safe-buffer:/usr/local/lib/node_modules/safe-buffer'
            - '~/n8n_dir/node_modules/safer-buffer:/usr/local/lib/node_modules/safer-buffer'
            - '~/n8n_dir/node_modules/selectn:/usr/local/lib/node_modules/selectn'
            - '~/n8n_dir/node_modules/setprototypeof:/usr/local/lib/node_modules/setprototypeof'
            - '~/n8n_dir/node_modules/sliced:/usr/local/lib/node_modules/sliced'
            - '~/n8n_dir/node_modules/smart-buffer:/usr/local/lib/node_modules/smart-buffer'
            - '~/n8n_dir/node_modules/socks:/usr/local/lib/node_modules/socks'
            - '~/n8n_dir/node_modules/socks-proxy-agent:/usr/local/lib/node_modules/socks-proxy-agent'
            - '~/n8n_dir/node_modules/source-map:/usr/local/lib/node_modules/source-map'
            - '~/n8n_dir/node_modules/sshpk:/usr/local/lib/node_modules/sshpk'
            - '~/n8n_dir/node_modules/statuses:/usr/local/lib/node_modules/statuses'
            - '~/n8n_dir/node_modules/stealthy-require:/usr/local/lib/node_modules/stealthy-require'
            - '~/n8n_dir/node_modules/stream-to-string:/usr/local/lib/node_modules/stream-to-string'
            - '~/n8n_dir/node_modules/string-format:/usr/local/lib/node_modules/string-format'
            - '~/n8n_dir/node_modules/string_decoder:/usr/local/lib/node_modules/string_decoder'
            - '~/n8n_dir/node_modules/superagent:/usr/local/lib/node_modules/superagent'
            - '~/n8n_dir/node_modules/superagent-proxy:/usr/local/lib/node_modules/superagent-proxy'
            - '~/n8n_dir/node_modules/supports-color:/usr/local/lib/node_modules/supports-color'
            - '~/n8n_dir/node_modules/toidentifier:/usr/local/lib/node_modules/toidentifier'
            - '~/n8n_dir/node_modules/torrent-search-api:/usr/local/lib/node_modules/torrent-search-api'
            - '~/n8n_dir/node_modules/tough-cookie:/usr/local/lib/node_modules/tough-cookie'
            - '~/n8n_dir/node_modules/tslib:/usr/local/lib/node_modules/tslib'
            - '~/n8n_dir/node_modules/tunnel-agent:/usr/local/lib/node_modules/tunnel-agent'
            - '~/n8n_dir/node_modules/tweetnacl:/usr/local/lib/node_modules/tweetnacl'
            - '~/n8n_dir/node_modules/type-check:/usr/local/lib/node_modules/type-check'
            - '~/n8n_dir/node_modules/type-is:/usr/local/lib/node_modules/type-is'
            - '~/n8n_dir/node_modules/universalify:/usr/local/lib/node_modules/universalify'
            - '~/n8n_dir/node_modules/unpipe:/usr/local/lib/node_modules/unpipe'
            - '~/n8n_dir/node_modules/uri-js:/usr/local/lib/node_modules/uri-js'
            - '~/n8n_dir/node_modules/util:/usr/local/lib/node_modules/util'
            - '~/n8n_dir/node_modules/util-deprecate:/usr/local/lib/node_modules/util-deprecate'
            - '~/n8n_dir/node_modules/uuid:/usr/local/lib/node_modules/uuid'
            - '~/n8n_dir/node_modules/vary:/usr/local/lib/node_modules/vary'
            - '~/n8n_dir/node_modules/verror:/usr/local/lib/node_modules/verror'
            - '~/n8n_dir/node_modules/word-wrap:/usr/local/lib/node_modules/word-wrap'
            - '~/n8n_dir/node_modules/wrap-fn:/usr/local/lib/node_modules/wrap-fn'
            - '~/n8n_dir/node_modules/x-ray:/usr/local/lib/node_modules/x-ray'
            - '~/n8n_dir/node_modules/x-ray-crawler:/usr/local/lib/node_modules/x-ray-crawler'
            - '~/n8n_dir/node_modules/x-ray-parse:/usr/local/lib/node_modules/x-ray-parse'
            - '~/n8n_dir/node_modules/x-ray-scraper:/usr/local/lib/node_modules/x-ray-scraper'
            - '~/n8n_dir/node_modules/xregexp:/usr/local/lib/node_modules/xregexp'
            - '~/n8n_dir/node_modules/yallist:/usr/local/lib/node_modules/yallist'
            - '~/n8n_dir/node_modules/yieldly:/usr/local/lib/node_modules/yieldly'
        image: 'n8nio/n8n:latest-rpi'
        environment:
            - N8N_BASIC_AUTH_ACTIVE=true
            - N8N_BASIC_AUTH_USER=username
            - N8N_BASIC_AUTH_PASSWORD=your_secret_n8n_password
            - EXECUTIONS_DATA_PRUNE=true
            - EXECUTIONS_DATA_MAX_AGE=120
            - EXECUTIONS_TIMEOUT=300
            - EXECUTIONS_TIMEOUT_MAX=500
            - GENERIC_TIMEZONE=Europe/Berlin
            - NODE_FUNCTION_ALLOW_EXTERNAL=torrent-search-api

Once configured this way run n8n and create a new workflow coping the one proposed.

Configure workflow

Transmission

In order to send command to transmission you must validate the Basic Auth. To do so: open the Start download node and edit the Credentials.

image.png

Perform the same operation choosing the new credentials also in node Start download new token.
In this automation we call transmission twice due to a security protocol in transmission system that prevents single click commands to be triggered, performing the request twice bypasses this security mechanism. https://en.wikipedia.org/wiki/Cross-site_request_forgery
We use the X-Transmission-Session-Id provided by the first request to authenticate the second request.

Telegram

In order to make the workflow work as expected you must create a telegram bot and configure the nodes (Torrent not found and Telegram1) to send your message once the workflow is complete.
Here's an easy guide to follow https://docs.n8n.io/nodes/n8n-nodes-base.telegram/
In those nodes you also should configure the Chat ID, you may use your telegram username or use a bot to retrieve your id.
You may chat with useridinfobot that sends you your id.

Ok google automation

Since right now we do not have a n8n client for mobile that can trigger automation using google assistant I decided to use an IFTTT automation to trigger the webhook.

I connect my IFTTT account with google assistant and pick the trigger.
Say a phrase with a text ingredient as in the picture below.

image.png

And configure the trigger this way.
scarica $ -> download $
or
metti in download $ -> put in download $

or some other trigger you may want.

image.png

Then configure your server to trigger the webhook of n8n.

image.png

Conclusion

In conclusion we provide a fully working automation that integrates in n8n a node library and provides an easy trigger to perform a complex operation.

Security concern

Giving the ability to trigger a download may be problematic for potential unwanted torrent malware download, so you may decide to authenticate the webhook request passing in the body another field with a shared token between the two endpoints.

Moreover the torrent-search-api library and its dependencies have some vulnerability that you may want to avoid on your own media-center, this will hopefully be patched soon in a further release of the library.

This is just an interesting proof of concept.

Quality of the download

You may want to introduce another block between torrent search and webhook trigger to search for a movie based on the words detected by google assistant, sometimes it misinterprets something and you may end up downloading potential copyrighted material.
Please use this automation only for free and open source movies and music.

Share Template

More Engineering workflow templates

Webhook node
Respond to Webhook node

Creating an API endpoint
Task: Create a simple API endpoint using the Webhook and Respond to Webhook nodes Why: You can prototype or replace a backend process with a single workflow Main use cases: Replace backend logic with a workflow
jon-n8n
Jonathan
Merge node

Joining different datasets
Task: Merge two datasets into one based on matching rules Why: A powerful capability of n8n is to easily branch out the workflow in order to process different datasets. Even more powerful is the ability to join them back together with SQL-like joining logic. Main use cases: Appending data sets Keep only new items Keep only existing items
jon-n8n
Jonathan
GitHub node
HTTP Request node
Merge node
+11

Back Up Your n8n Workflows To Github
This workflow will backup your workflows to Github. It uses the public api to export all of the workflow data using the n8n node. It then loops over the data checks in Github to see if a file exists that uses the workflow name. Once checked it will then update the file on Github if it exists, Create a new file if it doesn't exist and if it's the same it will ignore the file. Config Options repo_owner - Github owner repo_name - Github repository name repo_path - Path within the Github repository >This workflow has been updated to use the n8n node and the code node so requires at least version 0.198.0 of n8n
jon-n8n
Jonathan
HTTP Request node
+8

Scrape and store data from multiple website pages
This workflow allows extracting data from multiple pages website. The workflow: 1) Starts in a country list at https://www.theswiftcodes.com/browse-by-country/. 2) Loads every country page (https://www.theswiftcodes.com/albania/) 3) Paginates every page in the country page. 4) Extracts data from the country page. 5) Saves data to MongoDB. 6) Paginates through all pages in all countries. It uses getWorkflowStaticData('global') method to recover the next page (saved from the previous page), and it goes ahead with all the pages. There is a first section where the countries list is recovered and extracted. Later, I try to read if a local cache page is available and I recover the cached page from the disk. Finally, I save data to MongoDB, and we paginate all the pages in the country and for all the countries. I have applied a cache system to save a visited page to n8n local disk. If I relaunch workflow, we check if a cache file exists to discard non-required requests to the webpage. If the data present in the website changes, you can apply a Cron node to check the website once per week. Finally, before inserting data in MongoDB, the best way to avoid duplicates is to check that swift_code (the primary value of the collection) doesn't exist. I recommend using a proxy for all requests to avoid IP blocks. A good solution for proxy plus IP rotation is scrapoxy.io. This workflow is perfect for small data requirements. If you need to scrape dynamic data, you can use a Headless browser or any other service. If you want to scrape huge lists of URIs, I recommend using Scrapy + Scrapoxy.
mcolomer
Miquel Colomer
Google Sheets node
HTTP Request node
Item Lists node
+5

Google Maps Scraper
This workflow allows to scrape Google Maps data in an efficient way using SerpAPI. You'll get all data from Gmaps at a cheaper cost than Google Maps API. Add as input, your Google Maps search URL and you'll get a list of places with many data points such as: phone number website rating reviews address And much more. Full guide to implement the workflow is here: https://lempire.notion.site/Scrape-Google-Maps-places-with-n8n-b7f1785c3d474e858b7ee61ad4c21136?pvs=4
lucasperret
Lucas Perret
HTTP Request node
Merge node
+3

Backup n8n workflows to Google Drive
Temporary solution using the undocumented REST API for backups using Google drive. Please note that there are issues with this workflow. It does not support versioning, so please know that it will create multiple copies of the workflows so if you run this daily it will make the folder grow quickly. Once I figure out how to version in Gdrive I'll update it here.
djangelic
Angel Menendez

More Building Blocks workflow templates

Webhook node
Respond to Webhook node

Creating an API endpoint
Task: Create a simple API endpoint using the Webhook and Respond to Webhook nodes Why: You can prototype or replace a backend process with a single workflow Main use cases: Replace backend logic with a workflow
jon-n8n
Jonathan
Customer Datastore (n8n training) node

Very quick quickstart
Want to learn the basics of n8n? Our comprehensive quick quickstart tutorial is here to guide you through the basics of n8n, step by step. Designed with beginners in mind, this tutorial provides a hands-on approach to learning n8n's basic functionalities.
deborah
Deborah
HTTP Request node
Item Lists node

Pulling data from services that n8n doesn’t have a pre-built integration for
You still can use the app in a workflow even if we don’t have a node for that or the existing operation for that. With the HTTP Request node, it is possible to call any API point and use the incoming data in your workflow Main use cases: Connect with apps and services that n8n doesn’t have integration with Web scraping How it works This workflow can be divided into three branches, each serving a distinct purpose: 1.Splitting into Items (HTTP Request - Get Mock Albums): The workflow initiates with a manual trigger (On clicking 'execute'). It performs an HTTP request to retrieve mock albums data from "https://jsonplaceholder.typicode.com/albums." The obtained data is split into items using the Item Lists node, facilitating easier management. 2.Data Scraping (HTTP Request - Get Wikipedia Page and HTML Extract): Another branch of the workflow involves fetching a random Wikipedia page using an HTTP request to "https://en.wikipedia.org/wiki/Special:Random." The HTML Extract node extracts the article title from the fetched Wikipedia page. 3.Handling Pagination (The final branch deals with handling pagination for a GitHub API request): It sends an HTTP request to "https://api.github.com/users/that-one-tom/starred," with parameters like the page number and items per page dynamically set by the Set node. The workflow uses conditions (If - Are we finished?) to check if there are more pages to retrieve and increments the page number accordingly (Set - Increment Page). This process repeats until all pages are fetched, allowing for comprehensive data retrieval.
jon-n8n
Jonathan
Merge node

Joining different datasets
Task: Merge two datasets into one based on matching rules Why: A powerful capability of n8n is to easily branch out the workflow in order to process different datasets. Even more powerful is the ability to join them back together with SQL-like joining logic. Main use cases: Appending data sets Keep only new items Keep only existing items
jon-n8n
Jonathan
GitHub node
HTTP Request node
Merge node
+11

Back Up Your n8n Workflows To Github
This workflow will backup your workflows to Github. It uses the public api to export all of the workflow data using the n8n node. It then loops over the data checks in Github to see if a file exists that uses the workflow name. Once checked it will then update the file on Github if it exists, Create a new file if it doesn't exist and if it's the same it will ignore the file. Config Options repo_owner - Github owner repo_name - Github repository name repo_path - Path within the Github repository >This workflow has been updated to use the n8n node and the code node so requires at least version 0.198.0 of n8n
jon-n8n
Jonathan
Webhook node
Item Lists node
Respond to Webhook node

Convert JSON to an Excel file
Send a simple JSON array via HTTP POST and get an Excel file. The default filename is Export.xlsx. By adding the (optional) request ?filename=xyz you can specify the filename. NOTE: do not forget to change the webhook path!
dickhoning
Dick

Implement complex processes faster with n8n

Get started
red icon yellow icon red icon yellow icon

Automate without limits

Made with ♥ in Berlin
Impressum | Legal | Privacy

Report a vulnerability
© 2024 n8n | All rights reserved.