Google BigQuery and Scrape-It.Cloud integration

Make sure you have n8n installed and running:

• Install n8n on any platform using npm or Docker;
• Or let n8n host it for you on its cloud infrastructure.

In n8n, click the "Add workflow" button in the Workflows tab to create a new workflow.

Add the starting point – a trigger on when your workflow should run: on an app event, on schedule, on a webhook call, when called by another workflow, or manually. In some cases, the HTTP Request node might already serve as your starting point.

Drag the Google BigQuery node from the nodes panel on the left to the workflow canvas. Before dragging, choose the action Google BigQuery will perform. You can also swap steps and add the Scrape-It.Cloud node first, as the step 4.

Click on the Google BigQuery node, then click on the "Select credential" dropdown option next to the “Credential to connect with” field. Follow the authentication process to grant n8n access to your Google BigQuery. You can also share credentials you've created, and store sensitive credential information in an external vault using n8n's external secrets feature.

Add the HTTP Request node to your workflow canvas and authenticate it using a predefined credential type. This allows you to perform custom operations, without additional authentication setup.

Click on the HTTP node to configure its settings. With generic authentication, you'll typically need to provide endpoint URLs, headers, parameters, and any other authentication details specific to the service you're integrating with:

• Find the documentation of the Scrape-It.Cloud API and see if the API supports HTTP requests;
• Most APIs require some form of authentication and you can configure this in the HTTP Request mode (Basic Auth, Custom Auth, Digest Auth, Header Auth, OAuth1 API, OAuth2 API, Query Auth).

Connect the output of the Scrape-It.Cloud node to the input of the Google BigQuery node or vice versa.

Use n8n's core nodes to transform data the way you need and extend the integration:

• Splitting: IF and Switch
• Merging: Merge, Compare Datasets
• Looping: IF and Loop Over Items
• Waiting: Wait
• Creating sub-workflows: Execute Workflow and Execute Workflow Trigger
• Error handling: Stop And Error and Error Trigger
• Code: write custom JavaScript or Python and run it as a step in your workflow.

Once you've configured everything, make sure to save your workflow and activate it. Activating a workflow means that it will run automatically every time a trigger node receives input or meets a condition. By default, all newly created workflows are deactivated.

Run the workflow to see if everything is working as expected. Based on your configuration, data should flow from Google BigQuery to Scrape-It.Cloud or vice versa. Easily debug your workflow: you can check past executions to isolate the mistake and fix it.