n8n Workflow Automation

Overview

This guide shows you how to integrate SpiderIQ’s orchestrated campaigns with n8n, the powerful workflow automation platform. You’ll build a complete visual workflow that:

Creates orchestrated campaigns via SpiderIQ API
Loops through locations until complete
Fetches aggregated results
Exports to Google Sheets, Airtable, or your database

Prerequisites

n8n Instance

Self-hosted or n8n Cloud account

SpiderIQ Credentials

Your API token from SpiderIQ dashboard

Step 1: Create SpiderIQ Credentials

First, set up authentication for SpiderIQ in n8n.

Open Credentials

In n8n, go to Settings → Credentials → Add Credential

Select Type

Choose Header Auth (since SpiderIQ uses Bearer token)

Configure

Name: SpiderIQ API
Header Name: Authorization
Header Value: Bearer cli_xxx:sk_xxx:secret_xxx (your full token)

Save

Click Save to store the credential securely

Store your token securely in n8n credentials. Never hardcode it in workflow nodes.

Step 2: Create Campaign Node

Set up the HTTP Request node to create an orchestrated campaign.

HTTP Request Node Configuration

General Settings:

Setting	Value
Method	`POST`
URL	`https://spideriq.di-atomic.com/api/v1/jobs/spiderMaps/campaigns/submit`
Authentication	Predefined Credential Type → Header Auth
Credential	`SpiderIQ API`

Headers:

Header	Value
Content-Type	`application/json`

Body (JSON):

{
  "query": "{{ $json.query }}",
  "country_code": "{{ $json.country_code }}",
  "name": "n8n Campaign - {{ $now.format('yyyy-MM-dd') }}",
  "filter": {
    "mode": "population",
    "min_population": {{ $json.min_population || 50000 }}
  },
  "workflow": {
    "spidersite": {
      "enabled": true,
      "max_pages": 10,
      "crawl_strategy": "bestfirst",
      "enable_spa": true,
      "extract_company_info": true,
      "product_description": "{{ $json.product_description }}",
      "icp_description": "{{ $json.icp_description }}"
    },
    "spiderverify": {
      "enabled": true,
      "max_emails_per_business": 5
    },
    "filter_social_media": true,
    "filter_review_sites": true,
    "filter_directories": true,
    "filter_maps": true
  }
}

Response

The node outputs:

{
  "campaign_id": "camp_fr_restaurants_20251223_abc123",
  "status": "active",
  "total_locations": 42,
  "has_workflow": true
}

Step 3: Loop Until Complete

Create a loop that calls /next until all locations are processed.

Loop Structure

┌─────────────────────────────────────────┐
│  Loop                                    │
│  ┌─────────────────────────────────────┐│
│  │ HTTP Request: POST /next            ││
│  └─────────────────────────────────────┘│
│  ┌─────────────────────────────────────┐│
│  │ IF: has_more == true                ││
│  │   → Continue Loop                   ││
│  │   → Else: Exit Loop                 ││
│  └─────────────────────────────────────┘│
│  ┌─────────────────────────────────────┐│
│  │ Wait: 2 seconds                     ││
│  └─────────────────────────────────────┘│
└─────────────────────────────────────────┘

HTTP Request Node: Call /next

Configuration:

Setting	Value
Method	`POST`
URL	`https://spideriq.di-atomic.com/api/v1/jobs/spiderMaps/campaigns/{{ $json.campaign_id }}/next`
Authentication	Header Auth → `SpiderIQ API`

Response:

{
  "has_more": true,
  "campaign_id": "camp_fr_restaurants_20251223_abc123",
  "current_task": {
    "job_id": "550e8400-e29b-41d4-a716-446655440000",
    "search_string": "restaurants in Paris, France",
    "status": "queued"
  },
  "progress": {
    "completed": 5,
    "total": 42,
    "percentage": 11.9
  }
}

IF Node: Check has_more

Condition:

Value 1: {{ $json.has_more }}
Operation: equal
Value 2: true

Branches:

True: Continue to Wait node, then loop back
False: Exit loop, proceed to get results

Wait Node

Configuration:

Wait Time: 2 seconds
Reason: Rate limiting and allowing jobs to process

SpiderIQ processes jobs asynchronously across 112 workers. The /next endpoint returns immediately after queuing each job, so a short delay helps prevent overwhelming the API.

Step 4: Get Workflow Results

After the loop completes, fetch aggregated results.

HTTP Request Node: Get Results

Configuration:

Setting	Value
Method	`GET`
URL	`https://spideriq.di-atomic.com/api/v1/jobs/spiderMaps/campaigns/{{ $json.campaign_id }}/workflow-results`
Authentication	Header Auth → `SpiderIQ API`

Response Structure:

{
  "campaign_id": "camp_fr_restaurants_20251223_abc123",
  "total_businesses": 840,
  "total_valid_emails": 892,
  "workflow_progress": {
    "sites_completed": 485,
    "verifies_completed": 485,
    "emails_verified": 1250
  },
  "locations": [
    {
      "display_name": "Paris",
      "businesses": [
        {
          "business_name": "Le Petit Bistro",
          "business_phone": "+33 1 42 96 12 34",
          "domain": "lepetitbistro.fr",
          "emails_verified": [
            {"email": "contact@lepetitbistro.fr", "status": "valid"}
          ]
        }
      ]
    }
  ]
}

Step 5: Transform and Flatten Data

Use a Code node or Item Lists node to flatten the nested results.

Code Node: Flatten Results

JavaScript:

const results = $input.all()[0].json;
const leads = [];

for (const location of results.locations) {
  for (const business of location.businesses) {
    // Get valid emails only
    const validEmails = (business.emails_verified || [])
      .filter(e => e.status === 'valid')
      .map(e => e.email);

    if (validEmails.length > 0) {
      leads.push({
        campaign_id: results.campaign_id,
        location: location.display_name,
        business_name: business.business_name,
        business_phone: business.business_phone || '',
        business_address: business.business_address || '',
        business_rating: business.business_rating || 0,
        domain: business.domain || '',
        email: validEmails[0], // Primary email
        all_emails: validEmails.join(', '),
        lead_score: business.lead_scoring?.champ_score || 0,
        workflow_stage: business.workflow_stage
      });
    }
  }
}

return leads.map(lead => ({ json: lead }));

Output:

[
  {
    "campaign_id": "camp_fr_restaurants_20251223_abc123",
    "location": "Paris",
    "business_name": "Le Petit Bistro",
    "business_phone": "+33 1 42 96 12 34",
    "business_address": "123 Rue de Rivoli, Paris",
    "business_rating": 4.5,
    "domain": "lepetitbistro.fr",
    "email": "contact@lepetitbistro.fr",
    "all_emails": "contact@lepetitbistro.fr, reservations@lepetitbistro.fr",
    "lead_score": 78,
    "workflow_stage": "complete"
  }
]

Step 6: Export to Destination

Connect the flattened data to your preferred destination.

Option A: Google Sheets

Google Sheets Node Configuration:

Setting	Value
Operation	`Append or Update`
Document	Your Google Sheet
Sheet	`Leads`
Mapping	Define each column

Column Mapping:

Column	Value
A - Business Name	`{{ $json.business_name }}`
B - Phone	`{{ $json.business_phone }}`
C - Address	`{{ $json.business_address }}`
D - Rating	`{{ $json.business_rating }}`
E - Domain	`{{ $json.domain }}`
F - Email	`{{ $json.email }}`
G - Lead Score	`{{ $json.lead_score }}`
H - Location	`{{ $json.location }}`
I - Campaign ID	`{{ $json.campaign_id }}`

Option B: Airtable

Airtable Node Configuration:

Setting	Value
Operation	`Create`
Base	Your Airtable Base
Table	`Leads`

Field Mapping: Map each field from the Code node output to Airtable columns.

Option C: PostgreSQL / MySQL

Database Node Configuration:

Setting	Value
Operation	`Insert`
Table	`leads`

Use the Execute Query node for bulk inserts:

INSERT INTO leads (campaign_id, business_name, email, phone, rating, domain, location)
VALUES ($1, $2, $3, $4, $5, $6, $7)

Step 7: Schedule the Workflow

Set up automated execution with the Schedule Trigger node.

Schedule Trigger Configuration

Example Schedules:

Schedule	Cron Expression	Description
Daily at 8 AM	`0 8 * * *`	Run every morning
Weekly Monday	`0 9 * * 1`	Weekly on Monday at 9 AM
Every 6 hours	`0 /6 * *`	Four times per day

Manual Trigger for Testing

Add a Manual Trigger node in parallel for testing:

[Manual Trigger] ─┐
                  ├─→ [Set Parameters] → [Create Campaign] → ...
[Schedule Trigger]┘

Set Parameters Node

Configure campaign parameters:

{
  "query": "restaurants",
  "country_code": "FR",
  "min_population": 100000,
  "product_description": "Restaurant management software",
  "icp_description": "Restaurant owners seeking efficiency tools"
}

Complete Workflow

Here’s the full workflow structure:

[Schedule Trigger]
       │
       ▼
[Set Parameters]
       │
       ▼
[HTTP Request: Create Campaign]
       │
       ▼
[Loop] ◄──────────────────────┐
  │                           │
  ▼                           │
[HTTP Request: POST /next]    │
  │                           │
  ▼                           │
[IF: has_more == true?]       │
  │           │               │
  │ Yes       │ No            │
  ▼           ▼               │
[Wait 2s]  [Exit Loop]        │
  │                           │
  └───────────────────────────┘
       │
       ▼
[HTTP Request: GET /workflow-results]
       │
       ▼
[Code: Flatten Results]
       │
       ▼
[Google Sheets / Airtable / Database]
       │
       ▼
[Send Notification (Optional)]

Importable Workflow JSON

Copy this JSON and import it into n8n:

Complete Workflow JSON

{
  "name": "SpiderIQ Orchestrated Campaign",
  "nodes": [
    {
      "parameters": {
        "rule": {
          "interval": [{"field": "cronExpression", "expression": "0 8 * * *"}]
        }
      },
      "name": "Schedule Trigger",
      "type": "n8n-nodes-base.scheduleTrigger",
      "position": [250, 300]
    },
    {
      "parameters": {
        "values": {
          "string": [
            {"name": "query", "value": "restaurants"},
            {"name": "country_code", "value": "FR"},
            {"name": "product_description", "value": "Restaurant management software"},
            {"name": "icp_description", "value": "Restaurant owners seeking efficiency"}
          ],
          "number": [{"name": "min_population", "value": 100000}]
        }
      },
      "name": "Set Parameters",
      "type": "n8n-nodes-base.set",
      "position": [450, 300]
    },
    {
      "parameters": {
        "method": "POST",
        "url": "https://spideriq.di-atomic.com/api/v1/jobs/spiderMaps/campaigns/submit",
        "authentication": "predefinedCredentialType",
        "nodeCredentialType": "httpHeaderAuth",
        "sendHeaders": true,
        "headerParameters": {
          "parameters": [{"name": "Content-Type", "value": "application/json"}]
        },
        "sendBody": true,
        "bodyParameters": {
          "parameters": [
            {"name": "query", "value": "={{ $json.query }}"},
            {"name": "country_code", "value": "={{ $json.country_code }}"},
            {"name": "workflow", "value": "={\"spidersite\":{\"enabled\":true,\"max_pages\":10},\"spiderverify\":{\"enabled\":true}}"}
          ]
        }
      },
      "name": "Create Campaign",
      "type": "n8n-nodes-base.httpRequest",
      "position": [650, 300],
      "credentials": {"httpHeaderAuth": {"id": "1", "name": "SpiderIQ API"}}
    }
  ],
  "connections": {
    "Schedule Trigger": {"main": [[{"node": "Set Parameters", "type": "main", "index": 0}]]},
    "Set Parameters": {"main": [[{"node": "Create Campaign", "type": "main", "index": 0}]]}
  }
}

After importing, you’ll need to:

Create the SpiderIQ API credential
Complete the Loop and remaining nodes
Configure your destination (Sheets/Airtable/DB)

Error Handling

Add error handling with the Error Trigger workflow.

Error Trigger Workflow

Create a separate workflow that triggers on errors:

[Error Trigger]
       │
       ▼
[Get Error Details]
       │
       ▼
[Send Slack/Email Alert]

Retry Logic

Wrap API calls in a Loop with retry logic:

// In Code node before HTTP Request
const maxRetries = 3;
let attempt = 0;
let success = false;

while (attempt < maxRetries && !success) {
  try {
    // API call happens in next node
    success = true;
  } catch (error) {
    attempt++;
    await new Promise(r => setTimeout(r, 5000)); // Wait 5s
  }
}

Best Practices

Use Sub-Workflows

Break complex logic into sub-workflows for maintainability.

Add Logging

Use the Notion or Airtable node to log each run for debugging.

Set Timeouts

Configure execution timeouts for long campaigns.

Monitor Executions

Check n8n’s Execution History for failed runs.

Troubleshooting

401 Unauthorized Error

Verify your SpiderIQ token is correct
Check the token includes all three parts: cli_xxx:sk_xxx:secret_xxx
Ensure the Header Auth credential has Bearer prefix

Loop Never Exits

Check the IF node condition: {{ $json.has_more }} == true
Verify the /next response includes has_more field
Add a maximum iterations safety limit

Empty Results

Confirm the campaign completed (check /status)
Verify has_workflow: true in campaign response
Check if locations had any businesses

Timeout Errors

Increase workflow timeout in Settings
For large campaigns, consider splitting by region
Add longer Wait times between /next calls

Next Steps

Xano Integration

Prefer a no-code backend? Try Xano.

Orchestration Guide

Deep dive into workflow configuration.

API Reference

Complete campaign API documentation.

Campaign Status

Monitor campaign progress.

Guides

​Overview

​Prerequisites

n8n Instance

SpiderIQ Credentials

​Step 1: Create SpiderIQ Credentials

​Step 2: Create Campaign Node

​HTTP Request Node Configuration

​Response

​Step 3: Loop Until Complete

​Loop Structure

​HTTP Request Node: Call /next

​IF Node: Check has_more

​Wait Node

​Step 4: Get Workflow Results

​HTTP Request Node: Get Results

​Step 5: Transform and Flatten Data

​Code Node: Flatten Results

​Step 6: Export to Destination

​Option A: Google Sheets

​Option B: Airtable

​Option C: PostgreSQL / MySQL

​Step 7: Schedule the Workflow

​Schedule Trigger Configuration

​Manual Trigger for Testing

​Set Parameters Node

​Complete Workflow

​Importable Workflow JSON

​Error Handling

​Error Trigger Workflow

​Retry Logic

​Best Practices

Use Sub-Workflows

Add Logging

Set Timeouts

Monitor Executions

​Troubleshooting

​Next Steps

Xano Integration

Orchestration Guide

API Reference

Campaign Status

Overview

Prerequisites

Step 1: Create SpiderIQ Credentials

Step 2: Create Campaign Node

HTTP Request Node Configuration

Response

Step 3: Loop Until Complete

Loop Structure

HTTP Request Node: Call /next

IF Node: Check has_more

Wait Node

Step 4: Get Workflow Results

HTTP Request Node: Get Results

Step 5: Transform and Flatten Data

Code Node: Flatten Results

Step 6: Export to Destination

Option A: Google Sheets

Option B: Airtable

Option C: PostgreSQL / MySQL

Step 7: Schedule the Workflow

Schedule Trigger Configuration

Manual Trigger for Testing

Set Parameters Node

Complete Workflow

Importable Workflow JSON

Error Handling

Error Trigger Workflow

Retry Logic

Best Practices

Troubleshooting

Next Steps