agent: session-23 — Module A1 end-to-end tested and documented
- Fixed lead data loss bug in Enrichment Mapper (fullResponse overwrites input data — now references upstream node) - Full README with architecture diagram, input/output schemas, swap points, test results, pricing tiers - Live test results: validation ✓, mock enrichment ✓, Apollo path ✓ - Apollo free plan blocks people/match — documented as known limitation Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
jake
@@ -1,5 +1,186 @@
|
|||||||
# Lead Enrich Pipeline — Module A1
|
# Lead Enrich Pipeline — Module A1
|
||||||
|
|
||||||
> Full documentation coming after workflow build is complete. This placeholder exists so the directory structure is committed.
|
Webhook-triggered lead enrichment pipeline. Accepts a lead via POST, enriches it through Apollo (or mock data in test mode), grades the profile, and passes the result to a configurable downstream actions workflow.
|
||||||
|
|
||||||
See spec: `docs/superpowers/specs/2026-03-27-module-a1-lead-enrich-pipeline-design.md` in CORBEL-Furnished-House.
|
## Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
POST /webhook/lead-enrich
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Input Mapper ─── normalise fields
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Clean & Validate ─── trim, lowercase, email regex
|
||||||
|
│
|
||||||
|
├── INVALID ──► Dead Letter Response (422)
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Test Mode Check
|
||||||
|
│
|
||||||
|
├── TEST_MODE=true ──► Mock Enrichment (static data)
|
||||||
|
│
|
||||||
|
├── TEST_MODE=false ──► Apollo HTTP Request
|
||||||
|
│ │
|
||||||
|
│ ▼
|
||||||
|
│ Enrichment Mapper & Grader
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Success Response (200) + Call Actions Workflow
|
||||||
|
```
|
||||||
|
|
||||||
|
**Two workflows:**
|
||||||
|
- `lead-enrich-core.json` — the pipeline above (this is the product)
|
||||||
|
- `lead-actions-example.json` — reference recipe showing downstream routing by grade (CRM sync, Slack notification, welcome email, audit log)
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
1. Import `lead-enrich-core.json` into n8n (Settings → Import Workflow or via API)
|
||||||
|
2. Set `TEST_MODE=true` in your n8n environment variables
|
||||||
|
3. Activate the workflow
|
||||||
|
4. Test:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -X POST https://your-n8n-instance/webhook/lead-enrich \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"email": "test@example.com", "first_name": "Test", "last_name": "User", "company_name": "Acme", "source": "manual"}'
|
||||||
|
```
|
||||||
|
|
||||||
|
You should get a 200 response with mock enrichment data and a "Full Profile" grade.
|
||||||
|
|
||||||
|
## Input Schema
|
||||||
|
|
||||||
|
POST JSON body:
|
||||||
|
|
||||||
|
| Field | Type | Required | Notes |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `email` | string | Yes | Validated with regex. Lowercased automatically. |
|
||||||
|
| `first_name` | string | No | Passed to Apollo for matching accuracy |
|
||||||
|
| `last_name` | string | No | Passed to Apollo for matching accuracy |
|
||||||
|
| `company_name` | string | No | Passed to Apollo as `organization_name` |
|
||||||
|
| `source` | string | No | Defaults to "webhook". Use for tracking origin. |
|
||||||
|
|
||||||
|
Accepts fields at root level (`$json.email`) or nested in body (`$json.body.email`) — the Input Mapper handles both.
|
||||||
|
|
||||||
|
## Output Schema
|
||||||
|
|
||||||
|
200 response:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"lead": {
|
||||||
|
"email": "...",
|
||||||
|
"first_name": "...",
|
||||||
|
"last_name": "...",
|
||||||
|
"company_name": "...",
|
||||||
|
"source": "...",
|
||||||
|
"submitted_at": "ISO timestamp",
|
||||||
|
"raw_payload": { ... }
|
||||||
|
},
|
||||||
|
"enrichment": {
|
||||||
|
"provider": "apollo|mock",
|
||||||
|
"person": {
|
||||||
|
"title": "...",
|
||||||
|
"linkedin_url": "...",
|
||||||
|
"city": "...",
|
||||||
|
"state": "...",
|
||||||
|
"country": "...",
|
||||||
|
"seniority": "..."
|
||||||
|
},
|
||||||
|
"organisation": {
|
||||||
|
"name": "...",
|
||||||
|
"domain": "...",
|
||||||
|
"industry": "...",
|
||||||
|
"estimated_employees": 250,
|
||||||
|
"annual_revenue": 5000000,
|
||||||
|
"linkedin_url": "...",
|
||||||
|
"website_url": "..."
|
||||||
|
},
|
||||||
|
"enrichment_error": false,
|
||||||
|
"credits_consumed": 1,
|
||||||
|
"rate_limit_remaining": 95,
|
||||||
|
"credits_low": false
|
||||||
|
},
|
||||||
|
"grading": {
|
||||||
|
"profile_grade": "Full Profile|Partial Profile|Limited",
|
||||||
|
"fields_populated": 13,
|
||||||
|
"graded_at": "ISO timestamp"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
422 response (validation failure):
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"error": true,
|
||||||
|
"error_reason": "Invalid email format: ...",
|
||||||
|
"lead": { ... },
|
||||||
|
"timestamp": "ISO timestamp"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Profile Grading
|
||||||
|
|
||||||
|
| Grade | Criteria |
|
||||||
|
|---|---|
|
||||||
|
| **Full Profile** | Company name + job title + employee count all present |
|
||||||
|
| **Partial Profile** | Company name present but missing title or employee count |
|
||||||
|
| **Limited** | No company data, enrichment error, or no Apollo match |
|
||||||
|
|
||||||
|
## Environment Variables
|
||||||
|
|
||||||
|
Set these on your n8n instance (Docker env vars, not n8n Variables which require a paid plan):
|
||||||
|
|
||||||
|
| Variable | Required | Description |
|
||||||
|
|---|---|---|
|
||||||
|
| `TEST_MODE` | Yes | `"true"` for mock data, `"false"` for live Apollo |
|
||||||
|
| `APOLLO_API_KEY` | For live mode | Apollo People Enrichment API key. **Requires a paid Apollo plan** — the free tier does not include the `people/match` endpoint. |
|
||||||
|
|
||||||
|
See `.env.example` for the full list including downstream action credentials (HubSpot, Slack, SMTP).
|
||||||
|
|
||||||
|
## Apollo API Notes
|
||||||
|
|
||||||
|
- **Endpoint:** `POST https://api.apollo.io/api/v1/people/match`
|
||||||
|
- **Free tier limitation:** The `people/match` endpoint is **not available on the free plan**. You need at least a Basic plan ($49/month) or equivalent.
|
||||||
|
- **Rate limits:** 100 requests/minute on paid plans. The workflow reads `x-minute-requests-left` from response headers and sets `credits_low: true` when below 50.
|
||||||
|
- **No-match behaviour:** Apollo returns a successful response with `person: null`. No credits are consumed. The grader assigns "Limited" grade.
|
||||||
|
- **Retry logic:** The HTTP Request node retries up to 3 times with 5-second backoff on failure.
|
||||||
|
|
||||||
|
## Swap Points (for Client Deployment)
|
||||||
|
|
||||||
|
These are the nodes you'd modify when deploying for a client:
|
||||||
|
|
||||||
|
1. **Webhook path** — change `lead-enrich` to a client-specific path
|
||||||
|
2. **Apollo API key** — client provides their own key via env var
|
||||||
|
3. **Grading thresholds** — adjust in the Enrichment Mapper & Grader Code node
|
||||||
|
4. **Actions workflow** — replace `lead-actions-example` with client-specific routing
|
||||||
|
|
||||||
|
## Test Results (2026/03/28)
|
||||||
|
|
||||||
|
| Test | Result | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| Invalid email (bad format) | PASS — 422, correct error message | |
|
||||||
|
| Missing email | PASS — 422, "Email is missing or empty" | |
|
||||||
|
| Test mode (mock enrichment) | PASS — 200, Full Profile, 13 fields, provider=mock | |
|
||||||
|
| Live Apollo (valid email) | PASS — 200, Limited grade, provider=apollo | Apollo free plan blocks people/match endpoint. Workflow handles gracefully. |
|
||||||
|
| Lead data in response | PASS — lead object present in all paths | Bug fixed: upstream node reference for Apollo path |
|
||||||
|
|
||||||
|
## Files
|
||||||
|
|
||||||
|
```
|
||||||
|
A1-lead-enrich-pipeline/
|
||||||
|
├── README.md ← this file
|
||||||
|
├── .env.example ← required credentials
|
||||||
|
└── workflows/
|
||||||
|
├── lead-enrich-core.json ← the product (Workflow 1)
|
||||||
|
└── lead-actions-example.json ← reference recipe (Workflow 2)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Pricing (Template Marketplace)
|
||||||
|
|
||||||
|
| Tier | Price | What's included |
|
||||||
|
|---|---|---|
|
||||||
|
| Core template | £39–49 | Raw workflows + this documentation. Buyer wires own adapters. |
|
||||||
|
| Ready-made bundle | £79–99 | Pre-configured for Tally → Apollo → HubSpot + Sheets + Slack. |
|
||||||
|
| Done-for-you | £350–600 | Jake deploys, configures, tests, hands over. 7-day support window. |
|
||||||
|
|||||||
@@ -421,7 +421,7 @@
|
|||||||
],
|
],
|
||||||
"parameters": {
|
"parameters": {
|
||||||
"mode": "runOnceForAllItems",
|
"mode": "runOnceForAllItems",
|
||||||
"jsCode": "const items = $input.all();\nconst results = [];\n\nfor (const item of items) {\n const response = item.json;\n const lead = response.lead || item.json.lead;\n\n // Initialise enrichment structure\n const enrichment = {\n provider: 'apollo',\n person: {\n title: null,\n linkedin_url: null,\n city: null,\n state: null,\n country: null,\n seniority: null\n },\n organisation: {\n name: null,\n domain: null,\n industry: null,\n estimated_employees: null,\n annual_revenue: null,\n linkedin_url: null,\n website_url: null\n },\n enrichment_error: false,\n credits_consumed: 0,\n rate_limit_remaining: null,\n credits_low: false\n };\n\n // Check for HTTP errors via the response metadata\n const statusCode = response.statusCode || response.$response?.statusCode || 200;\n\n // Extract rate limit headers if available\n const headers = response.headers || response.$response?.headers || {};\n const rateRemaining = headers['x-minute-requests-left'];\n if (rateRemaining !== undefined) {\n enrichment.rate_limit_remaining = parseInt(rateRemaining, 10);\n enrichment.credits_low = enrichment.rate_limit_remaining < 50;\n }\n\n // Handle response based on status\n if (statusCode === 429) {\n enrichment.enrichment_error = true;\n } else if (statusCode >= 400) {\n enrichment.enrichment_error = true;\n } else {\n // Look for person data in the response body\n const body = response.body || response;\n const person = body.person || null;\n\n if (person && person !== null) {\n const org = person.organization || {};\n\n enrichment.person.title = person.title || null;\n enrichment.person.linkedin_url = person.linkedin_url || null;\n enrichment.person.city = person.city || null;\n enrichment.person.state = person.state || null;\n enrichment.person.country = person.country || null;\n enrichment.person.seniority = person.seniority || null;\n\n enrichment.organisation.name = org.name || null;\n enrichment.organisation.domain = org.primary_domain || null;\n enrichment.organisation.industry = org.industry || null;\n enrichment.organisation.estimated_employees = org.estimated_num_employees || null;\n enrichment.organisation.annual_revenue = org.annual_revenue || null;\n enrichment.organisation.linkedin_url = org.linkedin_url || null;\n enrichment.organisation.website_url = org.website_url || null;\n\n enrichment.credits_consumed = body.credits_consumed || 1;\n }\n // else: person is null \u2014 no match, all fields stay null, no error, no credits\n }\n\n // Grade the profile\n const hasCompany = enrichment.organisation.name !== null;\n const hasTitle = enrichment.person.title !== null;\n const hasEmployees = enrichment.organisation.estimated_employees !== null;\n\n let profileGrade;\n if (enrichment.enrichment_error || !hasCompany) {\n profileGrade = 'Limited';\n } else if (hasCompany && hasTitle && hasEmployees) {\n profileGrade = 'Full Profile';\n } else {\n profileGrade = 'Partial Profile';\n }\n\n // Count populated fields\n const personFields = Object.values(enrichment.person).filter(v => v !== null).length;\n const orgFields = Object.values(enrichment.organisation).filter(v => v !== null).length;\n\n const grading = {\n profile_grade: profileGrade,\n fields_populated: personFields + orgFields,\n graded_at: new Date().toISOString()\n };\n\n results.push({\n json: {\n lead,\n enrichment,\n grading\n }\n });\n}\n\nreturn results;"
|
"jsCode": "const items = $input.all();\nconst results = [];\n\nfor (const item of items) {\n const response = item.json;\n // Lead data lost when HTTP Request uses fullResponse — reference upstream\n let lead;\n try {\n lead = $('Test Mode Check').first().json.lead;\n } catch(e) {\n lead = response.lead || {};\n }\n\n // Initialise enrichment structure\n const enrichment = {\n provider: 'apollo',\n person: {\n title: null,\n linkedin_url: null,\n city: null,\n state: null,\n country: null,\n seniority: null\n },\n organisation: {\n name: null,\n domain: null,\n industry: null,\n estimated_employees: null,\n annual_revenue: null,\n linkedin_url: null,\n website_url: null\n },\n enrichment_error: false,\n credits_consumed: 0,\n rate_limit_remaining: null,\n credits_low: false\n };\n\n // Check for HTTP errors via the response metadata\n const statusCode = response.statusCode || response.$response?.statusCode || 200;\n\n // Extract rate limit headers if available\n const headers = response.headers || response.$response?.headers || {};\n const rateRemaining = headers['x-minute-requests-left'];\n if (rateRemaining !== undefined) {\n enrichment.rate_limit_remaining = parseInt(rateRemaining, 10);\n enrichment.credits_low = enrichment.rate_limit_remaining < 50;\n }\n\n // Handle response based on status\n if (statusCode === 429) {\n enrichment.enrichment_error = true;\n } else if (statusCode >= 400) {\n enrichment.enrichment_error = true;\n } else {\n // Look for person data in the response body\n const body = response.body || response;\n const person = body.person || null;\n\n if (person && person !== null) {\n const org = person.organization || {};\n\n enrichment.person.title = person.title || null;\n enrichment.person.linkedin_url = person.linkedin_url || null;\n enrichment.person.city = person.city || null;\n enrichment.person.state = person.state || null;\n enrichment.person.country = person.country || null;\n enrichment.person.seniority = person.seniority || null;\n\n enrichment.organisation.name = org.name || null;\n enrichment.organisation.domain = org.primary_domain || null;\n enrichment.organisation.industry = org.industry || null;\n enrichment.organisation.estimated_employees = org.estimated_num_employees || null;\n enrichment.organisation.annual_revenue = org.annual_revenue || null;\n enrichment.organisation.linkedin_url = org.linkedin_url || null;\n enrichment.organisation.website_url = org.website_url || null;\n\n enrichment.credits_consumed = body.credits_consumed || 1;\n }\n // else: person is null \u2014 no match, all fields stay null, no error, no credits\n }\n\n // Grade the profile\n const hasCompany = enrichment.organisation.name !== null;\n const hasTitle = enrichment.person.title !== null;\n const hasEmployees = enrichment.organisation.estimated_employees !== null;\n\n let profileGrade;\n if (enrichment.enrichment_error || !hasCompany) {\n profileGrade = 'Limited';\n } else if (hasCompany && hasTitle && hasEmployees) {\n profileGrade = 'Full Profile';\n } else {\n profileGrade = 'Partial Profile';\n }\n\n // Count populated fields\n const personFields = Object.values(enrichment.person).filter(v => v !== null).length;\n const orgFields = Object.values(enrichment.organisation).filter(v => v !== null).length;\n\n const grading = {\n profile_grade: profileGrade,\n fields_populated: personFields + orgFields,\n graded_at: new Date().toISOString()\n };\n\n results.push({\n json: {\n lead,\n enrichment,\n grading\n }\n });\n}\n\nreturn results;"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
|
|||||||
Reference in New Issue
Block a user