AI Voice Agent Builder's Kit

Step-by-Step Guide — Build a Working Voice Agent in Under 1 Hour

1. What You're Building

An AI phone agent that:

Answers your phone number 24/7 with natural voice
Searches your knowledge base (Obsidian, documents) during live calls
Triggers actions — outbound calls, SMS, reminders
Costs under $15/month (vs $99+ for prefab tools)

Prerequisites: You need a computer that can run Node.js and stay online. If it sleeps, the agent sleeps. For 24/7 uptime, run on a $5/mo VPS or Raspberry Pi.

2. Architecture Overview

YOU (calls Twilio number) ↓ Twilio (routes call) ↓ ElevenLabs Conversational AI │ ← Natural voice ↓ Webhook → Your Bridge Server │ ← POST with transcription ├─→ /search (query Obsidian) ├─→ /call (outbound via Twilio) └─→ /sms (send message) ↓ Return result → ElevenLabs → spoken response

Three services, one bridge, one goal.

3. Step 1: ElevenLabs Setup

Create an Agent

Go to elevenlabs.io → Sign up or log in
Navigate to Conversational AI → Agents
Click Create Agent → Name it (e.g., "Maya")
Pick a voice: Sarah (warm, natural) or Bella (bright, professional)
System prompt: "You are my personal assistant. Answer calls warmly, take messages, and help schedule appointments."

VIP Tip: Keep "first message" under 15 seconds. Example: "Hi, this is Maya. How can I help you today?"

Get Your Agent ID

Find your Agent ID (looks like agent_xxxxxxxxxxxxxxxxxxxxxxxx) in the dashboard URL or settings. Save it.

4. Step 2: Twilio Phone Number

Go to twilio.com → Sign up → Complete "quest" for $15 free credit
Buy a phone number (Phone Numbers → Manage → Buy a number)
Pick one with Voice + SMS capability (~$1.15/mo)

Configure the Webhook

You'll wire this after your bridge is running. For now, note down:

Account SID (starts with AC_)
Auth Token (secret — never share)
Phone Number

Wait on webhook config: Don't set the webhook URL yet. Your bridge needs to be live first.

5. Step 3: Build the Bridge Server

Create the Project

mkdir my-voice-agent cd my-voice-agent npm init -y npm install express axios dotenv

Server Code (server.js)

require('dotenv').config(); const express = require('express'); const axios = require('axios'); const app = express(); app.use(express.json()); // Health check app.get('/health', (req, res) => { res.json({ status: 'ok', agent: 'voice-agent' }); }); // Main webhook app.post('/maya/webhook', async (req, res) => { console.log('Webhook received:', req.body); const { tool_name, parameters } = req.body; try { if (tool_name === 'obsidian_search') { const results = await searchObsidian(parameters.query); return res.json({ success: true, result: results }); } if (tool_name === 'outbound_call') { await makeCall(parameters.phone, parameters.message); return res.json({ success: true, result: 'Calling now' }); } return res.json({ success: true, result: 'No action needed' }); } catch (err) { console.error(err); return res.status(500).json({ success: false, error: err.message }); } }); async function searchObsidian(query) { const resp = await axios.get('http://127.0.0.1:27123/search/', { params: { query }, headers: { 'Authorization': 'Bearer ' + process.env.OBSIDIAN_TOKEN } }); return resp.data.slice(0, 3).map(r => r.filename + ': ' + r.excerpt).join('\\n'); } async function makeCall(to, message) { const twilio = require('twilio')(process.env.TWILIO_SID, process.env.TWILIO_TOKEN); await twilio.calls.create({ to, from: process.env.TWILIO_PHONE, url: process.env.BRIDGE_URL + '/twiml?msg=' + encodeURIComponent(message) }); } app.listen(process.env.PORT || 8765, () => { console.log(`Bridge running on port ${process.env.PORT || 8765}`); });

Environment (.env)

PORT=8765 TWILIO_SID=AC_xxxxxxxx TWILIO_TOKEN=your_auth_token TWILIO_PHONE=+1XXXXXXXXXX BRIDGE_URL=https://your-tunnel-url.trycloudflare.com OBSIDIAN_TOKEN=your_obsidian_token

6. Step 4: Cloudflare Tunnel (Public URL)

Your computer is behind a router. You need a public URL for Twilio/ElevenLabs to reach it.

Install cloudflared:
# macOS brew install cloudflared # Ubuntu sudo apt install cloudflared
Run the tunnel:
cloudflared tunnel --url http://localhost:8765

Copy the URL (e.g., https://abc-123.trycloudflare.com)
Make it permanent: Sign up for Cloudflare Zero Trust (free), create a named tunnel with a fixed subdomain.

Heads up: Free tunnel URLs change on restart. For production, use a named tunnel or run as background service: nohup cloudflared tunnel --url http://localhost:8765 &

7. Step 5: Wire ElevenLabs → Your Bridge

ElevenLabs → Conversational AI → your agent → Settings
Paste webhook URL: https://your-tunnel-url.trycloudflare.com/maya/webhook
Add these External Tools:

Tool Name Method URL

obsidian_search POST /maya/webhook

outbound_call POST /maya/webhook

Tool Name	Method	URL
obsidian_search	POST	/maya/webhook
outbound_call	POST	/maya/webhook

CRITICAL — ElevenLabs Tool Schema Trick: Here's the exact JSON that passes validation:

{ "name": "obsidian_search", "description": "Search my notes", "parameters": { "type": "object", "properties": { "query": { "type": "string", "description": "What to search for" } }, "required": ["query"] } }

If the dashboard still rejects this, create via API instead.

8. Step 6: Wire Twilio → ElevenLabs

Twilio Console → Phone Numbers → Manage → Active numbers
Click your number → Voice & Fax
A call comes in → Webhook
https://api.elevenlabs.io/v1/convai/twilio/outbound?agent_id=YOUR_AGENT_ID // Method: POST

Now when someone calls your Twilio number, it routes to ElevenLabs, which talks to your bridge.

9. Step 7: Obsidian Knowledge Base

Obsidian → Settings → Community Plugins
Install "Local REST API" → Enable
Note port (default 27123) and generate API token
Add token to .env

Privacy: Local REST API only works on your machine. Notes never leave your computer unless the bridge explicitly sends data.

10. Step 8: Test the Full Stack

Start bridge: node server.js
Start tunnel: cloudflared tunnel --url http://localhost:8765
Call your Twilio number from your cell
Say: "Search my notes for projects I'm working on"
Check bridge console for the webhook hit

If nothing happens: Check Twilio logs. Common issues: wrong agent ID, cloudflared not running, missing /maya/webhook path.

11. Step 9: Outbound Calling

To make your agent call YOU back:

Add the outbound_call tool to your agent
Tell agent: "When asked, call me back with the outbound_call tool."

# Morning briefing — cron @ 7am daily 0 7 * * * curl -X POST http://localhost:8765/maya/webhook \ -H "Content-Type: application/json" \ -d '{"tool_name":"outbound_call","parameters":{"phone":"+1YOURNUMBER","message":"Good morning briefing"}}'

12. Monthly Cost Breakdown

Twilio number$1.15/mo

Incoming calls~$0.01/min

ElevenLabs voice (free tier)$0 (10k chars/mo)

Cloudflare tunnel$0

Your computer / VPS$0 (or $5/mo)

Total (light usage)~$3-8/mo

Total (heavy + VPS)~$15-25/mo

13. Troubleshooting

Problem	Cause	Fix
ElevenLabs rejects schema	API validation stricter than dashboard	Use exact JSON above. Create via API if dashboard fails.
Calls drop immediately	Webhook URL unreachable or wrong agent ID	Test bridge URL in browser. Check agent ID.
Agent doesn't use tools	Not enabled in prompt	Add to system prompt: "You have access to obsidian_search and outbound_call."
Obsidian returns nothing	Local REST API not enabled	Test: `curl http://127.0.0.1:27123/search/?query=test`
Cloudflared URL changes	Temporary tunnel	Create named tunnel in Zero Trust for fixed subdomain.
Twilio "application error"	Webhook returns 404/500	Check Twilio logs. Ensure /maya/webhook exists and returns JSON.
Voice robotic	Wrong model	Use "Turbo v2.5" for lowest latency. Pick Sarah/Bella.

14. Next Steps

Add more tools: Calendar booking, weather, expense logging
Improve prompt: Train with example conversations
Add fallback: Send transcript to your phone if agent can't answer
Measure ROI: Log calls → how many led to booked appointments

Questions? Email hello@jedaiflow.com — John responds personally.