REST API Documentation
Complete programmatic access to SiteGulp. Manage sites, start crawls, retrieve results, and configure webhooks via our REST API.
Quick Start
1Get Your API Key
Generate an API key from your Settings page. Keep it secure - it provides full access to your account.
2Make Your First Request
Include your API key in the Authorization header with every request.
curl -X GET "https://your-domain.com/api/v1/sites" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json"3Create a Site and Start Crawling
curl -X POST "https://your-domain.com/api/v1/sites" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "My Website",
"url": "https://example.com",
"maxPages": 100
}'curl -X POST "https://your-domain.com/api/v1/sites/SITE_ID/crawls" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "Initial Crawl",
"prompt": "Explore all main pages and capture screenshots"
}'Authentication
All API requests require authentication via Bearer token in the Authorization header:
Authorization: Bearer ak_your_api_key_hereRate Limiting
1,000 requests per hour per API key
Secure Keys
Keys are hashed with SHA-256 and never stored in plain text
Keep It Secret
Your API key provides full access to your account. Never expose it in client-side code.
API Endpoints
Sites
/api/v1/sites/api/v1/sites/api/v1/sites/:siteId/api/v1/sites/:siteId/api/v1/sites/:siteIdCrawls
/api/v1/sites/:siteId/crawls/api/v1/sites/:siteId/crawls/api/v1/crawls/:crawlId/api/v1/crawls/:crawlIdResults
/api/v1/crawls/:crawlId/results/api/v1/crawls/:crawlId/pages/api/v1/crawls/:crawlId/artifacts/api/v1/crawls/:crawlId/logsWebhooks
/api/v1/sites/:siteId/webhooks/api/v1/sites/:siteId/webhooks/api/v1/webhooks/:webhookId/api/v1/webhooks/:webhookId/api/v1/webhooks/:webhookId/api/v1/webhooks/:webhookId/testWebhooks
Webhook Events
Subscribe to events to receive real-time notifications when crawls change state.
crawl.startedFired when a crawl begins executioncrawl.completedFired when a crawl finishes successfullycrawl.failedFired when a crawl encounters an errorcrawl.cancelledFired when a crawl is cancelled*Subscribe to all eventsWebhook Payload
Example payload sent when a crawl completes
{
"event": "crawl.completed",
"timestamp": "2025-11-30T06:00:00.000Z",
"data": {
"crawlId": "abc123",
"projectId": "xyz789",
"projectName": "My Website",
"projectUrl": "https://example.com",
"crawlName": "Initial Crawl",
"status": "completed",
"pagesFound": 50,
"screenshotsTaken": 50,
"scriptsGenerated": 25,
"duration": 120000,
"completedAt": "2025-11-30T06:02:00.000Z"
}
}Webhook Signatures
Verify webhook authenticity using HMAC-SHA256 signatures
Every webhook includes a signature header for verification:
X-SiteGulp-Signature: t=1732946400,v1=abc123...Verify the signature by computing HMAC-SHA256 of the timestamp and payload using your webhook secret.
const crypto = require('crypto');
function verifyWebhookSignature(payload, signature, secret) {
const [timestampPart, signaturePart] = signature.split(',');
const timestamp = timestampPart.split('=')[1];
const receivedSignature = signaturePart.split('=')[1];
const signedPayload = `${timestamp}.${JSON.stringify(payload)}`;
const expectedSignature = crypto
.createHmac('sha256', secret)
.update(signedPayload)
.digest('hex');
return crypto.timingSafeEqual(
Buffer.from(receivedSignature),
Buffer.from(expectedSignature)
);
}Response Format
{
"success": true,
"data": {
"id": "site_abc123",
"name": "My Website",
"url": "https://example.com",
"createdAt": "2025-11-30T06:00:00.000Z"
},
"meta": {
"apiKey": "Production Key"
}
}Ready to Get Started?
Generate your API key and start automating your website crawls today.
Get Your API Key