> ## Documentation Index
> Fetch the complete documentation index at: https://docs.hitl.sh/llms.txt
> Use this file to discover all available pages before exploring further.

# Global Webhooks

> Configure global webhooks in HITL.sh to receive real-time notifications about all system events across your account

# Global Webhooks

Global webhooks in HITL.sh provide real-time notifications about all system events across your entire account. Unlike loop-specific webhooks, global webhooks capture events from all loops, making them ideal for centralized monitoring, analytics, and integration systems.

## What are Global Webhooks?

Global webhooks are HTTP endpoints that receive notifications about:

> [card] Global Webhook Events
>
> * **Account-wide Events**: User management, billing, and system updates
> * **All Loop Events**: Requests, responses, and loop status changes
> * **System Events**: Performance metrics, errors, and maintenance updates
> * **Integration Events**: API usage, rate limiting, and authentication events

## When to Use Global Webhooks

### Use Cases

> [card] Centralized Monitoring
>
> Monitor all HITL.sh activity from a single endpoint for comprehensive oversight.

> [card] Analytics Systems
>
> Collect data from all loops for unified analytics and reporting.

> [card] Audit Logging
>
> Maintain comprehensive audit trails for compliance and security purposes.

> [card] Integration Hubs
>
> Route events to multiple downstream systems and services.

### Comparison with Loop-Specific Webhooks

> [accordion] Global Webhooks
>
> * **Scope**: Account-wide, all loops
> * **Events**: All system events
> * **Use Case**: Centralized monitoring and analytics
> * **Complexity**: Higher, more event types to handle

> [accordion] Loop-Specific Webhooks
>
> * **Scope**: Single loop only
> * **Events**: Loop-specific events only
> * **Use Case**: Loop-specific integrations
> * **Complexity**: Lower, focused event handling

## Setting Up Global Webhooks

### 1. Create Webhook Endpoint

First, create an endpoint in your system to receive webhook notifications:

```python theme={null}
from flask import Flask, request, jsonify
import hmac
import hashlib

app = Flask(__name__)
WEBHOOK_SECRET = "your_webhook_secret"

@app.route('/webhooks/hitl/global', methods=['POST'])
def handle_global_webhook():
    # Verify webhook signature
    signature = request.headers.get('X-HITL-Signature')
    if not verify_signature(request.data, signature):
        return jsonify({"error": "Invalid signature"}), 401
    
    payload = request.json
    event_type = payload["event"]
    
    # Route to appropriate handler based on event type
    if event_type.startswith("request."):
        handle_request_event(payload)
    elif event_type.startswith("loop."):
        handle_loop_event(payload)
    elif event_type.startswith("user."):
        handle_user_event(payload)
    elif event_type.startswith("system."):
        handle_system_event(payload)
    
    return jsonify({"status": "success"}), 200

def verify_signature(payload, signature):
    expected_signature = hmac.new(
        WEBHOOK_SECRET.encode(),
        payload,
        hashlib.sha256
    ).hexdigest()
    
    return hmac.compare_digest(expected_signature, signature)
```

### 2. Configure Webhook in HITL.sh

Set up the global webhook through the API:

```python theme={null}
import requests

def create_global_webhook(webhook_url, events=None):
    if events is None:
        events = ["*"]  # Receive all events
    
    webhook_data = {
        "url": webhook_url,
        "events": events,
        "type": "global",  # Specify global webhook
        "secret": "your_webhook_secret",
        "enabled": True
    }
    
    response = requests.post(
        "https://api.hitl.sh/v1/api/webhooks",
        headers={
            "Authorization": f"Bearer {API_KEY}",
            "Content-Type": "application/json"
        },
        json=webhook_data
    )
    
    return response.json()
```

### 3. Test Webhook Configuration

Verify your webhook is working correctly:

```python theme={null}
def test_webhook(webhook_id):
    # Send a test event to verify webhook delivery
    response = requests.post(
        f"https://api.hitl.sh/v1/api/webhooks/{webhook_id}/test",
        headers={"Authorization": f"Bearer {API_KEY}"}
    )
    
    if response.status_code == 200:
        print("Webhook test successful")
        return True
    else:
        print(f"Webhook test failed: {response.text}")
        return False
```

## Event Types and Payloads

### Request Events

Events related to request lifecycle:

> [accordion] request.created
>
> ```json theme={null}
> {
>   "event": "request.created",
>   "timestamp": "2024-01-15T10:30:00Z",
>   "data": {
>     "request_id": "req_abc123",
>     "loop_id": "loop_xyz789",
>     "content_type": "text",
>     "priority": "normal",
>     "created_at": "2024-01-15T10:30:00Z"
>   }
> }
> ```

> [accordion] request.assigned
>
> ```json theme={null}
> {
>   "event": "request.assigned",
>   "timestamp": "2024-01-15T10:31:00Z",
>   "data": {
>     "request_id": "req_abc123",
>     "reviewer_id": "reviewer_def456",
>     "assigned_at": "2024-01-15T10:31:00Z"
>   }
> }
> ```

> [accordion] request.completed
>
> ```json theme={null}
> {
>   "event": "request.completed",
>   "timestamp": "2024-01-15T11:45:00Z",
>   "data": {
>     "request_id": "req_abc123",
>     "decision": "approved",
>     "reviewer_id": "reviewer_def456",
>     "completed_at": "2024-01-15T11:45:00Z"
>   }
> }
> ```

### Loop Events

Events related to loop management:

> [accordion] loop.created
>
> ```json theme={null}
> {
>   "event": "loop.created",
>   "timestamp": "2024-01-15T09:00:00Z",
>   "data": {
>     "loop_id": "loop_xyz789",
>     "name": "Content Moderation",
>     "created_by": "user_123",
>     "created_at": "2024-01-15T09:00:00Z"
>   }
> }
> ```

> [accordion] loop.updated
>
> ```json theme={null}
> {
>   "event": "loop.updated",
>   "timestamp": "2024-01-15T14:30:00Z",
>   "data": {
>     "loop_id": "loop_xyz789",
>     "changes": {
>       "required_approvals": {"from": 1, "to": 2},
>       "response_time": {"from": 3600, "to": 7200}
>     },
>     "updated_by": "user_123",
>     "updated_at": "2024-01-15T14:30:00Z"
>   }
> }
> ```

> [accordion] loop.archived
>
> ```json theme={null}
> {
>   "event": "loop.archived",
>   "timestamp": "2024-01-15T16:00:00Z",
>   "data": {
>     "loop_id": "loop_xyz789",
>     "archived_by": "user_123",
>     "archived_at": "2024-01-15T16:00:00Z"
>   }
> }
> ```

### User Events

Events related to user management:

> [accordion] user.invited
>
> ```json theme={null}
> {
>   "event": "user.invited",
>   "timestamp": "2024-01-15T10:00:00Z",
>   "data": {
>     "user_id": "user_456",
>     "email": "reviewer@example.com",
>     "invited_by": "user_123",
>     "invited_at": "2024-01-15T10:00:00Z"
>   }
> }
> ```

> [accordion] user.joined
>
> ```json theme={null}
> {
>   "event": "user.joined",
>   "timestamp": "2024-01-15T11:00:00Z",
>   "data": {
>     "user_id": "user_456",
>     "joined_at": "2024-01-15T11:00:00Z"
>   }
> }
> ```

### System Events

Events related to system operations:

> [accordion] system.performance
>
> ```json theme={null}
> {
>   "event": "system.performance",
>   "timestamp": "2024-01-15T12:00:00Z",
>   "data": {
>     "metrics": {
>       "active_requests": 150,
>       "average_response_time": 1800,
>       "reviewer_utilization": 0.75
>     },
>     "period": "1h"
>   }
> }
> ```

> [accordion] system.error
>
> ```json theme={null}
> {
>   "event": "system.error",
>   "timestamp": "2024-01-15T12:05:00Z",
>   "data": {
>     "error_code": "API_RATE_LIMIT",
>     "error_message": "API rate limit exceeded",
>     "affected_service": "content_moderation",
>     "severity": "warning"
>   }
> }
> ```

## Event Filtering

### Selective Event Subscription

Subscribe to specific event types to reduce noise:

```python theme={null}
def create_filtered_webhook(webhook_url):
    # Only receive request-related events
    events = [
        "request.created",
        "request.completed",
        "request.escalated"
    ]
    
    webhook_data = {
        "url": webhook_url,
        "events": events,
        "type": "global",
        "secret": "your_webhook_secret",
        "enabled": True
    }
    
    response = requests.post(
        "https://api.hitl.sh/v1/api/webhooks",
        headers={
            "Authorization": f"Bearer {API_KEY}",
            "Content-Type": "application/json"
        },
        json=webhook_data
    )
    
    return response.json()
```

### Event Pattern Matching

Use wildcards for flexible event filtering:

```python theme={null}
# Receive all request events
events = ["request.*"]

# Receive all events from specific loops
events = ["loop.xyz789.*"]

# Receive all events except system events
events = ["*", "!system.*"]
```

## Webhook Processing

### Event Routing

Route different event types to appropriate handlers:

```python theme={null}
def route_webhook_event(payload):
    event_type = payload["event"]
    
    # Route based on event category
    if event_type.startswith("request."):
        route_request_event(payload)
    elif event_type.startswith("loop."):
        route_loop_event(payload)
    elif event_type.startswith("user."):
        route_user_event(payload)
    elif event_type.startswith("system."):
        route_system_event(payload)
    else:
        handle_unknown_event(payload)

def route_request_event(payload):
    event_type = payload["event"]
    
    if event_type == "request.created":
        handle_new_request(payload["data"])
    elif event_type == "request.completed":
        handle_completed_request(payload["data"])
    elif event_type == "request.escalated":
        handle_escalated_request(payload["data"])
```

### Data Transformation

Transform webhook data for your systems:

```python theme={null}
def transform_request_data(webhook_data):
    """Transform webhook data to internal format"""
    return {
        "id": webhook_data["request_id"],
        "loop_id": webhook_data["loop_id"],
        "status": webhook_data.get("status", "unknown"),
        "created_at": webhook_data["created_at"],
        "priority": webhook_data.get("priority", "normal"),
        "content_type": webhook_data.get("content_type", "unknown")
    }

def transform_loop_data(webhook_data):
    """Transform loop event data"""
    return {
        "id": webhook_data["loop_id"],
        "name": webhook_data.get("name", ""),
        "status": webhook_data.get("status", "unknown"),
        "event_type": webhook_data["event"],
        "timestamp": webhook_data["timestamp"]
    }
```

## Error Handling and Retries

### Webhook Delivery Failures

Handle webhook delivery failures gracefully:

> [card] Retry Logic
>
> * Implement exponential backoff
> * Set maximum retry attempts
> * Log failed deliveries for investigation
> * Alert on persistent failures

> [card] Fallback Mechanisms
>
> * Store failed events for later processing
> * Implement alternative notification methods
> * Use polling as backup for critical events
> * Maintain event ordering when possible

### Dead Letter Queues

Implement dead letter queues for failed webhooks:

```python theme={null}
import redis
import json

class DeadLetterQueue:
    def __init__(self, redis_client):
        self.redis = redis_client
        self.queue_key = "hitl_webhook_dlq"
    
    def add_failed_event(self, event_data, error_reason):
        """Add failed webhook event to dead letter queue"""
        dlq_entry = {
            "event_data": event_data,
            "error_reason": error_reason,
            "failed_at": datetime.utcnow().isoformat(),
            "retry_count": 0
        }
        
        self.redis.lpush(self.queue_key, json.dumps(dlq_entry))
    
    def process_dlq(self):
        """Process events from dead letter queue"""
        while True:
            entry = self.redis.brpop(self.queue_key, timeout=1)
            if entry:
                dlq_entry = json.loads(entry[1])
                self.retry_event(dlq_entry)
    
    def retry_event(self, dlq_entry):
        """Retry processing a failed event"""
        try:
            # Attempt to process the event again
            process_webhook_event(dlq_entry["event_data"])
        except Exception as e:
            # Increment retry count
            dlq_entry["retry_count"] += 1
            
            if dlq_entry["retry_count"] < 3:
                # Re-add to queue for another attempt
                self.add_failed_event(
                    dlq_entry["event_data"],
                    str(e)
                )
            else:
                # Log final failure
                log_final_failure(dlq_entry)
```

## Security and Authentication

### Webhook Signature Verification

Always verify webhook signatures:

```python theme={null}
def verify_webhook_signature(payload, signature, secret):
    """Verify webhook signature using HMAC-SHA256"""
    expected_signature = hmac.new(
        secret.encode(),
        payload,
        hashlib.sha256
    ).hexdigest()
    
    return hmac.compare_digest(expected_signature, signature)

def secure_webhook_handler(request):
    """Secure webhook handler with signature verification"""
    # Get signature from headers
    signature = request.headers.get('X-HITL-Signature')
    if not signature:
        return jsonify({"error": "Missing signature"}), 401
    
    # Verify signature
    if not verify_webhook_signature(
        request.data, 
        signature, 
        WEBHOOK_SECRET
    ):
        return jsonify({"error": "Invalid signature"}), 401
    
    # Process webhook
    return process_webhook(request.json)
```

### Rate Limiting

Implement rate limiting to prevent abuse:

```python theme={null}
from flask_limiter import Limiter
from flask_limiter.util import get_remote_address

limiter = Limiter(
    app,
    key_func=get_remote_address,
    default_limits=["100 per minute"]
)

@app.route('/webhooks/hitl/global', methods=['POST'])
@limiter.limit("100 per minute")
def handle_global_webhook():
    # Webhook processing logic
    pass
```

## Monitoring and Analytics

### Webhook Performance Metrics

Track webhook delivery performance:

> [card] Delivery Metrics
>
> * Success rate percentage
> * Average delivery time
> * Failure rate by event type
> * Retry success rates

> [card] System Metrics
>
> * Webhook queue length
> * Processing latency
> * Error frequency
> * Resource utilization

### Alerting and Notifications

Set up alerts for webhook issues:

**1. Failure Alerts**

Alert when webhook delivery fails repeatedly.

**2. Performance Alerts**

Alert when webhook processing times exceed thresholds.

**3. Volume Alerts**

Alert when webhook volume is unusually high or low.

## Best Practices

### Webhook Design

> [card] Idempotency
>
> * Handle duplicate events gracefully
> * Use event IDs for deduplication
> * Implement idempotent processing logic

> [card] Event Ordering
>
> * Maintain event order when possible
> * Use timestamps for sequencing
> * Handle out-of-order events gracefully

### Performance Optimization

**1. Async Processing**

Process webhooks asynchronously to avoid blocking.

**2. Batch Processing**

Group related events for efficient processing.

**3. Connection Pooling**

Reuse HTTP connections for better performance.

## Troubleshooting

### Common Issues

> [accordion] Webhook Not Receiving Events
>
> * Verify webhook URL is accessible
> * Check webhook configuration in dashboard
> * Ensure webhook is enabled and active
> * Verify event subscriptions

> [accordion] Signature Verification Failures
>
> * Check webhook secret configuration
> * Verify signature header format
> * Ensure payload hasn't been modified
> * Test with webhook testing tool

> [accordion] High Failure Rates
>
> * Check endpoint availability and performance
> * Review error logs for specific failure reasons
> * Implement proper error handling
> * Consider implementing retry logic

## Next Steps

Ready to set up global webhooks for your HITL.sh account?

> [card] [Request Webhooks](https://docs.hitl.sh/api-reference/webhooks.md)
>
> Learn about per-request callback webhooks.

> [card] [API Reference](https://docs.hitl.sh/api-reference/introduction.md)
>
> Detailed API documentation for webhook management.

> [card] [Integration Examples](https://docs.hitl.sh/integrations/n8n.md)
>
> See how to use webhooks with popular integration platforms.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.