Responses

Responses are the structured decisions and feedback that human reviewers provide when completing HITL.sh requests. The response system supports six different types, each designed for specific use cases and providing different levels of structure and validation.

What is a Response?

A response in HITL.sh represents the human reviewer’s decision and contains:

Response Data: The actual decision or feedback in a structured format
Response Type: Which of the six supported response types was used
Reviewer Information: Who provided the response and when
Validation Status: Whether the response meets the configured requirements
Processing Metadata: Response time, platform used, and other tracking data

Responses are collected via the HITL mobile app and made available to your applications via API polling or webhooks. Each response type has its own data structure and validation rules.

Response Types Overview

HITL.sh supports six distinct response types, each optimized for different decision-making scenarios:

Text Response

Free-form feedback with character limits

Detailed explanations and qualitative feedback
Configurable length requirements
Perfect for open-ended reviews

Single Select

One choice from predefined options

Clear decision workflows
Visual options with colors and descriptions
Ideal for approve/reject scenarios

Multi Select

Multiple choices from option lists

Issue identification and categorization
Configurable selection limits
Great for checklists and audits

Rating Response

Numeric ratings on custom scales

Quality assessments and scoring
Custom labels and step increments
Perfect for performance evaluation

Number Response

Numeric input with validation

Pricing, quantities, measurements
Formatting with prefixes and suffixes
Range validation and decimal control

Response Configuration

When creating requests, you specify how reviewers should respond by setting the response_type and response_config:

Text Responses

For detailed feedback and explanations:

"response_type": "text",
"response_config": {
    "placeholder": "Explain your reasoning...",
    "min_length": 20,
    "max_length": 500,
    "required": True
}

Response Format:

{
  "response_data": "The content violates guideline 3.2 regarding promotional language. The phrase 'everyone should try it' is too promotional. Suggest rephrasing to 'this worked well for me' instead."
}

Single Select Responses

For structured decisions with predefined options:

"response_type": "single_select",
"response_config": {
    "options": [
        {
            "value": "approve",
            "label": "✅ Approve Content",
            "description": "Content meets all guidelines",
            "color": "#22c55e"
        },
        {
            "value": "reject",
            "label": "❌ Reject Content", 
            "description": "Content violates policies",
            "color": "#ef4444"
        }
    ],
    "required": True
}

Response Format:

{
  "response_data": {
    "selected_value": "reject",
    "selected_label": "❌ Reject Content"
  }
}

Multi Select Responses

For identifying multiple issues or aspects:

"response_type": "multi_select",
"response_config": {
    "options": [
        {"value": "spam", "label": "🚫 Spam Content"},
        {"value": "inappropriate", "label": "⚠️ Inappropriate Language"},
        {"value": "misleading", "label": "❌ Misleading Claims"}
    ],
    "min_selections": 1,
    "max_selections": 3,
    "required": True
}

Response Format:

{
  "response_data": {
    "selected_values": ["spam", "misleading"],
    "selected_labels": ["🚫 Spam Content", "❌ Misleading Claims"]
  }
}

Rating Responses

For quality assessments and scoring:

"response_type": "rating",
"response_config": {
    "scale_min": 1,
    "scale_max": 10,
    "scale_step": 0.5,
    "labels": {
        "1": "Poor Quality",
        "5": "Average",
        "10": "Excellent"
    },
    "required": True
}

Response Format:

{
  "response_data": {
    "rating": 7.5,
    "rating_label": "Good Quality"
  }
}

Number Responses

For quantitative input with validation:

"response_type": "number",
"response_config": {
    "min_value": 0,
    "max_value": 1000,
    "decimal_places": 2,
    "prefix": "$",
    "suffix": " USD",
    "required": True
}

Response Format:

{
  "response_data": {
    "number": 299.99,
    "formatted_value": "$299.99 USD"
  }
}

Response Processing

Accessing Response Data

Once a reviewer completes a request, you can access the response data via API:

import requests

# Get completed request with response
response = requests.get(
    f"https://api.hitl.sh/v1/api/requests/{request_id}",
    headers={"Authorization": f"Bearer {api_key}"}
)

request_data = response.json()["data"]["request"]

if request_data["status"] == "completed":
    # Extract response information
    response_data = request_data["response_data"]
    response_type = request_data["response_type"]
    reviewer = request_data["response_by_user"]
    response_time = request_data["response_time_seconds"]
    
    print(f"Response Type: {response_type}")
    print(f"Response Data: {response_data}")
    print(f"Completed by: {reviewer['name']} in {response_time:.1f}s")
    
    # Process based on response type
    process_response(response_type, response_data)

Response Processing Patterns

Handle different response types appropriately in your application:

def process_response(response_type, response_data):
    """Process responses based on their type"""
    
    if response_type == "text":
        # Process free-form feedback
        feedback = response_data
        save_feedback_for_review(feedback)
        
    elif response_type == "single_select":
        # Handle structured decision
        decision = response_data["selected_value"]
        if decision == "approve":
            approve_content()
        elif decision == "reject":
            reject_content()
        elif decision == "escalate":
            escalate_to_senior_team()
            
    elif response_type == "multi_select":
        # Handle multiple issues identified
        issues = response_data["selected_values"]
        for issue in issues:
            handle_content_issue(issue)
            
    elif response_type == "rating":
        # Handle quality score
        score = response_data["rating"]
        if score >= 8:
            mark_as_high_quality()
        elif score <= 3:
            flag_for_improvement()
        else:
            mark_as_acceptable()
            
    elif response_type == "number":
        # Handle numeric input
        value = response_data["number"]
        update_pricing_model(value)
        

Response Validation

HITL.sh automatically validates responses against the configured rules:

Validation Rules by Type

Text Validation

Response must be a non-empty string (if required)
Character count must be within min_length and max_length bounds
Cannot contain only whitespace if required

Select Validation

Selected values must exist in the configured options array
Single select allows exactly one selection
Multi select respects min_selections and max_selections limits
No duplicate selections allowed in multi select

Rating Validation

Value must be within scale_min and scale_max bounds
Must align with scale_step increments (e.g., only .0 and .5 for step=0.5)
Cannot be null if required

Number Validation

Value must be within min_value and max_value bounds
Decimal places cannot exceed configured limit
Negative numbers only allowed if allow_negative is true

Handling Validation Errors

The mobile app prevents invalid responses, but you should handle edge cases:

def validate_response_before_processing(request, response_data):
    """Additional validation before processing responses"""
    
    response_type = request["response_type"]
    response_config = request["response_config"]
    
    try:
        if response_type == "rating":
            rating = response_data["rating"]
            min_val = response_config["scale_min"]
            max_val = response_config["scale_max"]
            
            if not (min_val <= rating <= max_val):
                log_validation_error(f"Rating {rating} outside range [{min_val}, {max_val}]")
                return False
                
        elif response_type == "single_select":
            selected = response_data["selected_value"]
            valid_options = [opt["value"] for opt in response_config["options"]]
            
            if selected not in valid_options:
                log_validation_error(f"Invalid selection: {selected}")
                return False
                
        return True
        
    except KeyError as e:
        log_validation_error(f"Missing required field: {e}")
        return False

Response Analytics

Tracking Response Patterns

Monitor response patterns to improve your workflows:

def analyze_response_patterns(loop_id, days=30):
    """Analyze response patterns for a loop"""
    
    # Get recent requests for this loop
    response = requests.get(
        f"https://api.hitl.sh/v1/api/loops/{loop_id}/requests",
        headers={"Authorization": f"Bearer {api_key}"}
    )
    
    requests_data = response.json()["data"]["requests"]
    completed_requests = [r for r in requests_data if r["status"] == "completed"]
    
    # Analyze by response type
    response_type_counts = {}
    avg_response_times = {}
    
    for request in completed_requests:
        resp_type = request["response_type"]
        response_type_counts[resp_type] = response_type_counts.get(resp_type, 0) + 1
        
        if request.get("response_time_seconds"):
            if resp_type not in avg_response_times:
                avg_response_times[resp_type] = []
            avg_response_times[resp_type].append(request["response_time_seconds"])
    
    # Calculate averages
    for resp_type, times in avg_response_times.items():
        avg_time = sum(times) / len(times)
        print(f"{resp_type}: {response_type_counts[resp_type]} responses, avg {avg_time:.1f}s")

Response Quality Metrics

Track response quality and consistency:

def analyze_response_quality(requests_data):
    """Analyze quality metrics for responses"""
    
    # Group by response type
    by_type = {}
    for request in requests_data:
        if request["status"] != "completed":
            continue
            
        resp_type = request["response_type"]
        if resp_type not in by_type:
            by_type[resp_type] = []
        by_type[resp_type].append(request)
    
    # Analyze each type
    for resp_type, requests in by_type.items():
        print(f"\n{resp_type.upper()} Responses:")
        print(f"  Total: {len(requests)}")
        
        # Response time analysis
        times = [r["response_time_seconds"] for r in requests if r.get("response_time_seconds")]
        if times:
            print(f"  Avg response time: {sum(times)/len(times):.1f}s")
            print(f"  Response time range: {min(times):.1f}s - {max(times):.1f}s")
        
        # Type-specific analysis
        if resp_type == "single_select":
            analyze_single_select_distribution(requests)
        elif resp_type == "rating":
            analyze_rating_distribution(requests)

Best Practices

Choosing Response Types

Match Complexity to Need

Use simple response types (single select) for straightforward decisions. Reserve complex types (multi select, text) for situations requiring nuanced evaluation.

Consider Reviewer Experience

Remember that reviewers interact with responses on mobile devices. Keep options concise and touch-friendly.

Balance Structure and Flexibility

Structured responses (select, rating, number) are easier to process, but text responses provide richer feedback when needed.

Plan for Scale

Consider how you’ll process and analyze responses when choosing types. Structured responses are easier to aggregate and analyze.

Response Design Tips

Clear Options

For select responses, use descriptive labels and include helpful descriptions. Consider adding colors for visual clarity.

Reasonable Limits

Set appropriate character limits for text, selection limits for multi select, and ranges for numeric inputs.

Meaningful Defaults

Always provide sensible default responses that represent safe outcomes when requests timeout.

Consistent Language

Use consistent terminology across response options to avoid confusion and improve decision quality.

Next Steps

Response Types Guide

Detailed documentation of all six response types with examples

Request Creation

Learn how to configure responses when creating requests

Mobile App Guide

See how reviewers interact with different response types

Best Practices

Advanced strategies for optimizing response quality and processing

Getting Started

Integration Guides

Core Concepts

Mobile App

Requests

Responses

Integrations

Webhooks

​Responses

​What is a Response?

​Response Types Overview

Text Response

Single Select

Multi Select

Rating Response

Number Response

​Response Configuration

​Text Responses

​Single Select Responses

​Multi Select Responses

​Rating Responses

​Number Responses

​Response Processing

​Accessing Response Data

​Response Processing Patterns

​Response Validation

​Validation Rules by Type

​Handling Validation Errors

​Response Analytics

​Tracking Response Patterns

​Response Quality Metrics

​Best Practices

​Choosing Response Types

​Response Design Tips

Clear Options

Reasonable Limits

Meaningful Defaults

Consistent Language

​Next Steps

Response Types Guide

Request Creation

Mobile App Guide

Best Practices

Responses

What is a Response?

Response Types Overview

Response Configuration

Text Responses

Single Select Responses

Multi Select Responses

Rating Responses

Number Responses

Response Processing

Accessing Response Data

Response Processing Patterns

Response Validation

Validation Rules by Type

Handling Validation Errors

Response Analytics

Tracking Response Patterns

Response Quality Metrics

Best Practices

Choosing Response Types

Response Design Tips

Next Steps