Guide - Async Email Validation During Signup with checkboxHQ
Developer Guide: Async Email Validation During Signup with checkboxHQ
Overview
This guide shows you how to implement asynchronous email validation in your signup flow using checkboxHQ’s API. Async validation lets users sign up immediately while email verification happens in the background, providing a better user experience without compromising data quality.
Why Async Validation?
- Faster signups: Users don’t wait for API responses
- Better reliability: Network delays don’t block registration
- Cost-effective: Smart validation strategy (1-3 credits per email)
- Production-ready: Handle rate limits and failures gracefully
checkboxHQ API Overview
checkboxHQ provides three separate endpoints for email validation:
1. Disposable Email Check (/api/v1/verify_disposable/)
Cost: 1 credit per request
Purpose: Detect temporary/throwaway email addresses
// RequestPOST https://api.checkboxhq.com/api/v1/verify_disposable/Headers: x-api-key: YOUR_API_KEY{}
// Response{ "is_disposable": true, "is_valid_format": true, "domain": "guerrillamail.com", "provider": "GuerrillaMail"}2. DNS Verification (/api/v1/verify_dns/)
Cost: 2 credits per request
Purpose: Verify domain has email server (MX records) and website (A records)
// RequestPOST https://api.checkboxhq.com/api/v1/verify_dns/Headers: x-api-key: YOUR_API_KEY{}
// Response{ "has_mx_record": true, "mx_records": ["mail.example.com"], "has_a_record": true, "a_records": ["192.0.2.1"], "provider": "Example Email"}Understanding the response:
has_mx_record: false→ Email server not configured, email NOT deliverablehas_a_record: false→ No website, but email might still work- Both false → Likely junk/non-existent domain
3. Public Provider Check (/api/v1/verify_public_provider/)
Cost: 1 credit per request
Purpose: Identify free email providers (Gmail, Yahoo, etc.)
// RequestPOST https://api.checkboxhq.com/api/v1/verify_public_provider/Headers: x-api-key: YOUR_API_KEY{}
// Response{ "is_public_provider": true, "provider": "Gmail"}Rate Limits
- 100 requests per minute per API key
- HTTP 429 returned when exceeded
- Implement request throttling in production
Recommended Validation Strategy
For signup flows, use this cost-effective approach:
-
Always check disposable (1 credit)
- Reject if
is_disposable: true
- Reject if
-
Then verify DNS (2 credits)
- Reject if
has_mx_record: false - Warn if
has_a_record: false(optional)
- Reject if
-
Check public provider only if needed (1 credit)
- Only if you want to block free emails
- Skip for most B2C SaaS applications
Total cost per signup: 3 credits (or 1-2 if you skip DNS for known good domains)
Architecture Overview
User submits signup form ↓Create user account (status: pending) ↓Queue background validation job ↓Return success immediately ↓Background worker validates with checkboxHQ ↓Update user validation status ↓Take action if invalidImplementation
Step 1: Database Schema
CREATE TABLE users ( id SERIAL PRIMARY KEY, email VARCHAR(255) UNIQUE NOT NULL, password_hash VARCHAR(255) NOT NULL, name VARCHAR(255),
-- Validation tracking email_validation_status VARCHAR(50) DEFAULT 'pending', -- pending, validating, valid, invalid_disposable, invalid_no_mx, failed
-- checkboxHQ results is_disposable BOOLEAN, disposable_provider VARCHAR(255),
has_mx_record BOOLEAN, mx_records JSONB,
has_a_record BOOLEAN, a_records JSONB,
is_public_provider BOOLEAN, email_provider VARCHAR(255),
-- Metadata validation_error TEXT, account_status VARCHAR(50) DEFAULT 'active', created_at TIMESTAMP DEFAULT NOW(), validated_at TIMESTAMP);
-- Index for querying by validation statusCREATE INDEX idx_users_validation_statusON users(email_validation_status);
-- Index for finding invalid usersCREATE INDEX idx_users_disposableON users(is_disposable) WHERE is_disposable = true;Why this structure?
- Track validation progress through
email_validation_status - Store individual fields for fast queries (no JSON parsing)
- Keep full records in JSONB for debugging
- Separate validation status from account status
Step 2: Environment Configuration
from pydantic_settings import BaseSettings
class Settings(BaseSettings): # checkboxHQ API CHECKBOXHQ_API_KEY: str CHECKBOXHQ_BASE_URL: str = "https://api.checkboxhq.com"
# Database DATABASE_URL: str
# Redis for Celery (job queue) REDIS_URL: str = "redis://localhost:6379/0"
# Validation rules REJECT_DISPOSABLE: bool = True REJECT_INVALID_MX: bool = True REJECT_PUBLIC_PROVIDERS: bool = False # Usually False for B2C
# Rate limiting CHECKBOXHQ_RATE_LIMIT: int = 90 # Stay under 100/min
class Config: env_file = ".env"
settings = Settings()Create .env file:
CHECKBOXHQ_API_KEY=your_api_key_hereDATABASE_URL=postgresql://user:pass@localhost/dbnameREDIS_URL=redis://localhost:6379/0Step 3: Install Dependencies
pip install fastapi celery redis httpx sqlalchemy python-dotenv pydantic-settingsWhat each library does:
fastapi: Web framework for your APIcelery: Background job queue (processes validation async)redis: Message broker (Celery uses this to queue jobs)httpx: HTTP client for calling CheckBoxHQ APIsqlalchemy: Database ORMpydantic-settings: Environment variable management
Step 4: Signup Endpoint
from fastapi import APIRouter, HTTPException, Dependsfrom sqlalchemy.orm import Sessionfrom pydantic import BaseModel, EmailStrfrom ..database import get_dbfrom ..tasks import validate_email_asyncimport bcrypt
router = APIRouter()
class SignupRequest(BaseModel): email: EmailStr # Pydantic validates email format password: str name: str
class SignupResponse(BaseModel): id: int email: str name: str validation_status: str message: str
@router.post("/signup", response_model=SignupResponse)async def signup(request: SignupRequest, db: Session = Depends(get_db)): """ Create user account and queue email validation. User gets immediate access while validation runs in background. """
# 1. Check if email already exists existing_user = db.query(User).filter( User.email == request.email.lower() ).first()
if existing_user: raise HTTPException( status_code=400, detail="Email already registered" )
# 2. Hash password password_hash = bcrypt.hashpw( request.password.encode('utf-8'), bcrypt.gensalt() ).decode('utf-8')
# 3. Create user with pending validation user = User( email=request.email.lower(), password_hash=password_hash, name=request.name, email_validation_status="pending", account_status="active" # Allow immediate access )
db.add(user) db.commit() db.refresh(user)
# 4. Queue async validation (doesn't block response) validate_email_async.delay(user.id, user.email)
# 5. Return immediately return SignupResponse( id=user.id, email=user.email, name=user.name, validation_status="pending", message="Account created successfully. Email validation in progress." )Key points:
- User account created immediately
- Password hashed securely with bcrypt
- Email normalized to lowercase
- Validation queued with
.delay()(Celery async call) - User can start using the app right away
Step 5: Background Validation Task
from celery import Celeryimport httpxfrom sqlalchemy.orm import Sessionfrom config import settingsfrom models import Userimport loggingfrom datetime import datetimeimport time
# Initialize Celerycelery_app = Celery( 'checkboxhq_tasks', broker=settings.REDIS_URL, backend=settings.REDIS_URL)
# Configure Celerycelery_app.conf.update( task_serializer='json', result_serializer='json', accept_content=['json'], timezone='UTC', enable_utc=True,)
logger = logging.getLogger(__name__)
@celery_app.task( bind=True, max_retries=3, default_retry_delay=60 # Wait 60 seconds before retry)def validate_email_async(self, user_id: int, email: str): """ Validate email using checkboxHQ API.
Strategy: 1. Check if disposable (1 credit) 2. If not disposable, verify DNS (2 credits) 3. Optionally check public provider (1 credit)
Total: 3 credits per email (or 1 if disposable) """
db = get_db_session()
try: # Get user user = db.query(User).filter(User.id == user_id).first() if not user: logger.error(f"User {user_id} not found") return
# Update status user.email_validation_status = "validating" db.commit()
# === STEP 1: Check Disposable (1 credit) === logger.info(f"Checking disposable for {email}") disposable_result = check_disposable(email)
# Save disposable check results user.is_disposable = disposable_result['is_disposable'] user.disposable_provider = disposable_result.get('provider') db.commit()
# Reject if disposable if disposable_result['is_disposable'] and settings.REJECT_DISPOSABLE: user.email_validation_status = "invalid_disposable" user.validated_at = datetime.utcnow() db.commit()
handle_invalid_email(user, "disposable") logger.warning(f"Rejected disposable email: {email}") return
# === STEP 2: Verify DNS (2 credits) === logger.info(f"Verifying DNS for {email}")
# Rate limiting: Sleep if needed to stay under 100/min time.sleep(0.65) # ~92 requests/min to be safe
dns_result = verify_dns(email)
# Save DNS results user.has_mx_record = dns_result['has_mx_record'] user.mx_records = dns_result.get('mx_records') user.has_a_record = dns_result['has_a_record'] user.a_records = dns_result.get('a_records') user.email_provider = dns_result.get('provider') db.commit()
# Reject if no MX record if not dns_result['has_mx_record'] and settings.REJECT_INVALID_MX: user.email_validation_status = "invalid_no_mx" user.validated_at = datetime.utcnow() db.commit()
handle_invalid_email(user, "no_mx_record") logger.warning(f"Rejected email without MX: {email}") return
# Warn if no A record (domain has no website) if not dns_result['has_a_record']: logger.warning(f"No A record for {email} - domain has no website")
# === STEP 3: Check Public Provider (optional, 1 credit) === if settings.REJECT_PUBLIC_PROVIDERS: logger.info(f"Checking public provider for {email}")
time.sleep(0.65) # Rate limiting
public_result = check_public_provider(email)
user.is_public_provider = public_result['is_public_provider'] db.commit()
if public_result['is_public_provider']: user.email_validation_status = "invalid_public" user.validated_at = datetime.utcnow() db.commit()
handle_invalid_email(user, "public_provider") logger.warning(f"Rejected public provider: {email}") return
# === All checks passed === user.email_validation_status = "valid" user.validated_at = datetime.utcnow() db.commit()
logger.info(f"Email validated successfully: {email}")
except httpx.HTTPStatusError as e: if e.response.status_code == 429: # Rate limit exceeded - retry after delay logger.warning(f"Rate limit hit for {email}, retrying...") raise self.retry(countdown=120) # Wait 2 minutes else: logger.error(f"HTTP error validating {email}: {e}") raise self.retry(exc=e)
except Exception as e: logger.error(f"Unexpected error validating {email}: {e}") user.email_validation_status = "failed" user.validation_error = str(e) db.commit()
finally: db.close()
def check_disposable(email: str) -> dict: """Call checkboxHQ disposable check endpoint (1 credit)"""
response = httpx.post( f"{settings.CHECKBOXHQ_BASE_URL}/api/v1/verify_disposable/", json={"email": email}, headers={"x-api-key": settings.CHECKBOXHQ_API_KEY}, timeout=10.0 ) response.raise_for_status() return response.json()
def verify_dns(email: str) -> dict: """Call checkboxHQ DNS verification endpoint (2 credits)"""
response = httpx.post( f"{settings.CHECKBOXHQ_BASE_URL}/api/v1/verify_dns/", json={"email": email}, headers={"x-api-key": settings.CHECKBOXHQ_API_KEY}, timeout=10.0 ) response.raise_for_status() return response.json()
def check_public_provider(email: str) -> dict: """Call checkboxHQ public provider check endpoint (1 credit)"""
response = httpx.post( f"{settings.CHECKBOXHQ_BASE_URL}/api/v1/verify_public_provider/", json={"email": email}, headers={"x-api-key": settings.CHECKBOXHQ_API_KEY}, timeout=10.0 ) response.raise_for_status() return response.json()
def handle_invalid_email(user: User, reason: str): """ Take action when email fails validation.
Options: 1. Suspend account 2. Send notification email 3. Flag for manual review 4. Delete account """
if reason == "disposable": # Option: Suspend account user.account_status = "suspended" logger.info(f"Suspended account for disposable email: {user.email}")
# Option: Send notification # send_notification(user.email, "Disposable emails not allowed")
elif reason == "no_mx_record": # Email cannot receive messages user.account_status = "suspended" logger.info(f"Suspended account - no MX record: {user.email}")
elif reason == "public_provider": # Free email provider (if you're blocking them) user.account_status = "pending_review" logger.info(f"Flagged public provider account: {user.email}")
def get_db_session(): """Create database session""" from database import SessionLocal return SessionLocal()Key implementation details:
- Sequential checks: Disposable first (cheap), then DNS (expensive)
- Early exit: Stop if disposable, save 2 credits
- Rate limiting: 0.65s delay = ~92 requests/min (safe margin)
- Error handling: Retries on 429, fails gracefully otherwise
- Logging: Track every step for debugging
Step 6: Start Celery Worker
The worker processes validation jobs from the queue:
# Start worker in developmentcelery -A tasks.email_validation worker --loglevel=info
# Production: Use supervisor to keep it running# Create /etc/supervisor/conf.d/celery-worker.conf:[program:celery_worker]command=/path/to/venv/bin/celery -A tasks.email_validation worker --loglevel=infodirectory=/path/to/your/appuser=your-app-userautostart=trueautorestart=truestopwaitsecs=600stderr_logfile=/var/log/celery/worker-error.logstdout_logfile=/var/log/celery/worker.log# Start supervisorsudo supervisorctl rereadsudo supervisorctl updatesudo supervisorctl start celery_workerStep 7: Frontend Implementation (React)
import { useState } from 'react';import axios from 'axios';
function SignupForm() { const [formData, setFormData] = useState({ email: '', password: '', name: '' });
const [status, setStatus] = useState({ loading: false, success: false, error: null });
const handleSubmit = async (e) => { e.preventDefault(); setStatus({ loading: true, success: false, error: null });
try { const response = await axios.post( 'https://api.yourapp.com/api/v1/signup', formData );
// Success - user created setStatus({ loading: false, success: true, error: null });
// Store token if your API returns one if (response.data.token) { localStorage.setItem('token', response.data.token); }
// Show success message alert('Account created! Email validation in progress.');
// Redirect to dashboard window.location.href = '/dashboard';
} catch (error) { setStatus({ loading: false, success: false, error: error.response?.data?.detail || 'Signup failed. Please try again.' }); } };
return ( <div className="signup-form"> <h2>Create Account</h2>
<form onSubmit={handleSubmit}> <div className="form-group"> <label>Email</label> <input type="email" value={formData.email} onChange={(e) => setFormData({ ...formData, email: e.target.value })} required /> </div>
<div className="form-group"> <label>Name</label> <input type="text" placeholder="John Doe" value={formData.name} onChange={(e) => setFormData({ ...formData, name: e.target.value })} required /> </div>
<div className="form-group"> <label>Password</label> <input type="password" placeholder="••••••••" value={formData.password} onChange={(e) => setFormData({ ...formData, password: e.target.value })} required minLength={8} /> </div>
{status.error && ( <div className="alert alert-error"> {status.error} </div> )}
<button type="submit" disabled={status.loading} className="btn-primary" > {status.loading ? 'Creating account...' : 'Sign Up'} </button> </form>
<p className="signup-note"> Your email will be validated in the background. You can start using your account immediately. </p> </div> );}
export default SignupForm;Step 8: Show Validation Status in Dashboard
import { useState, useEffect } from 'react';import axios from 'axios';
function ValidationStatusBanner() { const [status, setStatus] = useState(null);
useEffect(() => { checkStatus();
// Poll every 5 seconds if validation is in progress const interval = setInterval(() => { if (status === 'pending' || status === 'validating') { checkStatus(); } }, 5000);
return () => clearInterval(interval); }, [status]);
const checkStatus = async () => { try { const response = await axios.get('/api/v1/user/me', { headers: { Authorization: `Bearer ${localStorage.getItem('token')}` } });
setStatus(response.data.email_validation_status);
} catch (error) { console.error('Failed to check status:', error); } };
// Don't show banner if valid if (!status || status === 'valid') { return null; }
const getBannerContent = () => { switch (status) { case 'pending': case 'validating': return { icon: '⏳', message: 'Verifying your email address...', className: 'banner-info' };
case 'invalid_disposable': return { icon: '⚠️', message: 'Your email appears to be temporary. Please update to a permanent email address.', className: 'banner-warning' };
case 'invalid_no_mx': return { icon: '❌', message: 'Your email domain cannot receive emails. Please update your email address.', className: 'banner-error' };
case 'failed': return { icon: '❌', message: 'Could not verify your email. Please contact support.', className: 'banner-error' };
default: return null; } };
const content = getBannerContent(); if (!content) return null;
return ( <div className={`validation-banner ${content.className}`}> <span className="banner-icon">{content.icon}</span> <span className="banner-message">{content.message}</span> {(status === 'invalid_disposable' || status === 'invalid_no_mx') && ( <button onClick={() => window.location.href = '/settings/email'} className="banner-action" > Update Email </button> )} </div> );}
export default ValidationStatusBanner;CSS:
.validation-banner { display: flex; align-items: center; padding: 12px 20px; margin-bottom: 20px; border-radius: 6px; gap: 12px;}
.banner-info { background-color: #e3f2fd; border-left: 4px solid #2196f3;}
.banner-warning { background-color: #fff3e0; border-left: 4px solid #ff9800;}
.banner-error { background-color: #ffebee; border-left: 4px solid #f44336;}
.banner-icon { font-size: 20px;}
.banner-message { flex: 1; font-size: 14px;}
.banner-action { padding: 6px 12px; background: white; border: 1px solid #ddd; border-radius: 4px; cursor: pointer;}Production Best Practices
1. Rate Limiting on Signup Endpoint
Protect your API quota from abuse:
from slowapi import Limiter, _rate_limit_exceeded_handlerfrom slowapi.util import get_remote_address
limiter = Limiter(key_func=get_remote_address)app.state.limiter = limiterapp.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler)
@router.post("/signup")@limiter.limit("5/minute") # 5 signups per minute per IPasync def signup(request: Request, signup_data: SignupRequest, ...): # Your signup logic passWhy? Prevents spam signups from burning through your checkboxHQ credits.
2. Smart Caching for Known Domains
Skip validation for trusted domains to save credits:
# Cache common trusted providersTRUSTED_DOMAINS = { 'gmail.com', 'yahoo.com', 'outlook.com', 'hotmail.com', 'icloud.com', 'protonmail.com', 'aol.com'}
def should_validate_email(email: str) -> bool: """Skip validation for trusted providers""" domain = email.split('@')[1].lower() return domain not in TRUSTED_DOMAINS
# In signup endpointif should_validate_email(request.email): validate_email_async.delay(user.id, user.email)else: # Mark as valid immediately user.email_validation_status = "valid" user.is_public_provider = True user.has_mx_record = True db.commit()Savings: ~1-3 credits per trusted domain signup
3. Monitoring & Alerts
Track validation metrics:
from prometheus_client import Counter, Histogram
# Define metricsvalidations_total = Counter( 'email_validations_total', 'Total email validations', ['status'] # valid, invalid_disposable, invalid_no_mx, failed)
validation_duration = Histogram( 'email_validation_duration_seconds', 'Time taken to validate email')
credit_usage = Counter( 'checkboxhq_credits_used', 'Credits consumed', ['endpoint'] # verify_disposable, verify_dns, verify_public_provider)
# Use in validation task@validation_duration.time()def validate_email_async(self, user_id: int, email: str): # Track disposable check credit_usage.labels(endpoint='verify_disposable').inc() disposable_result = check_disposable(email)
if disposable_result['is_disposable']: validations_total.labels(status='invalid_disposable').inc() return
# Track DNS check credit_usage.labels(endpoint='verify_dns').inc(2) # 2 credits dns_result = verify_dns(email)
# ... rest of validationDashboard queries:
- Total validations per day
- Success/failure rates
- Credit consumption rate
- Average validation time
4. Batch Validation for Existing Users
Validate old accounts that signed up before validation:
@celery_app.taskdef batch_validate_existing_users(): """ Run daily to validate unvalidated accounts. Process in batches to respect rate limits. """
db = get_db_session()
# Get 100 unvalidated users users = db.query(User).filter( User.email_validation_status == 'pending' ).limit(100).all()
for user in users: # Queue validation with 1 second delay between each validate_email_async.apply_async( args=[user.id, user.email], countdown=users.index(user) # Stagger by seconds )
db.close() logger.info(f"Queued {len(users)} users for validation")
# Schedule to run daily at 2 AMfrom celery.schedules import crontab
celery_app.conf.beat_schedule = { 'batch-validate-daily': { 'task': 'tasks.email_validation.batch_validate_existing_users', 'schedule': crontab(hour=2, minute=0), },}Run beat scheduler:
celery -A tasks.email_validation beat --loglevel=info5. Error Tracking with Sentry
Get notified when validations fail:
import sentry_sdk
sentry_sdk.init( dsn="your-sentry-dsn", traces_sample_rate=0.1)
@celery_app.taskdef validate_email_async(self, user_id: int, email: str): try: # Validation logic pass except Exception as e: # Log to Sentry sentry_sdk.capture_exception(e)
# Send alert if quota exceeded if "quota" in str(e).lower(): send_admin_alert("CheckBoxHQ quota exceeded!")
raise6. Graceful Degradation
Handle checkboxHQ API downtime:
@celery_app.task( bind=True, max_retries=3, autoretry_for=(httpx.HTTPError,), retry_backoff=True, retry_backoff_max=600, # Max 10 minutes retry_jitter=True)def validate_email_async(self, user_id: int, email: str): try: # Validation logic pass except httpx.HTTPError as e: # After 3 retries, mark as failed but don't block user if self.request.retries >= self.max_retries: user = db.query(User).get(user_id) user.email_validation_status = "failed" user.validation_error = "Service temporarily unavailable" user.account_status = "active" # Still allow access db.commit()
# Notify admins logger.error(f"Failed to validate {email} after retries") else: raiseResult: Users can still use your app even if CheckBoxHQ is down.
Testing Your Implementation
1. Manual Testing Checklist
Test these scenarios:
# ✅ Valid email (should pass)curl -X POST https://api.yourapp.com/api/v1/signup \ -H "Content-Type: application/json" \ -d '{ "email": "[email protected]", "password": "secure123", "name": "Test User" }'
# Expected: Account created, validation_status: pending → valid
# ❌ Disposable email (should be rejected)curl -X POST https://api.yourapp.com/api/v1/signup \ -H "Content-Type: application/json" \ -d '{ "email": "[email protected]", "password": "secure123", "name": "Disposable Test" }'
# Expected: Account created, validation_status: pending → invalid_disposable
# ❌ Invalid domain (no MX record)curl -X POST https://api.yourapp.com/api/v1/signup \ -H "Content-Type: application/json" \ -d '{ "email": "[email protected]", "password": "secure123", "name": "Invalid Domain" }'
# Expected: validation_status: pending → invalid_no_mx2. Automated Tests (pytest)
import pytestfrom fastapi.testclient import TestClientfrom app.main import appfrom time import sleep
client = TestClient(app)
def test_valid_email_signup(): """Test signup with valid email""" response = client.post("/api/v1/signup", json={ "password": "secure123", "name": "Test User" })
assert response.status_code == 200 assert response.json()["validation_status"] == "pending"
# Wait for async validation (or mock Celery) sleep(3)
# Check database assert user.email_validation_status == "valid"
def test_disposable_email_rejected(): """Test disposable email gets rejected""" response = client.post("/api/v1/signup", json={ "password": "secure123", "name": "Disposable User" })
assert response.status_code == 200
sleep(3)
assert user.email_validation_status == "invalid_disposable" assert user.is_disposable == True
def test_invalid_mx_rejected(): """Test email without MX record gets rejected""" response = client.post("/api/v1/signup", json={ "password": "secure123", "name": "No MX User" })
assert response.status_code == 200
sleep(3)
user = db.query(User).filter( ).first() assert user.email_validation_status == "invalid_no_mx" assert user.has_mx_record == False
def test_rate_limiting(): """Test signup rate limiting works"""
# Make 6 requests quickly (limit is 5/min) for i in range(6): response = client.post("/api/v1/signup", json={ "email": f"test{i}@example.com", "password": "secure123", "name": f"User {i}" })
if i < 5: assert response.status_code == 200 else: assert response.status_code == 429 # Rate limited3. Load Testing
Test concurrent signups:
import asyncioimport httpx
async def signup(session, email): response = await session.post( 'https://api.yourapp.com/api/v1/signup', json={ "email": email, "password": "test123", "name": "Load Test" } ) return response.status_code
async def load_test(): async with httpx.AsyncClient() as session: tasks = [ signup(session, f"test{i}@example.com") for i in range(100) # 100 concurrent signups ] results = await asyncio.gather(*tasks)
print(f"Success: {results.count(200)}") print(f"Rate limited: {results.count(429)}")
asyncio.run(load_test())Troubleshooting
Issue 1: Validations Stuck in “pending”
Symptoms: Status never changes from pending
Check:
# Is Celery worker running?ps aux | grep celery
# Is Redis running?redis-cli ping
# Check Celery logstail -f /var/log/celery/worker.log
# Check for failed taskscelery -A tasks.email_validation inspect activeFix:
# Restart Celery workersudo supervisorctl restart celery_worker
# Restart Redissudo systemctl restart redisIssue 2: HTTP 429 (Rate Limit Exceeded)
Symptoms: Validation fails with 429 error
Solutions:
- Increase delay between requests:
# In validate_email_asynctime.sleep(0.7) # ~85 requests/min- Use Celery rate limiting:
@celery_app.task(rate_limit='90/m') # Max 90 per minutedef validate_email_async(self, user_id: int, email: str): # Validation logic pass- Implement request queue:
from celery import group
# Queue multiple validations with controlled ratedef queue_validations(user_ids): jobs = group( validate_email_async.s(user_id, user.email) for user_id in user_ids ) # Celery handles rate limiting automatically jobs.apply_async()Issue 3: High Credit Consumption
Problem: Running out of credits too quickly
Solutions:
-
Cache trusted domains (shown earlier)
-
Skip re-validation:
# Don't validate if already validated recentlyif user.validated_at and (datetime.utcnow() - user.validated_at).days < 30: logger.info(f"Skipping validation for {email} - validated recently") return- Batch process strategically:
# Only validate during off-peak hoursfrom datetime import datetime
current_hour = datetime.utcnow().hourif 2 <= current_hour <= 6: # 2 AM - 6 AM UTC validate_email_async.delay(user_id, email)else: # Queue for later validate_email_async.apply_async( args=[user_id, email], eta=datetime.utcnow().replace(hour=2, minute=0) )Issue 4: Validation Takes Too Long
Problem: Users waiting too long to see results
Solution 1: Optimize validation logic
# Use async HTTP client for parallel requestsimport httpxfrom asyncio import gather
async def validate_email_fast(email: str): async with httpx.AsyncClient() as client: # Run both checks in parallel (if disposable check passes) disposable_task = client.post( f"{settings.CHECKBOXHQ_BASE_URL}/api/v1/verify_disposable/", json={"email": email}, headers={"x-api-key": settings.CHECKBOXHQ_API_KEY} )
disposable_result = await disposable_task data = disposable_result.json()
if not data['is_disposable']: # If not disposable, check DNS dns_result = await client.post( f"{settings.CHECKBOXHQ_BASE_URL}/api/v1/verify_dns/", json={"email": email}, headers={"x-api-key": settings.CHECKBOXHQ_API_KEY} ) return disposable_result, dns_result
return disposable_result, NoneSolution 2: Show partial results
# Update UI as each check completesuser.email_validation_status = "validating_disposable"db.commit()
# ... after disposable check ...user.email_validation_status = "validating_dns"db.commit()Cost Optimization
Calculate Your Monthly Costs
# Example: 1000 signups per month# Strategy: Disposable check → DNS check (if not disposable)
# Assuming 5% disposable ratedisposable_signups = 1000 * 0.05 = 50normal_signups = 1000 * 0.95 = 950
# Credits useddisposable_credits = 50 * 1 = 50 creditsdns_credits = 950 * 2 = 1900 creditstotal_credits = 50 + 1900 = 1950 credits
# With caching for common domains (50% cache hit rate)cached_signups = 950 * 0.5 = 475uncached_signups = 950 * 0.5 = 475
disposable_credits = 50 * 1 = 50dns_credits = 475 * 2 = 950total_with_cache = 50 + 950 = 1000 credits
# Cost savings: 1950 - 1000 = 950 credits (49% reduction)Optimization Strategies by Volume
Low volume (<100 signups/month)
- Validate all emails fully
- Don’t worry about caching
- Cost: ~300 credits/month
Medium volume (100-1000 signups/month)
- Cache trusted domains
- Skip validation for known good users (returning customers)
- Cost: ~1000 credits/month
High volume (1000+ signups/month)
- Aggressive caching
- Batch process during off-peak
- Consider webhooks for async updates
- Monitor disposable rate and adjust strategy
- Cost: ~3000 credits/month
Summary & Checklist
Implementation Checklist
- Database schema with validation tracking
- Environment variables configured (API key, Redis URL)
- Dependencies installed (FastAPI, Celery, Redis, etc.)
- Signup endpoint creates user immediately
- Background task validates with checkboxHQ
- Celery worker running (supervised in production)
- Rate limiting implemented (0.65s between API calls)
- Error handling for 429, timeouts, failures
- Frontend shows validation status
- Monitoring and logging configured
- Tests for valid, disposable, and invalid emails
Production Checklist
- Celery worker managed by supervisor/systemd
- Redis persistence enabled
- Sentry or error tracking configured
- Metrics collection (Prometheus/Grafana)
- Rate limiting on signup endpoint
- Domain caching for trusted providers
- Batch validation for existing users
- Daily backup of validation results
- Admin dashboard for monitoring
- Documentation for team
Next Steps
- Start simple: Implement basic validation with disposable + DNS checks
- Monitor costs: Track credit usage for first week
- Optimize gradually: Add caching after understanding patterns
- Test thoroughly: Try various email types
- Scale up: Add batch processing as you grow
Additional Resources
- checkboxHQ API Docs: https://api.checkboxhq.com/docs
- Celery Documentation: https://docs.celeryproject.org
- FastAPI Background Tasks: https://fastapi.tiangolo.com/tutorial/background-tasks/
- Redis Setup: https://redis.io/docs/getting-started/
- Supervisor: http://supervisord.org/
Need Help?
For checkboxHQ-specific questions:
- Email: [email protected]
- API Status: Check rate limits with
/api/v1/billing/usageendpoint
For implementation questions:
- Check Celery logs:
/var/log/celery/worker.log - Test API manually with curl
- Monitor Redis queue:
redis-cli LLEN celery
This guide provides a production-ready implementation for async email validation. Adjust the validation rules, caching strategy, and error handling based on your specific use case and signup volume.