Skip to content

Instantly share code, notes, and snippets.

@unclecode

unclecode/qa-testflight-guide.md

Last active December 17, 2025 05:10
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save unclecode/a863d5a33d1e599e7fa6d4b7dfe1d3dc to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save unclecode/a863d5a33d1e599e7fa6d4b7dfe1d3dc to your computer and use it in GitHub Desktop.
Download ZIP
Crawl4AI Cloud - QA TestFlight Testing Guide
Raw
qa-testflight-guide.md

Crawl4AI Cloud - QA TestFlight Program

Hello team! I've set up our internal QA testing program. Please read this carefully and follow the instructions.

Step 1: Get Access

  1. Accept GitHub invitation (check your email)
  2. Visit TestFlight repo: https://github.com/unclecode/crawl4ai-cloud-testflight
  3. Read the full README - it has 206+ test cases organized by category
  4. Test credentials will be shared separately via WhatsApp

What We're Testing - Two Main Areas

Area 1: Dashboard & Platform Features

Test the dashboard at: https://api.crawl4ai.com/dashboard

Authentication & Accounts:

  • Email/password registration
  • Email/password login
  • OAuth login (Google, GitHub)
  • Password reset / Forgot password
  • Magic link login
  • Linking/unlinking social accounts
  • Account settings management

API Key Management:

  • Create API keys
  • View API keys (hint display)
  • Delete API keys
  • Copy API keys

Billing & Subscriptions:

  • View pricing plans (Free, Plus, Pro)
  • Stripe checkout flow
  • Subscription management
  • Plan upgrades/downgrades
  • Cancellation flow
  • Reactivation after cancellation
  • Test Credit Card: 4242 4242 4242 4242 (CVC/expiry: any future date)

Usage Tracking:

  • Daily crawl quota display
  • Storage usage tracking
  • LLM token usage (if using managed LLM)
  • Current limits display (Free vs Paid)
  • Session usage tracking

Dashboard UI:

  • Quick Start panel (code examples)
  • Sessions page
  • Crawl history page
  • Storage management
  • Settings page
  • LLM Providers management (BYOK)
  • Responsive design (mobile/tablet)

Area 2: Crawler API (PRIMARY FOCUS)

This is our main priority - test thoroughly!

Authentication:

  • All API calls require: X-API-Key header
  • Create API keys from dashboard first

API Endpoints Available:

1. Sync Crawl (may not be in final release)

POST /v1/crawl
- Single URL
- Synchronous response (waits for result)
- Max timeout based on plan

2. Batch Crawl

POST /v1/crawl/batch
- Multiple URLs (max 10)
- Synchronous response
- Parallel processing

3. Async Crawl (MOST IMPORTANT)

POST /v1/crawl/async
- Multiple URLs (up to 100)
- Returns job ID
- Poll for status/results
- Production-ready

4. Job Management

GET    /v1/crawl/jobs/{job_id}    # Get job status
DELETE /v1/crawl/jobs/{job_id}    # Cancel job
GET    /v1/crawl/jobs             # List your jobs

5. Sessions API (CDP Browser Sessions)

POST   /v1/sessions               # Create session
GET    /v1/sessions/{id}          # Session status
DELETE /v1/sessions/{id}          # Close session
WS     /v1/cdp/{session_id}       # WebSocket proxy

6. Storage API

GET /v1/crawl/storage             # Storage usage
GET /v1/crawl/jobs/{id}/download  # Download results (S3)

Features to Test:

Basic Crawling:

  • Simple HTML extraction
  • Markdown conversion
  • Screenshot capture
  • Custom headers/cookies
  • JavaScript rendering
  • Page timeout configuration
  • User agent customization

Extraction Strategies:

LLM Extraction (AI-powered)

  • Managed service (use our API keys)
  • BYOK (bring your own key)
  • Different providers:
    • OpenAI (GPT-4o, GPT-4o-mini)
    • Anthropic (Claude Haiku, Claude Sonnet)
    • Groq (Llama models)
    • Google (Gemini)
    • Deepseek
  • Schema-based extraction (JSON schema)
  • Custom instructions
  • Token usage tracking

JSON CSS Extraction (selector-based, no LLM cost)

  • CSS selector patterns
  • Multiple field extraction
  • Nested selectors
  • Array extraction

JSON XPath Extraction

  • XPath expressions
  • Complex DOM queries

Rate Limits & Quotas:

Free Plan:

  • 1 concurrent request
  • 50 daily crawls
  • 10 requests/minute
  • 60s timeout
  • 100MB storage
  • No LLM managed service (BYOK only)

Paid Plan (Plus):

  • 5 concurrent requests
  • 500 daily crawls
  • Higher rate limits
  • 300s timeout
  • 1GB storage
  • LLM managed service (100K tokens/month)

Paid Plan (Pro):

  • 20 concurrent requests
  • 5000 daily crawls
  • Highest rate limits
  • 600s timeout
  • 10GB storage
  • LLM managed service (1M tokens/month)

Error Handling:

  • Rate limit errors (429)
  • Authentication errors (401)
  • Timeout errors
  • Invalid URL errors
  • Quota exceeded errors
  • Invalid extraction strategy errors
  • Session creation failures

SDKs Available

I'll share these SDKs separately:

  1. Python SDK - Install and test
  2. Node.js SDK - Install and test
  3. cURL examples - In the README

Test all three methods!

CRITICAL: Bug Reporting Requirements

When you find a bug (and you will!):

Step 1: Prepare Evidence

  1. Use Chrome browser ONLY (for consistency)
  2. Open DevTools (Press F12)
  3. Switch to Console tab
  4. Take screenshot showing:
    • The bug on the page
    • DevTools Console (to capture errors)
    • You can take ONE screenshot with both, or TWO separate screenshots

Step 2: Create GitHub Issue

  1. Go to Issues tab in TestFlight repo
  2. Click "New Issue" → "Bug Report"
  3. Fill in all required fields:
    • Test Case ID (if applicable)
    • Severity (Critical/High/Medium/Low)
    • Account Type (Free/Paid)
    • Steps to Reproduce (clear, numbered steps)
    • Expected Behavior
    • Actual Behavior
    • Screenshots (REQUIRED - upload your screenshots)
    • Browser & Version (should be Chrome)
    • Device / OS
    • Additional Context (console errors, network errors)

Step 3: Add Label

CRITICAL: After submitting, add the label: needs-triage

  • Without this label, your issue won't sync to our main repo!

Step 4: Submit

Your issue will automatically sync to the main repo for review.

Why screenshots with DevTools? Our AI (Claude) reads them to debug faster!

Testing Discipline

Please be systematic:

Work Methodically

  • Work through test cases one by one (206 total in README)
  • Don't skip categories
  • Document findings as you go

Test Comprehensively

  • Test BOTH Free and Paid accounts
  • Test different scenarios (valid/invalid inputs)
  • Test edge cases (empty strings, very long URLs, special characters)
  • Test error conditions (rate limits, timeouts)

Test on Multiple Devices

  • Desktop (required)
  • Mobile (if possible)
  • Tablet (if possible)

Browser Requirements

  • Chrome REQUIRED for all testing
  • Keep DevTools open while testing
  • Monitor Console for errors

Report Everything

  • Even small UI issues matter
  • Typos and grammar errors count
  • Confusing UX flows are bugs too
  • Performance issues (slow loading)

Label Discipline

  • ALWAYS add needs-triage label to issues
  • This is what triggers the sync to main repo

Priority Order

Focus on these areas in order:

  1. Async Crawl API (highest priority - production feature)

    • Job creation
    • Job polling
    • Job cancellation
    • Result retrieval
    • Error handling

  2. Dashboard Features

    • Authentication flows
    • API key management
    • Billing/subscription flows
    • Usage tracking accuracy

  3. LLM Extraction

    • Managed service (our keys)
    • BYOK (user keys)
    • Different providers
    • Token usage tracking

  4. Rate Limits & Quotas

    • Free vs Paid differences
    • Concurrent request limits
    • Daily quota enforcement
    • Rate limiting behavior

  5. Error Handling

    • Timeout handling
    • Quota exceeded messages
    • Invalid input handling
    • Network error recovery

  6. Sessions API (if time permits)

    • Session creation
    • CDP WebSocket connection
    • Session cleanup

  7. Batch/Sync Crawl (lower priority)

    • May not ship in final release

Test Payment Information

Stripe Test Mode Credit Card:

  • Card Number: 4242 4242 4242 4242
  • Expiry: Any future date (e.g., 12/26)
  • CVC: Any 3 digits (e.g., 123)
  • ZIP/Postal Code: Any valid ZIP code

Test Different Scenarios:

  • Successful payment
  • Upgrade from Free to Plus
  • Upgrade from Plus to Pro
  • Downgrade from Pro to Plus
  • Cancel subscription (grace period)
  • Reactivate canceled subscription
  • Payment failure (use card 4000 0000 0000 0002 for declined)

Common Test Scenarios

Scenario 1: New User Journey

  1. Register new account (email/password)
  2. Verify email (check magic link)
  3. Login to dashboard
  4. Create first API key
  5. Try Quick Start example
  6. View usage stats
  7. Upgrade to paid plan
  8. Test increased limits

Scenario 2: OAuth User Journey

  1. Login with Google/GitHub
  2. Link additional OAuth provider
  3. Unlink OAuth provider
  4. Test login with remaining provider

Scenario 3: API Testing

  1. Get API key from dashboard
  2. Test sync crawl (simple URL)
  3. Test batch crawl (multiple URLs)
  4. Test async crawl (job workflow)
  5. Test LLM extraction
  6. Test CSS extraction
  7. Monitor rate limits
  8. Test quota enforcement

Scenario 4: Sessions Testing

  1. Create browser session
  2. Connect via WebSocket (if you know how)
  3. Verify session shows in dashboard
  4. Delete session
  5. Verify cleanup

Scenario 5: Error Testing

  1. Test with invalid API key (401)
  2. Exceed rate limit (429)
  3. Exceed daily quota
  4. Invalid URL format
  5. Timeout scenario (very slow URL)
  6. Invalid extraction strategy

Questions & Support

Ask in WhatsApp Group

  • Technical questions about features
  • Clarification on expected behavior
  • Test credential issues
  • Access problems

Report in GitHub Issues

  • Bugs and errors
  • UI/UX problems
  • Feature requests (as separate issue type)

Milestone

Onboarding

  • Accept GitHub invites
  • Read this guide and README
  • Familiarize with dashboard
  • Run first tests

Main Testing

  • Work through test cases systematically
  • Report bugs as you find them
  • Test different account types
  • Test SDKs

Edge Cases

  • Test error conditions
  • Test performance limits
  • Mobile testing
  • Integration testing

Final Pass

  • Re-test fixed bugs
  • Final smoke tests
  • Documentation review

Success Metrics

We're aiming for:

  • All 206 test cases executed
  • All P0 (Critical) bugs found and fixed
  • All P1 (High) bugs found and fixed
  • 90%+ of P2 (Medium) bugs found
  • Platform stable for beta launch

Thank You!

Your thorough testing will help us launch a rock-solid product. Every bug you find makes Crawl4AI Cloud better for our users.

Remember the essentials:

  • Chrome browser only
  • DevTools screenshots with every bug
  • needs-triage label on all issues
  • Test systematically
  • Report everything

Let's make this the best web crawling service available!

Questions? Ask in the WhatsApp group.

Ready? Accept your GitHub invite and start testing!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment