ERC3 REST API Specification

Overview

The ERC3 REST API provides endpoints for benchmark evaluation management, task execution, and specialized store/demo operations. This document describes the actual HTTP endpoints used by the ERC3 Python client under the hood.

Base URL: Configurable (defaults to https://erc.timetoact-group.at)

HTTP Patterns

Authentication

Method: API key passed in request body (not HTTP headers)
Format: Keys must start with key- prefix
Example: {"account_key": "key-abc123def456..."}

Request Format

Method: All endpoints use POST requests
Content-Type: application/json
Body: JSON payload with endpoint-specific fields

Response Format

Success: JSON response with 200 status code

Error: Structured error format

{
  "status": 400,
  "error": "Error description",
  "code": "ERROR_CODE"
}

URL Structure

Core API: {base_url}/{endpoint}
Store API: {base_url}/store/{task_id}/{endpoint}
Demo API: {base_url}/demo/{task_id}/{endpoint}

Core API Endpoints

Authentication

Get API Key

POST /get_key

Retrieves an API key for a given email address.

Request:

{
  "email": "user@example.com"
}

Response:

{
  "key": "key-abc123def456...",
  "public_key": "public-key-string"
}

Benchmarks

List Benchmarks

POST /benchmarks/list

Retrieves a list of available benchmarks.

Request:

{}

Response:

{
  "benchmarks": [
    {
      "id": "benchmark-1",
      "short_description": "Description of benchmark 1",
      "status": "active",
      "tasks": 10
    },
    {
      "id": "benchmark-2",
      "short_description": "Description of benchmark 2",
      "status": "active",
      "tasks": 15
    }
  ]
}

View Benchmark Details

POST /benchmarks/view

Retrieves detailed information about a specific benchmark.

Request:

{
  "benchmark": "benchmark-1"
}

Response:

{
  "id": "benchmark-1",
  "description": "Detailed description of the benchmark",
  "status": "active",
  "specs": [
    {
      "id": "spec-1",
      "name": "Specification 1",
      "description": "Description of spec 1"
    }
  ],
  "routes": [
    {
      "path": "/api/v1/route1",
      "method": "GET",
      "description": "Route description"
    }
  ]
}

Sessions

Search Sessions

POST /sessions/search

Searches for sessions based on various criteria.

Request:

{
  "account_key": "key-abc123def456...",
  "workspace": "workspace-name",
  "status": ["open", "done", "evaluated"],
  "benchmark": "benchmark-1"
}

Response:

{
  "sessions": [
    {
      "session_id": "session-123",
      "benchmark": "benchmark-1",
      "workspace": "workspace-name",
      "status": "open",
      "created_at": "2024-01-01T00:00:00Z",
      "updated_at": "2024-01-01T01:00:00Z"
    }
  ]
}

Start Session

POST /sessions/start

Creates and starts a new evaluation session.

Request:

{
  "account_key": "key-abc123def456...",
  "benchmark": "benchmark-1",
  "workspace": "workspace-name",
  "name": "My Evaluation Session",
  "architecture": "x86_64"
}

Response:

{
  "session_id": "session-123",
  "task_count": 10
}

Get Session Status

POST /sessions/status

Retrieves the current status of a session.

Request:

{
  "session_id": "session-123"
}

Response:

{
  "status": "running",
  "benchmark": "benchmark-1",
  "failed": 0,
  "new": 2,
  "running": 5,
  "completed": 3,
  "tasks": [
    {
      "task_id": "task-1",
      "status": "completed",
      "score": 0.95
    }
  ],
  "name": "My Evaluation Session",
  "architecture": "x86_64",
  "flags": ["sgr", "local"]
}

Submit Session

POST /sessions/submit

Submits a completed session for evaluation.

Request:

{
  "session_id": "session-123"
}

Response:

{
  "status": "submitted"
}

Tasks

Start Task

POST /tasks/start

Starts execution of a specific task within a session.

Request:

{
  "task_id": "task-1",
  "benchmark": "benchmark-1",
  "spec_id": "spec-1"
}

Response:

{
  "task_id": "task-1",
  "session_id": "session-123",
  "status": "in_progress"
}

Complete Task

POST /tasks/complete

Marks a task as completed and retrieves evaluation results.

Request:

{
  "task_id": "task-1"
}

Response:

{
  "status": "completed",
  "eval": {
    "score": 0.95,
    "logs": "Task completed successfully"
  }
}

View Task Details

POST /tasks/view

Retrieves detailed information about a task, including logs since a specific time.

Request:

{
  "task_id": "task-1",
  "since": 1640995200
}

Response:

{
  "task_id": "task-1",
  "text": "Task description and content",
  "session_id": "session-123",
  "spec": "spec-1",
  "status": "completed",
  "logs": [
    {
      "timestamp": "2024-01-01T12:00:00Z",
      "level": "INFO",
      "message": "Task started"
    }
  ],
  "score": 0.95,
  "benchmark": "benchmark-1",
  "error_message": null
}

Log Task Usage

POST /tasks/log

Logs resource usage and duration for a task.

Request:

{
  "task_id": "task-1",
  "model": "gpt-4",
  "usage": {
    "prompt_tokens": 1000,
    "completion_tokens": 500,
    "total_tokens": 1500
  },
  "duration_sec": 45.5
}

Response:

{}

Store API Endpoints

Base URL Pattern: {base_url}/store/{task_id}

All store endpoints use a tool-based dispatch pattern where the tool field specifies the actual operation.

Product Management

List Products

POST /products/list

Retrieves a paginated list of products.

Request:

{
  "tool": "/products/list",
  "offset": 0,
  "limit": 20
}

Response:

{
  "products": [
    {
      "sku": "PROD-001",
      "name": "Product Name",
      "available": 10,
      "price": 2999
    }
  ],
  "next_offset": 20
}

Shopping Cart Management

View Basket

POST /basket/view

Retrieves current shopping cart contents.

Request:

{
  "tool": "/basket/view"
}

Response:

{
  "items": [
    {
      "sku": "PROD-001",
      "quantity": 2,
      "price": 2999
    }
  ],
  "subtotal": 5998,
  "coupon": "SAVE10",
  "discount": 600,
  "total": 5398
}

Add to Basket

POST /basket/add

Adds items to the shopping cart.

Request:

{
  "tool": "/basket/add",
  "sku": "PROD-001",
  "quantity": 1
}

Response:

{
  "line_count": 2,
  "item_count": 3
}

Remove from Basket

POST /basket/remove

Removes items from the shopping cart.

Request:

{
  "tool": "/basket/remove",
  "sku": "PROD-001",
  "quantity": 1
}

Response:

{
  "line_count": 1,
  "item_count": 1
}

Checkout

POST /basket/checkout

Processes the current shopping cart for checkout.

Request:

{
  "tool": "/basket/checkout"
}

Response:

{
  "items": [
    {
      "sku": "PROD-001",
      "quantity": 1,
      "price": 2999
    }
  ],
  "subtotal": 2999,
  "coupon": null,
  "discount": 0,
  "total": 2999
}

Coupon Management

Apply Coupon

POST /coupon/apply

Applies a discount coupon to the current basket.

Request:

{
  "tool": "/coupon/apply",
  "coupon": "SAVE10"
}

Response:

{}

Remove Coupon

POST /coupon/remove

Removes the currently applied coupon.

Request:

{
  "tool": "/coupon/remove"
}

Response:

{}

Demo API Endpoints

Base URL Pattern: {base_url}/demo/{task_id}

Demo endpoints use the same tool-based dispatch pattern as store endpoints.

Secret Management

Get Secret

POST /secret

Retrieves a secret value for the current demo task.

Request:

{
  "tool": "/secret"
}

Response:

{
  "value": "secret-value-123"
}

Answer Submission

Submit Answer

POST /answer

Submits an answer for the current demo task.

Request:

{
  "tool": "/answer",
  "answer": "The answer to the demo question"
}

Response:

{}

Error Handling

Standard Error Response Format

All endpoints return errors in a consistent format:

{
  "status": 400,
  "error": "Detailed error message",
  "code": "ERROR_TYPE"
}

Common Error Codes

Status Code	Error Code	Description
400	BAD_REQUEST	Invalid request parameters or format
401	UNAUTHORIZED	Invalid or missing API key
403	FORBIDDEN	Insufficient permissions for requested action
404	NOT_FOUND	Requested resource does not exist
429	RATE_LIMITED	API rate limit exceeded
500	INTERNAL_ERROR	Server-side error occurred

Authentication Errors

API key validation errors:

{
  "status": 401,
  "error": "Invalid API key format. Key must start with 'key-'",
  "code": "INVALID_KEY_FORMAT"
}

{
  "status": 401,
  "error": "API key not found or expired",
  "code": "KEY_NOT_FOUND"
}

Validation Errors

Request validation failures:

{
  "status": 400,
  "error": "Missing required field: 'session_id'",
  "code": "MISSING_REQUIRED_FIELD"
}

{
  "status": 400,
  "error": "Invalid status value. Must be one of: ['open', 'done', 'evaluated']",
  "code": "INVALID_ENUM_VALUE"
}

Usage Examples

Complete Session Workflow

# 1. Get API key
curl -X POST https://erc.timetoact-group.at/get_key \
  -H "Content-Type: application/json" \
  -d '{"email": "user@example.com"}'

# 2. List available benchmarks
curl -X POST https://erc.timetoact-group.at/benchmarks/list \
  -H "Content-Type: application/json" \
  -d '{}'

# 3. Start a new session
curl -X POST https://erc.timetoact-group.at/sessions/start \
  -H "Content-Type: application/json" \
  -d '{
    "account_key": "key-abc123def456...",
    "benchmark": "benchmark-1",
    "workspace": "my-workspace",
    "name": "Test Session",
    "architecture": "x86_64"
  }'

# 4. Check session status
curl -X POST https://erc.timetoact-group.at/sessions/status \
  -H "Content-Type: application/json" \
  -d '{"session_id": "session-123"}'

# 5. Submit completed session
curl -X POST https://erc.timetoact-group.at/sessions/submit \
  -H "Content-Type: application/json" \
  -d '{"session_id": "session-123"}'

Store API Example

# View shopping basket
curl -X POST "https://erc.timetoact-group.at/store/task-123/basket/view" \
  -H "Content-Type: application/json" \
  -d '{"tool": "/basket/view"}'

# Add item to basket
curl -X POST "https://erc.timetoact-group.at/store/task-123/basket/add" \
  -H "Content-Type: application/json" \
  -d '{
    "tool": "/basket/add",
    "sku": "PROD-001",
    "quantity": 2
  }'

Demo API Example

# Get secret for demo task
curl -X POST "https://erc.timetoact-group.at/demo/task-456/secret" \
  -H "Content-Type: application/json" \
  -d '{"tool": "/secret"}'

# Submit answer
curl -X POST "https://erc.timetoact-group.at/demo/task-456/answer" \
  -H "Content-Type: application/json" \
  -d '{
    "tool": "/answer",
    "answer": "This is my answer to the demo question"
  }'

Implementation Notes

Tool-Based Routing

Store and Demo APIs use a tool-based dispatch system where the tool field in the request body specifies the actual endpoint path. This allows for a generic POST endpoint that routes to specific functionality based on the tool name.

Session-Based Architecture

The API uses a session-based model where:

Sessions contain multiple tasks
Tasks represent individual evaluation units
Both sessions and tasks have lifecycle states
Tasks can be executed independently or as part of a session

Resource Usage Tracking

The /tasks/log endpoint enables detailed resource usage tracking, including:

Model usage (tokens, API calls)
Execution duration
Custom metrics

This data is typically used for cost analysis and performance optimization.

Generated from ERC3 package version 1.0.4 source code analysis

timurkhakhalev/reference.md

ERC3 REST API Specification

Overview

HTTP Patterns

Authentication

Request Format

Response Format

URL Structure

Core API Endpoints

Authentication

Get API Key

Benchmarks

List Benchmarks

View Benchmark Details

Sessions

Search Sessions

Start Session

Get Session Status

Submit Session

Tasks

Start Task

Complete Task

View Task Details

Log Task Usage

Store API Endpoints

Product Management

List Products

Shopping Cart Management

View Basket

Add to Basket

Remove from Basket

Checkout

Coupon Management

Apply Coupon

Remove Coupon

Demo API Endpoints

Secret Management

Get Secret

Answer Submission

Submit Answer

Error Handling

Standard Error Response Format

Common Error Codes

Authentication Errors

Validation Errors

Usage Examples

Complete Session Workflow

Store API Example

Demo API Example

Implementation Notes

Tool-Based Routing

Session-Based Architecture

Resource Usage Tracking