MinIO AIStor: Synchronous vs Asynchronous Replication Dataflow

replication-dataflow.md

MinIO: Synchronous vs Asynchronous Replication

Synchronous Replication

Client waits for replication to complete before receiving response.

┌────────┐         ┌─────────────────────────────┐         ┌────────────┐
│        │  PUT    │      SOURCE CLUSTER         │         │   REMOTE   │
│ Client ├────────►│                             │         │   TARGET   │
│        │         │  1. Write to local disk     │         │            │
│        │         │           │                 │         │            │
│        │         │           ▼                 │         │            │
│        │         │  2. Replicate to remote ────┼────────►│  3. Write  │
│        │         │           │                 │         │            │
│        │         │           │◄── WAIT ────────┼─────────┤  4. ACK    │
│        │         │           ▼                 │         │            │
│        │◄────────┤  5. Return response         │         │            │
│        │   OK    │                             │         │            │
└────────┘         └─────────────────────────────┘         └────────────┘

Timeline:
─────────────────────────────────────────────────────────────────────────►
│ Request │ Local │ ████ BLOCKED (network I/O) ████ │ Response │
│         │ Write │                                  │          │

Asynchronous Replication

Client receives response immediately; replication happens in background.

┌────────┐         ┌─────────────────────────────┐         ┌────────────┐
│        │  PUT    │      SOURCE CLUSTER         │         │   REMOTE   │
│ Client ├────────►│                             │         │   TARGET   │
│        │         │  1. Write to local disk     │         │            │
│        │         │           │                 │         │            │
│        │         │           ▼                 │         │            │
│        │◄────────┤  2. Return response         │         │            │
│        │   OK    │           │                 │         │            │
└────────┘         │           ▼                 │         │            │
                   │  3. Queue replication task  │         │            │
                   │           │                 │         │            │
                   │    ═══════╪═══════════      │         │            │
                   │    ║ BACKGROUND WORKER ║    │         │            │
                   │    ═══════╪═══════════      │         │            │
                   │           │                 │         │            │
                   │           ▼                 │         │            │
                   │  4. Replicate to remote ────┼────────►│  5. Write  │
                   │                             │         │            │
                   │  6. If failed → MRF queue   │         │            │
                   │     (retry later)           │         │            │
                   └─────────────────────────────┘         └────────────┘

Timeline:
─────────────────────────────────────────────────────────────────────────►
│ Request │ Local │ Response │
│         │ Write │          │
                             ════════════════════════════════════════════►
                             ║ Background: Queue → Worker → Remote PUT ║

Comparison

Aspect	Synchronous	Asynchronous
Response timing	After remote write	After local write
Consistency	Strong	Eventual
Client latency	High	Low
Throughput	Lower	Higher
On failure	Error to client	Retry via MRF queue
Use case	Compliance, RPO=0	Backups, DR, high-throughput

Key Code Path

scheduleReplication()                    # cmd/bucket-replication.go:2722
         │
         ▼
  dsc.Synchronous()?
        /     \
      YES      NO
       │        │
       ▼        ▼
replicateObject()    queueReplicaTask()
   (inline)              │
                         ▼
                   Worker Pool
                   (100 workers)
                         │
                         ▼
                  replicateObject()
                   (background)