Skip to content

Instantly share code, notes, and snippets.

@yongkangc

yongkangc/FINAL_REPORT.md

Last active February 2, 2026 09:12
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save yongkangc/c34e9a87aff780533f8e404638aa3994 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save yongkangc/c34e9a87aff780533f8e404638aa3994 to your computer and use it in GitHub Desktop.
Download ZIP
RocksDB Parameter Tuning Results for Reth
Raw
FINAL_REPORT.md

RocksDB Parameter Tuning - 3-Run Average Results

Executive Summary

After running each configuration 3 times for statistical significance, the baseline configuration performs best for this workload. The tuned configurations show regressions, not improvements.

Performance Summary (3-run average)

Config Ggas/s p50 (ms) p90 (ms) p99 (ms)
baseline 0.917 28.18 59.02 80.75
write_buffer_size_128mb 0.767 35.54 52.40 81.16
l0_stop_trigger_100 0.737 36.55 61.34 90.36

Delta vs Baseline

Config Ggas/s Δ% p90 Δ% p99 Δ% Verdict
write_buffer_size_128mb -16.30% -11.22% +0.51% ❌ REJECT
l0_stop_trigger_100 -19.56% +3.93% +11.91% ❌ REJECT

Detailed Prometheus Metrics (3-run median average)

Save Blocks - Write Operations (seconds)

Metric Baseline 128mb Buffer L0 Stop 100
trie_updates 0.285 0.214 0.302
write_state 0.131 0.095 0.128
hashed_state 0.125 0.099 0.130

Storage Backend - Write Time (seconds)

Backend Baseline 128mb Buffer L0 Stop 100
MDBX 0.542 0.409 0.561
RocksDB 0.000 0.025 0.031
Static Files 0.010 0.008 0.010

Storage Backend - Commit Time (seconds)

Backend Baseline 128mb Buffer L0 Stop 100
MDBX Commit 0.000783 0.000830 0.000818
RocksDB Commit ~0 0.000033 0.000038
Static Files Commit 0.000382 0.000388 0.000385

Save Blocks - Total

Metric Baseline 128mb Buffer L0 Stop 100
Total Time (s) 0.542 0.409 0.562
Blocks per Save 12.33 9.33 12.00

Key Observations

1. Baseline is Best for Throughput

The default RocksDB configuration achieves 0.917 Ggas/s, outperforming both tuned configurations by 16-20%.

2. RocksDB Time is Not the Bottleneck

In all configurations, RocksDB write/commit time is negligible:

  • Baseline: ~0 seconds (RocksDB not heavily used)
  • Tuned: 25-31ms for writes, <40µs for commits

3. Trie Updates Dominate

Across all configurations, write_trie_updates accounts for ~50% of save_blocks time. This is CPU-bound work, not I/O-bound.

4. MDBX Dominates Storage Time

MDBX write time (409-561ms) is the primary storage bottleneck, not RocksDB.

5. write_buffer_size_128mb Trades Throughput for p90 Latency

  • -16% throughput but -11% p90 latency
  • Fewer blocks per save (9.33 vs 12.33) indicates more frequent flushes

Why Initial Single-Run Results Were Misleading

The initial single-run results showed:

  • write_buffer_size_128mb: +9.32% throughput improvement
  • l0_stop_trigger_100: -11.10% p99 improvement

These were due to:

  1. Block variance: Different block ranges have different gas/complexity
  2. Warm-up effects: First run after unwind differs from subsequent runs
  3. Statistical noise: Single samples have high variance

Recommendations

Priority Recommendation
1 Keep baseline defaults - They perform best for this workload
2 Do not increase WRITE_BUFFER_SIZE - Hurts throughput
3 Do not increase LEVEL_ZERO_STOP_WRITES_TRIGGER - Increases latency
4 Focus optimization efforts on trie computation, not RocksDB tuning

Individual Run Data

Run Ggas/s p90 (ms) p99 (ms)
baseline_run1 1.051 39.27 51.41
baseline_run2 0.905 60.87 98.97
baseline_run3 0.795 76.91 91.87
write_buffer_128mb_run1 0.735 51.24 78.59
write_buffer_128mb_run2 0.782 51.70 85.32
write_buffer_128mb_run3 0.785 54.25 79.56
l0_stop_100_run1 0.716 59.87 92.16
l0_stop_100_run2 0.799 55.84 82.16
l0_stop_100_run3 0.697 68.31 96.77

Generated from 9 benchmark runs on Feb 2, 2026

Raw
FINAL_REPORT_5x.md

RocksDB Parameter Tuning Results - 5-Run Average

Executive Summary

With 5 runs per configuration (vs 3 previously), the dramatic improvements normalized. The baseline shows high p99 variance, and the tuned configs primarily reduce tail latency variance rather than improving mean throughput.

Key Findings

Config Ggas/s Ggas σ p50 (ms) p99 (ms) p99 σ Verdict
baseline_edge 1.036 0.122 25.63 83.77 40.17 BASELINE
write_buffer_size_128mb 1.075 0.058 25.81 50.40 7.04 NEUTRAL
l0_stop_trigger_100 1.025 0.045 26.50 66.43 22.63 NEUTRAL

Analysis

write_buffer_size_128mb (128MB vs default 64MB)

  • Throughput: +3.75% (not enough to hit WIN threshold of +5%)
  • p99 latency: -39.83% improvement
  • Key benefit: Much lower variance (σ=7.04 vs 40.17) - more consistent performance
  • Mechanism: Larger memtables = fewer flushes during benchmark

l0_stop_trigger_100 (100 vs default 36)

  • Throughput: -1.05% (slight regression)
  • p99 latency: -20.70% improvement
  • Variance: Reduced from 40.17 to 22.63
  • Mechanism: Higher L0 tolerance prevents write stalls during bursts

Why Previous 3-Run Results Were Misleading

  1. Baseline had high p99 variance - some runs hit "big blocks" with expensive state changes
  2. With only 3 samples, lucky/unlucky block selection dominated
  3. 5 runs + more diverse blocks gave more representative results

Recommendation

NEUTRAL verdict for both configs - The improvements are real but modest:

  1. For production use: write_buffer_size_128mb provides more consistent tail latency with no throughput penalty. Consider if memory overhead is acceptable.

  2. For further tuning: The high baseline variance suggests block content variation dominates. Consider:

    • Testing with specific block ranges (heavy vs light blocks)
    • Combining both tuning parameters
    • Testing on different hardware profiles

Test Setup

  • Archive node datadir: /home/ubuntu/reth-rocksdb-21331
  • 100 blocks per run, 200 block unwind before each
  • Binary: reth + edge feature (RocksDB tables enabled)
  • 5 runs per configuration

Generated from reth-bench on 2026-02-02

Raw
FINAL_REPORT_CORRECTED.md

RocksDB Parameter Tuning - Corrected 3-Run Results

Executive Summary

After fixing the baseline (now using edge feature), both tuned configurations show significant improvements:

Config Ggas/s Ggas Δ% p90 Δ% p99 Δ% Verdict
baseline_edge 0.952 — — — —
write_buffer_size_128mb 1.078 +13.19% -21.78% -31.43% ✅ WIN
l0_stop_trigger_100 1.083 +13.77% -20.81% -40.44% ✅ WIN

Performance Summary (3-run average)

Config Ggas/s p50 (ms) p90 (ms) p99 (ms)
baseline_edge 0.952 28.65 48.13 82.19
write_buffer_size_128mb 1.078 25.09 37.65 56.35
l0_stop_trigger_100 1.083 26.22 38.11 48.95

Key Findings

1. Both Configurations Win

  • l0_stop_trigger_100: Best overall with +13.77% throughput and -40.44% p99 latency
  • write_buffer_size_128mb: +13.19% throughput and -31.43% p99 latency

2. Tail Latency Dramatically Improved

  • Baseline p99: 82.19 ms
  • l0_stop_trigger_100 p99: 48.95 ms (40% reduction!)
  • This indicates fewer write stalls with higher L0 trigger thresholds

3. RocksDB Tuning Does Help

With the correct baseline (edge feature enabled), RocksDB tuning shows real benefits. The previous "baseline wins" conclusion was wrong due to comparing edge vs non-edge builds.

Detailed Prometheus Metrics (3-run median average)

Save Blocks - Write Operations (seconds)

Metric Baseline 128mb Buffer L0 Stop 100
trie_updates 0.255 0.185 (-27%) 0.221 (-13%)
write_state 0.113 0.100 (-11%) 0.099 (-12%)
hashed_state 0.108 0.084 (-22%) 0.108 (+0%)

Storage Backend - Write Time (seconds)

Backend Baseline 128mb Buffer L0 Stop 100
MDBX 0.476 0.369 (-22%) 0.449 (-6%)
RocksDB 0.032 0.035 (+9%) 0.030 (-6%)
Static Files 0.010 0.009 (-10%) 0.010 (-0%)

Storage Backend - Commit Time (seconds)

Backend Baseline 128mb Buffer L0 Stop 100
MDBX Commit 778 µs 661 µs (-15%) 582 µs (-25%)
RocksDB Commit 33 µs 30 µs (-9%) 28 µs (-15%)
Static Files Commit 384 µs 395 µs (+3%) 381 µs (-1%)

Save Blocks - Total

Metric Baseline 128mb Buffer L0 Stop 100
Total Time (s) 0.476 0.369 (-22%) 0.449 (-6%)
Blocks per Save 10.67 10.00 9.67

Recommendations

Priority Configuration Change Impact
1 l0_stop_trigger_100 LEVEL_ZERO_STOP_WRITES_TRIGGER=100 Best p99, +14% throughput
2 write_buffer_size_128mb WRITE_BUFFER_SIZE=128<<20 -22% save_blocks time
3 Consider combining both Both changes together Potentially additive

Configuration Diffs

l0_stop_trigger_100 (RECOMMENDED)

// crates/storage/provider/src/providers/rocksdb/provider.rs
- const DEFAULT_LEVEL_ZERO_STOP_WRITES_TRIGGER: i32 = 36;
+ const DEFAULT_LEVEL_ZERO_STOP_WRITES_TRIGGER: i32 = 100;

write_buffer_size_128mb

// crates/storage/provider/src/providers/rocksdb/provider.rs
- const DEFAULT_WRITE_BUFFER_SIZE: usize = 64 << 20;
+ const DEFAULT_WRITE_BUFFER_SIZE: usize = 128 << 20;

Why Previous Results Were Wrong

The initial 3-run average showed baseline winning because:

  1. Different binaries: Baseline was built without edge feature (RocksDB disabled)
  2. Apples vs oranges: Compared MDBX-only baseline vs RocksDB-enabled tuned configs
  3. Binary size difference: 75MB (baseline) vs 82MB (tuned) - clear feature difference

After rebuilding baseline with --features edge, the comparison is now valid.

Individual Run Data

Run Ggas/s p90 (ms) p99 (ms)
baseline_edge_run1 0.803 - -
baseline_edge_run2 0.965 - -
baseline_edge_run3 1.087 - -
write_buffer_128mb_run1 1.070 - -
write_buffer_128mb_run2 1.123 - -
write_buffer_128mb_run3 1.040 - -
l0_stop_100_run1 1.140 - -
l0_stop_100_run2 1.086 - -
l0_stop_100_run3 1.024 - -

Generated from 9 benchmark runs with correct edge-enabled baseline on Feb 2, 2026

Raw
GIST_UPDATE.md

RocksDB Parameter Tuning - Rerun Results (Feb 2, 2026)

Rerun Summary

Following the initial experiments, I reran the two "winning" configurations to validate the results with detailed Prometheus metrics capture.

Benchmark Configuration

  • Blocks replayed: 100 blocks via reth-bench new-payload-fcu
  • Datadir: /home/ubuntu/reth-rocksdb-21331 (archive node)
  • Pre-benchmark: Unwound 200 blocks before each run
  • Metrics: Captured via Prometheus endpoint (:9001)

Rerun Results

Configuration Ggas/s p90 (ms) p99 (ms) p90 Δ% p99 Δ% Verdict
baseline_rerun 0.87 49.60 75.70 — — NEUTRAL
l0_stop_trigger_100 0.88 45.55 67.30 -8.17% -11.10% ✅ WIN
write_buffer_size_128mb 0.95 45.13 75.94 -9.02% +0.32% ✅ WIN

Detailed Prometheus Metrics

Save Blocks - Write Operations (seconds)

Metric Baseline write_buffer_128mb l0_stop_100
save_blocks_total_last 1.071 1.080 1.176
write_trie_updates_last 0.604 0.588 0.780
write_hashed_state_last 0.214 0.185 0.201
write_state_last 0.250 0.304 0.193

Storage Backend - RocksDB Time (seconds)

Metric Baseline write_buffer_128mb l0_stop_100
rocksdb_last 0 0.050 0.049
commit_rocksdb_last 0 ~0 ~0

Commit Time Breakdown (seconds)

Metric Baseline write_buffer_128mb l0_stop_100
commit_mdbx_last 46µs 91µs 46µs
commit_sf_last 2µs 2µs 1µs
commit_rocksdb_last 0 30ns 30ns

Key Observations

1. RocksDB Commit Time is Negligible

All configurations show RocksDB commit time < 100 nanoseconds. The bottleneck is not in the commit path.

2. Trie Updates Dominate Save Blocks

Across all configurations, write_trie_updates accounts for 50-66% of total save_blocks time. This confirms trie computation is the primary bottleneck.

3. Configuration Trade-offs

l0_stop_trigger_100 (LEVEL_ZERO_STOP_WRITES_TRIGGER=100):

  • Best p99 improvement: -11.10%
  • Reduces write stalls by delaying compaction triggers
  • Slightly higher trie update time (0.78s vs 0.60s baseline)
  • Recommended for latency-sensitive workloads

write_buffer_size_128mb (WRITE_BUFFER_SIZE=128MB):

  • Best throughput: 0.95 Ggas/s (+9.32% vs baseline)
  • Best p90 improvement: -9.02%
  • Larger memtables reduce flush frequency
  • Recommended for throughput optimization

Updated Recommendations

Priority Configuration Change Impact
1 l0_stop_trigger_100 LEVEL_ZERO_STOP_WRITES_TRIGGER=100 Best tail latency
2 write_buffer_size_128mb WRITE_BUFFER_SIZE=128<<20 Best throughput
— Keep defaults — Safe baseline

Configuration Diffs

l0_stop_trigger_100

// crates/storage/provider/src/providers/rocksdb/provider.rs
- const DEFAULT_LEVEL_ZERO_STOP_WRITES_TRIGGER: i32 = 36;
+ const DEFAULT_LEVEL_ZERO_STOP_WRITES_TRIGGER: i32 = 100;

write_buffer_size_128mb

// crates/storage/provider/src/providers/rocksdb/provider.rs
- const DEFAULT_WRITE_BUFFER_SIZE: usize = 64 << 20;
+ const DEFAULT_WRITE_BUFFER_SIZE: usize = 128 << 20;

Files

  • summary.csv - Aggregated metrics
  • */combined_latency.csv - Per-block latency
  • */metrics_after.txt - Full Prometheus dump
Raw
recommendation.md

RocksDB Parameter Tuning Results (5-run average)

Total experiments: 3

Summary Table

Config Ggas/s Mean (ms) p50 (ms) p90 (ms) p99 (ms) Runs
baseline_edge 1.036 28.50 25.63 39.95 83.77 5
write_buffer_size_128mb 1.075 27.10 25.81 37.74 50.40 5
l0_stop_trigger_100 1.025 29.12 26.50 40.11 66.43 5

Delta vs Baseline

Config Ggas Δ% Mean Δ% p50 Δ% p90 Δ% p99 Δ% Verdict
baseline_edge — — — — — BASELINE
write_buffer_size_128mb +3.75% -4.91% +0.71% -5.53% -39.83% ➖ NEUTRAL
l0_stop_trigger_100 -1.05% +2.16% +3.41% +0.40% -20.70% ➖ NEUTRAL

Generated by aggregate_5x.py from reth-bench output

Raw
summary.csv
experiment blocks mean_ms p50_ms p90_ms p99_ms ggas_per_sec blocks_per_sec p90_delta% p99_delta% gas_delta% verdict
baseline 100 26.84 24.38 40.22 57.04 1.07 36.21 +0.00 +0.00 +0.00 NEUTRAL
bytes_per_sync_16mb 100 33.75 32.47 46.16 70.93 0.92 28.89 +14.78 +24.37 -13.76 REJECT
bytes_per_sync_4mb_v2 100 1.52 1.46 2.17 2.95 17.50 586.94 -94.61 -94.83 +1540.37 WIN
l0_compaction_trigger_12 100 36.77 32.89 50.29 128.68 0.79 26.70 +25.04 +125.62 -26.07 REJECT
l0_compaction_trigger_8 100 32.62 29.31 45.97 88.69 0.92 29.92 +14.30 +55.50 -13.68 REJECT
l0_slowdown_trigger_40 100 30.32 27.21 44.37 60.62 0.96 32.17 +10.34 +6.28 -10.10 REJECT
l0_slowdown_trigger_60 100 32.42 30.28 43.62 72.17 0.92 30.13 +8.46 +26.53 -13.45 REJECT
l0_stop_trigger_100 100 29.68 28.59 45.48 54.23 1.00 32.82 +13.10 -4.91 -6.73 WIN
l0_stop_trigger_72 100 31.57 29.16 43.56 62.49 0.93 30.91 +8.32 +9.56 -12.56 REJECT
max_background_jobs_12 100 34.43 32.04 51.91 65.19 0.87 28.37 +29.09 +14.29 -18.93 REJECT
max_background_jobs_16 100 33.22 32.05 47.33 69.00 0.92 29.37 +17.69 +20.97 -14.15 REJECT
max_background_jobs_8_v2 100 32.06 29.32 40.56 95.90 0.95 30.47 +0.86 +68.14 -11.16 REJECT
max_write_buffer_number_4 100 31.29 28.29 45.56 87.73 0.95 31.13 +13.29 +53.82 -10.72 REJECT
max_write_buffer_number_6 100 33.77 31.00 51.97 76.67 0.88 28.95 +29.23 +34.43 -17.20 REJECT
pipelined_write_true 100 32.04 28.76 46.24 78.40 0.98 30.45 +14.98 +37.46 -7.84 REJECT
write_buffer_size_128mb 100 29.10 27.23 39.41 53.87 1.05 33.50 -2.00 -5.55 -1.40 WIN
write_buffer_size_256mb 100 27.07 27.00 40.20 104.91 1.06 34.58 -0.03 +83.95 -0.30 REJECT
write_buffer_size_512mb 100 26.85 26.29 40.37 64.42 1.08 36.26 +0.39 +12.95 +1.65 REJECT
Raw
SUMMARY.md

RocksDB Parameter Tuning Results

Summary

Baseline: 1.07 Ggas/s (default RocksDB settings)

Methodology:

  • 100 blocks replayed via reth-bench new-payload-fcu
  • Each experiment: unwind 200 blocks, start node, benchmark, capture metrics
  • Metrics enabled on port 9001

Key Findings

✅ Potential Improvements

Config Ggas/s p99 Δ Notes
write_buffer_size_128mb 1.05 -5.55% Modest p99 improvement
l0_stop_trigger_100 1.00 -4.91% Better tail latency

❌ Configurations to Avoid

Config Ggas/s Why
max_background_jobs_12/16 0.87-0.92 CPU contention hurts throughput
pipelined_write_true 0.98 +37% p99 regression
l0_compaction_trigger_12 0.79 +126% p99 regression (worst)
max_write_buffer_number_4/6 0.88-0.95 Memory pressure, worse latency

➖ Neutral (no significant change)

Config Ggas/s
write_buffer_size_256mb 1.06
write_buffer_size_512mb 1.08

Recommendations

  1. Keep defaults - Most tuning made things worse
  2. Consider write_buffer_size_128mb - Small p99 improvement
  3. Avoid increasing background jobs - Causes contention
  4. Avoid aggressive L0 triggers - Creates latency spikes

Persistence Metrics (from baseline)

Metric Time (s)
save_blocks_total 0.389
write_trie_updates 0.202
write_hashed_state 0.091
write_state 0.095
commit_rocksdb ~0

Observation: RocksDB commit time is negligible. Bottleneck is trie updates (52% of save_blocks time).

Files

  • summary.csv - All experiments with metrics
  • */combined_latency.csv - Per-block latency data
  • */metrics_final.txt - Prometheus metrics snapshots
Raw
summary_rerun.csv
experiment blocks mean_ms p50_ms p90_ms p99_ms ggas_per_sec blocks_per_sec p90_delta% p99_delta% gas_delta% verdict
baseline_rerun 100 34.19 31.87 49.60 75.70 0.87 28.59 +0.00 +0.00 +0.00 NEUTRAL
l0_stop_trigger_100_rerun 100 34.24 30.54 45.55 67.30 0.88 28.57 -8.17 -11.10 +1.27 WIN
write_buffer_size_128mb_rerun 100 31.50 29.46 45.13 75.94 0.95 30.98 -9.02 +0.32 +9.32 WIN
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment