RavenDB Embedded Faster Tests

README.md

RavenDB Embedded with Tests (Reqnroll)

On my original test setup with .NET Framework 4.7.2 and RavenDB 6, it took about 9 minutes to execute my 760 tests.

After upgrading to .NET 10, that by itself reduced the total time to 6-7 minutes.

I spent a couple days looking at how I was executing my tests and using the RavenDB Embedded server and found some ways to improve that even further.

As of right now, that is down to 2.4 minutes. (a 4X reduction).

Test Setup

• Reqnroll
• NUnit 3
• RavenDB 6.2 (Embedded)
• .NET 10
• 6-core CPU
• 64GB RAM

Optimizations

• Ensure DocumentStore is scoped and lifecycle is tied to each test (scenario)
• Create a global DocumentStore to handle server-wide operations (disposed at end of test run)
• Create a new database per test
• When the test ends, delete it (fire-and-forget) otherwise disk usage balloons
• Lazily initialize the DocumentStore so tests that don't need it, don't instantiate it
• Clean/delete any existing RavenDB directory in case there are files leftover
• Leverage Reqnroll + NUnit parallelization
• Run in memory
• Disable topology updates/cache (single node)
• Use Developer license (uses all available cores)

While my test harness is using Reqnroll and NUnit, you can apply these patterns to other test frameworks.

Lifecycle and Scopes

• DbTestHarness -- Singleton, created by Reqnroll
• DbContext -- Scoped (scenario/test), injected by Reqnroll through Microsoft.DependencyInjection
• TestingContext -- Scoped, but static CreateServices is invoked once for the test run

Test Hooks

• [BeforeTestRun] -- happens before all tests run
• [BeforeScenario] -- happens before a standalone test runs
• [AfterScenario] -- happens after the test executes
• [AfterTestRun] -- happens at the end

Dependencies/services can be added using Reqnroll.Microsoft.Extensions.DependencyInjection and the [ScenarioDependencies] attribute.

Notes

Things I tried that didn't seem to make a difference:

• Use explicit List of indexes and ExecuteIndexes (no discernible difference)
• Create a Snapshot backup on first test and restore for each subsequent test without re-initializing docstore (test run took >20 min!)
• Don't delete databases (eats up disk quickly)
• Delete batches of databases at a time (<10s difference)

Definitely the biggest win was parallelization, which cut the 6.4 mins into 2.4 mins on my machine (6-core, 64GB RAM).

DbContext.cs

using Newtonsoft.Json;
using Raven.Client.Documents;
using Raven.Client.Documents.Conventions;
using Raven.Client.Documents.Indexes;
using Raven.Client.Documents.Session;
using Raven.Client.Json.Serialization.NewtonsoftJson;
using Raven.Embedded;
using Reqnroll;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text.RegularExpressions;
using System.Threading;

namespace Tests.Support
{
    public class DbContext : IDisposable
    {
        private readonly ScenarioContext _context;

        public Lazy<IDocumentStore> Store;

        public string Database { get; }

        public DbContext(ScenarioContext context)
        {
            _context = context;

            Store = new (GetDocumentStore, LazyThreadSafetyMode.ExecutionAndPublication);     
            Database = Guid.NewGuid().ToString("N").ToLowerInvariant();
        }

        private IDocumentStore GetDocumentStore()
        {
            var documentStore = EmbeddedServer.Instance.GetDocumentStore(new DatabaseOptions(Database)
            {
                Conventions = new DocumentConventions()
                {
                    DisableTopologyCache = true,
                    DisableTopologyUpdates = true,
                    MaxNumberOfRequestsPerSession = 100,
                    HttpCompressionAlgorithm = Raven.Client.Http.HttpCompressionAlgorithm.Zstd,
                    Serialization = new NewtonsoftJsonSerializationConventions()
                    {
                        CustomizeJsonSerializer = serializer =>
                        {
                            serializer.TypeNameHandling = TypeNameHandling.None;
                            serializer.NullValueHandling = NullValueHandling.Ignore;
                        }
                    }
                }
            }).Initialize();

            // Wait for non-stale results during unit test execution
            // otherwise some tests may intermittently fail
            documentStore.OnBeforeQuery += (sender, beforeQueryExecutedArgs) =>
            {
                if (_context.ScenarioInfo.CombinedTags.Contains("SkipWaitForNonStaleResults"))
                {
                    return;
                }
                beforeQueryExecutedArgs.QueryCustomization.WaitForNonStaleResults();
            };

            IndexCreation.CreateIndexes(typeof(Your_Index).Assembly, documentStore);

            SeedDb(documentStore);

            return documentStore;

        }

        private static void SeedDb(IDocumentStore store)
        {
            using (var session = store.OpenSession())
            {
                // seed
            }
        }

        public void Dispose()
        {
            GC.SuppressFinalize(this);
            if (Store.IsValueCreated)
            {
                Store.Value.Dispose();
            }
        }
    }
}

DbTestHarness.cs

using Newtonsoft.Json;
using Raven.Client.Documents;
using Raven.Client.Documents.Conventions;
using Raven.Client.Json.Serialization.NewtonsoftJson;
using Raven.Client.ServerWide.Operations;
using Raven.Client.Util;
using Raven.Embedded;
using Reqnroll;
using Reqnroll.UnitTestProvider;
using System;
using System.Diagnostics;
using System.IO;
using System.Linq;
using System.Security.Cryptography.X509Certificates;
using System.Threading.Tasks;

namespace Tests.Support
{

    [Binding]
    public class DbTestHarness
    {
        private static IDocumentStore GlobalStore { get; set; }

        [BeforeTestRun]
        public static void BeforeAll()
        {
            if (Directory.Exists("RavenDB"))
            {
                Directory.Delete("RavenDB", true);
            }

            EmbeddedServer.Instance.StartServer(new ServerOptions()
            {
                FrameworkVersion = "8.x",
                Licensing =
                {
                    DisableAutoUpdate = true,
                    DisableAutoUpdateFromApi = true,
                    EulaAccepted = true,
                    ThrowOnInvalidOrMissingLicense = true,
                    License = /* Developer License */
                },
                CommandLineArgs = ["--RunInMemory=true"]
            });

            GlobalStore = GetGlobalDocumentStore();

            if (Debugger.IsAttached)
            {
                EmbeddedServer.Instance.OpenStudioInBrowser();
            }
        }

        [AfterScenario]
        public static void TeardownDatabase(ScenarioContext scenario, DbContext db, IReqnrollOutputHelper logger)
        {
            var parameters = new DeleteDatabasesOperation.Parameters
            {
                DatabaseNames = [db.Database],
                HardDelete = true,
                TimeToWaitForConfirmation = null
            };

            GlobalStore.Maintenance.Server.SendAsync(new DeleteDatabasesOperation(parameters))
                .ContinueWith(t => { 
                    if (t.IsFaulted)
                    {
                        logger.WriteLine($"[Error] Failed to delete {db.Database} databases for scenario '{scenario.ScenarioInfo.Title}': {t.Exception.Message}");
                    }
                }, TaskScheduler.Default);
        }

        [AfterTestRun]
        public static void AfterAll()
        {
            EmbeddedServer.Instance.Dispose();
        }

        public static IDocumentStore GetGlobalDocumentStore()
        {
            var embeddedServerUrl = AsyncHelpers.RunSync(() => EmbeddedServer.Instance.GetServerUriAsync());

            var documentStore = new DocumentStore()
            {
                Urls = [embeddedServerUrl.AbsoluteUri],
                Conventions =
                {
                    DisableTopologyCache = true
                }
            };

            documentStore.Initialize();

            return documentStore;
        }
    }
}

NUnitConfig.cs

using NUnit.Framework;

[assembly: Parallelizable(ParallelScope.Children)]

TestServices.cs

using Microsoft.Extensions.DependencyInjection;
using Raven.Client.Documents;
using Raven.Client.Documents.Session;
using Reqnroll.Assist;
using Reqnroll.Microsoft.Extensions.DependencyInjection;
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;

namespace Tests.Support;

public class TestingContext
{
    [ScenarioDependencies]
    public static IServiceCollection CreateServices()
    {
      var services = new ServiceCollection();

      services.AddScoped<DbContext>();
      services.AddScoped(ctx => ctx.GetRequiredService<DbContext>().Store.Value);
      services.AddScoped(ctx => ctx.GetRequiredService<IDocumentStore>().OpenSession());
      services.AddScoped(ctx => ctx.GetRequiredService<IDocumentStore>().OpenAsyncSession());
    }
}