A Test Failure That Looked Like a Race Condition

test-investigation_blog-post.md

A Test Failure That Looked Like a Race Condition

Generated by Opus 4.6

The Symptom

Our integration tests started failing. Not all of them, not consistently, and not when run individually. Only when multiple tests ran together, some would fail - some with unexpected HTTP status codes, others with assertion errors where the API returned different data than expected.

It looked like a classic race condition. It appeared out of nowhere. It worked on other developers' machines. It worked in CI pipelines. Only my local machine was affected.

The Investigation

Our integration tests use WebApplicationFactory<Program> to spin up an in-memory test server for each test. Each test creates its own factory, runs HTTP requests against it, and disposes the factory when done. The tests share a single database via a collection fixture and run sequentially.

The failing tests all had one thing in common: they called endpoints decorated with [Authorize]. The authentication was silently failing, causing the API to behave as if no user was logged in - returning 401 on protected endpoints, or serving different data because the user identity was missing.

The breakthrough came from xUnit v3's [CaptureConsole] attribute (applied via [assembly: CaptureConsole]). In xUnit v3, console output from the test host is not captured by default. Adding this attribute made the ASP.NET Core logs visible in the test output, and they revealed everything:

info: Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler[1]
      Failed to validate the token.
      System.AggregateException: An error occurred while writing to logger(s).
      (Cannot access a disposed object. Object name: 'EventLogInternal'.)

info: Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler[7]
      router was not authenticated. Failure message: An error occurred
      while writing to logger(s). (Cannot access a disposed object.
      Object name: 'EventLogInternal'.)

info: Microsoft.AspNetCore.Authorization.DefaultAuthorizationService[2]
      Authorization failed. These requirements were not met:
      DenyAnonymousAuthorizationRequirement: Requires an authenticated user.

The authentication wasn't failing because of a bad token. It was failing because the logging infrastructure crashed while trying to write a JWT validation log entry.

The Root Cause

Here is what was happening:

1. On Windows, ASP.NET Core registers the EventLog logging provider by default. This provider writes to the Windows Event Log through an EventLogInternal handle.

2. Our JWT bearer handler (the router authentication scheme) validates tokens on every request. During validation, Microsoft.IdentityModel logs diagnostic messages through IdentityModelEventSource, which bridges to the standard ILogger infrastructure.

3. When a test finishes and its WebApplicationFactory is disposed, the host's ILoggerFactory is disposed, which disposes all logging providers, including EventLogLoggerProvider. This closes the EventLogInternal OS handle.

4. Microsoft.IdentityModel maintains a static LogHelper that was initialized with a logger from the first factory's DI container. When subsequent test factories try to validate JWT tokens, the IdentityModel logging still references the original (now disposed) logger. The disposed EventLogInternal handle throws ObjectDisposedException.

5. This exception propagates as an AggregateException from the logging layer. The JwtBearerHandler treats it as a token validation failure. The [Authorize] middleware sees an unauthenticated user and returns 401.

The critical detail: the token validation itself would have succeeded. The authentication only failed because the logging of the validation result crashed.

Why It Only Affected My Machine

The Windows EventLog provider only creates EventLogInternal handles when the Event Log source is registered on the machine. On my machine, the .NET Runtime source was registered (verified with [System.Diagnostics.EventLog]::SourceExists('.NET Runtime')). On my colleague's machine and in the CI pipeline (Linux containers), it wasn't - so the EventLog provider was essentially a no-op, and the disposed handle issue never triggered.

The Fix

Remove logging providers that tests don't need. In the WebApplicationFactory setup:

var webAppFactory = new WebApplicationFactory(configuration)
    .WithWebHostBuilder(builder =>
    {
        builder.ConfigureLogging(logging =>
        {
            logging.ClearProviders();
            logging.AddConsole();
        });

        builder.ConfigureServices(services =>
        {
            // ... test service overrides
        });
    });

This eliminates the EventLog provider entirely, keeping only console output for test diagnostics. No disposed handles, no ghost failures.

Takeaways

• Not every intermittent failure is a race condition. This looked like shared state between tests, and it was - but the shared state was a static logger reference in Microsoft.IdentityModel, not anything in our code.
• Platform-specific defaults can bite you. The EventLog provider is a Windows-only default. Tests that pass on Linux CI can fail on Windows dev machines (or vice versa) due to platform-specific logging providers.
• xUnit v3's [CaptureConsole] attribute is invaluable. Without it, the ASP.NET Core host logs are invisible in test output. The authentication failure reason was logged clearly - we just couldn't see it. If you're using xUnit v3 with WebApplicationFactory, add [assembly: CaptureConsole] to your test project.
• Strip down your test host. Test hosts don't need EventLog, EventSource, or Debug logging providers. ClearProviders() + AddConsole() gives you clean diagnostics without side effects.