I have a .NET 5 solution with an API project and two separate test projects (one is bare unit tests, other is integration/e2e tests) based on XUnit.
As part of the e2e tests I seed the database with some test data.
Till yesterday, all tests succeeded. Today, I have added a few more tests to my suite and the tests started to behave inconsistently:
- Running the full suite locally from Visual Studio succeeded, so I had pushed confidently to Azure DevOps for pipelining
- Running
dotnet testlocally succeedes - Running the newly added tests (3) individually of course succeeds
- On Azure DevOps, some old tests fail. Interestingly, two consecutive runs yielded the same tests failing. I couldn't run a third execution because I drained all the pipelines budget
Note that today Azure DevOps is experiencing an incident in the European area
The errors are different. In one case a REST method invoking a database COUNT that is supposed to return 6 returns 0(!), while in another case I have exception
System.ArgumentException : An item with the same key has already been added. Key: 1
at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value)
at Microsoft.EntityFrameworkCore.InMemory.Storage.Internal.InMemoryTable`1.Create(IUpdateEntry entry)
at Microsoft.EntityFrameworkCore.InMemory.Storage.Internal.InMemoryStore.ExecuteTransaction(IList`1 entries, IDiagnosticsLogger`1 updateLogger)
at Microsoft.EntityFrameworkCore.InMemory.Storage.Internal.InMemoryDatabase.SaveChanges(IList`1 entries)
at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChanges(IList`1 entriesToSave)
at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChanges(DbContext _, Boolean acceptAllChangesOnSuccess)
at Microsoft.EntityFrameworkCore.Storage.NonRetryingExecutionStrategy.Execute[TState,TResult](TState state, Func`3 operation, Func`3 verifySucceeded)
at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChanges(Boolean acceptAllChangesOnSuccess)
at Microsoft.EntityFrameworkCore.DbContext.SaveChanges(Boolean acceptAllChangesOnSuccess)
at Microsoft.EntityFrameworkCore.DbContext.SaveChanges()
at Web.TestUtilities.Seeder.Seed(SendGridManagerContext dbContext) in D:\a\1\s\TestUtilities\Seeder.cs:line 27
at Web.WebApplicationFactory.InitDatabase() in D:\a\1\s\WebApplicationFactory.cs:line 164
at TestFixtureBase..ctor(WebApplicationFactory testFactory, ITestOutputHelper outputHelper) in D:\a\1\s\TestFixtureBase.cs:line 27
at Web.Tests.ControllerTests..ctor(WebApplicationFactory testFactory, ITestOutputHelper outputHelper) in D:\a\1\s\Tests\ControllerTests.cs:line 19
(I redacted the stack trace a little)
Judging from the relationship between the two errors, I suspect that the method I seed the database is not correct.
Here is the showdown.
I created a WebApplicationFactory class
public class MyWebApplicationFactory : WebApplicationFactory<Startup>
{
public ITestOutputHelper Output { protected get; set; }
protected override void ConfigureWebHost(IWebHostBuilder builder)
{
base.ConfigureWebHost(builder);
builder.UseUrls("https://localhost:5001")
.ConfigureLogging(logging => logging
.ClearProviders()
.AddXUnit(Output)
.AddSimpleConsole())
.ConfigureTestServices(services =>
{
services.AddLogging(log =>
log.AddXUnit(Output ?? throw new Exception(
$"{nameof(Output)} stream must be set prior to invoking configuration. It should be done in the test base fixture")));
services.Remove(services.SingleOrDefault(service =>
service.ServiceType == typeof(DbContextOptions<MyDbContext>)));
services.AddDbContext<MyDbContext>(options =>
options.UseInMemoryDatabase("sg_inmemory"));
services.Configure<JwtBearerOptions>(.......
services.AddSingleton(.......
})
;
}
protected override IHostBuilder CreateHostBuilder()
{
return base.CreateHostBuilder()
.ConfigureLogging(log =>
log.AddXUnit()
);
}
public HttpClient CreateHttpClientAuthenticatedUsingMicrosoftOidc()
{
}
public async Task<HttpClient> CreateHttpClientAuthenticatedUsingPrivateOidcAsync(
CancellationToken cancellationToken = default)
{
}
public void InitDatabase()
{
using var scope = Services.CreateScope();
using var dbContext = scope.ServiceProvider.GetRequiredService<MyDbContext>();
dbContext.Database.EnsureCreated();
dbContext.Seed(); //Extension method defined elsewhere
}
public void DestroyDatabase()
{
using var scope = Services.CreateScope();
using var dbContext = scope.ServiceProvider.GetRequiredService<MyDbContext>();
dbContext.Database.EnsureDeleted();
}
}
To really seed the database I created my own extension method
public static void Seed(this MyDbContext dbContext)
{
using var tx = new TransactionScope();
dbContext.MyEntity.AddRange(GetSeedData());
dbContext.SaveChanges();
tx.Complete();
}
And to invoke DB init/destroy cycle on testing I leverage constructor and disposer
public class ControllerTests : TestFixtureBase
{
public ControllerTests(MyWebApplicationFactory testFactory, ITestOutputHelper outputHelper)
: base(testFactory, outputHelper)
{
}
// Let's talk about test code later
}
This class inherits from the same fixture
public abstract class TestFixtureBase : IClassFixture<MyWebApplicationFactory>, IDisposable
{
private CancellationTokenSource _cancellationTokenSource => Debugger.IsAttached
? new CancellationTokenSource()
: new CancellationTokenSource(TimeSpan.FromMinutes(2));
protected MtWebApplicationFactory TestFactory { get; }
protected HttpClient PublicHttpClient => TestFactory.CreateClient();
protected CancellationToken CancellationToken => _cancellationTokenSource.Token;
protected TestFixtureBase(MyWebApplicationFactory testFactory,
ITestOutputHelper outputHelper)
{
TestFactory = testFactory;
TestFactory.Output = outputHelper;
TestFactory.InitDatabase();
}
~TestFixtureBase() => Dispose(false);
public void Dispose()
{
Dispose(disposing: true);
GC.SuppressFinalize(this);
}
protected virtual void Dispose(bool disposing)
{
if (disposing)
{
TestFactory.DestroyDatabase();
PublicHttpClient.Dispose();
_cancellationTokenSource.Dispose();
}
}
}
Newly added tests resemble existing tests. Please note that the tests that are failing are old tests that passed before my push, and I only added functionality
[Fact]
public async Task TestStatistics_ReturnsNull() // tests that REST controller returns empty JSON array []
{
var client = TestFactory.CreateHttpClientAuthenticatedUsingMicrosoftOidc();
var message = await client.GetAsync(
"/api/v1/secure/Controller/statistics/?Environment.IsNull=true",
CancellationToken); //shall return empty
message.EnsureSuccessStatusCode();
var result =
await message.Content.ReadFromJsonAsync<List<StatisticsDTO>>(
cancellationToken: CancellationToken);
Assert.NotNull(result);
Assert.Empty(result);
}
[Fact]
public async Task TestCount_NoFilter()
{
var client = TestFactory.CreateHttpClientAuthenticatedUsingMicrosoftOidc();
var message = await client.GetAsync("/api/v1/secure/Controller/count",
CancellationToken);
message.EnsureSuccessStatusCode();
var result = await message.Content.ReadFromJsonAsync<int>();
int expected = Seeder.GetSeedData().Count(); //The seeder generates 6 mock records today
Assert.Equal(expected, result); //Worked till last push
}
My investigation and question
I suspect that due to the asynchronicity there could be some point where the test runs exactly after another disposing test has cleared the database, and for the duplicate key error I am having a hard time understanding it.
It is my educated guess that I am doing something wrong in seeding the database.
Currently my requirement is that the REST application is started with some mocked records ready in its in-memory database. While the database is in memory and is ephemeral, I tried to do the educated exercise to clear up the databse as this code will be part of examples to share with fellow developer-students in order to teach them correct patterns. Please allow me to insist on clearing an in memory database.
And finally, there is nothing special in the pipeline code (I did dotnet restore in the beginning)
- task: DotNetCoreCLI@2
displayName: "Execute tests"
inputs:
command: 'test'
projects: |
*Tests/*.csproj
arguments: '--no-restore'
options.UseInMemoryDatabase(Guid.NewGuid().ToString())? – Fabio