How to tackle huge amount of memory allocation on ASP.NET Core 2.0 application EF Core usage?

Question

I stumbled across a very strange issue. Whenever the web application starts, dotnet.exe has a decent memory usage (about 300M). However, when it touches some parts (I sense it is related to EF Core usage), it allocated a huge amount of memory in short amount of time (about 8GB in 2-3 seconds).

This memory usage takes about 10-15 seconds, after that the memory settles at about 600M and it operates normally.

I tried both dottrace and builtin Diagnostics Tools to understand what allocates so much memory, but cannot find anything meaningful:

Dottrace for the most memory consuming thread, but I could not catch a snapshot of the memory while being very high (it only shows me about ~1GB in total and about 800M managed memory).

VS Diagnostic Tools delta between baseline and immediately after the memory spiked

How can I can I get to the root cause of this memory allocation? It is strange that it does not seem to be a leak, since the memory is eventually deallocated.

Question: How to tackle huge amount of memory allocation on ASP.NET Core 2.0 application EF Core usage?

I think the issue is indeed related to number of injected services, but first I will provide more about the application architecture. I rely on a bunch of generic repositories that are injected in a scoped data access which is creates a wrapper upon the data context and help with saving multiple information (for various repositories) in a single transaction if needed:

Repository<T> : IRepository<T>
    <- DbContext

ScopedDataAccess : IScopedDataAccess
    <- DbContext
    <- logging service
    <- dozens of IRepository<T>

Everything is "scoped":

services.AddScoped<IScopedDataAccess, ScopedDataAccess>();
services.AddScoped(typeof(IRepository<>), typeof(Repository<>));

I removed about half of the injected repositories in ScopedDataAccess and the required memory reduced to about a half.

What is more strange is that the Diagnostic Tools shows a decrease of memory without being directly tied to a GC kicking in (see the following graph, GC is the upper yellow sign):

Also, I double checked that I have stopped all async jobs (e.g. Quartz).

What is actually happening in your application? Memory is allocated as necessary, so assuming you aren't leaking memory, then that means you're doing something that actually requires large amounts of memory. That could be pulling vast quantities of database results (i.e. millions of records at a time), maybe large file uploads (without spooling to the filesystem), or even just seemingly innocuous things like over-selecting in a query for a table that houses binary blob data. If you don't explicitly exclude the blob column, then you're actually downloading all that data into memory. — Chris Pratt
I have checked using the SQL profiler and there is virtually no activity there. I read no custom files during the initialization. However, from the trace data it looks like a thread is generating (emitting) some metadata related to dependency injection / EF Core. The allocation is not done explicitly in my code, because I can use step by step and see the memory being allocated by something outside my code base. — Alexei - check Codidact
That doesn't mean it's not initiated by your codebase. The examples I gave were just that: examples. It's certainly not an exhaustive list of possible culprits. The general idea is that you need to look at areas of your code where you'd actually be using large amounts of memory. Any type of file handling is a potential. Otherwise, you simply want to look at large amounts of object allocations. That happens most often with EF queries, simply because EF materializes the result in an object graph, but you could be instantiating your own large objects, just as well. — Chris Pratt
@ChrisPratt - I have identified a specific service that seems to be related to memory allocation and if I commented out all functionality, but left one injected dependency which is not used at all. About 3GB are still allocated and eventually deallocated. So, my assumption is that it is related to DI somehow. — Alexei - check Codidact
Well, it's not DI, per se. It could be a particular service you're registering with the DI container, but dependency injection itself is not the issue. Update your question with what you've found (and continue to do so). — Chris Pratt

Alexei - check Codidact Alexei - check Codidact · Accepted Answer · 2019-04-22T09:11:49

Not a full answer, but I have done the following and greatly reduce the memory (and CPU usage):

simplified dependency graph by splitting large services requiring lots of injected services
upgraded to ASP.NET Core 2.1

The last step had the most visible effect and my Diagnostics Tools now shows a much friendlier graph:

How to tackle huge amount of memory allocation on ASP.NET Core 2.0 application EF Core usage?

1 Answers