Entity Framework ChangeTracker Stream and Save to Query Afterwards

Question

I want to validate when certain Entity is Modified/Updated/Inserted/Deleted in Entity Framework (with database). (eg ProductType Entity table)

** Would like Change Tracker to be saved and queried later, as 'the track changes are lost if they are not saved before the DbContext object is destroyed.' Only need ChangeTracker to be retained for couple hours most to analyze, and be saved in MessageQueue if needed. MemoryCache can also work.

1) Someone recommend using LoggerFactory:

optionsBuilder.UseLoggerFactory(loggerFactory) 

https://www.entityframeworktutorial.net/efcore/logging-in-entityframework-core.aspx

However, it can be hard to parse as updates, modifications, deletes can be done with a Join, aliases, brackets, with intricate SQL statements often generated by EF. So text parsing may not be accurate.

INSERT INTO [Students] ([DateOfBirth], [GradeId], [Height], [Photo], [Stud
entName], [Weight])
VALUES (@p0, @p1, @p2, @p3, @p4, @p5);
SELECT [StudentID]
FROM [Students]
WHERE @@ROWCOUNT = 1 AND [StudentID] = scope_identity();
info: Microsoft.EntityFrameworkCore.Database.Command[200101]
Executed DbCommand (68ms) [Parameters=[@p0='' (DbType = DateTime2), @p1=''
(DbType = Int32), @p2='0', @p3='' (Size = 8000) (DbType = Binary), @p4='Steve'
(Size = 4000), @p5='0'], CommandType='Text', CommandTimeout='30']
SET NOCOUNT ON;
INSERT INTO [Students] ([DateOfBirth], [GradeId], [Height], [Photo], [Stud
entName], [Weight])
VALUES (@p0, @p1, @p2, @p3, @p4, @p5);
SELECT [StudentID]
FROM [Students]
WHERE @@ROWCOUNT = 1 AND [StudentID] = scope_identity();
A data reader was disposed.
dbug: Microsoft.EntityFrameworkCore.Database.Transaction[2002

2) Option 2 is ChangeTracker, Others recommended using ChangeTracker before SaveChanges statement, because its much cleaner (see below query). However executing this before every SaveChanges statement, considering we have 500 transactions /per second affects application performance speed.

So is there any method to stream and retain ChangeTracker history to a log after save changes, where one can query easier what entities changed? Its optimal to find out after transaction is complete, without blocking transactions.

var Entries = context.ChangeTracker
           .Entries()
           .Where(x => x.State == EntityState.Modified || x.State == EntityState.Deleted|| x.State == EntityState.Added)
           .Select(x =>x.Entity)
           .ToList();

https://entityframework.net/change-tracker

Answer 1

@DerrikRodgers, time required to check if Entity is an instance of a certain type is not comparable to a transaction time. 500 transactions/second will work absolutely fine having code you provided.

public void ReportChanges()
{
  var entities=ChangeTracker
           .Entries()
           .Where(x => x.Entity is Product)
           .Where(x => x.State == EntityState.Modified || x.State == EntityState.Deleted || x.State == EntityState.Added)
           .GroupBy(x=>x.State, x => x.Entity as Product)
           .ToList();
  ... // Producer/Consumer to do not block current thread and control concurrency
}

Get performance baseline with a synthetic test, something like this:

var cc = new ApplicationDbContext();
for (int i = 0; i < 100; i++)
    cc.Users.Add(new ApplicationUser() { Id = i });

var sw = Stopwatch.StartNew();
for (int i = 0; i < 1000; i++)
    cc.ReportChanges();
sw.Stop();
Console.WriteLine($"1000 times to report 100 entities took {sw.ElapsedMilliseconds}ms. Rate {1000*100/(sw.ElapsedMilliseconds/1000.0)} items/s");
Console.ReadKey();

// 1000 times to report 100 entities took 461ms. Rate 216919.739696312 items/s

Another question is what you are going to do with the filtered entities. That thing might be slow. For example if you'll try to log them all and you don't use asynchronous sinks/targets in your logger, this will be slow. In this case you can implement Producer/Consumer pattern and pipe filtered entities to another consumer which will dispatch them in a different thread and do something long. Rx.NET might be very helpful in this case.

Instead of producer/consumer you can just start a Task, but that might lead to thread pool starvation and I'd recommend explicitly control parallelism of your 'slow' operations.