I was seriously considering ditching Entity Framework in favor or a light-weight micro ORM such as Dapper, but then I made a couple lines code change which resulted in reduction of bulk load time from 03:27:44 (yes, that's three hours) to 00:02:19. Here's that simple change that produced 90 times performance increase.

To oversimplify things a bit, I had module with multiple metrics. One-to-many, essentially. One module can have multiple metrics and one metric is used in only one module:

public class Module  
{
  public int ModuleId { get; set; }
  public string Name { get; set; }  
  public virtual List<Metrics> Metrics { get; set; }
}

public class Metrics  
{
    public int MetricsId { get; set; }

    public int DateId { get; set; }
    public virtual DimDate Date { get; set; }

    public int? ModuleId { get; set; }
    public virtual Module Module { get; set; }

    public int? MaintainabilityIndex { get; set; }
    public int? CyclomaticComplexity { get; set; }        
    public int? CodeCoverage { get; set; }                
}

During data collection large amount is data is inserted into the database. There are multiple modules each with its own metrics for a given date. The skeleton of my insert code was this:

var module = new Module();  
module.Name = name;

metrics = new Metrics();  
metrics.Module = module;  
metrics.Date = dimDate;  
metrics.CodeCoverage = coverage;

module.Metrics.Add(metrics);  

This worked ok, but as we starting pilling up more data the load speed began decreasing. I improved that with better change tracking, but as size of data kept on increasing the bulk load process came to a screeching halt with System.OutOfMemoryExceptions.

I started debugging it with Sql Profiler and found no activity at all. The loader was simply running doing no SQL calls while taking more and more memory. And here's the line where it was hanging:

// ...
module.Metrics.Add(metrics);  

It turned out that before inserting a module I checked if it already exists and a typical module had over hundred thousands of metrics. Adding one more extra metrics even with change tracking off was taking EF forever.

The solution was this one-line change to insert metrics directly:

var module = new Module();  
module.Name = name;

metrics = new Metrics();  
metrics.Module = module;  
metrics.Date = dimDate;  
metrics.CodeCoverage = coverage;

_context.Metrics.Add(metrics); // Insert metrics directly  

I was surprised by magnitude of difference that it produced, although I still don't understand why it was taking EF so much time to add one more element into a child collection especially with change tracking off.