Entity Framework: Joining in memory data with DbSet

转载自：https://ilmatte.wordpress.com/2013/01/06/entity-framework-joining-in-memory-data-with-dbset/

The argument of this post is relevant to people using Entity Framework and needing to filter data coming from a Database with a list of in-memory data.

In this article I will try to start summarizing what is well explained in a good article by Michael Hompus, adapting his example to Entity Framework Code First and adding a second issue for distracted programmers.
If you want his clear explanation I suggest you to go for his post:

http://blog.hompus.nl/2010/08/26/joining-an-iqueryable-with-an-ienumerable/

I will start from Michael’s article with the difference that my example will use Entity Framework Code First.
I will try to underline the 2 issues involved with this topic.

It could happen that you want to filter some data bases on a list of values and you want to filter them while querying, in order to avoid loading unuseful data in memory.

In my example I suppose that you already know Entity Framework Code First.
I explicitly invoke a DatabaseInitializer to be sure to create a freshly new database.
I previously created a DbContext with a single Dbset of Customer entities:

public class CustomerContext : DbContext
{
public DbSet Customers { get; set; }
}
and I created the Customer entity:

public class Customer
{
public int Id { get; set; }

public string Name { get; set; }

public string Surname { get; set; }
}
I created a Console application to test the Linq queries I want to analyze.
I want to filter Customers base on their Ids. I want only three of them:

private static void MainMethod()
{
try
{
var customerIds = new List {1,5,7};
using (var context = new CustomerContext())
{
var initializer = new DropCreateDatabaseAlways();
initializer.InitializeDatabase(context);

var customers = from customer in context.Customers
join customerId in customerIds
on customer.Id equals customerId
select customer;
var result = customers.ToList();
}
}
catch (Exception exception)
{
Console.WriteLine(exception.ToString());
}
}
and this is the resulting query:

SELECT
[Extent1].[Id] AS [Id],
[Extent1].[Name] AS [Name],
[Extent1].[Surname] AS [Surname]
FROM [dbo].[Customers] AS [Extent1]
INNER JOIN (SELECT
[UnionAll1].[C1] AS [C1]
FROM (SELECT
cast(1 as bigint) AS [C1]
FROM ( SELECT 1 AS X ) AS [SingleRowTable1]
UNION ALL
SELECT
cast(5 as bigint) AS [C1]
FROM ( SELECT 1 AS X ) AS [SingleRowTable2]) AS [UnionAll1]
UNION ALL
SELECT
cast(7 as bigint) AS [C1]
FROM ( SELECT 1 AS X ) AS [SingleRowTable3]) AS [UnionAll2] ON [Extent1].[Id] = [UnionAll2].[C1]
As you can see, a UNION ALL statement is issued for any item in the collection.
It’s time for the first issue:

1) Sql Server has a maximum limit of depth for nested subqueries: 32 (http://msdn.microsoft.com/en-us/library/ms189575%28v=sql.105%29.aspx)
Then, if your in-memory collection gets too big you will get the following exception:

System.Data.EntityCommandExecutionException: An error occurred while executing the command definition. See the inner exception for details. —> System.Data.SqlClient.SqlException: Some part of your SQL statement is nested too deeply. Rewrite the query or break it up into smaller queries.

As Michael, I will use Enumerable.Range to create a list of the desired length, modifying the MainMethod as in the following snippet:

private static void MainMethod()
{
try
{
var customerIds = Enumerable.Range(1, 50);
using (var context = new CustomerContext())
{
var initializer = new DropCreateDatabaseAlways();
initializer.InitializeDatabase(context);

var customers = from customer in context.Customers
join customerId in customerIds
on customer.Id equals customerId
select customer;
var result = customers.ToList();
}
}
catch (Exception exception)
{
Console.WriteLine(exception.ToString());
}
}
If you run you’re console application now you will get the exception.

If you had to write the desired SQL on your own you probably would have opted
for a simple: where …. in (…).

This would avoid us incurring in the max limit of nested statement.
If you want to obtain such a result as generated SQL you should modify your Linq
query to use the Contains method as in the following version of: MainMethod:

private static void MainMethod()
{
try
{
var customerIds = Enumerable.Range(1, 50);
using (var context = new CustomerContext())
{
var initializer = new DropCreateDatabaseAlways();
initializer.InitializeDatabase(context);

var customers = from customer in context.Customers
where customerIds.Contains(customer.Id)
select customer;
var result = customers.ToList();
}
}
catch (Exception exception)
{
Console.WriteLine(exception.ToString());
}
}
Now the resulting query, easier to read than the previous one, is the following:

{SELECT
[Extent1].[Id] AS [Id],
[Extent1].[Name] AS [Name],
[Extent1].[Surname] AS [Surname]
FROM [dbo].[Customers] AS [Extent1]
WHERE [Extent1].[Id] IN (1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50)}
Obviously there’s something strange if you’re forced to filter with a very big in-memory collection but that’s it.

2)
It’s now time for the second issue. Go back for a while to the original version of our method. I will show it again here:

private static void MainMethod()
{
try
{
var customerIds = new List {1,5,7};
using (var context = new CustomerContext())
{
var initializer = new DropCreateDatabaseAlways();
initializer.InitializeDatabase(context);

var customers = from customer in context.Customers
join customerId in customerIds
on customer.Id equals customerId
select customer;
var result = customers.ToList();
}
}
catch (Exception exception)
{
Console.WriteLine(exception.ToString());
}
}
The query seems very obvious for people used to SQL but we must always know what kind of collection we are using.
Let’s rewrite the previous query with method chaining syntax like in the following snippet:

private static void MainMethod()
{
try
{
var customerIds = new List {1,5,7};
using (var context = new CustomerContext())
{
var initializer = new DropCreateDatabaseAlways();
initializer.InitializeDatabase(context);

var customers = context.Customers
.Join(customerIds,
customer => customer.Id,
customerId => customerId,
(customer, customerId) => customer)
.Select(customer => customer);
var result = customers.ToList();
}
}
catch (Exception exception)
{
Console.WriteLine(exception.ToString());
}
}
If we run our console application we will obtain the same query with both versions of our method.
The same query we saw at the beginning:

SELECT
[Extent1].[Id] AS [Id],
[Extent1].[Name] AS [Name],
[Extent1].[Surname] AS [Surname]
FROM [dbo].[Customers] AS [Extent1]
INNER JOIN (SELECT
[UnionAll1].[C1] AS [C1]
FROM (SELECT
cast(1 as bigint) AS [C1]
FROM ( SELECT 1 AS X ) AS [SingleRowTable1]
UNION ALL
SELECT
cast(5 as bigint) AS [C1]
FROM ( SELECT 1 AS X ) AS [SingleRowTable2]) AS [UnionAll1]
UNION ALL
SELECT
cast(7 as bigint) AS [C1]
FROM ( SELECT 1 AS X ) AS [SingleRowTable3]) AS [UnionAll2] ON [Extent1].[Id] = [UnionAll2].[C1]
The method syntax is more explicit about what’s happening: the IQueryable.Join method is being invoked.
This means that Linq To Entities IQueryable Provider plays its role in generating the resulting SQL: converting the Linq join into a SQL inner join.

The query syntax implies very specific roles to the joined collection depending on their position in the query: the left one is called: the outer collection and the right one: the inner collection.
If we inadvertently revert the order of our 2 lists, happening to put the in-memory list to the left side like in the following snippet:

private static void MainMethod()
{
try
{
var customerIds = new List {1,5,7};
using (var context = new CustomerContext())
{
var initializer = new DropCreateDatabaseAlways();
initializer.InitializeDatabase(context);

var customers = from customerId in customerIds
join customer in context.Customers
on customerId equals customer.Id
select customer;
var result = customers.ToList();
}
}
catch (Exception exception)
{
Console.WriteLine(exception.ToString());
}
}
the method invoked will be: IEnumerable.Join and the SQL sent to the Database will be the following (you can see it with Sql Server Profiler):

SELECT
[Extent1].[Id] AS [Id],
[Extent1].[Name] AS [Name],
[Extent1].[Surname] AS [Surname]
FROM [dbo].[Customers] AS [Extent1]
As you can see, our filter condition simply disappeared: no join nor where…in condition but still the ‘result’ variable will contain only the desired results.

If the left operand in a join statement is of type IEnumberable, the Enumerable.Join extension method will be chosen during method overload resolution.
This means that the whole Customers table will be loaded in memory and then filtered via Linq To Objects…and this is not what we want.

So we definitely need to pay attention when joining in-memory collections with IQueryable collections and remember to always put the IQueryable to the left side.