-
Notifications
You must be signed in to change notification settings - Fork 3.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Translate GroupBy followed by FirstOrDefault over group #12088
Comments
?? |
Is there any update? Any alternative or planned fix? @ajcvickers @shanselman |
Is there a workaround for this that would allow for database-evaluated |
Any workaround while it's being fixed? I am in the process of migrating .NET Core from 2.1 to 3.1 and EF Core from 2.1 to 3.1. |
Another breaking change for migrating to ef core 3.1 and nobody knows when it will be supported :( @smitpatel @ajcvickers |
@suadev - It was getting client eval'ed before. It is part of same breaking change disabling client eval. The perf was bad for this case. If your group had 100 element each and you are selecting only first, we still got data for all 100 elements to client side to compute the first. |
Very important feature IMHO.
|
This query pattern shows up quite a bit in my under-way 1000 table LINQ to SQL migration project. Example use case: You have a list of blogs. For each blog you want to display a single highlighted post (the most upvoted or most recent one). So you might group by blog and take the first item from each group. This can be translated with This query could result in the
This query could result in the
But any way to support this would be better than no way so that our migration can proceed. The Depending on the scenario the performance difference can be significant. SQL Server is unable to automatically switch between theses forms. |
@smitpatel @ajcvickers Is there any update on this issue or work around you could suggest for my case #21619 please? If I distil my use-case it comes down to a simple multi-column group-by statement and a projection:
The Items projection just doesn't want to work. |
I've got a similar use case that seems to be related to this issue. I use code like this in several places: // This works fine
query.Select(price => new Customer {
Name = price.Payer.Name,
Code = price.Payer.Code,
City = price.Payer.City,
ParentCode = price.Payer.ParentCode,
ParentLevel = CustomerLevel.Corporate,
CustomerLevel = CustomerLevel.Payer
}).Distinct().ToListAsync(); That works fine. The query variable could potentially have a variety of expressions and joins with no issue. As soon as I add a call to // This throws error
// query is of type IQueryable<Price>
query.Select(price => new Customer {
Name = price.Payer.Name,
Code = price.Payer.Code,
City = price.Payer.City,
ParentCode = price.Payer.ParentCode,
ParentLevel = CustomerLevel.Corporate,
CustomerLevel = CustomerLevel.Payer
}).Distinct().OrderBy(cust => cust.Name).ToListAsync(); Also, placement of the // This also throws error
// query is of type IQueryable<Price>
query
.OrderBy(price => price.payer.Name)
.Select(price => new Customer {
Name = price.Payer.Name,
Code = price.Payer.Code,
City = price.Payer.City,
ParentCode = price.Payer.ParentCode,
ParentLevel = CustomerLevel.Corporate,
CustomerLevel = CustomerLevel.Payer
}).Distinct().ToListAsync(); |
I don't think you can assume that - I believe this worked fine in EF 6 / LINQ to SQL so I wouldn't assume was using EF Core 2.x (I would never assume that - it was barely functional.)
Will EF 6 be supported on .Net 6? |
@NetMage - The comment from suadev indicated that they ran into another breaking change when upgrading to EF Core 3.1. So yes it is fair to assume it is coming from earlier version of EF Core. When upgrading from EF6/LINQ to SQL, the concept of running into breaking change has no meaning. Changing from one product to another product will require code changes and there are no "breaking changes" in such change. |
- ClientProjections is alternate when ProjectionMember binding is not possible using indexes - ClientProjections is separate from SelectExpression.Projection where former holds complex client side mapped projections and later holds only scalars - What it enables - Single result subquery is not joined right away - pendingCollections are removed - Ability to bind with above subquery translations after projection has converted to index based binding (enabling pending selector future change) - Ability to bind client projections smoothly like ProjectionMember binding so suppose translations post-client eval where it is supported - Grouping.FirstOrDefault can add subquery projection which applies separately so we can combine aggregate/non-aggregate projection on grouping - Introduce CollectionResultExpression which holds info on how to create collection for a subquery - ApplyProjection converts projectionMember/Index based bindings to actual scalar projection and updates shaper in same pass - Unique collectionId are assigned by shaper - Change in flow to projection to let sqlTranslator determines translatibility before processing for client side - Regression in collection over single result subquery - Earlier we applied single result right away copying over pending collection to outer which allowed single result subquery to convert apply to join - Now all client projections apply at same time so pending collection (equivalent subquery) is applied inside single result subquery so we fail to convert apply to join Ground work for #20291, #12088, #13805 Resolves #23571
- ClientProjections is alternate when ProjectionMember binding is not possible using indexes - ClientProjections is separate from SelectExpression.Projection where former holds complex client side mapped projections and later holds only scalars - What it enables - Single result subquery is not joined right away - pendingCollections are removed - Ability to bind with above subquery translations after projection has converted to index based binding (enabling pending selector future change) - Ability to bind client projections smoothly like ProjectionMember binding so suppose translations post-client eval where it is supported - Grouping.FirstOrDefault can add subquery projection which applies separately so we can combine aggregate/non-aggregate projection on grouping - Introduce CollectionResultExpression which holds info on how to create collection for a subquery - ApplyProjection converts projectionMember/Index based bindings to actual scalar projection and updates shaper in same pass - Unique collectionId are assigned by shaper - Change in flow to projection to let sqlTranslator determines translatibility before processing for client side - Regression in collection over single result subquery - Earlier we applied single result right away copying over pending collection to outer which allowed single result subquery to convert apply to join - Now all client projections apply at same time so pending collection (equivalent subquery) is applied inside single result subquery so we fail to convert apply to join Ground work for #20291, #12088, #13805 Resolves #23571
Wondering what all of you have been using as a workaround up until this is fixed? Last time i got away with it by caching all the results and doing to query in the app instead of linq-to-sql. This time I really have to do it like @reathh shows. So if anyone got a workaround, that would be great.
|
I got away with something like this before: await _db.RetailerRates
.GroupBy(x => x.Price)
.Select(x => new { CreatedAt = x.Max(y => y.CreatedAt), Price = x.Key})
.Join(_db.RetailerRates, x => x, x => new { x.CreatedAt, x.Price }, (_, retailerRate) => retailerRate);
.ToListAsync(); Disclaimer: This solution depends on the uniqueness of your Price/CreatedAt combination. If not unique, then you might end up with multiple entities for each Price/CreatedAt combination. You could do some extra filtering after materializing the query to accomodate for this 🙂 |
@Ruud-cb Try to use the codes below, await _db.RetailerRates
.Select(r => r.Price)
.Distinct()
.SelectMany(p => _db.RetailerRates.Where(r => r.Price == p).OrderByDescending(r => r.CreatedAt).Take(1))
.ToListAsync() For SQL Server, it will generate the sql with |
I think .Distinct() will materialize all distinct prices? |
@DK8ALREC Yes, it works as same as |
Yes, my point was that I think your approach would make 2 database queries, and it will materialize all distinct Prices before making the "final query", which could potentially be a lot of rows I guess. |
No, it won't. You just need to take a try :) |
- ClientProjections is alternate when ProjectionMember binding is not possible using indexes - ClientProjections is separate from SelectExpression.Projection where former holds complex client side mapped projections and later holds only scalars - What it enables - Single result subquery is not joined right away - pendingCollections are removed - Ability to bind with above subquery translations after projection has converted to index based binding (enabling pending selector future change) - Ability to bind client projections smoothly like ProjectionMember binding so suppose translations post-client eval where it is supported - Grouping.FirstOrDefault can add subquery projection which applies separately so we can combine aggregate/non-aggregate projection on grouping - Introduce CollectionResultExpression which holds info on how to create collection for a subquery - ApplyProjection converts projectionMember/Index based bindings to actual scalar projection and updates shaper in same pass - Unique collectionId are assigned by shaper - Change in flow to projection to let sqlTranslator determines translatibility before processing for client side - Regression in collection over single result subquery - Earlier we applied single result right away copying over pending collection to outer which allowed single result subquery to convert apply to join - Now all client projections apply at same time so pending collection (equivalent subquery) is applied inside single result subquery so we fail to convert apply to join Ground work for #20291, #12088, #13805 Resolves #23571
- ClientProjections is alternate when ProjectionMember binding is not possible using indexes - ClientProjections is separate from SelectExpression.Projection where former holds complex client side mapped projections and later holds only scalars - What it enables - Single result subquery is not joined right away - pendingCollections are removed - Ability to bind with above subquery translations after projection has converted to index based binding (enabling pending selector future change) - Ability to bind client projections smoothly like ProjectionMember binding so suppose translations post-client eval where it is supported - Grouping.FirstOrDefault can add subquery projection which applies separately so we can combine aggregate/non-aggregate projection on grouping - Introduce CollectionResultExpression which holds info on how to create collection for a subquery - ApplyProjection converts projectionMember/Index based bindings to actual scalar projection and updates shaper in same pass - Unique collectionId are assigned by shaper - Change in flow to projection to let sqlTranslator determines translatibility before processing for client side - Regression in collection over single result subquery - Earlier we applied single result right away copying over pending collection to outer which allowed single result subquery to convert apply to join - Now all client projections apply at same time so pending collection (equivalent subquery) is applied inside single result subquery so we fail to convert apply to join Ground work for #20291, #12088, #13805 Resolves #23571
- Allow expanding navigations after GroupBy operator applied before reducing it to non-grouping Fixes #22609 - Translate FirstOrDefault over grouping element Fixes #12088 - Add ability to select N element over grouping element Fixes #13805 Overall approach: A grouping element (the range variable you get after applying GroupBy operator) is of type `IGrouping<TKey, TElement>` which implements `IEnumerable<TElement>`. Hence we treat this enumerable as if it is queryable during nav expansion phase. During translation phase we inject ShapedQueryExpression in place of the grouping element which is being enumerated. What this allows us is to expand navigation just like any other query root and translate a subquery similar to other subqueries to facilitate reusing same code for the tasks. During translation phase in relational layer, since aggregate operation can be lifted into projection for SelectExpression containing SQL GROUP BY. This code path works in 2 ways, when translating we try to combine predicate/distinct into the aggregate operation (so in future when we support custom aggregate operators, we don't have to understand the shape of it to modify it later. When adding this scalar subquery to SelectExpression, we try to pattern match it to see if we can lift it. Further during lifting, we also lift any additional joins in the subquery (which implies there were some joins expanded on grouping element before aggregate) including the navigation expanded from owned navigations. A pending TODO is to de-dupe navigation expanded. It is not straight forward since aliases of table would have changed when previous was lifted. Given every enumerable grouping element act as query root, every time we replace it inside a lambda expression, we need to create a copy of the root. Navigation expansion and individual queryableMethodTranslatingEV does this. So each root act and translate independently from each other. Bug fixes: - Fix a bug in identifying single result in InMemory to convert it to enumerable - Null out _groupingParameter in InMemoryQueryExpression once the projection to reduce it has been applied - Throw better error message when translating Min/Max over an entity type for InMemory
- Allow expanding navigations after GroupBy operator applied before reducing it to non-grouping Fixes #22609 - Translate FirstOrDefault over grouping element Fixes #12088 - Add ability to select N element over grouping element Fixes #13805 Overall approach: A grouping element (the range variable you get after applying GroupBy operator) is of type `IGrouping<TKey, TElement>` which implements `IEnumerable<TElement>`. Hence we treat this enumerable as if it is queryable during nav expansion phase. During translation phase we inject ShapedQueryExpression in place of the grouping element which is being enumerated. What this allows us is to expand navigation just like any other query root and translate a subquery similar to other subqueries to facilitate reusing same code for the tasks. During translation phase in relational layer, since aggregate operation can be lifted into projection for SelectExpression containing SQL GROUP BY. This code path works in 2 ways, when translating we try to combine predicate/distinct into the aggregate operation (so in future when we support custom aggregate operators, we don't have to understand the shape of it to modify it later. When adding this scalar subquery to SelectExpression, we try to pattern match it to see if we can lift it. Further during lifting, we also lift any additional joins in the subquery (which implies there were some joins expanded on grouping element before aggregate) including the navigation expanded from owned navigations. A pending TODO is to de-dupe navigation expanded. It is not straight forward since aliases of table would have changed when previous was lifted. Given every enumerable grouping element act as query root, every time we replace it inside a lambda expression, we need to create a copy of the root. Navigation expansion and individual queryableMethodTranslatingEV does this. So each root act and translate independently from each other. Bug fixes: - Fix a bug in identifying single result in InMemory to convert it to enumerable - Null out _groupingParameter in InMemoryQueryExpression once the projection to reduce it has been applied - Throw better error message when translating Min/Max over an entity type for InMemory
- Allow expanding navigations after GroupBy operator applied before reducing it to non-grouping Fixes #22609 - Translate FirstOrDefault over grouping element Fixes #12088 - Add ability to select N element over grouping element Fixes #13805 Overall approach: A grouping element (the range variable you get after applying GroupBy operator) is of type `IGrouping<TKey, TElement>` which implements `IEnumerable<TElement>`. Hence we treat this enumerable as if it is queryable during nav expansion phase. During translation phase we inject ShapedQueryExpression in place of the grouping element which is being enumerated. What this allows us is to expand navigation just like any other query root and translate a subquery similar to other subqueries to facilitate reusing same code for the tasks. During translation phase in relational layer, since aggregate operation can be lifted into projection for SelectExpression containing SQL GROUP BY. This code path works in 2 ways, when translating we try to combine predicate/distinct into the aggregate operation (so in future when we support custom aggregate operators, we don't have to understand the shape of it to modify it later. When adding this scalar subquery to SelectExpression, we try to pattern match it to see if we can lift it. Further during lifting, we also lift any additional joins in the subquery (which implies there were some joins expanded on grouping element before aggregate) including the navigation expanded from owned navigations. A pending TODO is to de-dupe navigation expanded. It is not straight forward since aliases of table would have changed when previous was lifted. Given every enumerable grouping element act as query root, every time we replace it inside a lambda expression, we need to create a copy of the root. Navigation expansion and individual queryableMethodTranslatingEV does this. So each root act and translate independently from each other. Bug fixes: - Fix a bug in identifying single result in InMemory to convert it to enumerable - Null out _groupingParameter in InMemoryQueryExpression once the projection to reduce it has been applied - Throw better error message when translating Min/Max over an entity type for InMemory
- Allow expanding navigations after GroupBy operator applied before reducing it to non-grouping Fixes #22609 - Translate FirstOrDefault over grouping element Fixes #12088 - Add ability to select N element over grouping element Fixes #13805 Overall approach: A grouping element (the range variable you get after applying GroupBy operator) is of type `IGrouping<TKey, TElement>` which implements `IEnumerable<TElement>`. Hence we treat this enumerable as if it is queryable during nav expansion phase. During translation phase we inject ShapedQueryExpression in place of the grouping element which is being enumerated. What this allows us is to expand navigation just like any other query root and translate a subquery similar to other subqueries to facilitate reusing same code for the tasks. During translation phase in relational layer, since aggregate operation can be lifted into projection for SelectExpression containing SQL GROUP BY. This code path works in 2 ways, when translating we try to combine predicate/distinct into the aggregate operation (so in future when we support custom aggregate operators, we don't have to understand the shape of it to modify it later. When adding this scalar subquery to SelectExpression, we try to pattern match it to see if we can lift it. Further during lifting, we also lift any additional joins in the subquery (which implies there were some joins expanded on grouping element before aggregate) including the navigation expanded from owned navigations. A pending TODO is to de-dupe navigation expanded. It is not straight forward since aliases of table would have changed when previous was lifted. Given every enumerable grouping element act as query root, every time we replace it inside a lambda expression, we need to create a copy of the root. Navigation expansion and individual queryableMethodTranslatingEV does this. So each root act and translate independently from each other. Bug fixes: - Fix a bug in identifying single result in InMemory to convert it to enumerable - Null out _groupingParameter in InMemoryQueryExpression once the projection to reduce it has been applied - Throw better error message when translating Min/Max over an entity type for InMemory
- Allow expanding navigations after GroupBy operator applied before reducing it to non-grouping Fixes #22609 - Translate FirstOrDefault over grouping element Fixes #12088 - Add ability to select N element over grouping element Fixes #13805 Overall approach: A grouping element (the range variable you get after applying GroupBy operator) is of type `IGrouping<TKey, TElement>` which implements `IEnumerable<TElement>`. Hence we treat this enumerable as if it is queryable during nav expansion phase. During translation phase we inject ShapedQueryExpression in place of the grouping element which is being enumerated. What this allows us is to expand navigation just like any other query root and translate a subquery similar to other subqueries to facilitate reusing same code for the tasks. During translation phase in relational layer, since aggregate operation can be lifted into projection for SelectExpression containing SQL GROUP BY. This code path works in 2 ways, when translating we try to combine predicate/distinct into the aggregate operation (so in future when we support custom aggregate operators, we don't have to understand the shape of it to modify it later. When adding this scalar subquery to SelectExpression, we try to pattern match it to see if we can lift it. Further during lifting, we also lift any additional joins in the subquery (which implies there were some joins expanded on grouping element before aggregate) including the navigation expanded from owned navigations. A pending TODO is to de-dupe navigation expanded. It is not straight forward since aliases of table would have changed when previous was lifted. Given every enumerable grouping element act as query root, every time we replace it inside a lambda expression, we need to create a copy of the root. Navigation expansion and individual queryableMethodTranslatingEV does this. So each root act and translate independently from each other. Bug fixes: - Fix a bug in identifying single result in InMemory to convert it to enumerable - Null out _groupingParameter in InMemoryQueryExpression once the projection to reduce it has been applied - Throw better error message when translating Min/Max over an entity type for InMemory
- Allow expanding navigations after GroupBy operator applied before reducing it to non-grouping Fixes #22609 - Translate FirstOrDefault over grouping element Fixes #12088 - Add ability to select N element over grouping element Fixes #13805 Overall approach: A grouping element (the range variable you get after applying GroupBy operator) is of type `IGrouping<TKey, TElement>` which implements `IEnumerable<TElement>`. Hence we treat this enumerable as if it is queryable during nav expansion phase. During translation phase we inject ShapedQueryExpression in place of the grouping element which is being enumerated. What this allows us is to expand navigation just like any other query root and translate a subquery similar to other subqueries to facilitate reusing same code for the tasks. During translation phase in relational layer, since aggregate operation can be lifted into projection for SelectExpression containing SQL GROUP BY. This code path works in 2 ways, when translating we try to combine predicate/distinct into the aggregate operation (so in future when we support custom aggregate operators, we don't have to understand the shape of it to modify it later. When adding this scalar subquery to SelectExpression, we try to pattern match it to see if we can lift it. Further during lifting, we also lift any additional joins in the subquery (which implies there were some joins expanded on grouping element before aggregate) including the navigation expanded from owned navigations. A pending TODO is to de-dupe navigation expanded. It is not straight forward since aliases of table would have changed when previous was lifted. Given every enumerable grouping element act as query root, every time we replace it inside a lambda expression, we need to create a copy of the root. Navigation expansion and individual queryableMethodTranslatingEV does this. So each root act and translate independently from each other. Bug fixes: - Fix a bug in identifying single result in InMemory to convert it to enumerable - Null out _groupingParameter in InMemoryQueryExpression once the projection to reduce it has been applied - Throw better error message when translating Min/Max over an entity type for InMemory
- Allow expanding navigations after GroupBy operator applied before reducing it to non-grouping Fixes #22609 - Translate FirstOrDefault over grouping element Fixes #12088 - Add ability to select N element over grouping element Fixes #13805 Overall approach: A grouping element (the range variable you get after applying GroupBy operator) is of type `IGrouping<TKey, TElement>` which implements `IEnumerable<TElement>`. Hence we treat this enumerable as if it is queryable during nav expansion phase. During translation phase we inject ShapedQueryExpression in place of the grouping element which is being enumerated. What this allows us is to expand navigation just like any other query root and translate a subquery similar to other subqueries to facilitate reusing same code for the tasks. During translation phase in relational layer, since aggregate operation can be lifted into projection for SelectExpression containing SQL GROUP BY. This code path works in 2 ways, when translating we try to combine predicate/distinct into the aggregate operation (so in future when we support custom aggregate operators, we don't have to understand the shape of it to modify it later. When adding this scalar subquery to SelectExpression, we try to pattern match it to see if we can lift it. Further during lifting, we also lift any additional joins in the subquery (which implies there were some joins expanded on grouping element before aggregate) including the navigation expanded from owned navigations. A pending TODO is to de-dupe navigation expanded. It is not straight forward since aliases of table would have changed when previous was lifted. Given every enumerable grouping element act as query root, every time we replace it inside a lambda expression, we need to create a copy of the root. Navigation expansion and individual queryableMethodTranslatingEV does this. So each root act and translate independently from each other. Bug fixes: - Fix a bug in identifying single result in InMemory to convert it to enumerable - Null out _groupingParameter in InMemoryQueryExpression once the projection to reduce it has been applied - Throw better error message when translating Min/Max over an entity type for InMemory
No description provided.
The text was updated successfully, but these errors were encountered: