System.NullReferenceException thrown

[henrikdahl8240]

I have an OData query like:
...Bikes(1)/ns.MountainBike?$expand=Children($select=P1,P2,ns.A/P3,ns.B/P4)

Some of the instances in Children are of type A, none are of type B.

Unfortunately this exception is being thrown:

System.NullReferenceException: Object reference not set to an instance of an object.
   at lambda_method36509(Closure , QueryContext , DbDataReader , ResultContext , ResultCoordinator )
   at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.CustomShaperCompilingExpressionVisitor.
<PopulateCollection>g__ProcessCurrentElementRow|7_1[TCollection,TElement,TRelatedEntity](<>c__DisplayClass7_0`3& )
   at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.CustomShaperCompilingExpressionVisitor.PopulateCollection[TCollection,TElement,TRelatedEntity](Int32 collectionId, QueryContext queryContext, DbDataReader dbDataReader, ResultCoordinator resultCoordinator, Func`3 parentIdentifier, Func`3 outerIdentifier, Func`3 selfIdentifier, IReadOnlyList`1 parentIdentifierValueComparers, IReadOnlyList`1 outerIdentifierValueComparers, IReadOnlyList`1 selfIdentifierValueComparers, Func`5 innerShaper)
   at lambda_method36510(Closure , QueryContext , DbDataReader , ResultContext , Int32[] , ResultCoordinator )
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1.Enumerator.MoveNext()
   at Microsoft.AspNet.OData.EnableQueryAttribute.SingleOrDefault(IQueryable queryable, IWebApiActionDescriptor actionDescriptor)
   at Microsoft.AspNet.OData.EnableQueryAttribute.ExecuteQuery(Object responseValue, IQueryable singleResultCollection, IWebApiActionDescriptor actionDescriptor, Func`2 modelFunction, IWebApiRequestMessage request, Func`2 createQueryOptionFunction)
   at Microsoft.AspNet.OData.EnableQueryAttribute.OnActionExecuted(Object responseValue, IQueryable singleResultCollection, IWebApiActionDescriptor actionDescriptor, IWebApiRequestMessage request, Func`2 modelFunction, Func`2 createQueryOptionFunction, Action`1 createResponseAction, Action`3 createErrorAction)
   at Microsoft.AspNet.OData.EnableQueryAttribute.OnActionExecuted(ActionExecutedContext actionExecutedContext)
   at Microsoft.AspNetCore.Mvc.Filters.ActionFilterAttribute.OnActionExecutionAsync(ActionExecutingContext context, ActionExecutionDelegate next)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.
    <InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.
        <InvokeFilterPipelineAsync>g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.
            <InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.
                <Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)

If I do this query instead, it works fine:
...Bikes(1)/ns.MountainBike?$expand=Children($select=P1,P2,ns.A/P3)

If I do this query, I get the same exception:
...Bikes(1)/ns.MountainBike?$expand=Children($select=ns.B/P4)

Therefore it looks to me like if I use a cast for a type of which no instances match, I get the exception.

Is this enough information, because it takes a lot of time for me to make a concrete sample which demonstrates it?

I use Microsoft.AspNetCore.OData 7.4.1 and .Net 5.0 preview 5.

[henrikdahl8240]

I have made one more observations: If I query the whole collection of bikes, i.e. do not specify the key, I get the same exception, i.e.:
...Bikes/ns.MountainBike?$expand=Children($select=P1,P2,ns.A/P3,ns.B/P4)

BUT if the Bikes collection is empty, i.e. the Bikes table in the database has no rows, the query executes correctly without any exception!!

To me it looks immature, i.e. simple cases you should expect to occur over and over again have not been taken into account.

[henrikdahl8240]

As far as I know, I have now published a tiny sample to Github: https://github.com/henrikdahl8240/CastProblem20200624

You may see the few simple steps in todo.txt, which you can just copy/paste to your machine.

Can you see the problem now?

[henrikdahl8240]

@xuzhg In case your nightly build already includes a fix for this, I would like to try it already today. Your nightly builds have earlier proved very valuable. I just thought, that perhaps you have already fixed it some time ago.

[henrikdahl8240]

Unfortunately .NET 5.0 Preview 6 results in the same exception.

I hope you may take a look into it. There are only a few lines of code in the sample and it will take you very short time to familiarize yourself with it.

[xuzhg]

Is it related to this: #2142?

[henrikdahl8240]

@xuzhg The problematic case does not involve any complex type.

You can easily download the sample I uploaded to GitHub and it will take you very short time to study because there are not more than ten simple classes.

If you fix it today and your nighly build will include the correction, I can try it later today.

It's very nice you pay attention to the issue, Sam.

Concerning complex types, which are modelled as owned types in EF Core I actually have another problem, that I cannot $select it without using AsNoTracking(). Perhaps there is a good reason for this.

[xuzhg]

@xuzhg The problematic case does not involve any complex type.

You can easily download the sample I uploaded to GitHub and it will take you very short time to study because there are not more than ten simple classes.

If you fix it today and your nighly build will include the correction, I can try it later today.

It's very nice you pay attention to the issue, Sam.

Concerning complex types, which are modelled as owned types in EF Core I actually have another problem, that I cannot $select it without using AsNoTracking(). Perhaps there is a good reason for this.

I can't spend more time this week on this issue. Next week is ok. Is it ok for you at that time to reminder me? (Lol)

[henrikdahl8240]

@xuzhg OK. I look forward for your next response. I assume it is a tiny issue to correct.

I would like to add, that exactly the same construction works perfectly when making the server using .NET Framework 4.7 and Microsoft.OData 7.5.0 instead of .NET 5.0 Preview 6.

[henrikdahl8240]

@xuzhg A colleague of yours, Arthur Vickers, can reproduce the problem, but he has asked for at hint. Do you think, that you may provide this hint, so it can be clarified?
You can see it here: dotnet/efcore#21398 (comment)

His text was:

I can reproduce this, but I don't know how to get the IQueryable that OData is generating. Does anyone know where to set a breakpoint or something else that can be done to figure out what query OData is generating?

[xuzhg]

@henrikdahl8240 I run your sample and add the logger into it, and output the EFCore logging as:

For your (wrong) request (It failed at my side)

http://localhost:5000/odata/BikeShop_GoodCategorys/CastProblem20200624.BikeShop_GoodCategory_NonRoot?$expand=CategoryChildren($select=ID,Version),Contents($select=ID,Version,CastProblem20200624.BikeShop_Good_Z3950Target/ISIL_Number,CastProblem20200624.BikeShop_Good__Local/Preferred_PropertyUri_For_BibliographicalRecord)

info: Microsoft.EntityFrameworkCore.Infrastructure[10403]
      Entity Framework Core 5.0.0-preview.5.20278.2 initialized 'CastProblemDbContext' using provider 'Microsoft.EntityFrameworkCore.SqlServer' with options: using lazy-loading proxies CommandTimeout=600
info: Microsoft.EntityFrameworkCore.Database.Command[20101]
      Executed DbCommand (75ms) [Parameters=[@__TypedProperty_1='?' (Size = 4000)], CommandType='Text', CommandTimeout='600']
      SELECT [b].[ID], [b].[Discriminator], [b].[Version], [b].[CategoryParentID], [t].[c], [t].[c0], [t].[ID], [t].[c1], [t].[Version], CASE
          WHEN [b1].[Discriminator] = CAST(8 AS tinyint) THEN N'CastProblem20200624.BikeShop_Good__Local__BikeShop_BibliographicalRecordInventory'
          WHEN [b1].[Discriminator] = CAST(7 AS tinyint) THEN N'CastProblem20200624.BikeShop_Good__Local__BikeShop'
          WHEN [b1].[Discriminator] = CAST(0 AS tinyint) THEN N'CastProblem20200624.BikeShop_Good_Z3950Target'
          ELSE N'CastProblem20200624.BikeShop_Good'
      END, [b1].[ID], [b1].[Version], CASE
          WHEN [b1].[Discriminator] = CAST(0 AS tinyint) THEN N'ISIL_Number'
          ELSE NULL
      END, [b1].[ISIL_Number], NULL, [b1].[BikeShopID], [b1].[CategoryID], [b1].[Discriminator], [b1].[Preferred_PropertyUri_For_BibliographicalRecord], [b1].[Target_BikeShopID], [b1].[Target_BikeShop_BibliographicalRecordInventoryID]
      FROM [BikeShop_GoodCategorys] AS [b]
      LEFT JOIN (
          SELECT @__TypedProperty_1 AS [c], N'ID' AS [c0], [b0].[ID], N'Version' AS [c1], [b0].[Version], [b0].[CategoryParentID]
          FROM [BikeShop_GoodCategorys] AS [b0]
          WHERE [b0].[Discriminator] = CAST(1 AS tinyint)
      ) AS [t] ON [b].[ID] = [t].[CategoryParentID]
      LEFT JOIN [BikeShop_Goods] AS [b1] ON [b].[ID] = [b1].[CategoryID]
      WHERE [b].[Discriminator] = CAST(1 AS tinyint)
      ORDER BY [b].[ID], [t].[ID], [b1].[ID]

For your working request (It also works fine at my side)

http://localhost:5000/odata/BikeShop_GoodCategorys/CastProblem20200624.BikeShop_GoodCategory_NonRoot?$expand=CategoryChildren($select=ID,Version),Contents

The EFCore log:

info: Microsoft.EntityFrameworkCore.Infrastructure[10403]
      Entity Framework Core 5.0.0-preview.5.20278.2 initialized 'CastProblemDbContext' using provider 'Microsoft.EntityFrameworkCore.SqlServer' with options: using lazy-loading proxies CommandTimeout=600
info: Microsoft.EntityFrameworkCore.Database.Command[20101]
      Executed DbCommand (78ms) [Parameters=[@__TypedProperty_2='?' (Size = 4000), @__TypedProperty_1='?' (Size = 4000)], CommandType='Text', CommandTimeout='600']
      SELECT [b].[ID], [b].[Discriminator], [b].[Version], [b].[CategoryParentID], [t].[c], [t].[c0], [t].[ID], [t].[c1], [t].[Version], @__TypedProperty_2, [b1].[ID], [b1].[BikeShopID], [b1].[CategoryID], [b1].[Discriminator], [b1].[Version], [b1].[ISIL_Number], [b1].[Preferred_PropertyUri_For_BibliographicalRecord], [b1].[Target_BikeShopID], [b1].[Target_BikeShop_BibliographicalRecordInventoryID], CAST(1 AS bit)
      FROM [BikeShop_GoodCategorys] AS [b]
      LEFT JOIN (
          SELECT @__TypedProperty_1 AS [c], N'ID' AS [c0], [b0].[ID], N'Version' AS [c1], [b0].[Version], [b0].[CategoryParentID]
          FROM [BikeShop_GoodCategorys] AS [b0]
          WHERE [b0].[Discriminator] = CAST(1 AS tinyint)
      ) AS [t] ON [b].[ID] = [t].[CategoryParentID]
      LEFT JOIN [BikeShop_Goods] AS [b1] ON [b].[ID] = [b1].[CategoryID]
      WHERE [b].[Discriminator] = CAST(1 AS tinyint)
      ORDER BY [b].[ID], [t].[ID], [b1].[ID]

And get result:

{
    "@odata.context": "http://localhost:5000/odata/$metadata#BikeShop_GoodCategorys/CastProblem20200624.BikeShop_GoodCategory_NonRoot(CategoryChildren(ID,Version),Contents())",
    "value": [
        {
            "@odata.etag": "W/\"YmluYXJ5J0FBQUFBQUFBQjlNPSc=\"",
            "ID": "bec5a44a-2862-4bfa-6562-08d81cb3c80b",
            "Version": "AAAAAAAAB9M=",
            "CategoryParentID": "7da7553e-2bc0-4e03-6561-08d81cb3c80b",
            "CategoryChildren": [],
            "Contents": [
                {
                    "@odata.type": "#CastProblem20200624.BikeShop_Good_Z3950Target",
                    "ID": "cc13309e-1f93-4ed5-edf4-08d81cb3c814",
                    "Version": "AAAAAAAAB9Q=",
                    "CategoryID": "bec5a44a-2862-4bfa-6562-08d81cb3c80b",
                    "BikeShopID": "1791029c-5437-42d9-474e-08d81cb3c7f8",
                    "ISIL_Number": "C"
                }
            ]
        }
    ]
}

I am on-call this week and will put eyes on it.

[henrikdahl8240]

@xuzhg You also get the same result as I in the "good" case.

It sounds very nice.

If I would also like to add a logger, for instance in some other situation, is it just a matter of adding the logger in the usual .NET Core way or is there some special OData setting-up, which I must do in order to include the query as you have done?

[xuzhg]

@henrikdahl8240 That's only related to Logger at EF Core side.
you can refer: https://docs.microsoft.com/en-us/ef/core/miscellaneous/logging?tabs=v3

What I did:

public class Startup
{
        public static readonly ILoggerFactory MyLoggerFactory
            = LoggerFactory.Create(builder => { builder.AddConsole(); });

      ....

        public void ConfigureServices(IServiceCollection services)
        {
            services.AddDbContext<CastProblemDbContext>(options =>
            {
                options.UseLazyLoadingProxies();
                options.UseSqlServer(Configuration.GetConnectionString("CastProblemDbContext"), opts =>
                {
                    opts.CommandTimeout((int)TimeSpan.FromMinutes(10).TotalSeconds);
                });
                options.UseLoggerFactory(MyLoggerFactory);
            });

...
}

[xuzhg]

@henrikdahl8240 Can I close this issue?

[henrikdahl8240]

Yes, you and others can read about details of the issue at dotnet/efcore#21398.

Thank you for putting efforts into this, which let me to continue.

[henrikdahl8240]

@xuzhg Should you be able to put some more contextual exception on your side, it would probably be beneficial to many people, for instance as I suggest in the peer posting. In the current situation the user is basically left with no directed indication at all.