For awhile I've been experiencing an issue in Azure Functions where the app crashes without logs and will not restart on it's own. By looking at the eventlog.xml I've been able to track down an AggregateException stemming from some NullReferenceException in a GRPC startup.
Application: w3wp.exe CoreCLR Version: 6.0.2623.60508 .NET Version: 6.0.26 Description: The process was terminated due to an unhandled exception. Exception Info: System.AggregateException: One or more errors occurred. (One or more errors occurred. (Object reference not set to an instance of an object.)) --->
System.AggregateException: One or more errors occurred. (Object reference not set to an instance of an object.) --->
System.NullReferenceException: Object reference not set to an instance of an object. at Microsoft.Azure.WebJobs.Script.Grpc.AspNetCoreGrpcServer.ShutdownAsync() in /_/src/WebJobs.Script.Grpc/Server/AspNetCoreGrpcServer.cs:line 60 at Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcInitializationService.OuterStopAsync(CancellationToken cancellationToken) in /_/src/WebJobs.Script/Workers/Rpc/RpcInitializationService.cs:line 77 at Microsoft.Azure.WebJobs.Script.HostedServiceManager.StopAsync(CancellationToken cancellationToken) in /_/src/WebJobs.Script/HostedServiceManager.cs:line 29 at Microsoft.AspNetCore.Hosting.HostedServiceExecutor.ExecuteAsync(Func`2 callback, Boolean throwOnFirstFailure)
--- End of inner exception stack trace ---
at Microsoft.AspNetCore.Hosting.HostedServiceExecutor.ExecuteAsync(Func`2 callback, Boolean throwOnFirstFailure)
at Microsoft.AspNetCore.Hosting.WebHost.StopAsync(CancellationToken cancellationToken) at Microsoft.AspNetCore.Hosting.WebHostExtensions.WaitForTokenShutdownAsync(IWebHost host, CancellationToken cancellationToken)
at Microsoft.AspNetCore.Hosting.WebHostExtensions.RunAsync(IWebHost host, CancellationToken token, String startupMessage)
at Microsoft.AspNetCore.Hosting.WebHostExtensions.RunAsync(IWebHost host, CancellationToken token, String startupMessage) at Microsoft.AspNetCore.Hosting.WebHostExtensions.RunAsync(IWebHost host, CancellationToken token)
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.Wait()
at Microsoft.Azure.WebJobs.Script.WebHost.Program.Main(String[] args) in /_/src/WebJobs.Script.WebHost/Program.cs:line 30
I believe that the cause is something within the IBM.EntityFrameworkCore package v6.0.0.300 targeting DB2 AS400, because this does not happen anywhere else that I have similar setups for SQL Server. This has not been reproducible, it just happens sometimes. Has anyone run into this issue, or know of any work arounds?
Have currently reached out to Microsoft Support without much luck, when I thought this was an IIS issue. Have not tried much as the logs around this are few and far between, and seems to be some issue at a lower level. I did see a work around that I have yet to try where someone with a smiliar w3wp.exe exception wrapped their Startup in try/catch and exited with -1 to get the server to restart after a fail. Might give that a go, but would like to get to the bottom of this problem.
Expectation is that the app runs without randomly dying and not restarting itself after this failure.