Azure function suddenly stopped working on Azure

Question

I have an azure function with a ServiceBus Trigger running locally and on azure just fine. Suddenly, the Azure deployment started failing with the following stack trace:

System.MissingMethodException : Method not found: 'Microsoft.Azure.WebJobs.IWebJobsExtensionBuilder Microsoft.Azure.WebJobs.WebJobsExtensionBuilderExtensions.ConfigureOptions(Microsoft.Azure.WebJobs.IWebJobsExtensionBuilder, System.Action`3)'.

  at Microsoft.Extensions.Hosting.ServiceBusHostBuilderExtensions.AddServiceBus(IWebJobsBuilder builder,Action`1 configure)

  at Microsoft.Extensions.Hosting.ServiceBusHostBuilderExtensions.AddServiceBus(IWebJobsBuilder builder)

  at Microsoft.Azure.WebJobs.ServiceBus.ServiceBusWebJobsStartup.Configure(IWebJobsBuilder builder)

  at Microsoft.Azure.WebJobs.WebJobsBuilderExtensions.ConfigureStartup(IWebJobsStartup startup,WebJobsBuilderContext context,IWebJobsBuilder builder) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Hosting\WebJobsBuilderExtensions.cs : 166

  at Microsoft.Azure.WebJobs.WebJobsBuilderExtensions.ConfigureAndLogUserConfiguredServices(IWebJobsStartup startup,WebJobsBuilderContext context,IWebJobsBuilder builder,ILoggerFactory loggerFactory) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Hosting\WebJobsBuilderExtensions.cs : 130

  at Microsoft.Azure.WebJobs.WebJobsBuilderExtensions.UseWebJobsStartup(IWebJobsBuilder builder,Type startupType,WebJobsBuilderContext context,ILoggerFactory loggerFactory) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Hosting\WebJobsBuilderExtensions.cs : 115

  at Microsoft.Azure.WebJobs.WebJobsBuilderExtensions.UseExternalStartup(IWebJobsBuilder builder,IWebJobsStartupTypeLocator startupTypeLocator,WebJobsBuilderContext context,ILoggerFactory loggerFactory) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Hosting\WebJobsBuilderExtensions.cs : 213

  at Microsoft.Azure.WebJobs.Script.ScriptHostBuilderExtensions.<>c__DisplayClass7_0.b__1(HostBuilderContext context,IWebJobsBuilder webJobsBuilder) at D:\a\1\s\src\WebJobs.Script\ScriptHostBuilderExtensions.cs : 204

  at Microsoft.Extensions.Hosting.WebJobsHostBuilderExtensions.<>c__DisplayClass5_0.b__1(HostBuilderContext context,IServiceCollection services) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Hosting\WebJobsHostBuilderExtensions.cs : 54

  at Microsoft.Extensions.Hosting.HostBuilder.CreateServiceProvider()

  at Microsoft.Extensions.Hosting.HostBuilder.Build()

  at Microsoft.Azure.WebJobs.Script.WebHost.DefaultScriptHostBuilder.BuildHost(Boolean skipHostStartup,Boolean skipHostConfigurationParsing) at D:\a\1\s\src\WebJobs.Script.WebHost\DefaultScriptHostBuilder.cs : 59

  at async Microsoft.Azure.WebJobs.Script.WebHost.WebJobsScriptHostService.UnsynchronizedStartHostAsync(ScriptHostStartupOperation activeOperation,Int32 attemptCount,JobHostStartupMode startupMode) at D:\a\1\s\src\WebJobs.Script.WebHost\WebJobsScriptHostService.cs : 242

I have tried the following with no luck:

Deploy my functions again from VS 2019 - still fails
Create a new Function App - still fails
Went through the troubleshooting guide but no luck, my connection strings and my azure storage hasn't changed

My packages haven't changed since the last working deployment:

What puzzles me is that this is working just fine locally with no errors or warnings.

I am running out of ideas so any help is much appreciated.

UPDATE:

Below are the runtimes and SDKs installed on the Azure deployment

D:\home>dotnet --list-runtimes Microsoft.AspNetCore.All 2.1.22 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.All]

Microsoft.AspNetCore.All 2.2.14 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.All]

Microsoft.AspNetCore.App 2.1.22 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]

Microsoft.AspNetCore.App 2.2.14 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]

Microsoft.AspNetCore.App 3.0.3 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]

Microsoft.AspNetCore.App 3.1.8 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]

Microsoft.AspNetCore.App 3.1.11 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]

Microsoft.NETCore.App 1.0.16 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]

Microsoft.NETCore.App 1.1.13 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]

Microsoft.NETCore.App 2.0.9 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]

Microsoft.NETCore.App 2.1.22 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]

Microsoft.NETCore.App 2.2.14 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]

Microsoft.NETCore.App 3.0.3 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]

Microsoft.NETCore.App 3.1.11 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]

D:\home>dotnet --list-sdks

1.1.14 [D:\Program Files (x86)\dotnet\sdk]

2.1.518 [D:\Program Files (x86)\dotnet\sdk]

2.2.109 [D:\Program Files (x86)\dotnet\sdk]

3.1.111 [D:\Program Files (x86)\dotnet\sdk]

And here are those installed locally where the function is running just fine:

PS C:\Users\Admin> dotnet --list-runtimes

Microsoft.AspNetCore.All 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]

Microsoft.AspNetCore.All 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]

Microsoft.AspNetCore.All 2.1.23 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]

Microsoft.AspNetCore.All 2.1.24 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]

Microsoft.AspNetCore.App 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]

Microsoft.AspNetCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]

Microsoft.AspNetCore.App 2.1.23 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]

Microsoft.AspNetCore.App 2.1.24 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]

Microsoft.AspNetCore.App 3.1.10 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]

Microsoft.AspNetCore.App 3.1.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]

Microsoft.AspNetCore.App 5.0.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]

Microsoft.AspNetCore.App 5.0.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]

Microsoft.NETCore.App 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

Microsoft.NETCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

Microsoft.NETCore.App 2.1.23 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

Microsoft.NETCore.App 2.1.24 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

Microsoft.NETCore.App 3.1.10 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

Microsoft.NETCore.App 3.1.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

Microsoft.NETCore.App 5.0.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

Microsoft.NETCore.App 5.0.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

Microsoft.WindowsDesktop.App 3.1.10 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Microsoft.WindowsDesktop.App 3.1.11 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Microsoft.WindowsDesktop.App 5.0.1 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Microsoft.WindowsDesktop.App 5.0.2 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

PS C:\Users\Admin> dotnet --list-sdks

2.1.520 [C:\Program Files\dotnet\sdk]

2.1.700 [C:\Program Files\dotnet\sdk]

2.1.701 [C:\Program Files\dotnet\sdk]

3.1.405 [C:\Program Files\dotnet\sdk]

5.0.101 [C:\Program Files\dotnet\sdk]

5.0.102 [C:\Program Files\dotnet\sdk]

Can you go in to your service and see what framework versions are installed? So if your service is hello.azurewebsites.net, go to hello.scm.azurewebsites.net, then go in to the console and run dotnet --list-runtimes and dotnet --list-sdks, do it locally as well where it's working. I'm just wondering if there is some incompatibility like you are running 5 on your local machine, but the server is in 3.1. I've had this weird stuff happen before and it was caused by the folks maintaining our subscription and they hadn't updated our SDKs. — Andy
Another thing you could try is making sure that when you publish, all the files are removed from your remote directory: More info here: eugenechiang.com/2020/11/25/… — Andy
Thanks @Andy. Please see my updates regarding the runtimes and SDKs installed. I have also tried publishing with "delete files in destination" enabled, the issue persists. I also tried chasing the files manually into storage and delete them with no luck. — GETah
What are you targeting? 3.1? 5? It looks like your server doesn't have 5. It goes up to 3.1. Can you compile your function as .netcore3.1? Make sure your nuget packages are also 3.1 as well. then slowly change them to the latest version — Andy
@Andy thanks a lot! I downgraded my installation and removed all traces of .net core 5. I am now getting the same failure locally as well which is going to be much easier to debug. Would it make sense that you turn your comments into an answer so I can accept? — GETah

Andy Andy · Accepted Answer · 2021-02-11T14:56:20

When I see goofy stuff like this, I like to verify what runtimes and SDKs I am running locally and what is running remotely. To do this, we need to log in to our service's Kudu portal:

If your service domain is hello.azurewebsites.net, then go to hello.scm.azurewebsites.net.

At the top of the site there is a "Debug Console" option. Click that and go to CMD and type in dotnet --list-sdks and also dotnet --list-runtimes

If you are using nuget packages in the version 5 range, make sure your server has the v5 SDK/framework.

Also, stay consistent in your nuget packages. If you are using asp.net core 3.1, do not use v5 nuget packages. Make sure your packages are all v3.

I am going to link the information from your comment about not mixing .NET Core 3.1 and .NET5 packages when working specifically with Azure Functions. Apparently Azure Functions haven't been fully vetted with .NET5 which is why your server is only showing .NET Core 3.1.

Azure function suddenly stopped working on Azure

1 Answers