IIS app pool keeps crashing

Question

our app pool out of nowhere keeps crashing, and I've disabled the rapid fail-fail protection so our site doesnt shut down, but this is not really the way to find the error.

The event log is fileld with errors like: A process serving application pool (name) suffered a fatal communication error with the Windows Process Activation Service. The process id was '8480'. The data field contains the error number.

I've made a DebugDialog analyses of when it crashes, but this tells me nothing?

Thread 1 - System ID 9060
Entry point   0x00000000 
Create time   01-11-2016 13:32:06 
Time spent in user mode   0 Days 00:00:00.00 
Time spent in kernel mode   0 Days 00:00:00.00 




Function   Source 
ntdll!NtRemoveIoCompletion+c    
KERNELBASE!GetQueuedCompletionStatus+2a    
nativerd!NOTIFICATION_THREAD::ThreadProc+4f    
kernel32!BaseThreadInitThunk+24    
ntdll!__RtlUserThreadStart+2f    
ntdll!_RtlUserThreadStart+1b    




Back to Top 


Thread 2 - System ID 3372
Entry point   0x00000000 
Create time   01-11-2016 13:32:06 
Time spent in user mode   0 Days 00:00:00.093 
Time spent in kernel mode   0 Days 00:00:00.031 




.NET Call Stack



Function 


Full Call Stack



Function   Source 
ntdll!NtWaitForWorkViaWorkerFactory+c    
ntdll!TppWorkerThread+259    
kernel32!BaseThreadInitThunk+24    
ntdll!__RtlUserThreadStart+2f    
ntdll!_RtlUserThreadStart+1b    




Back to Top 


Thread 3 - System ID 8928
Entry point   0x00000000 
Create time   01-11-2016 13:32:06 
Time spent in user mode   0 Days 00:00:00.00 
Time spent in kernel mode   0 Days 00:00:00.015 




.NET Call Stack



Function 


Full Call Stack



Function   Source 
ntdll!NtWaitForWorkViaWorkerFactory+c    
ntdll!TppWorkerThread+259    
kernel32!BaseThreadInitThunk+24    
ntdll!__RtlUserThreadStart+2f    
ntdll!_RtlUserThreadStart+1b    




Back to Top 


Thread 4 - System ID 8396
Entry point   0x00000000 
Create time   01-11-2016 13:32:06 
Time spent in user mode   0 Days 00:00:00.531 
Time spent in kernel mode   0 Days 00:00:00.078 




Function   Source 
ntdll!NtRemoveIoCompletion+c    
KERNELBASE!GetQueuedCompletionStatus+2a    
w3tp!THREAD_POOL_DATA::ThreadPoolThread+36    
w3tp!THREAD_POOL_DATA::ThreadPoolThread+2b    
w3tp!THREAD_MANAGER::ThreadManagerThread+4d    
kernel32!BaseThreadInitThunk+24    
ntdll!__RtlUserThreadStart+2f    
ntdll!_RtlUserThreadStart+1b    




Back to Top 


Thread 5 - System ID 2884
Entry point   0x00000000 
Create time   01-11-2016 13:32:06 
Time spent in user mode   0 Days 00:00:00.156 
Time spent in kernel mode   0 Days 00:00:00.015 




Function   Source 
ntdll!NtRemoveIoCompletion+c    
KERNELBASE!GetQueuedCompletionStatus+2a    
w3tp!THREAD_POOL_DATA::ThreadPoolThread+36    
w3tp!THREAD_POOL_DATA::ThreadPoolThread+2b    
w3tp!THREAD_MANAGER::ThreadManagerThread+18    
kernel32!BaseThreadInitThunk+24    
ntdll!__RtlUserThreadStart+2f    
ntdll!_RtlUserThreadStart+1b    




Back to Top 


Thread 6 - System ID 8136
Entry point   0x00000000 
Create time   01-11-2016 13:32:06 
Time spent in user mode   0 Days 00:00:00.312 
Time spent in kernel mode   0 Days 00:00:00.015 




Function   Source 
ntdll!NtRemoveIoCompletion+c    
KERNELBASE!GetQueuedCompletionStatus+2a    
w3tp!THREAD_POOL_DATA::ThreadPoolThread+36    
w3tp!THREAD_POOL_DATA::ThreadPoolThread+2b    
w3tp!THREAD_MANAGER::ThreadManagerThread+18    
kernel32!BaseThreadInitThunk+24    
ntdll!__RtlUserThreadStart+2f    
ntdll!_RtlUserThreadStart+1b    




Back to Top 


Thread 7 - System ID 1516
Entry point   0x00000000 
Create time   01-11-2016 13:32:06 
Time spent in user mode   0 Days 00:00:00.734 
Time spent in kernel mode   0 Days 00:00:02.281 




.NET Call Stack



Function 


Full Call Stack



Function   Source 
ntdll!NtRemoveIoCompletion+c    
KERNELBASE!GetQueuedCompletionStatus+2a    
w3tp!THREAD_POOL_DATA::ThreadPoolThread+36    
w3tp!THREAD_POOL_DATA::ThreadPoolThread+2b    
w3tp!THREAD_MANAGER::ThreadManagerThread+4d    
kernel32!BaseThreadInitThunk+24    
ntdll!__RtlUserThreadStart+2f    
ntdll!_RtlUserThreadStart+1b    




Back to Top 


Thread 8 - System ID 5912
Entry point   0x00000000 
Create time   01-11-2016 13:32:06 
Time spent in user mode   0 Days 00:00:00.00 
Time spent in kernel mode   0 Days 00:00:00.00 




Function   Source 
ntdll!NtWaitForWorkViaWorkerFactory+c    
ntdll!TppWorkerThread+259    
kernel32!BaseThreadInitThunk+24    
ntdll!__RtlUserThreadStart+2f    
ntdll!_RtlUserThreadStart+1b    




Back to Top 


Thread 9 - System ID 1512
Entry point   0x00000000 
Create time   01-11-2016 13:32:06 
Time spent in user mode   0 Days 00:00:00.00 
Time spent in kernel mode   0 Days 00:00:00.00 




Function   Source 
ntdll!NtWaitForMultipleObjects+c    
KERNELBASE!WaitForMultipleObjectsEx+cc    
kernel32!WaitForMultipleObjects+19    
iisfreb!LOSSY_QUEUE::WaitForDequeue+51    
iisfreb!FREB_LOG_FILE_MANAGER::WriterThread+b9    
kernel32!BaseThreadInitThunk+24    
ntdll!__RtlUserThreadStart+2f    
ntdll!_RtlUserThreadStart+1b    




Back to Top 


Thread 10 - System ID 7856
Entry point   0x00000000 
Create time   01-11-2016 13:32:06 
Time spent in user mode   0 Days 00:00:00.00 
Time spent in kernel mode   0 Days 00:00:00.00 




Function   Source 
ntdll!NtWaitForMultipleObjects+c    
KERNELBASE!WaitForMultipleObjectsEx+cc    
clr!DebuggerRCThread::MainLoop+98    
clr!DebuggerRCThread::ThreadProc+cb    
clr!DebuggerRCThread::ThreadProcStatic+b9    
kernel32!BaseThreadInitThunk+24    
ntdll!__RtlUserThreadStart+2f    
ntdll!_RtlUserThreadStart+1b    




Back to Top 


Thread 11 - System ID 8432
Entry point   0x00000000 
Create time   01-11-2016 13:32:06 
Time spent in user mode   0 Days 00:00:01.437 
Time spent in kernel mode   0 Days 00:00:00.062 




Function   Source 
ntdll!NtWaitForSingleObject+c    
KERNELBASE!WaitForSingleObjectEx+99    
clr!CLREventBase::Reset+145    
clr!CLREventBase::Reset+18d    
clr!CLREventBase::WaitEx+152    
clr!CLREventBase::Wait+1a    
clr!SVR::gc_heap::gc_thread_function+5d    
clr!SVR::gc_heap::gc_thread_stub+71    
kernel32!BaseThreadInitThunk+24    
ntdll!__RtlUserThreadStart+2f    
ntdll!_RtlUserThreadStart+1b    




Back to Top 


Thread 12 - System ID 7740
Entry point   0x00000000 
Create time   01-11-2016 13:32:06 
Time spent in user mode   0 Days 00:00:01.125 
Time spent in kernel mode   0 Days 00:00:00.062 




Function   Source 
ntdll!NtWaitForSingleObject+c    
KERNELBASE!WaitForSingleObjectEx+99    
clr!CLREventBase::Reset+145    
clr!CLREventBase::Reset+18d    
clr!CLREventBase::WaitEx+152    
clr!CLREventBase::Wait+1a    
clr!SVR::gc_heap::gc_thread_function+129    
clr!SVR::gc_heap::gc_thread_stub+71    
kernel32!BaseThreadInitThunk+24    
ntdll!__RtlUserThreadStart+2f    
ntdll!_RtlUserThreadStart+1b

Alot of threads look like this but theres more in Thread 37:

Thread 37 - System ID 4880
Entry point   0x00000000 
Create time   01-11-2016 13:44:54 
Time spent in user mode   0 Days 00:00:02.984 
Time spent in kernel mode   0 Days 00:00:00.062 




This thread is not fully resolved and may or may not be a problem. Further analysis of these threads may be required.



.NET Call Stack



Function 
System.RuntimeMethodHandle.InvokeMethod(System.Object, System.Object[], System.Signature, Boolean) 
System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr, System.Web.RequestNotificationStatus ByRef) 


Full Call Stack



Function   Source 
ntdll!NtTerminateProcess    
KERNELBASE!TerminateProcess+23    
clr!EEPolicy::HandleFatalStackOverflow+1b5    
clr!EEPolicy::HandleStackOverflow+1b0    
clr!COMPlusFrameHandler+9b    
ntdll!ExecuteHandler2+26    
ntdll!ExecuteHandler+24    
ntdll!KiUserExceptionDispatcher+f    
clr!_EH_prolog3_catch+1e    
clr!ArrayNative::ArrayCopy+c    
mscorlib_ni+3af760    
clr!CallDescrWorkerInternal+34    
clr!CallDescrWorkerWithHandler+6b    
clr!CallDescrWorkerReflectionWrapper+55    
clr!RuntimeMethodHandle::InvokeMethod+7eb    
mscorlib_ni+3a22ed    
mscorlib_ni+3a1e76    
System_Data_SqlXml_ni+1223c7    
System_Data_SqlXml_ni+121df8    
0x21c51279    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c51cb2    
0x21c6945d    
0x21c67322    
System_Data_SqlXml_ni+11f0fc    
System_Data_SqlXml_ni+11f00d    
System_Xml_ni+4ee44a    
System_Xml_ni+4edcd3    
0x1a60d177    
0x1a60ce3b    
0x1a60c584    
0x1a60ac71    
0x1a609b81    
0x1a608780    
System_Web_ni+1ebc0d    
System_Web_ni+1ea54c    
System_Web_ni+1ea5f9    
System_Web_ni+1ea5f9    
System_Web_ni+1ea5f9    
System_Web_ni+1ea5f9    
System_Web_ni+1ea5f9    
System_Web_ni+1ea5f9    
System_Web_ni+1ea5f9    
System_Web_ni+1ea5f9    
System_Web_ni+1ea5f9    
System_Web_ni+1f3785    
System_Web_ni+1f2ff3    
System_Web_ni+1f2f29    
System_Web_ni+1f2ed7    
System_Web_ni+1f2ebb    
System_Web_ni+1f5ead    
System_Web_ni+1cfce2    
System_Web_ni+1de623    
System_Web_ni+1d0630    
System_Web_ni+1dd182    
System_Web_ni+1d1490    
System_Web_ni+1d117f    
webengine4!W3_MGD_HANDLER::ProcessNotification+5a    
webengine4!W3_MGD_HANDLER::DoWork+2cc    
webengine4!RequestDoWork+2ea    
webengine4!CMgdEngHttpModule::OnExecuteRequestHandler+18    
iiscore!NOTIFICATION_CONTEXT::RequestDoWork+17b    
iiscore!NOTIFICATION_CONTEXT::CallModulesInternal+3d7    
iiscore!NOTIFICATION_CONTEXT::CallModules+2b    
iiscore!NOTIFICATION_MAIN::DoStateRequestExecuteHandler+40    
iiscore!NOTIFICATION_MAIN::DoWork+11a    
iiscore!W3_CONTEXT_BASE::ContinueNotificationLoop+1d    
iiscore!W3_CONTEXT_BASE::IndicateCompletion+8c    
webengine4!W3_MGD_HANDLER::IndicateCompletion+38    
webengine4!MgdIndicateCompletion+22    
System_Web_ni+22510d    
System_Web_ni+1d1633    
System_Web_ni+1d117f    
clr!UM2MThunk_WrapperHelper+10    
clr!UM2MThunk_Wrapper+76    
clr!Thread::DoADCallBack+b3    
clr!UM2MDoADCallBack+92    
0x022fd9af    
webengine4!W3_MGD_HANDLER::ProcessNotification+5a    
webengine4!ProcessNotificationCallback+32    
clr!UnManagedPerAppDomainTPCount::DispatchWorkItem+1ce    
clr!ThreadpoolMgr::ExecuteWorkRequest+42    
clr!ThreadpoolMgr::WorkerThreadStart+390    
clr!Thread::intermediateThreadProc+4d    
kernel32!BaseThreadInitThunk+24    
ntdll!__RtlUserThreadStart+2f    
ntdll!_RtlUserThreadStart+1b

Niels Sparenberg Niels Sparenberg · Accepted Answer · 2016-11-02T10:11:14

I found out how to get a more detailed stacktrace. I downloaded Windbg found in the Windows Software Development Kit: https://developer.microsoft.com/da-dk/windows/hardware/windows-driver-kit

Open the crash dump file in Windbg and run the following commands:

.loadby sos.dll clr
!clrstack

This will give you are more detailed stacktrace of the problem. In this case it was a function called over and over again.

IIS app pool keeps crashing

1 Answers