Crash Dump Analysis Patterns (Part 3)

Crash Dump Analysis Patterns (Part 3)

Another pattern Iobservefrequently is False Positive Dump.We get dumpspointingin a wrong direction ornot useful for analysisand this usually happens whenwrong toolwas selected or right onewas not properly configured for capturing crash dumps. Here is one example I investigated in detail.

The customer experienced frequent spooler crashes. The dump was sent for investigation to find an offending component: usually it is a printer driver. WinDbgrevealed the following exception thread stack (parameters are not shown here for readability):

KERNEL32!RaiseException+0x56
KERNEL32!OutputDebugStringA+0x55
KERNEL32!OutputDebugStringW+0x39
HPZUI041!ConvertTicket+0x3c90
HPZUI041!DllGetClassObject+0x5d9b
HPZUI041!DllGetClassObject+0x11bb

The immediate responseis to point to HPZUI041.DLL butif we look at parameters to KERNEL32!OutputDebugStringA we would see that the string passed to it is a valid NULL-terminated string:

0:010> da 000d0040
000d0040 ".Lower DWORD of elapsed time = 3"
000d0060 "750000."

If we disassemble OutputDebugStringA up to RaiseException callwe would see:

0:010> u KERNEL32!OutputDebugStringA
KERNEL32!OutputDebugStringA+0x55
KERNEL32!OutputDebugStringA:
push ebp
mov ebp,esp
push 0FFFFFFFFh
push offset KERNEL32!'string'+0x10
push offset KERNEL32!_except_handler3
mov eax,dword ptr fs:[00000000h]
push eax
mov dword ptr fs:[0],esp
push ecx
push ecx
sub esp,228h
push ebx
push esi
push edi
mov dword ptr [ebp-18h],esp
and dword ptr [ebp-4],0
mov edx,dword ptr [ebp+8]
mov edi,edx
or ecx,0FFFFFFFFh
xor eax,eax
repne scas byte ptr es:[edi]
not ecx
mov dword ptr [ebp-20h],ecx
mov dword ptr [ebp-1Ch],edx
lea eax,[ebp-20h]
push eax
push 2
push 0
push 40010006h
call KERNEL32!RaiseException

There is no jumps in the code prior to KERNEL32!RaiseException call and this means that raising exceptionwas expected. Also MSDN documentation says:

“If the application has no debugger, the system debugger displays the string. If the application has no debugger and the system debugger is not active, OutputDebugString does nothing.”

So spoolsv.exe might have been monitored by a debugger which caught that exception and instead of dismissing it dumped the spooler process.

If we look at ‘analyze -v’ output we could see the following:

Comment: 'Userdump generated complete user-mode minidump
with Exception Monitor function on WS002E0O-01-MFP'

ERROR_CODE: (NTSTATUS) 0x40010006 -
Debugger printed exception on control C.

Now we see thatdebugger was User Mode Process Dumper you can downloadfrom Microsoft web site:

How to use the Userdump.exe tool to create a dump file

If we download it,install it and write a small console program in Visual C++ to reproduce this crash:

#include "stdafx.h"
#include
int _tmain(int argc, _TCHAR* argv[])
{
OutputDebugString(_T("Sample string"));
return 0;
}

and if we compile it in Release mode andconfigure Process Dumper applet in Control Panel to include TestOutputDebugString.exe with the following properties:

and then run our program we would see Process Dumper catching KERNEL32!RaiseExceptionand saving the dump.

Even if weselect to ignore exceptions that occur inside kernel32.dll this tool still dumpsourprocess. Now we can see that the customer most probably enabled ‘All Exceptions’ check box too. What the customer should have done is to use default rules like on the picture below:

Or select exception codes manually. In this case no dump is generated even if we manually select all of them. Just to check thatthe latter configuration still catches access violationswe can add a line of code dereferencing NULL pointer and Process Dumper will catch it and save the dump.

Conclusion: the customer should have used NTSD as a default postmortem debugger from the start. Then if crash happened wewould have seen the real offending component or could have applied other patterns and requested additional dumps.

- DmitryVostokov-