逃出「Driver Verifier」导致的无限蓝屏

没啥用的前言

我的电脑自从买来就故障不断,无故卡死、无故蓝屏、自动重启。好在频率都不高,忍一忍凑合用。最近一段时间蓝屏的频率激增到了忍无可忍的地步,作为一个Geek,我自然要想办法排查一下原因。蓝屏最常见的错误码是KMODE_EXCEPTION_NOT_HANDLED,IRQL_LESS_OR_NOT_EQUAL,PAGE_FAULT_IN_NONPAGED_AREA,这将凶手指向了驱动。然而「Windows Debugger」却分析不到可疑的驱动,有几次分析甚至将故障原因指向硬件。这让找真凶的过程异常艰难。好在微软提供了一个自带工具「Driver Verifier」,它可以检测驱动程序在运行过程中存在的问题,这给“查案”带来了一丝转机。

「Driver Verifier」的应用逻辑如下:

当被检测的驱动程序出现行为异常时,立刻、主动触发一次蓝屏(错误码为DRIVER_VEFRIFIER_DETECTED_VIOLATION,以保留错误现场完整的调用栈帧和中断序列。

BSOD

Driver Verifier is a diagnostic tool built into Windows 10, it is designed to verify both native Microsoft drivers and third party drivers. Driver Verifier’s verification process involves putting heavy stress on drivers with the intention of making bad, outdated, incompatible or misbehaving drivers fail. The required result is a BSOD (Blue Screen of Death) which will generate a crash dump for debugging purposes.

我开启了「Driver Verifier」的标准配置,应用到了所有驱动程序,并打开了「完全内存转储」,增加「Windows Debugger」可用的信息量。配置完后重启,POST自检后马上就触发了蓝屏DRIVER_VEFRIFIER_DETECTED_VIOLATION,好家伙,原来真的是驱动搞的鬼?

我重启电脑,准备进入系统用「Windows Debugger」调试,然而,POST自检后马上又进入DRIVER_VEFRIFIER_DETECTED_VIOLATION。这时我才意识到,我陷入了无限蓝屏(Boot loop)

逃出无限蓝屏

法1 WinRE(无效)

重启进入恢复模式,选择打开命令提示符。运行verifier,从图形界面尝试关掉「Driver Verifier」,选择删除现有配置后返回No action taken,提示成功。重启后依旧蓝屏。 怀疑是RE的路径问题,cdC:\Windows\System32重新执行,依旧没效果

法2 系统还原(无效)

理论上可行,然而我并没有创建系统还原点,pass😅

法3 安全模式(有效)

在安全模式下,Windows指挥加载必要的驱动,换句话说,有问题的驱动不会被加载。理论上可以正常进入系统;Windows 10并不提供从「Windows Boot Manager」直接进入安全模式的方法,所以要绕点路。 先进入恢复模式,在命令提示符输入

1
bcdedit /set {default} safeboot minimal

将安全模式设为默认启动选项,重启,没有蓝屏,终于成功进系统了😭

打开终端,将「Driver Verifier」关掉

1
verifier /reset

输入msconfig,将Boot->Boot Option的安全模式取消,重启正常进入系统。

调试

使用「Windows Debugger」分析刚刚的内存转储文件,凶手出来了,是谷歌的安卓模拟器驱动😶

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
Microsoft (R) Windows Debugger Version 10.0.22549.1000 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [C:\WINDOWS\MEMORY.DMP]
Kernel Bitmap Dump File: Full address space is available


************* Path validation summary **************
Response                         Time (ms)     Location
Deferred                                       srv*
Symbol search path is: srv*
Executable search path is: 
Windows 10 Kernel Version 19041 MP (16 procs) Free x64
Product: WinNt, suite: TerminalServer SingleUserTS
Edition build lab: 19041.1.amd64fre.vb_release.191206-1406
Machine Name:
Kernel base = 0xfffff800`1401e000 PsLoadedModuleList = 0xfffff800`14c482b0
Debug session time: Sun Mar 27 00:29:33.307 2022 (UTC + 8:00)
System Uptime: 0 days 0:00:05.019
Loading Kernel Symbols
...............................................................
......................................
Loading User Symbols

Loading unloaded module list
...
For analysis of this file, run !analyze -v
nt!KeBugCheckEx:
fffff800`144152e0 48894c2408      mov     qword ptr [rsp+8],rcx ss:0018:ffffea87`e5e06520=00000000000000c4
5: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

DRIVER_VERIFIER_DETECTED_VIOLATION (c4)
A device driver attempting to corrupt the system has been caught.  This is
because the driver was specified in the registry as being suspect (by the
administrator) and the kernel has enabled substantial checking of this driver.
If the driver attempts to corrupt the system, BugChecks 0xC4, 0xC1 and 0xA will
be among the most commonly seen crashes.
Arguments:
Arg1: 0000000000000062, A driver has forgotten to free its pool allocations prior to unloading.
Arg2: ffffb90f949ffcc0, name of the driver having the issue.
Arg3: ffffb90f9492b570, verifier internal structure with driver information.
Arg4: 0000000000000002, total # of (paged+nonpaged) allocations that weren't freed.
	Type !verifier 3 drivername.sys for info on the allocations
	that were leaked that caused the bugcheck.

Debugging Details:
------------------

Unable to load image \SystemRoot\system32\DRIVERS\gvm.sys, Win32 error 0n2

KEY_VALUES_STRING: 1

    Key  : Analysis.CPU.mSec
    Value: 3140

    Key  : Analysis.DebugAnalysisManager
    Value: Create

    Key  : Analysis.Elapsed.mSec
    Value: 14338

    Key  : Analysis.Init.CPU.mSec
    Value: 499

    Key  : Analysis.Init.Elapsed.mSec
    Value: 10589

    Key  : Analysis.Memory.CommitPeak.Mb
    Value: 83

    Key  : WER.OS.Branch
    Value: vb_release

    Key  : WER.OS.Timestamp
    Value: 2019-12-06T14:06:00Z

    Key  : WER.OS.Version
    Value: 10.0.19041.1


FILE_IN_CAB:  MEMORY.DMP

BUGCHECK_CODE:  c4

BUGCHECK_P1: 62

BUGCHECK_P2: ffffb90f949ffcc0

BUGCHECK_P3: ffffb90f9492b570

BUGCHECK_P4: 2

IMAGE_NAME:  gvm.sys

MODULE_NAME: gvm

FAULTING_MODULE: fffff80956390000 gvm

VERIFIER_DRIVER_ENTRY: dt nt!_MI_VERIFIER_DRIVER_ENTRY ffffb90f9492b570
Symbol nt!_MI_VERIFIER_DRIVER_ENTRY not found.

BLACKBOXNTFS: 1 (!blackboxntfs)


PROCESS_NAME:  System

STACK_TEXT:  
ffffea87`e5e06518 fffff800`149ebe34     : 00000000`000000c4 00000000`00000062 ffffb90f`949ffcc0 ffffb90f`9492b570 : nt!KeBugCheckEx
ffffea87`e5e06520 fffff800`149fb169     : ffffb90f`9492b570 ffffea87`e5e06610 ffffb90f`949ffb30 ffffb90f`94460040 : nt!VerifierBugCheckIfAppropriate+0xe0
ffffea87`e5e06560 fffff800`144a85e0     : ffffb90f`9492b570 ffffb90f`949ffbd0 00000000`00000001 ffffb90f`94460020 : nt!VfPoolCheckForLeaks+0x49
ffffea87`e5e065a0 fffff800`149dd502     : 00000000`00274000 ffffb90f`949ffb30 fffff800`14c3aeb0 fffff800`14c3aeb0 : nt!VfTargetDriversRemove+0x11a174
ffffea87`e5e06620 fffff800`146c7553     : ffffb90f`949ffb30 ffffea87`e5e06750 00000000`00000001 00000000`ffffffff : nt!VfDriverUnloadImage+0x3e
ffffea87`e5e06650 fffff800`1477e001     : 00000000`00000000 fffff800`ffffffff ffff241c`00000001 ffffa303`e1ed1240 : nt!MiUnloadSystemImage+0x2eb
ffffea87`e5e067f0 fffff800`1477df2e     : ffffb90f`8c4e4a60 ffffea87`e5e06a40 00000000`00000000 00000000`00000000 : nt!MmUnloadSystemImage+0x41
ffffea87`e5e06820 fffff800`146913e0     : ffffb90f`8c4e4a60 ffffea87`e5e06a40 ffffb90f`8c4e4a60 fffff800`1465b7df : nt!IopDeleteDriver+0x4e
ffffea87`e5e06870 fffff800`14226357     : 00000000`00000000 00000000`00000000 ffffea87`e5e06a40 ffffb90f`8c4e4a90 : nt!ObpRemoveObjectRoutine+0x80
ffffea87`e5e068d0 fffff800`1422627e     : 00000000`00000008 00000000`00000000 ffffb90f`8c4e4a60 00000000`c0000365 : nt!ObfDereferenceObjectWithTag+0xc7
ffffea87`e5e06910 fffff800`1474814a     : 00000000`00000008 00000000`00000008 00000000`ffffffa1 00000000`00001000 : nt!HalPutDmaAdapter+0xe
ffffea87`e5e06940 fffff800`14a6bda3     : ffffb90f`945ff658 ffffb90f`945ff658 ffffea87`e5e06b80 00000000`00000000 : nt!IopLoadDriver+0x76a
ffffea87`e5e06b10 fffff800`14a6072a     : ffffffff`ffffffa1 ffffa303`e51fefd0 00000000`00000000 fffff800`116d7750 : nt!IopInitializeSystemDrivers+0x157
ffffea87`e5e06bb0 fffff800`147a485b     : fffff800`116d7750 fffff800`14c64fe8 fffff800`147a4820 fffff800`116d7750 : nt!IoInitSystem+0x2e
ffffea87`e5e06be0 fffff800`14373995     : ffffb90f`89ec3080 fffff800`147a4820 fffff800`116d7750 a1a64fe8`ce8b48d7 : nt!Phase1Initialization+0x3b
ffffea87`e5e06c10 fffff800`1441c938     : fffff800`11a4c180 ffffb90f`89ec3080 fffff800`14373940 058b4800`0040b7e8 : nt!PspSystemThreadStartup+0x55
ffffea87`e5e06c60 00000000`00000000     : ffffea87`e5e07000 ffffea87`e5e01000 00000000`00000000 00000000`00000000 : nt!KiStartSystemThread+0x28


IMAGE_VERSION:  1.7.0.0

STACK_COMMAND:  .cxr; .ecxr ; kb

FAILURE_BUCKET_ID:  0xc4_62_VRF_LEAKED_POOL_IMAGE_gvm.sys

OS_VERSION:  10.0.19041.1

BUILDLAB_STR:  vb_release

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 10

FAILURE_ID_HASH:  {a0c4665d-e60a-f4d0-aeeb-0605adf8a6ef}

Followup:     MachineOwner
---------

5: kd> lmvm gvm
Browse full module list
start             end                 module name
fffff809`56390000 fffff809`56604000   gvm        (no symbols)           
    Loaded symbol image file: gvm.sys
    Image path: \SystemRoot\system32\DRIVERS\gvm.sys
    Image name: gvm.sys
    Browse all global symbols  functions  data
    Timestamp:        Wed Sep 23 03:38:16 2020 (5F6A52A8)
    CheckSum:         00065EDA
    ImageSize:        00274000
    File version:     1.7.0.0
    Product version:  1.7.0.0
    File flags:       2 (Mask 3F) Pre-release
    File OS:          40004 NT Win32
    File type:        3.7 Driver
    File date:        00000000.00000000
    Translations:     0409.04b0
    Information from resource tables:
        CompanyName:      Google LLC
        ProductName:      Android Emulator Hypervisor Driver for AMD Processors
        InternalName:     Android Emulator Hypervisor Driver for AMD Processors
        OriginalFilename: gvm.sys
        ProductVersion:   1.7
        FileVersion:      1.7
        FileDescription:  Android Emulator Hypervisor Driver for AMD Processors
        LegalCopyright:   Copyright (c) 2019 Google LLC

好吧,这个驱动只是阻止了系统启动,并不一定就是真凶。为了进一步排查,我又重新打开了「Driver Verifier」,当然这次长了记性,设置它的启动模式为「启动失败时自动关闭」。

1
verifier /bootmode resetonbootfail

接下来的2天内我打算继续运行「Driver Verifier」,看看还有没有其他的幕后黑手。看看这一串蓝屏记录,我是真的能忍🙃 eventlog

Reference

Licensed under CC BY-NC-SA 4.0
使用 Hugo 构建
主题 StackJimmy 设计