03/2014 ~ 賈格的神秘古堡

2014-03-26

Windbg内核调试之五: 一次利用Dump文件调试Deadlock的实战

http://www.cnblogs.com/Sonic2007/archive/2010/09/01/1541988.html

[CaseStudy][WinDbg] Hang

[Scenario]
User 發生了Hang Issue

[開始查案]
根據收到的Dump 通常會發生Hang 應該是resource被佔住或是deadlock之類的
所以第一步

!locks

0: kd> !locks
**** DUMP OF ALL RESOURCE OBJECTS ****
KD: Scanning for held locks...

Resource @ Ntfs!NtfsData (0x88b01474)    Exclusively owned
     Threads: 84acd020-01<*> 
KD: Scanning for held locks..

Resource @ 0x851d756c    Exclusively owned
    Contention Count = 41
    NumberOfSharedWaiters = 29
     Threads: 84e79d48-02<*> 85c98790-01    863ec940-01    86557030-01    
              84bc9d48-01    8603aa60-01    84e7b8f0-01    861073b8-01    
              8639f5b0-01    84e42648-01    84acd020-01    84c1da58-01    
              84cf3d48-01    84c48030-01    84c54a88-01    84e3f078-01    
              8602ec58-01    86203d48-01    85ecd660-01    86202640-01    
              8620f310-01    862bc030-01    862a9d48-01    84e79410-01    
              86434c78-01    86a5fa60-01    863f82b8-01    8645d998-01    
              8605f030-01    86a46a08-01    
KD: Scanning for held locks..............................................................

Resource @ 0x85d8ae68    Exclusively owned
    Contention Count = 2
     Threads: 8605f030-01<*> 
KD: Scanning for held locks.

Resource @ 0x85d88840    Exclusively owned
    Contention Count = 1
    NumberOfExclusiveWaiters = 1
     Threads: 8605f030-01<*> 
     Threads Waiting On Exclusive Access:
              860d88d8       

KD: Scanning for held locks...................................................................................................................................................................................................................................

Resource @ 0x84d8ff34    Shared 1 owning threads
    Contention Count = 2
    NumberOfExclusiveWaiters = 1
     Threads: 84acd420-01<*> 
     Threads Waiting On Exclusive Access:
              84e79d48       

KD: Scanning for held locks..............................
10293 total locks, 5 locks currently held

Threads: 84e79d48-02<*> 打*代表這條Thread 佔了這個resource
從上面資訊來看
最下面的Resource
Resource @ 0x84d8ff34 Shared 1 owning threads
卡住了Thread 84e79d48

而Thread 84e79d48 的Resource
Resource @ 0x851d756c Exclusively owned
卡住了Thread 8605f030
Resource @ 0x85d88840 Exclusively owned
Resource @ 0x85d8ae68 Exclusively owned

所以才造成System Hang

讓我們來看看罪魁禍首 Thread 84e79d48到底在幹嘛

0: kd> !thread 84acd420
THREAD 84acd420  Cid 0004.001c  Teb: 00000000 Win32Thread: 00000000 WAIT: (WrQueue) UserMode Non-Alertable
    82948600  QueueObject
Not impersonating
DeviceMap                 89a088d8
Owning Process            84a41a20       Image:         System
Attached Process          N/A            Image:         N/A
Wait Start TickCount      23593          Ticks: 1024 (0:00:00:15.974)
Context Switch Count      1966           IdealProcessor: 0             
UserTime                  00:00:00.000
KernelTime                00:00:00.483
Win32 Start Address nt!ExpWorkerThread (0x8288999e)
Stack Init 8ad0bfd0 Current 8ad0bc10 Base 8ad0c000 Limit 8ad09000 Call 0
Priority 14 BasePriority 13 UnusualBoost 1 ForegroundBoost 0 IoPriority 2 PagePriority 5
ChildEBP RetAddr  Args to Child              
8ad0bc28 8288a69d 84acd420 8293af08 82937d20 nt!KiSwapContext+0x26 (FPO: [Uses EBP] [0,0,4])
8ad0bc60 828894f7 84acd4e0 84acd420 82948600 nt!KiSwapThread+0x266
8ad0bc88 8288a1ed 84acd420 84acd4e0 00000000 nt!KiCommitThreadWait+0x1df
8ad0bcec 82889a83 82948600 00000001 00000000 nt!KeRemoveQueueEx+0x4f8
8ad0bd50 82a15f5e 00000000 a9364519 00000000 nt!ExpWorkerThread+0xe5
8ad0bd90 828bd219 8288999e 00000000 00000000 nt!PspSystemThreadStartup+0x9e
00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x19

結果看來很正常看不出什麼東西
這時候來看看Resource吧

對Resource @ 0x84d8ff34位置下dt
dt 0x84d8ff34 _eresource
Why _eresource 我想應該是ntdll的變數吧
總之我們看第7個變數
+0x018 OwnerEntry : _OWNER_ENTRY
所以一樣對它下dt
dt 0x84d8ff34 _eresource OwnerEntry.
要記得打"." 才能看更多資訊

//
// Checking of resource of 0x84d8ff34.
//
0: kd> dt 0x84d8ff34 _eresource
ntdll!_ERESOURCE
   +0x000 SystemResourcesList : _LIST_ENTRY [ 0x84d8f884 - 0x84d8fed4 ]
   +0x008 OwnerTable       : 0x84d8e158 _OWNER_ENTRY
   +0x00c ActiveCount      : 0n1
   +0x00e Flag             : 4
   +0x010 SharedWaiters    : (null) 
   +0x014 ExclusiveWaiters : 0x84d8f838 _KEVENT
   +0x018 OwnerEntry       : _OWNER_ENTRY
   +0x020 ActiveEntries    : 1
   +0x024 ContentionCount  : 2
   +0x028 NumberOfSharedWaiters : 0
   +0x02c NumberOfExclusiveWaiters : 1
   +0x030 Address          : (null) 
   +0x030 CreatorBackTraceIndex : 0
   +0x034 SpinLock         : 0
//
// no thread exclusively owns this resource.
//
0: kd> dt 0x84d8ff34 _eresource OwnerEntry.
ntdll!_ERESOURCE
   +0x018 OwnerEntry  : 
      +0x000 OwnerThread : 0
      +0x004 IoPriorityBoosted : 0y0
      +0x004 OwnerReferenced : 0y0
      +0x004 OwnerCount  : 0y000000000000000000000000000000 (0)
      +0x004 TableSize   : 0

沒東西
來看看OwnerTable吧

dt 0x84d8e158 _OWNER_ENTRY
看到TableSize = 7 所以後面可能有7個Entry
我們可以先用 ?? sizeof(_OWNER_ENTRY) 看OWNER_ENTRY的size

unsigned int 8

所以我們知道下一個Entry 就要再+8

dt 0x84d8e160 _OWNER_ENTRY

0: kd> dt 0x84d8e158 _OWNER_ENTRY
ntdll!_OWNER_ENTRY
   +0x000 OwnerThread      : 0
   +0x004 IoPriorityBoosted : 0y1
   +0x004 OwnerReferenced  : 0y1
   +0x004 OwnerCount       : 0y000000000000000000000000000001 (0x1)
   +0x004 TableSize        : 7
0: kd> dt 0x84d8e160 _OWNER_ENTRY
ntdll!_OWNER_ENTRY
   +0x000 OwnerThread      : 0x84acd420  ---> thread of 0x84acd420 shared owns this resource 
   +0x004 IoPriorityBoosted : 0y0
   +0x004 OwnerReferenced  : 0y0
   +0x004 OwnerCount       : 0y000000000000000000000000000001 (0x1)
   +0x004 TableSize        : 4

從第二個Entry看到了 OwnerThread 0x84acd420
結果還是這條System Thread呀

所以案子到這邊查不下去了
我們僅知道System Thread這個Resource沒有釋放
所以Lock住其他需要Exclusively owned的Resource 也造成了Hang

劉栩安的日常生活 - 3M to 6M

6個月的安安身高69公分體重8.1公斤頭圍44公分

很開心安安的身高追上來了

來看看這三個月間的紀錄吧 (整理時發現這三個月拍了不少照片呢~)

安安的心情都寫在衣服上了 - 我要抱抱

3M17D

睡夢中的來追我阿

3M19D

啊～好睏喔睡著的安安

很懷念他以前好睡的模式

現在有點睡眠障礙呢

安安穿上從美國買來的Carter's

3M24D

安吸的不是手指是推理!!!!

是不是半夜偷看神探伽利略XD

3M25D

戴上新帽子安安的咚吱咚吱開心的勒~

大概從這時候就會發現安安的頭蠻大的

帽子戴起來都很緊繃~

安安四個月了開始會嘗試坐起來~

四個月傳說中的收口水儀式~

4M1D

一家三口第一次外出旅行長榮很貼心地幫忙準備嬰兒所需用品 at 長榮礁溪鳳凰酒店.

很多人說安安跟媽媽都很甜喔~

請叫我頭好壯壯黃飛鴻~

笑容真的很甜~

抬腳腳運動~

4M16D

第一次翻身囉其實我們很少讓安安趴著今晚想說訓練一下沒想到居然真的翻了

4M19D

今年的除夕團圓飯多了安安喔雖然他是戰力外

吃完團圓飯之後安安穿上過年限定的戰袍出征囉~

4M20D

大年初一每次看我們吃東西嘴巴都一直動的安安第一次嘗嘗米精的味道

想不到一試成主顧

大年初一下午來志清湖散散心 — at 長庚大學志清湖

4M21D

大年初二親戚們來拜年安安換上另一套服裝見客囉

大家都說安安可愛極了~

4M22D

大年初三好天氣是個適合出遊的日子我們選擇到陽明山賞花下山順道去美麗華晃晃 at 陽明山

下山去逛美麗華鋼鐵人特展

安安 : 喔喔喔~~~

4M26D

台中新社賞櫻 at 新社區公所

賞櫻不忘聞香

5M5D

例行趴趴練習目標為前方5公分處長頸鹿流口水了啦

5M5D

自從餵了南瓜泥之後安安超愛吃飯時老是高興地跳來跳去常常吃得滿嘴都是副作用：黃色大便又多又臭

5M6D

開始嘗試餵紅蘿蔔泥好像不太好吃耶~

5M15D

到了亂哺口水的年紀整天哺來哺去的口水噴的到處都是

5M19D

應該是要長牙了今天安安咬我的手時我開始感覺到被硬硬的東西切斷的錯覺打開嘴巴一看兩個白白的牙齒就要冒出頭來囉～

老婆一直說安安哭哭很像郝劭文有像嗎?

5M20D

正所謂阿公阿婆疼孫安安立馬就擁有人生第一台車～

我一直要安安的阿公阿婆不要這麼著急

按部就班地長大就好

阿婆常常偷偷違反規則ㄟ~

這部車被我要求收起來了等安安7個月大之後再開箱

5M26D

去月心同事家作客恰好月心同事也一起來拜訪

安安是哥哥相當的主動唷

兩小無猜的互動很可愛呢~

以上是迷糊老爸不負責報導
請大家敬請期待下一期6Mto9M的成長紀錄~

[CaseStudy][WinDbg] Crash - Break instruction exception

[Scenario]
User 從Agent打開log 立即發生了Crash

[開始查案]
根據收到的Dump

0:000> !analyze -v

FAULTING_IP: 

+ff79600

00000000`00000000 ??              ???

EXCEPTION_RECORD:  ffffffffffffffff -- (.exr 0xffffffffffffffff)

ExceptionAddress: 0000000000000000

   ExceptionCode: 80000003 (Break instruction exception)

  ExceptionFlags: 00000000

NumberParameters: 0

STACK_TEXT:  

00000000`0011f9f8 000007fe`fd8b1430 : 00000000`01e975d0 00000000`7752300a 00000000`01e975d0 00000000`01e9d868 : ntdll!NtWaitForMultipleObjects+0xa

00000000`0011fa00 00000000`77511220 : 00000000`0011fb40 00000000`0011fb30 00000000`00000000 00000000`00000000 : KERNELBASE!WaitForMultipleObjectsEx+0xe8

00000000`0011fb00 00000001`40034e09 : 00000000`00000000 7fffffff`fffffffe 00000001`4012d390 00000000`00000000 : kernel32!WaitForMultipleObjects+0xb0

00000000`0011fb90 00000001`4002afc7 : 00000001`40137fc0 00000000`00000000 00000000`00000000 00000000`001c3a00 : XXX!CWindowManagerClient::Run+0xe9

...(Ignore)

只知道原因是Break instruction exception 但是從這個Stack看起來卻沒有幫助因為這條Stack就是顯示主程式起來之後在等結束

正常我們會從FOLLOWUP_IP來看預計下一步會執行的點

FOLLOWUP_IP: 

XXX!CWindowManagerClient::Run+e9 
00000001`40034e09 3d81000000      cmp     eax,81h

0:000> uf 00000001`40034e09
XXX!CWindowManagerClient::Run+0xd1 [XXX\windowmanagerclient.cpp @ 168]:
  168 00000001`40034df1 41b901000000    mov     r9d,1
  168 00000001`40034df7 4533c0          xor     r8d,r8d
  168 00000001`40034dfa 488d542468      lea     rdx,[rsp+68h]
  168 00000001`40034dff 418d4901        lea     ecx,[r9+1]
  168 00000001`40034e03 ff1537080a00    call    qword ptr [XXX!_imp_WaitForMultipleObjects (00000001`400d5640)]
  169 00000001`40034e09 3d81000000      cmp     eax,81h
  169 00000001`40034e0e 7775            ja      XXX!CWindowManagerClient::Run+0x165 (00000001`40034e85)

可以看出就是在00000001`40034e03 Call _imp_WaitForMultipleObjects時出了問題

但是這樣還是看不出什麼端倪

所以先列出所有的thread 看是否有異狀

0:000> ~*knb
...(Ignore)
...
22  Id: 6b98.1eb0 Suspend: 0 Teb: 000007ff`fff7c000 Unfrozen
 # Child-SP          RetAddr           Call Site
00 00000000`09cb89c8 000007fe`fd8b1430 ntdll!NtWaitForMultipleObjects+0xa
01 00000000`09cb89d0 00000000`77522ce3 KERNELBASE!WaitForMultipleObjectsEx+0xe8
02 00000000`09cb8ad0 00000000`77599105 kernel32!WaitForMultipleObjectsExImplementation+0xb3
03 00000000`09cb8b60 00000000`77599287 kernel32!WerpReportFaultInternal+0x215
04 00000000`09cb8c00 00000000`775992df kernel32!WerpReportFault+0x77
05 00000000`09cb8c30 00000000`775994fc kernel32!BasepReportFault+0x1f
06 00000000`09cb8c60 00000000`60201ad5 kernel32!UnhandledExceptionFilter+0x1fc
07 00000000`09cb8d40 00000000`60203547 XXX!_invalid_parameter+0xc5 [f:\sp\vctools\crt_bld\self_64_amd64\crt\src\invarg.c @ 88]
08 00000000`09cb9300 00000000`6006758b XXX!wcscpy_s+0x97 [f:\sp\vctools\crt_bld\self_64_amd64\crt\src\tcscpy_s.inl @ 30]
09 00000000`09cb9340 00000000`600502d7 XXX!CLogQueryEntry::FillDlpData+0x11bb [XXX\logcl_logquery.cpp @ 3211]
0a 00000000`09cb9510 00000000`60049349 XXX!CLogQueryEntry::OnLogviewVirus+0x17f7 [XXX\logcl_logquery.cpp @ 900]

結果在第22條thread 發現了UnhandledException
很明顯
在做wcscpy_s 時參數錯誤
來看看定義

errno_t wcscpy_s(
   wchar_t *strDestination,
   size_t numberOfElements,
   const wchar_t *strSource 
);

這題的答案是
wcscpy_s 時的Src size 比Des大所以造成Crash

Compile Error when we build the GoogleTest in VS2012

我嘗試要在VS2012上試試Unit Test

Step 1 :
去googletest 網站download googletest project
https://code.google.com/p/googletest/

目前版號是gtest-1.7.0

Step 2：
解壓縮zip

Step 3 :
用VS2012 開啟專案
.\gtest-1.7.0\msvc\

Step 4 :
Rebuild projects

根據Primer上的說明欲先玩googletest 必先compile之
To write a test program using Google Test, you need to compile Google Test into a library and link your test with it.

但是正常情況下會compile error
錯誤原因為 :
Error 5 error C2977: 'std::tuple' : too many template arguments d:\coding\gtest-1.7.0\include\gtest\gtest-printers.h

[Root Cause]
http://stackoverflow.com/questions/12558327/google-test-in-visual-studio-2012

VC++ 2012 does not (and will never) support variadic templates; consequently, its standard library implementation attempts to fake them using preprocessor-generated overloads and specializations. As this blog post notes, the number of faux variadic template parameters defaults to 5 – the problem is that gtest is trying to instantiate std::tuple<> with as many as 10 template arguments.

As the blog post goes on to mention, you can define the _VARIADIC_MAX macro project-wide to support up through 10 parameters; you'll probably want to do this by way of your project's preprocessor definitions setting.

[Solution]
打開4個Project的Property
properties -> C\C++ Preprocessor -> Processor Definitions -> <edit...>

Add "_VARIADIC_MAX=10"
Save and Rebuild all projects

[Reference] F-Droid 收集Android App Open Source 的網站

F-Droid 收集Android App Open Source 的網站

https://f-droid.org/repository/browse/

賈格的神秘古堡

[公告] 網誌搬家囉搬到 Medium 囉 : https://medium.com/juggernauts-agile-stories

Settings

Tags

About Juggernaut

2014-03-26

[Reference] 教你怎麼看Windbg

Windbg内核调试之五: 一次利用Dump文件调试Deadlock的实战

[CaseStudy][WinDbg] Hang

劉栩安的日常生活 - 3M to 6M

2014-03-22

[CaseStudy][WinDbg] Crash - Break instruction exception

2014-03-17

Compile Error when we build the GoogleTest in VS2012

2014-03-07

[Reference] F-Droid 收集Android App Open Source 的網站

搜尋此網誌

Total Pageviews

Popular Posts

Categories

標籤

Blog Archive