记一次某制造业ERP系统 CPU打爆事故分析

一:背景1.讲故事前些天有位朋友微信找到我,说他的程序出现了CPU阶段性爆高,过了一会就下去了,咨询下这个爆高阶段程序内部到底发生了什么? 画个图大概是下面这样,你懂的 。

记一次某制造业ERP系统 CPU打爆事故分析

文章插图
按经验来说,这种情况一般是程序在做 CPU 密集型运算,所以让朋友在 CPU 高的时候间隔 5~10s 抓两个 dump 下来,然后就是用 WinDbg 分析 。
二:WinDbg 分析1. CPU 真的爆高吗耳听为虚,眼见为实,我们用 !tp 观察下当前的CPU情况 。
0:000> !tpCPU utilization: 100%Worker Thread: Total: 16 Running: 2 Idle: 14 MaxLimit: 32767 MinLimit: 2Work Request in Queue: 0--------------------------------------Number of Timers: 2--------------------------------------Completion Port Thread:Total: 2 Free: 2 MaxFree: 4 CurrentLimit: 2 MaxLimit: 1000 MinLimit: 2果不其然,CPU直接打满,接下来就是看看当前有几个CPU逻辑核,这么不够扛 。。。
0:000> !cpuidCPF/M/SManufacturerMHz 06,106,6<unavailable>2700 16,106,6<unavailable>2700我去,一个生产环境居然只有两个核 。。。果然这大环境下公司活着都不够滋润 。
2. 到底是谁引发的既然是阶段性爆高,最简单粗暴的就是看下各个线程栈,使用 ~*e !clrstack 命令即可,因为只有两核,所以理论上两个线程就可以把 CPU 干趴下,扫了一下线程栈,果然有对号入座的,输出信息如下:
0:000> ~*e !clrstackOS Thread Id: 0x146c (42)Child SPIP Call Site00000089abcfca18 00007ffc4baffdb4 [InlinedCallFrame: 00000089abcfca18] System.Drawing.SafeNativeMethods+Gdip.IntGdipDisposeImage(System.Runtime.InteropServices.HandleRef)00000089abcfca18 00007ffbdd4a7a48 [InlinedCallFrame: 00000089abcfca18] System.Drawing.SafeNativeMethods+Gdip.IntGdipDisposeImage(System.Runtime.InteropServices.HandleRef)00000089abcfc9f0 00007ffbdd4a7a48 DomainNeutralILStubClass.IL_STUB_PInvoke(System.Runtime.InteropServices.HandleRef)00000089abcfcaa0 00007ffbdd52ad0a System.Drawing.SafeNativeMethods+Gdip.GdipDisposeImage(System.Runtime.InteropServices.HandleRef)00000089abcfcae0 00007ffbdd52ac3f System.Drawing.Image.Dispose(Boolean)00000089abcfcb30 00007ffbdd556b5a System.Drawing.Image.Dispose()00000089abcfcb60 00007ffbe39397c7 NPOI.SS.Util.SheetUtil.GetCellWidth(NPOI.SS.UserModel.ICell, Int32, NPOI.SS.UserModel.DataFormatter, Boolean)00000089abcfcc00 00007ffbe3939654 NPOI.SS.Util.SheetUtil.GetCellWidth(NPOI.SS.UserModel.ICell, Int32, NPOI.SS.UserModel.DataFormatter, Boolean)00000089abcfcd30 00007ffbe39382e1 NPOI.SS.Util.SheetUtil.GetColumnWidth(NPOI.SS.UserModel.ISheet, Int32, Boolean)00000089abcfcdc0 00007ffbe39380bc NPOI.XSSF.UserModel.XSSFSheet.AutoSizeColumn(Int32, Boolean)...OS Thread Id: 0x1c8c (46)Child SPIP Call Site00000089ad43dba8 00007ffc4baffdb4 [InlinedCallFrame: 00000089ad43dba8] System.Drawing.SafeNativeMethods+Gdip.IntGdipDisposeImage(System.Runtime.InteropServices.HandleRef)00000089ad43dba8 00007ffbdd4a7a48 [InlinedCallFrame: 00000089ad43dba8] System.Drawing.SafeNativeMethods+Gdip.IntGdipDisposeImage(System.Runtime.InteropServices.HandleRef)00000089ad43db80 00007ffbdd4a7a48 DomainNeutralILStubClass.IL_STUB_PInvoke(System.Runtime.InteropServices.HandleRef)00000089ad43dc30 00007ffbdd52ad0a System.Drawing.SafeNativeMethods+Gdip.GdipDisposeImage(System.Runtime.InteropServices.HandleRef)00000089ad43dc70 00007ffbdd52ac3f System.Drawing.Image.Dispose(Boolean)00000089ad43dcc0 00007ffbdd556b5a System.Drawing.Image.Dispose()00000089ad43dcf0 00007ffbe39397c7 NPOI.SS.Util.SheetUtil.GetCellWidth(NPOI.SS.UserModel.ICell, Int32, NPOI.SS.UserModel.DataFormatter, Boolean)00000089ad43dd90 00007ffbe3939654 NPOI.SS.Util.SheetUtil.GetCellWidth(NPOI.SS.UserModel.ICell, Int32, NPOI.SS.UserModel.DataFormatter, Boolean)00000089ad43dec0 00007ffbe39382e1 NPOI.SS.Util.SheetUtil.GetColumnWidth(NPOI.SS.UserModel.ISheet, Int32, Boolean)00000089ad43df50 00007ffbe39380bc NPOI.XSSF.UserModel.XSSFSheet.AutoSizeColumn(Int32, Boolean)...00000089ad43e460 00007ffbe115b193 System.Web.Mvc.ControllerActionInvoker.InvokeActionMethod(System.Web.Mvc.ControllerContext, System.Web.Mvc.ActionDescriptor, System.Collections.Generic.IDictionary`2<System.String,System.Object>)...00000089abcfd310 00007ffbe115b147 System.Web.Mvc.Async.AsyncControllerActionInvoker+c.b__9_0(System.IAsyncResult, ActionInvocation)...

经验总结扩展阅读