Linux服務(wù)器高I/O等待延遲問(wèn)題查找
作者：董桉遠(yuǎn)
最近負(fù)責(zé)調(diào)查某一個(gè)服務(wù)器卡頓的問(wèn)題，記錄一下自己調(diào)查過(guò)程中學(xué)到的東西。Linux服務(wù)器高I/O等待延遲問(wèn)題查找……
0. 首先是top查看一下系統(tǒng)狀況
發(fā)現(xiàn)兩個(gè)參數(shù)異常，一是平均負(fù)載高，一是cpu %wa一直在50%以上。
查了一下%wa參數(shù)的意義：
- wa -- iowait
- Amount of time the CPU has been waiting for I/O to complete.
1. 查看磁盤(pán)讀寫(xiě)數(shù)據(jù)
用iostat 查看磁盤(pán)狀況
- $ iostat -x 2 5
- avg-cpu: %user %nice %system %iowait %steal %idle
- 3.66 0.00 47.64 48.69 0.00 0.00
- Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
- sda 44.50 39.27 117.28 29.32 11220.94 13126.70 332.17 65.77 462.79 9.80 2274.71 7.60 111.41
- dm-0 0.00 0.00 83.25 9.95 10515.18 4295.29 317.84 57.01 648.54 16.73 5935.79 11.48 107.02
- dm-1 0.00 0.00 57.07 40.84 228.27 163.35 8.00 93.84 979.61 13.94 2329.08 10.93 107.02
2.查看進(jìn)程狀態(tài)
進(jìn)程狀態(tài)表
- PROCESS STATE CODES
- D uninterruptible sleep (usually IO)
- R running or runnable (on run queue)
- S interruptible sleep (waiting for an event to complete)
- T stopped, either by a job control signal or because it is being traced.
- W paging (not valid since the 2.6.xx kernel)
- X dead (should never be seen)
- Z defunct ("zombie") process, terminated but not reaped by its parent.
查看到進(jìn)程中狀態(tài)D（disk sleep）的進(jìn)程
- # for x in `seq 1 1 10`; do ps -eo state,pid,cmd | grep "^D"; echo "----"; sleep 5; done
查看到有kjournald這個(gè)進(jìn)程
查看這個(gè)進(jìn)程是做什么的？
kjournald是ext3文件系統(tǒng)記錄文件系統(tǒng)日志的進(jìn)程。
查看進(jìn)程相關(guān)狀態(tài)信息
- # cat /proc/487/io
- rchar: 48752567
- wchar: 549961789
- syscr: 5967
- syscw: 67138
- read_bytes: 49020928
- write_bytes: 549961728
- cancelled_write_bytes: 0
查看都往哪寫(xiě)東西了
- # lsof -p 487
引起kjournald 狀態(tài)D的原因還在查。。。。
責(zé)任編輯：牛小雨
來(lái)源：
董桉遠(yuǎn)的博客
