_ _
__ _ _ _ (_) __| |
/ _` || | | || | / _` | / _ \| '__|
| (_| || |_| || || (_| || __/| |
\__, | \__,_||_| \__,_| \___||_|
|___/
Do you struggle to improve system performance or to find root cause that makes system abnormal?
Guider is made to measure the amount of system resource usage and to trace system behavior.
You can analyze your performance issues effectively with this tool.
Guider pursues characteristics as below.
- Easy to use: just run without any setting and package installation
- Measure correctly: count, time in from us, size in from byte
- Provide all features: enough functions for experiment and analysis
- Submit the report in detail: show as much information as possible
It usually supports all platforms based on the Linux kernel as shown below.
- Android
- distro (Ubuntu, CentOS, RHEL, Linux Mint, Arch Linux, ...)
- webOS
- ccOS
- Tizen
- Windows (only for drawing, reporting, networking)
The features of Guider are as follows.
# ./guider.py top -a
[Top Info] [Time: 71406.120] [Interval: 1.0] [Ctxt: 52687] [Life: +0/-0] [IRQ: 12517] [Core: 24] [Task: 326/433] [Load: 0.2/0.4/0.5] [RAM: 63876] [Swap: 65491]
[Cycle: 2G / Inst: 6G / IPC: 2.45 / CacheMiss: 77K(6%) / BranchMiss: 857K(0%) / Clock: 22G / MinFlt: 4 / MajFlt: 0]
==========================================================================================================================================================
ID | CPU (Usr/Ker/Blk/IRQ)| Mem (Diff/ User/Cache/Kern)| Swap (Diff/ I/O )|NrPgRclm | BlkRW | NrFlt | NrBlk | NrSIRQ | NrMlk | NrDrt | Network |
----------------------------------------------------------------------------------------------------------------------------------------------------------
Total | 6 %( 4 / 0 / 0 / 0 )|11074( 0/ 905/50751/1146)| 0 ( 0 / 0/0 )| 0/0 | 0/0 | 0 | 0 | 1658 | 0 | 7 | 1K/11K |
----------------------------------------------------------------------------------------------------------------------------------------------------------
Core/0 | 1 %( 0 / 0 / 0 / 0 )| | powersave | 0-0 | ? C | 1288 Mhz [1171-2441]|
Core/1 | 1 %( 0 / 0 / 0 / 0 )| | powersave | 0-1 | ? C | 1530 Mhz [1171-2441]|
Core/2 | 1 %( 0 / 0 / 0 / 0 )| | powersave | 0-2 | ? C | 1171 Mhz [1171-2441]|
Core/3 | 1 %( 0 / 0 / 0 / 0 )| | powersave | 0-3 | ? C | 1173 Mhz [1171-2441]|
Core/4 | 8 %( 1 / 2 / 0 / 0 )|##### | powersave | 0-4 | ? C | 1171 Mhz [1171-2441]|
Core/5 | 1 %( 0 / 0 / 0 / 0 )| | powersave | 0-5 | ? C | 1175 Mhz [1171-2441]|
Core/6 | 1 %( 0 / 0 / 0 / 0 )| | powersave | 1-0 | ? C | 2330 Mhz [1171-2441]|
Core/7 | 1 %( 0 / 0 / 0 / 0 )| | powersave | 1-1 | ? C | 2342 Mhz [1171-2441]|
Core/8 | 1 %( 0 / 0 / 0 / 0 )| | powersave | 1-2 | ? C | 2367 Mhz [1171-2441]|
Core/9 | 1 %( 0 / 0 / 0 / 0 )| | powersave | 1-3 | ? C | 2246 Mhz [1171-2441]|
Core/10|100 %(99 / 0 / 0 / 0 )|###################################################################| powersave | 1-4 | ? C | 2246 Mhz [1171-2441]|
Core/11| 1 %( 0 / 0 / 0 / 0 )| | powersave | 1-5 | ? C | 2290 Mhz [1171-2441]|
Core/12| 1 %( 0 / 0 / 0 / 0 )| | powersave | 0-0 | ? C | 1171 Mhz [1171-2441]|
Core/13| 4 %( 3 / 0 / 0 / 0 )|## | powersave | 0-1 | ? C | 1953 Mhz [1171-2441]|
Core/14| 1 %( 0 / 0 / 0 / 0 )| | powersave | 0-2 | ? C | 1952 Mhz [1171-2441]|
Core/15| 1 %( 0 / 0 / 0 / 0 )| | powersave | 0-3 | ? C | 1171 Mhz [1171-2441]|
Core/16| 12 %( 2 / 3 / 0 / 0 )|######## | powersave | 0-4 | ? C | 1953 Mhz [1171-2441]|
Core/17| 1 %( 0 / 0 / 0 / 0 )| | powersave | 0-5 | ? C | 1846 Mhz [1171-2441]|
Core/18| 1 %( 0 / 0 / 0 / 0 )| | powersave | 1-0 | ? C | 2278 Mhz [1171-2441]|
Core/19| 1 %( 0 / 0 / 0 / 0 )| | powersave | 1-1 | ? C | 2243 Mhz [1171-2441]|
Core/20| 1 %( 0 / 0 / 0 / 0 )| | powersave | 1-2 | ? C | 2247 Mhz [1171-2441]|
Core/21| 1 %( 0 / 0 / 0 / 0 )| | powersave | 1-3 | ? C | 2246 Mhz [1171-2441]|
Core/22| 1 %( 0 / 0 / 0 / 0 )| | powersave | 1-4 | ? C | 2440 Mhz [1171-2441]|
Core/23| 1 %( 0 / 0 / 0 / 0 )| | powersave | 1-5 | ? C | 2393 Mhz [1171-2441]|
==========================================================================================================================================================
Process ( PID/ PPID/ Nr/ Pri)| CPU(Usr/Ker/Dly)| Mem(RSS/Txt/Shr/Swp)| Blk( RD / WR /NrFlt)| Yld | Prmt | FD | LifeTime| WaitChannel |
----------------------------------------------------------------------------------------------------------------------------------------------------------
yes ( 2075/ 9085/ 1/C 0)| 99( 99/ 0/ 0)| 8( 0/ 0/ 0/ 0)| 0( -/ -/ 0)| 0| 0| 256| 0: 3:43| RUNNING |
a.out ( 2082/ 9085/ 3/C 0)| 16( 6/ 10/ -)| 30( 1/ 0/ 1/ 0)| 0( -/ -/ 0)| 0| 0| 256| 0: 3:36| futex_wait_queue_me |
guider ( 2182/ 9085/ 1/C 0)| 3( 3/ 0/ 0)| 101( 62/ 2/ 5/ 0)| 0( -/ -/ 0)| 1| 1|1024| 0: 0: 2| RUNNING |
bash ( 6960/ 6959/ 1/C 0)| 0( 0/ 0/ -)| 24( 6/ 0/ 3/ 0)| 0( -/ -/ 0)| 0| 0| 256| 20:26:27| do_wait |
vi ( 7200/ 7197/ 1/C 0)| 0( 0/ 0/ -)| 58( 8/ 1/ 5/ 0)| 0( -/ -/ 0)| 0| 0| 64| 20:24:23|poll_schedule_timeout|
bash ( 7916/ 6959/ 1/C 0)| 0( 0/ 0/ -)| 24( 6/ 0/ 3/ 0)| 0( -/ -/ 0)| 0| 0| 256| 19:57:58| do_wait |
nmbd ( 2960/ 1/ 1/C 0)| 0( 0/ 0/ -)| 91( 4/ 3/ 4/ 0)| 0( -/ -/ 0)| 0| 0| 64| 1K:20:52|poll_schedule_timeout|
udevd ( 2222/ 1/ 1/C 0)| 0( 0/ 0/ -)| 21( 2/ 0/ 1/ 0)| 0( -/ -/ 0)| 0| 0| 64| 1K:20:53| ep_poll |
kworker/14:1H ( 3288/ 2/ 1/C-20)| 0( 0/ 0/ -)| 0( 0/ 0/ 0/ -)| 0( -/ -/ 0)| 0| 0| 64| 1K:20:51| worker_thread |
bioset ( 1265/ 2/ 1/C-20)| 0( 0/ 0/ -)| 0( 0/ 0/ 0/ -)| 0( -/ -/ 0)| 0| 0| 64| 1K:20:54| rescuer_thread |
kworker/22:1H ( 1787/ 2/ 1/C-20)| 0( 0/ 0/ -)| 0( 0/ 0/ 0/ -)| 0( -/ -/ 0)| 0| 0| 64| 1K:20:53| worker_thread |
/usr/sbin/apach ( 7221/ 3817/ 1/C 0)| 0( 0/ 0/ -)| 250( 51/ 0/ 12/ 0)| 0( -/ -/ 0)| 0| 0| 64| 20:23:39| inet_csk_accept |
sshd ( 1992/ 1977/ 1/C 0)| 0( 0/ 0/ 0)| 131( 4/ 0/ 2/ 0)| 0( -/ -/ 0)| 3| 1| 64| 0: 4:39|poll_schedule_timeout|
netns ( 130/ 2/ 1/C-20)| 0( 0/ 0/ -)| 0( 0/ 0/ 0/ -)| 0( -/ -/ 0)| 0| 0| 64| 1K:20:54| rescuer_thread |
kworker/20:2 ( 1962/ 2/ 1/C 0)| 0( 0/ 0/ 0)| 0( 0/ 0/ 0/ -)| 0( -/ -/ 0)| 1| 0| 64| 1K:20:53| worker_thread |
kworker/13:1H ( 3286/ 2/ 1/C-20)| 0( 0/ 0/ -)| 0( 0/ 0/ 0/ -)| 0( -/ -/ 0)| 0| 0| 64| 1K:20:51| worker_thread |
bash ( 9085/ 9084/ 1/C 0)| 0( 0/ 0/ -)| 23( 5/ 0/ 3/ 0)| 0( -/ -/ 0)| 0| 0| 256| 1K:47:24| do_wait |
ext4-rsv-conver ( 2079/ 2/ 1/C-20)| 0( 0/ 0/ -)| 0( 0/ 0/ 0/ -)| 0( -/ -/ 0)| 0| 0| 64| 1K:20:53| rescuer_thread |
kworker/12:1H ( 3285/ 2/ 1/C-20)| 0( 0/ 0/ -)| 0( 0/ 0/ 0/ -)| 0( -/ -/ 0)| 0| 0| 64| 1K:20:51| worker_thread |
---more---
# ./guider.py threadtop
[Top Info] [Time: 7175936.960] [Interval: 1.0] [Ctxt: 11934] [Life: +176/-177] [IRQ: 25109] [Core: 24] [Task: 389/493] [RAM: 63876] [Swap: 65491] (Unit: %/MB/NR)
[Cycle: 46G / Inst: 34G / IPC: 0.75 / CacheMiss: 23M(7%) / BranchMiss: 290M(4%) / Clock: 23G / MinFlt: 154,525 / MajFlt: 0]
==========================================================================================================================================================
ID | CPU (Usr/Ker/Blk/IRQ)| Mem (Diff/ User/Cache/Kern)| Swap (Diff/ I/O )|NrPgRclm | BlkRW | NrFlt | NrBlk | NrSIRQ | NrMlk | NrDrt | Network |
----------------------------------------------------------------------------------------------------------------------------------------------------------
Total | 87 %(80 / 3 / 0 / 0 )| 5416( -32/ 1198/56102/1160)| 0 ( 0 / 0/0 )| 0/0 | 0/11 | 0 | 0 | 31709 | 0 | 17635 | 2K/15K |
==========================================================================================================================================================
Thread ( TID/ PID/ Nr/ Pri)| CPU(Usr/Ker/Dly)| Mem(RSS/Txt/Shr/Swp)| Blk( RD / WR /NrFlt)| Yld | Prmt | FD | LifeTime| WaitChannel |
----------------------------------------------------------------------------------------------------------------------------------------------------------
cc1 (32544/32544/ 1/C 0)| 100(100/ 0/ 0)| 47( 40/ 12/ 13/ 0)| 0( -/ -/ 0)| 0| 1| 64| 0: 0: 2| RUNNING |
cc1 (32646/32646/ 1/C 0)| 100(100/ 0/ 0)| 45( 38/ 12/ 12/ 0)| 0( -/ -/ 0)| 0| 3| 64| 0: 0: 2| RUNNING |
cc1 (32738/32738/ 1/C 0)| 100(100/ 0/ 0)| 44( 37/ 12/ 12/ 0)| 0( -/ -/ 0)| 0| 3| 64| 0: 0: 1| RUNNING |
cc1 ( 319/ 319/ 1/C 0)| 100( 99/ 1/ 0)| 39( 34/ 12/ 12/ 0)| 0( -/ -/ 0)| 0| 2| 64| 0: 0: 1| RUNNING |
cc1 ( 352/ 352/ 1/C 0)| 100( 99/ 2/ 0)| 39( 32/ 12/ 12/ 0)| 0( -/ -/ 0)| 0| 1| 64| 0: 0: 1| RUNNING |
cc1 (32637/32637/ 1/C 0)| 100(100/ 0/ 0)| 48( 41/ 12/ 12/ 0)| 0( -/ -/ 0)| 0| 29| 64| 0: 0: 2| RUNNING |
cc1 (32746/32746/ 1/C 0)| 100(100/ 0/ 0)| 42( 36/ 12/ 12/ 0)| 0( -/ -/ 0)| 0| 1| 64| 0: 0: 1| RUNNING |
*cc1 ( 413/ 413/ 1/C 0)| 68( 67/ 1/ -)| 39( 32/ 12/ 12/ 0)| 0( -/ -/ 0)| -| -| 64| 0: 0: 0| RUNNING |
*cc1 ( 436/ 436/ 1/C 0)| 55( 54/ 1/ -)| 36( 30/ 12/ 12/ 0)| 0( -/ -/ 0)| -| -| 64| 0: 0: 0| RUNNING |
rs:main Q:Reg ( 2773/ 2702/ 4/C 0)| 43( 33/ 10/ 1)| 244( 5/ 0/ 2/ 0)| 0( -/ -/ 0)| 2291| 26| 64| 1K:18:54| futex_wait_queue_me |
*cc1 ( 469/ 469/ 1/C 0)| 35( 33/ 2/ -)| 32( 24/ 12/ 12/ 0)| 0( -/ -/ 0)| -| -| 64| 0: 0: 0| RUNNING |
*cc1 ( 490/ 490/ 1/C 0)| 29( 28/ 1/ -)| 0( 0/ 0/ -/ -)| 0( -/ -/ 0)| -| -| -| 0: 0: 0| |
*cc1 ( 479/ 479/ 1/C 0)| 29( 29/ 0/ -)| 34( 24/ 12/ 12/ 0)| 0( -/ -/ 0)| -| -| 64| 0: 0: 0| RUNNING |
*cc1 ( 495/ 495/ 1/C 0)| 28( 28/ 0/ -)| 30( 21/ 12/ 9/ 0)| 0( -/ -/ 0)| -| -| 64| 0: 0: 0| RUNNING |
*cc1 ( 500/ 500/ 1/C 0)| 27( 26/ 1/ -)| 30( 21/ 12/ 9/ 0)| 0( -/ -/ 0)| -| -| 64| 0: 0: 0| RUNNING |
*cc1 ( 508/ 508/ 1/C 0)| 18( 18/ 0/ -)| 28( 18/ 12/ 9/ 0)| 0( -/ -/ 0)| -| -| 64| 0: 0: 0| RUNNING |
*as ( 510/ 510/ 1/C 0)| 12( 12/ 0/ -)| 13( 10/ 1/ -/ -)| 0( -/ -/ 0)| -| -| -| 0: 0: 0| |
rsyslogd ( 2780/ 2702/ 4/C 0)| 12( 7/ 5/ 0)| 244( 5/ 0/ 2/ 0)| 0( -/ -/ 0)| 2699| 232| 64| 1K:18:54| do_syslog |
guider (31674/31674/ 1/C 0)| 7( 7/ 0/ 0)| 116( 77/ 2/ 5/ 0)| 0( -/ -/ 0)| 1| 2|1024| 0: 0: 6| RUNNING |
*cc1 ( 528/ 528/ 1/C 0)| 7( 7/ 0/ -)| 23( 14/ 12/ 9/ 0)| 0( -/ -/ 0)| -| -| 64| 0: 0: 0| RUNNING |
*cc1 ( 536/ 536/ 1/C 0)| 2( 2/ 0/ -)| 23( 12/ 12/ 9/ 0)| 0( -/ -/ 0)| -| -| 64| 0: 0: 0| RUNNING |
*as ( 537/ 537/ 1/C 0)| 1( 1/ 0/ -)| 9( 7/ 1/ -/ -)| 0( -/ -/ 0)| -| -| -| 0: 0: 0| |
----------------------------------------------------------------------------------------------------------------------------------------------------------
[+]sh ( 525/ 525/ 1/C 0)| 0( 0/ 0/ -)| 13( 2/ 0/ -/ -)| 0( -/ -/ 0)| -| -| -| 0: 0: 1| - |
[+]arm-linux-gnu ( 527/ 527/ 1/C 0)| 0( 0/ 0/ -)| 5( 1/ 0/ -/ -)| 0( -/ -/ 0)| -| -| -| 0: 0: 1| - |
[+]cc1 ( 436/ 436/ 1/C 0)| 55( 54/ 1/ -)| 36( 30/ 12/ 12/ 0)| 0( -/ -/ 0)| -| -| -| 0: 0: 1| - |
[+]arm-linux-gnu ( 434/ 434/ 1/C 0)| 0( 0/ 0/ -)| 5( 1/ 0/ -/ -)| 0( -/ -/ 0)| -| -| -| 0: 0: 1| - |
[+]sh ( 432/ 432/ 1/C 0)| 0( 0/ 0/ -)| 13( 2/ 0/ -/ -)| 0( -/ -/ 0)| -| -| -| 0: 0: 1| - |
[+]sh ( 498/ 498/ 1/C 0)| 0( 0/ 0/ -)| 13( 2/ 0/ -/ -)| 0( -/ -/ 0)| -| -| -| 0: 0: 1| - |
[+]arm-linux-gnu ( 499/ 499/ 1/C 0)| 0( 0/ 0/ -)| 5( 1/ 0/ -/ -)| 0( -/ -/ 0)| -| -| -| 0: 0: 1| - |
[+]cc1 ( 495/ 495/ 1/C 0)| 28( 28/ 0/ -)| 30( 21/ 12/ 9/ 0)| 0( -/ -/ 0)| -| -| -| 0: 0: 1| - |
[+]cc1 ( 490/ 490/ 1/C 0)| 29( 28/ 1/ -)| 0( 0/ 0/ -/ -)| 0( -/ -/ 0)| -| -| -| 0: 0: 1| - |
[+]sh ( 491/ 491/ 1/C 0)| 0( 0/ 0/ -)| 13( 2/ 0/ -/ -)| 0( -/ -/ 0)| -| -| -| 0: 0: 1| - |
# ./guider.py filetop -g init
[Top File Info] [Time: 7176036.720] [Proc: 322] [FD: 1323] [File: 400] (Unit: %/MB/NR)
==========================================================================================================================================================
PROC ( ID / Pid / Nr / Pri)| FD | PATH |
----------------------------------------------------------------------------------------------------------------------------------------------------------
init ( 1/ 0/ 1/C 0)| 12| SOCKET: 4 DEVICE: 3 PIPE: 2 EVENT: 2 NORMAL: 1 PROC: 0 |
| 11| /var/log/upstart/mysql.log.1 (deleted) |
| 10| socket:[13414] (@/com/ubuntu/upstart) |
| 9| socket:[23593] (@/com/ubuntu/upstart) |
| 8| socket:[6241] |
| 7| socket:[3098] (@/com/ubuntu/upstart) |
| 6| anon_inode:inotify |
| 5| anon_inode:inotify |
| 4| pipe:[3097] |
| 3| pipe:[3097] |
| 2| /dev/null |
| 1| /dev/null |
| 0| /dev/null |
----------------------------------------------------------------------------------------------------------------------------------------------------------
# ./guider.py stacktop -g syslog
[Top Info] [Time: 7176163.830] [Interval: 1.0] [Ctxt: 2914] [Life: +13/-12] [IRQ: 5103] [Core: 24] [Task: 328/435] [RAM: 63876] [Swap: 65491] (Unit: %/MB/NR)
[Cycle: 2G / Inst: 3G / IPC: 1.34 / CacheMiss: 6M(34%) / BranchMiss: 4M(0%) / Clock: 23G / MinFlt: 53,257 / MajFlt: 0]
==========================================================================================================================================================
ID | CPU (Usr/Ker/Blk/IRQ)| Mem (Diff/ User/Cache/Kern)| Swap (Diff/ I/O )|NrPgRclm | BlkRW | NrFlt | NrBlk | NrSIRQ | NrMlk | NrDrt | Network |
----------------------------------------------------------------------------------------------------------------------------------------------------------
Total | 6 %( 3 / 1 / 0 / 0 )| 4913(-204/ 974/56824/1165)| 0 ( 0 / 0/0 )| 0/0 | 0/42 | 0 | 0 | 3713 | 0 | 90901 | 2K/13K |
==========================================================================================================================================================
Thread ( TID/ PID/ Nr/ Pri)| CPU(Usr/Ker/Dly)| Mem(RSS/Txt/Shr/Swp)| Blk( RD / WR /NrFlt)| Yld | Prmt | FD | LifeTime| WaitChannel |
----------------------------------------------------------------------------------------------------------------------------------------------------------
rsyslogd ( 2702/ 2702/ 4/C 0)| 0( 0/ 0/ -)| 244( 5/ 0/ 2/ 0)| 0( -/ -/ 0)| 0| 0| 64| 1K:22:40|poll_schedule_timeout|
100% | poll_schedule_timeout+0x43/0x70 <- do_select+0x711/0x7f0 <- core_sys_select+0x196/0x280 <-
SyS_select+0xa6/0xe0 <- entry_SYSCALL_64_fastpath+0x1a/0xa5
----------------------------------------------------------------------------------------------------------------------------------------------------------
rsyslogd ( 2779/ 2702/ 4/C 0)| 0( 0/ 0/ -)| 244( 5/ 0/ 2/ 0)| 0( -/ -/ 0)| 0| 0| 64| 1K:22:40|poll_schedule_timeout|
100% | poll_schedule_timeout+0x43/0x70 <- do_select+0x711/0x7f0 <- core_sys_select+0x196/0x280 <-
SyS_select+0xa6/0xe0 <- entry_SYSCALL_64_fastpath+0x1a/0xa5
----------------------------------------------------------------------------------------------------------------------------------------------------------
rsyslogd ( 2780/ 2702/ 4/C 0)| 0( 0/ 0/ 0)| 244( 5/ 0/ 2/ 0)| 0( -/ -/ 0)| 116| 0| 64| 1K:22:40| do_syslog |
99% | do_syslog+0x446/0x4c0 <- kmsg_read+0x3f/0x50 <- proc_reg_read+0x3d/0x60 <- __vfs_read+0x23/0x110 <-
vfs_read+0x91/0x130 <- SyS_read+0x41/0xa0 <- entry_SYSCALL_64_fastpath+0x1a/0xa5
----------------------------------------------------------------------------------------------------------------------------------------------------------
# ./guider.py perftop -g yes
[Top Info] [Time: 7181955.420] [Interval: 1.0] [Ctxt: 121] [Life: +0/-0] [IRQ: 1947] [Core: 24] [Task: 317/424] [RAM: 63876] [Swap: 65491] (Unit: %/MB/NR)
==========================================================================================================================================================
ID | CPU (Usr/Ker/Blk/IRQ)| Mem (Diff/ User/Cache/Kern)| Swap (Diff/ I/O )|NrPgRclm | BlkRW | NrFlt | NrBlk | NrSIRQ | NrMlk | NrDrt | Network |
----------------------------------------------------------------------------------------------------------------------------------------------------------
Total | 5 %( 4 / 0 / 0 / 0 )| 3783( 0/ 875/58078/1140)| 0 ( 0 / 0/0 )| 0/0 | 0/0 | 0 | 0 | 2023 | 0 | 0 | 1K/3K |
==========================================================================================================================================================
Process ( PID/ PPID/ Nr/ Pri)| CPU(Usr/Ker/Dly)| Mem(RSS/Txt/Shr/Swp)| Blk( RD / WR /NrFlt)| Yld | Prmt | FD | LifeTime| WaitChannel |
----------------------------------------------------------------------------------------------------------------------------------------------------------
yes (22371/ 9085/ 1/R 90)| 99( 99/ 0/ 0)| 8( 0/ 0/ 0/ 0)| 0( -/ -/ 0)| 0| 0| 256| 1:34:11| RUNNING |
| [Cycle: 2G / Inst: 6G / IPC: 2.82 / CacheMiss: 11K(98%) / BranchMiss: 26K(0%) / Clock: 972M / MinFlt: 0 / MajFlt: 0]
----------------------------------------------------------------------------------------------------------------------------------------------------------
# ./guider.py memtop
[Top Info] [Time: 7176233.650] [Interval: 1.0] [Ctxt: 289] [Life: +2/-2] [IRQ: 1397] [Core: 24] [Task: 323/430] [RAM: 63876] [Swap: 65491] (Unit: %/MB/NR)
[Cycle: 127M / Inst: 121M / IPC: 0.95 / CacheMiss: 280K(23%) / BranchMiss: 437K(1%) / Clock: 22G / MinFlt: 714 / MajFlt: 0]
==========================================================================================================================================================
ID | CPU (Usr/Ker/Blk/IRQ)| Mem (Diff/ User/Cache/Kern)| Swap (Diff/ I/O )|NrPgRclm | BlkRW | NrFlt | NrBlk | NrSIRQ | NrMlk | NrDrt | Network |
----------------------------------------------------------------------------------------------------------------------------------------------------------
Total | 1 %( 0 / 0 / 0 / 0 )| 4706( 1/ 865/57151/1154)| 0 ( 0 / 0/0 )| 0/0 | 0/0 | 0 | 0 | 484 | 0 | 35 | 1K/4K |
==========================================================================================================================================================
Process ( PID/ PPID/ Nr/ Pri)| CPU(Usr/Ker/Dly)| Mem(RSS/Txt/Shr/Swp)| Blk( RD / WR /NrFlt)| Yld | Prmt | FD | LifeTime| WaitChannel |
----------------------------------------------------------------------------------------------------------------------------------------------------------
guider (22307/ 9085/ 1/C 0)| 2( 2/ 0/ 0)| 101( 62/ 2/ 5/ 0)| 0( -/ -/ 0)| 1| 0|1024| 0: 0: 5| RUNNING |
(1)[STACK] | SIZE: 0M / RSS: 0M / PSS: 0M / SWAP: 0M / HUGE: 0M / LOCK: 0K / SDRT: 0K / PDRT: 140K / NOPM: 0K|
(1)[SHM] | SIZE: 0M / RSS: 0M / PSS: 0M / SWAP: 0M / HUGE: 0M / LOCK: 0K / SDRT: 0K / PDRT: 0K / NOPM: 0K|
(19)[FILE] | SIZE: 44M / RSS: 6M / PSS: 2M / SWAP: 0M / HUGE: 0M / LOCK: 0K / SDRT: 0K / PDRT: 632K / NOPM: 31M|
(3)[ETC] | SIZE: 0M / RSS: 0M / PSS: 0M / SWAP: 0M / HUGE: 0M / LOCK: 0K / SDRT: 0K / PDRT: 0K / NOPM: 0K|
(12)[ANON] | SIZE: 56M / RSS: 56M / PSS: 56M / SWAP: 0M / HUGE: 0M / LOCK: 0K / SDRT: 0K / PDRT: 56M / NOPM: 0K|
----------------------------------------------------------------------------------------------------------------------------------------------------------
# ./guider.py reporttop -j . -u
# cat guider.report
{
"task": {
"nrThread": 397,
"nrBlocked": 0,
"nrCtx": 4290,
"nrProc": 292
},
"mem": {
"kernel": 1432,
"anonDiff": -1,
"pgRclmFg": 0,
"cache": 35332,
"slabDiff": 0,
"free": 26929,
"anon": 698,
"pgDirty": 28,
"file": 31751,
"freeDiff": -1,
"pgRclmBg": 0,
"total": 64391,
"slab": 3581,
"fileDiff": -1
"procs": {
"1954": {
"text": 0,
"pid": 1954,
"rank": 2,
"comm": "ruby1.9.1",
"runtime": "110:43:32",
"rss": 104
},
"27977": {
"text": 0,
"pid": 27977,
"rank": 10,
"comm": "/usr/sbin/apach",
"runtime": "34:46:1",
"rss": 36
},
"31050": {
"text": 0,
"pid": 31050,
"rank": 8,
"comm": "/usr/sbin/apach",
"runtime": "10:21:44",
"rss": 38
},
},
"storage": {
"total": {
"read": 0,
"mount": {},
"favail": 133443655,
"free": 1141633,
"write": 1,
"usage": 1152423,
"total": 2294056,
"usageper": 50
},
"/dev/sdb1": {
"read": 0,
"mount": {
"path": "/mnt/hdd1",
"fs": "ext4",
"option": "rw,relatime,data=ordered"
},
"favail": 50709466,
"free": 293649,
"write": 0,
"usage": 645251,
"total": 938900,
"usageper": 68
},
"/dev/sda1": {
"read": 0,
"mount": {
"path": "/",
"fs": "ext4",
"option": "rw,relatime,errors=remount-ro,data=ordered"
},
"favail": 26727712,
"free": 374921,
"write": 0,
"usage": 41335,
"total": 416256,
"usageper": 9
},
},
"system": {
"load5m": 2.38,
"uptime": 4191643.92,
"nrSoftIrq": 7405,
"nrIrq": 7289,
"load15m": 0.84,
"interval": 1.029999999795109,
"pid": 14578,
"load1m": 9.39
},
"event": {
"CPU_INTENSIVE": {
"14592": {
"kernel": 0,
"runtime": "0:0:47",
"pid": 14592,
"rank": 3,
"comm": "yes",
"user": 99,
"total": 100
},
"14593": {
"kernel": 0,
"runtime": "0:0:46",
"pid": 14593,
"rank": 10,
"comm": "yes",
"user": 99,
"total": 100
},
"14594": {
"kernel": 0,
"runtime": "0:0:46",
"pid": 14594,
"rank": 12,
"comm": "yes",
"user": 100,
"total": 100
},
},
"swap": {
"usage": 76,
"total": 65491,
"usageDiff": 0
},
"net": {
"inbound": 1479,
"outbound": 392
},
"cpu": {
"kernel": 0,
"iowait": 0,
"nrCore": 24,
"idle": 8,
"user": 91,
"irq": 0,
"total": 92,
"procs": {
"14592": {
"kernel": 0,
"runtime": "0:0:47",
"pid": 14592,
"rank": 3,
"comm": "yes",
"user": 99,
"total": 100
},
"14593": {
"kernel": 0,
"runtime": "0:0:46",
"pid": 14593,
"rank": 10,
"comm": "yes",
"user": 99,
"total": 100
},
"14594": {
"kernel": 0,
"runtime": "0:0:46",
"pid": 14594,
"rank": 12,
"comm": "yes",
"user": 100,
"total": 100
},
},
"block": {
"read": 0,
"write": 0,
"procs": {},
"nrMajFlt": 0,
"ioWait": 0
}
}
# ./guider.py cpulimit -g 22371:50
_ _
__ _ _ _ (_) __| | ___ _ __
/ _` || | | || | / _` | / _ \| '__|
| (_| || |_| || || (_| || __/| |
\__, | \__,_||_| \__,_| \___||_|
|___/
[Info] priority of 22376 task is changed to -20(C)
[Info] limited cpu usage of yes(22371) process to 50%, it used 50%
[Info] limited cpu usage of yes(22371) process to 50%, it used 50%
[Info] limited cpu usage of yes(22371) process to 50%, it used 50%
[Info] limited cpu usage of yes(22371) process to 50%, it used 50%
# ./guider.py setsched r:90:22371
_ _
__ _ _ _ (_) __| | ___ _ __
/ _` || | | || | / _` | / _ \| '__|
| (_| || |_| || || (_| || __/| |
\__, | \__,_||_| \__,_| \___||_|
|___/
[Info] priority of 22371 task is changed to 90(R)
# ./guider.py kill -stop 12345
_ _
__ _ _ _ (_) __| | ___ _ __
/ _` || | | || | / _` | / _ \| '__|
| (_| || |_| || || (_| || __/| |
\__, | \__,_||_| \__,_| \___||_|
|___/
[Info] sent signal SIGSTOP to 10594 process
# ./guider.py wsstop -g yes
[Top Info] [Time: 7176629.490] [Interval: 1.0] [Ctxt: 195] [Life: +0/-0] [IRQ: 2688] [Core: 24] [Task: 327/434] [RAM: 63876] [Swap: 65491] (Unit: %/MB/NR)
[Cycle: 2G / Inst: 6G / IPC: 2.75 / CacheMiss: 202K(19%) / BranchMiss: 325K(0%) / Clock: 23G / MinFlt: 4 / MajFlt: 0]
==========================================================================================================================================================
ID | CPU (Usr/Ker/Blk/IRQ)| Mem (Diff/ User/Cache/Kern)| Swap (Diff/ I/O )|NrPgRclm | BlkRW | NrFlt | NrBlk | NrSIRQ | NrMlk | NrDrt | Network |
----------------------------------------------------------------------------------------------------------------------------------------------------------
Total | 5 %( 4 / 0 / 0 / 0 )| 4719( 0/ 856/57152/1149)| 0 ( 0 / 0/0 )| 0/0 | 0/0 | 0 | 0 | 2410 | 0 | 2 | 1K/5K |
==========================================================================================================================================================
Process ( PID/ PPID/ Nr/ Pri)| CPU(Usr/Ker/Dly)| Mem(RSS/Txt/Shr/Swp)| Blk( RD / WR /NrFlt)| Yld | Prmt | FD | LifeTime| WaitChannel |
----------------------------------------------------------------------------------------------------------------------------------------------------------
yes (22371/ 9085/ 1/R 90)| 99( 99/ 0/ 0)| 8( 0/ 0/ 0/ 0)| 0( -/ -/ 0)| 0| 0| 256| 0: 5:25| RUNNING |
(1)[STACK] | SIZE: 0M / RSS: 0M / PSS: 0M / SWAP: 0M / HUGE: 0M / LOCK: 0K / SDRT: 0K / PDRT: 8K / NOPM: 0K|
| WSS: [ 8K] -> 4K -> 4K -> 4K -> 4K -> 4K -> 4K -> 4K -> 4K -> 4K -> 4K -> 4K
(4)[FILE] | SIZE: 7M / RSS: 1M / PSS: 0M / SWAP: 0M / HUGE: 0M / LOCK: 0K / SDRT: 0K / PDRT: 40K / NOPM:2048K|
| WSS: [ 1M] -> 1M -> 1M -> 1M -> 1M -> 1M -> 1M -> 1M -> 1M -> 1M -> 1M -> 1M
(3)[ETC] | SIZE: 0M / RSS: 0M / PSS: 0M / SWAP: 0M / HUGE: 0M / LOCK: 0K / SDRT: 0K / PDRT: 0K / NOPM: 0K|
| WSS: [ 4K] -> 4K -> 4K -> 4K -> 4K -> 0 -> 0 -> 0 -> 0 -> 0 -> 4K -> 4K
(5)[ANON] | SIZE: 0M / RSS: 0M / PSS: 0M / SWAP: 0M / HUGE: 0M / LOCK: 0K / SDRT: 0K / PDRT: 48K / NOPM: 0K|
| WSS: [ 48K] -> 4K -> 4K -> 4K -> 4K -> 4K -> 4K -> 4K -> 4K -> 4K -> 4K -> 4K
----------------------------------------------------------------------------------------------------------------------------------------------------------
# ./guider.py record -a -e m,b
[Thread Info] [ Elapsed: 2.050 ] [ Start: 2849868.198 ] [ Running: 112 ] [ CtxSwc: 3357 ] [ LogSize: 4054 KB ] [ Unit: Sec/MB/NR ]
==========================================================================================================================================================
__________Thread Info___________|_____________CPU Info______________|______SCHED Info______|________BLOCK Info________|_____________MEM Info_____________|
| | | | |
Name( Tid/ Pid)|LF|Usage( %)|Delay( Max)|Pri| IRQ | Yld| Lose|Steal| Mig| Read( MB/ Cnt)|WCnt( MB)| Sum(Usr/Buf/Ker)|Rcl|Wst|DRcl(Nr)|
==========================================================================================================================================================
# CPU: 12
CORE/0(-----/-----)|--| 0.00( 0.1)| 0.00( 0.00)| 0| 0.00| 7| -| -| -| 0.00( 0/ 1)| 0( 0)| 0( 0/ 0/ 0)| 0| 0|0.00( 0)|
CORE/1(-----/-----)|--| 0.00( 0.1)| 0.10( 0.00)| 0| 0.00| 147| -| -| -| 0.00( 0/ 0)| 0( 0)| 0( 0/ 0/ 0)| 0| 0|0.00( 0)|
CORE/2(-----/-----)|--| 0.00( 0.1)| 0.16( 0.00)| 0| 0.00| 211| -| -| -| 0.00( 0/ 0)| 0( 0)| 0( 0/ 0/ 0)| 0| 0|0.00( 0)|
CORE/3(-----/-----)|--| 0.00( 0.1)| 0.11( 0.00)| 0| 0.00| 181| -| -| -| 0.00( 0/ 0)| 32( 0)| 0( 0/ 0/ 0)| 0| 0|0.00( 0)|
CORE/4(-----/-----)|--| 0.00( 0.1)| 0.11( 0.00)| 0| 0.00| 232| -| -| -| 0.00( 0/ 0)| 0( 0)| 0( 0/ 0/ 0)| 0| 0|0.00( 0)|
CORE/5(-----/-----)|--| 0.30( 14.8)| 0.18( 0.00)| 0| 0.00| 179| -| -| -| 1.26( 6/ 495)| 19( 0)| 61( 57/ 0/ 3)| 0| 0|0.00( 0)|
CORE/6(-----/-----)|--| 0.00( 0.0)| 0.35( 0.00)| 0| 0.00| 57| -| -| -| 0.00( 0/ 0)| 0( 0)| 0( 0/ 0/ 0)| 0| 0|0.00( 0)|
CORE/7(-----/-----)|--| 0.00( 0.0)| 0.60( 0.00)| 0| 0.00| 100| -| -| -| 0.00( 0/ 0)| 0( 0)| 0( 0/ 0/ 0)| 0| 0|0.00( 0)|
CORE/8(-----/-----)|--| 0.00( 0.0)| 0.44( 0.00)| 0| 0.00| 59| -| -| -| 0.00( 0/ 0)| 0( 0)| 0( 0/ 0/ 0)| 0| 0|0.00( 0)|
CORE/9(-----/-----)|--| 0.00( 0.0)| 1.94( 0.00)| 0| 0.00| 37| -| -| -| 0.00( 0/ 0)| 0( 0)| 0( 0/ 0/ 0)| 0| 0|0.00( 0)|
CORE/10(-----/-----)|--| 0.07( 3.4)| 0.00( 0.00)| 0| 0.00| 2| -| -| -| 0.00( 0/ 0)| 0( 0)| 0( 0/ 0/ 0)| 0| 0|0.00( 0)|
CORE/11(-----/-----)|--| 0.00( 0.0)| 2.05( 0.00)| 0| 0.00| 39| -| -| -| 0.00( 0/ 0)| 0( 0)| 0( 0/ 0/ 0)| 0| 0|0.00( 0)|
----------------------------------------------------------------------------------------------------------------------------------------------------------
# Hot: 4
synergyc( 3604/ 3602)| | 0.17( 8.5)| 0.00( 0.00)| 0| 0.00| 3| 14| 3| 0| 0.00( 0/ 0)| 0( 0)| 0( 0/ 0/ 0)| 0| 0|0.00( 0)|
arm-starfish-li(16087/16087)| | 0.13( 6.3)| 0.00( 0.00)| 0| 0.00| 0| 20| 157| 4| 1.26( 6/ 496)| 0( 0)| 61( 57/ 0/ 3)| 0| 0|0.00( 0)|
guider(16088/16088)| | 0.07( 3.4)| 0.00( 0.00)|R90| 0.00| 2| 0| 2| 0| 0.00( 0/ 0)| 0( 0)| 0( 0/ 0/ 0)| 0| 0|0.00( 0)|
----------------------------------------------------------------------------------------------------------------------------------------------------------
# ./guider.py record -t
[Thread Syscall Info] (Unit: Sec/NR)
==========================================================================================================================================================
Name( Tid) Syscall( ID) Elapsed Count Error Min Max Avg
==========================================================================================================================================================
arm-linux-gnuea( 3000)
close( 3) 0.039396 69 0 0.000001 0.005353 0.000571
stat( 4) 0.011521 74 0 0.000001 0.009423 0.000156
fchmod( 91) 0.000046 3 0 0.000002 0.000039 0.000015
getpriority(140) 0.000017 33 0 0.000000 0.000001 0.000001
lgetxattr(192) 0.000014 3 0 0.000003 0.000008 0.000005
recvfrom( 45) 0.000004 1 0 0.000004 0.000004 0.000004
----------------------------------------------------------------------------------------------------------------------------------------------------------
guider( 3001)
pause( 34) 0.283474 1 1 0.283474 0.283474 0.283474
select( 23) 0.100122 1 0 0.100122 0.100122 0.100122
write( 1) 0.000234 6 0 0.000031 0.000059 0.000039
open( 2) 0.000084 7 0 0.000007 0.000038 0.000012
ioctl( 16) 0.000009 14 14 0.000001 0.000001 0.000001
fstat( 5) 0.000006 14 0 0.000001 0.000001 0.000000
lseek( 8) 0.000006 21 0 0.000000 0.000001 0.000000
close( 3) 0.000005 7 0 0.000000 0.000001 0.000001
rt_sigaction( 13) 0.000001 1 0 0.000001 0.000001 0.000001
----------------------------------------------------------------------------------------------------------------------------------------------------------
mysqld( 3237)
futex(202) 0.000000 1 0 0.000000 0.000000 0.000000
----------------------------------------------------------------------------------------------------------------------------------------------------------
mysqld( 3238)
futex(202) 0.000002 1 0 0.000002 0.000002 0.000002
----------------------------------------------------------------------------------------------------------------------------------------------------------
screen( 9045)
select( 23) 0.000082 4 0 0.000004 0.000069 0.000021
----------------------------------------------------------------------------------------------------------------------------------------------------------
# ./guider.py record -e b
[Thread Block Info] (Unit: KB/NR)
==========================================================================================================================================================
ID OPT NrDev TOTAL SEQUENTIAL( %) FS PATH
[ACCESS] COUNT
==========================================================================================================================================================
TOTAL READ 8:33 170,624 158,356( 92.8) ext4 /dev/sdc1
[ 4K - 7K] 2,024
[ 8K - 15K] 21
[ 16K - 31K] 43
[ 32K - 63K] 44
[ 64K - 127K] 155
[ 128K - 255K] 112
[ 256K - 511K] 510
[ 512K - 1023K] 2
WRITE 8:33 8 4( 50.0) ext4 /dev/sdc1
[ 4K - 7K] 2
----------------------------------------------------------------------------------------------------------------------------------------------------------
cron(3115) READ 8:33 644 576( 89.4) ext4 /dev/sdc1
[ 4K - 7K] 161
----------------------------------------------------------------------------------------------------------------------------------------------------------
cat(10392) READ 8:33 604 404( 66.9) ext4 /dev/sdc1
[ 4K - 7K] 110
[ 16K - 31K] 1
[ 32K - 63K] 1
[ 64K - 127K] 1
----------------------------------------------------------------------------------------------------------------------------------------------------------
bash(9085) READ 8:33 28 4( 14.3) ext4 /dev/sdc1
[ 4K - 7K] 7
----------------------------------------------------------------------------------------------------------------------------------------------------------
cat(10395) READ 8:33 169,348 157,384( 92.9) ext4 /dev/sdc1
[ 4K - 7K] 1,746
[ 8K - 15K] 21
[ 16K - 31K] 42
[ 32K - 63K] 43
[ 64K - 127K] 154
[ 128K - 255K] 112
[ 256K - 511K] 510
[ 512K - 1023K] 2
----------------------------------------------------------------------------------------------------------------------------------------------------------
kworker/u50:0(10304) WRITE 8:33 8 4( 50.0) ext4 /dev/sdc1
[ 4K - 7K] 2
----------------------------------------------------------------------------------------------------------------------------------------------------------
# ./guider.py record -e L
[Thread Futex Lock Info] [ Elapsed : 1.225 ] (Unit: Sec/NR)
==========================================================================================================================================================
Name( Tid/ Pid) Elapsed Process Block NrBlock CallMax Lock LockMax NrLock NrWait LBlock NrLBlock LastStat
==========================================================================================================================================================
mysqld( 3236/ 3208) 0.469 0.000 0.469 1 0.469 0.000 0.000 0 1 0.000 0 Wait
----------------------------------------------------------------------------------------------------------------------------------------------------------
mysqld( 3237/ 3208) 0.890 0.000 0.890 1 0.890 0.000 0.000 0 1 0.000 0 Wait
----------------------------------------------------------------------------------------------------------------------------------------------------------
mysqld( 3238/ 3208) 1.075 0.000 1.075 1 1.075 0.000 0.000 0 1 0.000 0 Wait
----------------------------------------------------------------------------------------------------------------------------------------------------------
[Thread File Lock Info] (Unit: Sec/NR)
==========================================================================================================================================================
Name( Tid) Wait Lock nrTryLock nrLocked
==========================================================================================================================================================
smbd( 2631) 0.000 0.000 3 3
----------------------------------------------------------------------------------------------------------------------------------------------------------
# ./guider.py record -s . -K openfile:getname::**string
[Thread KERNEL Event Info]
==========================================================================================================================================================
Event Comm( Tid ) Usage Count ProcMax ProcMin InterMax InterMin
==========================================================================================================================================================
openfile TOTAL( - ) 0.000729 1012 0.000013 0.000001 1.979834 0.000109
ps(10728) 0.000640 968 0.000013 0.000000 0.001636 0.000006
python2(10727) 0.000038 26 0.000004 0.000001 1.979834 0.000020
tmux( 6959) 0.000031 9 0.000006 0.000003 0.299492 0.201316
PassengerAgent(23183) 0.000008 5 0.000002 0.000001 0.007375 0.000109
sendmail-mta( 3419) 0.000007 2 0.000006 0.000001 0.000077 0.000077
PassengerAgent(10729) 0.000003 1 0.000003 0.000003 0.000000 0.000000
smbd(11086) 0.000002 1 0.000002 0.000002 0.000000 0.000000
----------------------------------------------------------------------------------------------------------------------------------------------------------
[Thread KERNEL Event History]
==========================================================================================================================================================
EVENT TYPE TIME COMM( TID) CALLER ELAPSED ARG
==========================================================================================================================================================
openfile EXIT 0.063942 tmux( 6959) porch_do_sys_open 0.000003 1>"/proc/7969/cmdline"
openfile ENTER 0.137626 python2(10727) -
openfile EXIT 0.137628 python2(10727) porch_do_sys_open 0.000002 1>"/sys/kernel/debug/tracing/trace"
openfile ENTER 0.363431 tmux( 6959) -
openfile EXIT 0.363437 tmux( 6959) porch_do_sys_open 0.000006 1>"/proc/7197/cmdline"
openfile ENTER 0.510452 smbd(11086) -
openfile EXIT 0.510454 smbd(11086) porch_do_sys_open 0.000002 1>"/var/log/samba/log.jhkim-z97x-ud3h"
openfile ENTER 0.564845 tmux( 6959) -
openfile EXIT 0.564848 tmux( 6959) porch_do_sys_open 0.000003 1>"/proc/7969/cmdline"
openfile ENTER 0.864255 tmux( 6959) -
openfile EXIT 0.864258 tmux( 6959) porch_do_sys_open 0.000003 1>"/proc/7197/cmdline"
openfile ENTER 1.065571 tmux( 6959) -
openfile EXIT 1.065574 tmux( 6959) porch_do_sys_open 0.000003 1>"/proc/7969/cmdline"
openfile ENTER 1.364980 tmux( 6959) -
openfile EXIT 1.364984 tmux( 6959) porch_do_sys_open 0.000004 1>"/proc/7197/cmdline"
openfile ENTER 1.437128 sendmail-mta( 3419) -
openfile EXIT 1.437134 sendmail-mta( 3419) porch_do_sys_open 0.000006 1>"/proc/loadavg"
openfile ENTER 1.437205 sendmail-mta( 3419) -
openfile EXIT 1.437206 sendmail-mta( 3419) porch_do_sys_open 0.000001 1>"/proc/loadavg"
openfile ENTER 1.566369 tmux( 6959) -
openfile EXIT 1.566372 tmux( 6959) porch_do_sys_open 0.000003 1>"/proc/7969/cmdline"
openfile ENTER 1.865776 tmux( 6959) -
openfile EXIT 1.865779 tmux( 6959) porch_do_sys_open 0.000003 1>"/proc/7197/cmdline"
openfile ENTER 1.955265 PassengerAgent(10729) -
openfile EXIT 1.955268 PassengerAgent(10729) porch_do_sys_open 0.000003 1>"/dev/fd"
----------------------------------------------------------------------------------------------------------------------------------------------------------
# ./guider.py record -f -s .
# ./guider.py guider.dat -o . -a
[Function CPU Info] [Cnt: 394] [Interval: 8ms] (USER)
==========================================================================================================================================================
__Usage__|___________________Function____________________|_____________________________________________Binary_____________________________________________
==========================================================================================================================================================
99.0% | cpuTest | /media/disk/work/test/a.out
+ 100.0% | <- startTest [/media/disk/work/test/a.out] <- main [/media/disk/work/test/a.out]
<- __libc_start_main [/lib/x86_64-linux-gnu/libc-2.19.so]
----------------------------------------------------------------------------------------------------------------------------------------------------------
0.5% | memset | /lib/x86_64-linux-gnu/libc-2.19.so
+ 100.0% | <- startTest [/media/disk/work/test/a.out] <- main [/media/disk/work/test/a.out]
<- __libc_start_main [/lib/x86_64-linux-gnu/libc-2.19.so]
----------------------------------------------------------------------------------------------------------------------------------------------------------
0.3% | _int_malloc | /lib/x86_64-linux-gnu/libc-2.19.so
----------------------------------------------------------------------------------------------------------------------------------------------------------
0.3% | 00007f756e3e7ee4 | ??
+ 100.0% | <- 000000000044676f [/media/disk/work/test/a.out]
----------------------------------------------------------------------------------------------------------------------------------------------------------
[Function CPU Info] [Cnt: 394] [Interval: 8ms] (KERNEL)
==========================================================================================================================================================
__Usage__|____________________________________________________________________Function____________________________________________________________________
==========================================================================================================================================================
100.0% | hrtimer_interrupt
+ 99.5% | <- local_apic_timer_interrupt <- smp_apic_timer_interrupt <- apic_timer_interrupt
+ 0.3% | <- local_apic_timer_interrupt <- smp_apic_timer_interrupt <- apic_timer_interrupt <- do_page_fault <- page_fault
+ 0.3% | <- local_apic_timer_interrupt <- smp_apic_timer_interrupt <- apic_timer_interrupt <- __do_fault <- handle_mm_fault <- __do_page_fault
<- do_page_fault <- page_fault
----------------------------------------------------------------------------------------------------------------------------------------------------------
# ./guider.py record -f -e m -s .
# ./guider.py guider.dat -o . -a
[Function Page Info] [Total: 11416KB] [Alloc: 11444KB(817)] [Free: 188KB(47)] (USER)
==========================================================================================================================================================
Usage ( Usr / Buf / Ker )|___________________Function____________________|________________LifeTime________________|______________Binary_______________
==========================================================================================================================================================
10256K( 2048/ 0/ 8208)| memset | AVR: 1.563 / MIN: 1.560 / MAX: 1.568 | /lib/x86_64-linux-gnu/libc-2.19.so
+ 10256K( 2048/ 0/ 8208)| <- startTest [/media/disk/work/test/a.out] <- main [/media/disk/work/test/a.out]
<- __libc_start_main [/lib/x86_64-linux-gnu/libc-2.19.so]
----------------------------------------------------------------------------------------------------------------------------------------------------------
960K( 956/ 0/ 4)| _int_malloc | AVR: 1.559 / MIN: 1.554 / MAX: 1.560 | /lib/x86_64-linux-gnu/libc-2.19.so
----------------------------------------------------------------------------------------------------------------------------------------------------------
56K( 16/ 0/ 40)| 00007f756e3e81e7 | AVR: 1.569 / MIN: 1.568 / MAX: 1.569 | ??
----------------------------------------------------------------------------------------------------------------------------------------------------------
44K( 36/ 0/ 8)| sysmalloc | AVR: 1.560 / MIN: 1.558 / MAX: 1.568 | /lib/x86_64-linux-gnu/libc-2.19.so
----------------------------------------------------------------------------------------------------------------------------------------------------------
12K( 12/ 0/ 0)| elf_machine_rela_relative | AVR: 1.568 / MIN: 1.568 / MAX: 1.568 | /lib/x86_64-linux-gnu/ld-2.19.so
+ 12K( 12/ 0/ 0)| <- dl_main [/lib/x86_64-linux-gnu/ld-2.19.so] <- _dl_sysdep_start [/lib/x86_64-linux-gnu/ld-2.19.so]
----------------------------------------------------------------------------------------------------------------------------------------------------------
8K( 8/ 0/ 0)| realloc | AVR: 1.568 / MIN: 1.568 / MAX: 1.568 | /lib/x86_64-linux-gnu/ld-2.19.so
+ 4K( 4/ 0/ 0)| <- _dl_map_object [/lib/x86_64-linux-gnu/ld-2.19.so]
----------------------------------------------------------------------------------------------------------------------------------------------------------
8K( 4/ 0/ 4)| dl_main | AVR: 1.568 / MIN: 1.568 / MAX: 1.568 | /lib/x86_64-linux-gnu/ld-2.19.so
+ 8K( 4/ 0/ 4)| <- _dl_sysdep_start [/lib/x86_64-linux-gnu/ld-2.19.so]
----------------------------------------------------------------------------------------------------------------------------------------------------------
[Function Page Info] [Total: 11416KB] [Alloc: 11444KB(817)] [Free: 188KB(47)] (KERNEL)
==========================================================================================================================================================
Usage ( Usr / Buf / Ker )|___________________Function____________________|__________________________________LifeTime__________________________________
==========================================================================================================================================================
8192K( 0/ 0/ 8192)| do_huge_pmd_anonymous_page | AVR: 1.563 / MIN: 1.562 / MAX: 1.564
+ 8192K( 0/ 0/ 8192)| <- handle_mm_fault <- __do_page_fault <- do_page_fault <- page_fault
----------------------------------------------------------------------------------------------------------------------------------------------------------
3084K( 3084/ 0/ 0)| handle_mm_fault | AVR: 1.563 / MIN: 1.554 / MAX: 1.569
+ 3076K( 3076/ 0/ 0)| <- __do_page_fault <- do_page_fault <- page_fault
+ 4K( 4/ 0/ 0)| <- __get_user_pages <- get_user_pages <- copy_strings.isra.17 <- copy_strings_kernel <- do_execve_common.isra.23
<- SyS_execve <- stub_execve
+ 4K( 4/ 0/ 0)| <- __do_page_fault <- do_page_fault <- page_fault <- load_elf_binary <- search_binary_handler
<- do_execve_common.isra.23 <- SyS_execve <- stub_execve
----------------------------------------------------------------------------------------------------------------------------------------------------------
# ./guider.py record -F
[File Usage Info] [ File: 281 ] [ RAM: 78056(KB) ] [ Keys: Foward/Back/Save/Quit ]
==========================================================================================================================================================
__RAM(KB)___|_File(KB)_|__%___|_____________________________________________________Library & Process_____________________________________________________
==========================================================================================================================================================
7,616 | 7,616 | 100 | /run/samba/locking.tdb [Proc: 10] [Link: 1]
| smbd ( 2937) | smbd ( 9178) | smbd (21387) | smbd ( 3356) |
| smbd ( 2828) | smbd ( 2417) | smbd ( 3862) | smbd ( 2631) |
| smbd (11086) | smbd ( 729) |
----------------------------------------------------------------------------------------------------------------------------------------------------------
6,076 | 8,452 | 71 | /usr/lib/apache2/modules/libphp5.so [Proc: 11] [Link: 1]
| /usr/sbin/apach (13071) | /usr/sbin/apach (13073) | /usr/sbin/apach ( 3817) | /usr/sbin/apach ( 9111) |
| /usr/sbin/apach (20085) | /usr/sbin/apach ( 7221) | /usr/sbin/apach ( 345) | /usr/sbin/apach ( 346) |
| /usr/sbin/apach ( 7222) | /usr/sbin/apach (14278) | /usr/sbin/apach ( 9715) |
----------------------------------------------------------------------------------------------------------------------------------------------------------
5,784 | 9,828 | 58 | /usr/sbin/smbd [Proc: 10] [Link: 1]
| smbd ( 2937) | smbd ( 9178) | smbd (21387) | smbd ( 3356) |
| smbd ( 2828) | smbd ( 2417) | smbd ( 3862) | smbd ( 2631) |
| smbd (11086) | smbd ( 729) |
----------------------------------------------------------------------------------------------------------------------------------------------------------
4,800 | 25,880 | 18 | /var/lib/gems/1.9.1/gems/passenger-5.1.0/buildout/support-binaries/PassengerAgent [Proc: 3] [Link: 1]
| PassengerAgent (23161) | PassengerAgent (23176) | PassengerAgent (23191) |
----------------------------------------------------------------------------------------------------------------------------------------------------------
3,612 | 12,016 | 30 | /usr/sbin/mysqld [Proc: 1] [Link: 1]
| mysqld ( 3208) |
----------------------------------------------------------------------------------------------------------------------------------------------------------
2,988 | 2,988 | 100 | /usr/lib/libpython2.7.so.1.0 [Proc: 6] [Link: 1]
| vi (18865) | vi (28546) | vi ( 7200) | vi (22546) |
| vi ( 8826) | vi ( 8135) |
----------------------------------------------------------------------------------------------------------------------------------------------------------
2,228 | 2,884 | 77 | /usr/bin/python3.2mu [Proc: 1] [Link: 1]
| guider (22637) |
----------------------------------------------------------------------------------------------------------------------------------------------------------
2,016 | 2,016 | 100 | /usr/lib/libruby-1.9.1.so.1.9.1 [Proc: 1] [Link: 1]
| ruby1.9.1 (23294) |
----------------------------------------------------------------------------------------------------------------------------------------------------------
# ./guider.py draw guider.out
Input a command as below to start tracing for all threads.
# ./guider.py record -a
And input 'Ctrl + c' on keyboard to finish tracing.
Input a command as below to start monitoring for all processes.
$ ./guider.py top -a
And Input 'Ctrl + c' on keyboard to finish monitoring.
Input a command as below to see more instructions.
$ ./guider.py -h
Visit a below link to see the output of guider.
- https://github.com/iipeace/guider/wiki
- linux kernel (>= 2.6)
- python (>= 2.7)
If you can run 'pip' on your system then just input a command as below
# pip install guider
Then you can use 'guider' command
Otherwise, download the source from https://github.com/iipeace/guider
Then you can just run "guider.py" but it is a little bit heavy
If you want to run guider lightly then input a command as below
# make && make install
Then you can use 'guider' command
Enable kernel options as below to take advantage of all the features
And if CONFIG_STRICT_MEMORY_RWX is enabled then disable it
CONFIG_RING_BUFFER
CONFIG_FTRACE
CONFIG_TRACING
CONFIG_TRACING_SUPPORT
CONFIG_EVENT_TRACING
CONFIG_NOP_TRACER
CONFIG_TRACEPOINTS
CONFIG_DYNAMIC_FTRACE
CONFIG_HAVE_DYNAMIC_FTRACE
CONFIG_FTRACE_SYSCALLS
CONFIG_HAVE_SYSCALL_TRACEPOINTS
CONFIG_TRACE_IRQFLAGS
CONFIG_TRACE_IRQFLAGS_SUPPORT
CONFIG_STACKTRACE
CONFIG_STACKTRACE_SUPPORT
CONFIG_USER_STACKTRACE_SUPPORT
CONFIG_FUNCTION_TRACER
CONFIG_FUNCTION_GRAPH_TRACER
CONFIG_UPROBES
CONFIG_UPROBE_EVENT
CONFIG_KPROBES
CONFIG_KPROBE_EVENTS
CONFIG_TASKSTATS
CONFIG_TASK_DELAY_ACCT
CONFIG_TASK_XACCT
CONFIG_TASK_IO_ACCOUNTING
CONFIG_PERF_EVENTS
CONFIG_HW_PERF_EVENT
[analysis]
top [process]
bgtop [background]
threadtop [thread]
filetop [file]
stacktop [stack]
perftop [PMU]
memtop [memory]
disktop [storage]
wsstop [WSS]
reptop [report]
record [thread]
funcrecord [function]
filerecord [file]
sysrecord [system]
draw [image]
cpudraw [cpu]
memdraw [memory]
vssdraw [vss]
rssdraw [rss]
leakdraw [leak]
iodraw [io]
mem [page]
strace [syscall]
[control]
kill [signal]
setsched [priority]
getaffinity [affinity]
setaffinity [affinity]
server [server]
client [client]
cpulimit [cpu]
[test]
alloctest [mem]
[communication]
list
start
send
event
- Use comma(,) as delimiter for multiple option values
[record]
-e [enable_options - characters]
[common] {m(em)|b(lock)|e(ncoding)}
[function] {h(eap)|L(ock)|p(ipe)|g(raph)}
[thread] {i(rq)|l(ock)|n(et)|p(ipe)|
P(ower)|r(eset)|g(raph)}
[top] {t(hread)|wf(C)|s(tack)|w(ss)|d(isk)|
P(erf)|G(pu)|i(rq)|ps(S)|u(ss)|W(chan)|
I(mage)|a(ffinity)|r(eport)|a(ffinity)|
h(andler)|f(loat)|R(file)}
-d [disable_options - characters]
[common] {c(pu)|e(ncoding)}
[thread] {a(ll)}
[function] {a(ll)|u(ser)}
[top] {p(rint)|P(erf)|W(chan)|n(net)}
-s [save_traceData - path]
-u [run_inBackground]
-W [wait_forSignal]
-b [set_bufferSize - KB]
-D [trace_threadDependency]
-t [trace_syscall - syscalls]
-T [set_fontPath - path]
-j [set_reportPath - path]
-U [set_userEvent - name:func|addr:file]
-K [set_kernelEvent - name:func|addr{:%reg/argtype:rettype}]
-C [set_commandScriptPath - path]
-w [set_customRecordCommand - BEFORE|AFTER|STOP:file{:value}]
-x [set_addressForLocalServer - {ip:port}]
-X [set_requestToRemoteServer - {req@ip:port}]
-N [set_reportToRemoteServer - req@ip:port]
-M [set_objdumpPath - file]
[analysis]
-o [save_outputData - path]
-S [sort - c(pu)/m(em)/b(lock)/w(fc)/p(id)/
n(ew)/r(untime)/f(ile)/s(yscall)]
-O [set_coreFilter - cores]
-P [group_perProcessBasis]
-p [show_preemptInfo - tids]
-l [set_addr2linePath - files]
-r [set_targetRootPath - dir]
-I [set_inputValue - file|addr]
-q [configure_taskList]
-Z [convert_textToImage]
-L [set_graphLayout - CPU|MEM|IO{:proportion}]
-m [set_terminalSize - {rows:cols}]
[common]
-a [show_allInfo]
-Q [print_allRowsInaStream]
-i [set_interval - sec]
-R [set_repeatCount - {interval:}count]
-g [set_filter - comms|tids{:files}]
-A [set_arch - arm|aarch64|x86|x64]
-c [set_customEvent - event:cond]
-E [set_errorLogPath - file]
-H [set_functionDepth]
-k [set_killList - comms|tids{:CONT}]
-z [set_cpuAffinity - mask:tids|ALL{:CONT}]
-Y [set_schedPriority - policy:prio|times{:tid|ALL:CONT}]
-v [verbose]
[ thread mode examples ]
- record and report CPU events of threads
# ./guider.py record -s .
- record and save specific resource events of threads in the background
# ./guider.py record -s . -e m, b, i -u
- record and save specific resource events excluding CPU of threads in the background
# ./guider.py record -s . -e m, b, i -d c -u
- record and save specific system call events of specific threads
# ./guider.py record -s . -t sys_read, write -g 1234
- record and save lock events of threads
# ./guider.py record -s . -e L
- record and save specific user function events of threads
# ./guider.py record -s . -U evt1:func1:/tmp/a.out, evt2:0x1234:/tmp/b.out
- record and save specific kernel function events of threads
# ./guider.py record -s . -K evt1:func1, evt2:0x1234
- record and save specific kernel function events with register values
# ./guider.py record -s . -K strace32:func1:%bp/u32.%sp/s64, strace:0x1234:$stack:NONE
- record and save specific kernel function events with the return value
# ./guider.py record -s . -K openfile:getname::**string, access:0x1234:NONE:*string
- execute special commands and record and save CPU events of threads
# ./guider.py record -s . -w BEFORE:/tmp/started:1, BEFORE:ls
- report all possible information from trace data
# ./guider.py guider.dat -o . -a -i
- report stats on a specific interval from trace data
# ./guider.py guider.dat -o . -R 3
- report stats including preemption of specific threads from trace data
# ./guider.py guider.dat -o . -p 1234, 4567
- report stats including specific threads involved in the specific processes from trace data
# ./guider.py guider.dat -o . -P -g 1234, 4567
- draw graph and chart from trace data
# ./guider.py draw guider.dat
[ function mode examples ]
- record and report CPU function events of threads
# ./guider.py record -f -s .
- record and save CPU function events of specific threads having TID bigger than 1024
# ./guider.py record -f -s . -g 1024\<
- record and save specific function events of threads except for user-mode
# ./guider.py record -f -s . -d u -c sched/sched_switch
- record and save specific resource function events specific threads
# ./guider.py record -f -s . -e m, b, h -g 1234
- record and save specific function events of threads with specific argument condition
# ./guider.py record -f -s . -c softirq_entry:vec══1
- record and save segmentation fault function events of threads
# ./guider.py record -f -s . -K segflt:bad_area -e p
- record and save blocking function events of threads
# ./guider.py record -f -s . -K block:schedule
- execute special commands and record and save CPU function events of threads
# ./guider.py record -s . -w BEFORE:/tmp/started:1, BEFORE:ls
- report all possible information from trace data using specific toolchain tools
# ./guider.py guider.dat -o . -r /home/target/root -l $(which arm-addr2line) -a
- report all possible information with remote server support
# ./guider.py guider.dat -o . -a -X 10.97.20.53:5555 -x 1234
- report all possible information about only lower than 3 function levels
# ./guider.py guider.dat -o . -H 3
[ top mode examples ]
- show resource usage of processes in real-time
# ./guider.py top
- show resource usage of processes on the fixed-size terminal in real-time
# ./guider.py top -m
- show files opened via processes in real-time
# ./guider.py top -e F
- show specific files opened via specific processes in real-time
# ./guider.py top -e F -g init, lightdm : home, var
- show performance stats of specific processes in real-time
# ./guider.py top -e P -g init, lightdm
- show resource usage of processes by sorting memory in real-time
# ./guider.py top -S m
- show resource usage of processes by sorting file in real-time
# ./guider.py top -S f
- show resource usage of processes only 5 times in real-time
# ./guider.py top -R 5
- show resource usage of processes only 5 times per 3-sec interval in real-time
# ./guider.py top -R 3, 5
- show resource usage including block of threads per 2-sec interval in real-time
# ./guider.py top -e t, b -i 2 -a
- show resource usage of specific processes/threads involved in specific process group in real-time
# ./guider.py top -g 1234,4567 -P
- save resource usage of processes and write to the specific file in real-time
# ./guider.py top -o . -e p
- save and print resource usage of processes in real-time
# ./guider.py top -o . -Q
- save resource usage of processes in the background
# ./guider.py top -o . -u
- report system stats in the background
# ./guider.py reporttop -j . -u
- save resource usage of processes and report system stats if some events occur
# ./guider.py top -o . -e r, R
- save resource usage of processes and report system status to the specific image
# ./guider.py top -o . -e r, I
- save resource usage of processes and report to file if specific conditions met
# ./guider.py top -o . -e R
- show resource usage of processes and execute special commands every interval
# ./guider.py top -w AFTER:/tmp/touched:1, AFTER:ls
- show storage usage in real-time
# ./guider.py disktop
- trace memory working set of specific processes
# ./guider.py wsstop -g chrome
- draw graph and chart to specific files
# ./guider.py draw guider.out
- draw graph and chart for specific process group to specific files
# ./guider.py draw guider.out -g chrome
- draw CPU and memory graphs of specific processes to a specific file proportionally
# ./guider.py draw guider.out -g chrome -L cpu:5, mem:5
- draw VSS graph and chart for specific processes to specific files
# ./guider.py vssdraw guider.out -g chrome
- draw leak graph and chart to specific files
# ./guider.py leakdraw guider.out
- show and report resource usage of processes to specific server
# ./guider.py top -e r -N [email protected]:5555
- show and report resource usage of processes to specific clients that asked it
# ./guider.py top -x 5555
- handle report data from the server
# ./guider.py top -x 5555 -X
- show resource usage of processes and set condition file path for the report
# ./guider.py top -I guider.json
[ file mode examples ]
- trace memory usage of files mapped to processes
# ./guider.py record -F -o .
- trace memory usage of files mapped to processes each interval
# ./guider.py record -F -i
[ etc examples ]
- check the property of specific pages
# ./guider.py mem -g 1234 -I 0x7abc1234-0x7abc6789
- convert a text file to an image file
# ./guider.py guider.out -Z
- wait for the signal to start
# ./guider.py record│top -W
- show guider processes
# ./guider.py list
- send the signal to all guider processes
# ./guider.py send
# ./guider.py kill
- send the stop signal to all guider processes
# ./guider.py stop
- send specific signals to specific processes
# ./guider.py send -9 1234, 4567
# ./guider.py kill -kill 1234, 4567
- change the priority of tasks
# ./guider.py setsched c:-19, r:90:1217, i:0:1209
- change the priority of tasks in a group
# ./guider.py setsched c:-19, r:90:1217 -P
- update priority of all tasks shown to real-time 90
# ./guider.py top -Y r:90:ALL
- update priority of all tasks shown to the deadline policy
# ./guider.py top -Y d:1000000/20000000/20000000:ALL
- update the priority of a task continuously to real-time 90
# ./guider.py top -Y r:90:1234:CONT
- update CPU affinity of all tasks shown
# ./guider.py top -z f:ALL
- update CPU affinity of tasks continuously
# ./guider.py top -z f:1234:CONT
- limit CPU usage of specific processes
# ./guider.py cpulimit -g 1234:40, 5678:10
- limit CPU usage of specific threads
# ./guider.py cpulimit -g 1234:40, 5678:10 -e t