Klient si mě objednal, abych zkontroloval jeho systém, protože se mu zdála reakce databází pomalá.
Udělal jsem kontrolu jedné databáze tzv. healthcheck a měl jsem již pochyby o systému, neboť databáze sama o sobě byla nastavená normálně.
Přikládám proto seznam základních utilit, které vám pomohou odhalit ihned problém z linux based operačního systému.
Užitečné utility – použití níže
vmstat Monitors processes, CPU, memory, or disk I/O bottlenecks.
watch Periodically runs another command.
ps Identifies highest CPU- and memory-consuming sessions. Used to identify Oracle sessions consuming the most system resources.
top Identifies sessions consuming the most resources.
mpstat Reports CPU statistics.
sar Displays CPU, memory, disk I/O, and network usage, both current and historical.
free Displays free and used memory.
df Reports on free disk space.
du Displays disk usage.
iostat Displays disk I/O statistics.
netstat Reports on network statistics.
IOSTAT
[oracle@oracle ~]$ iostat -x 1 Linux 2.6.32-358.el6.x86_64 (oracle.dbserver.cz) 10/09/2014 _x86_64_ (4 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 1.00 0.00 0.25 28.57 0.00 70.18
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 60.00 70.00 43.00 47.00 840.00 800.00 18.22 11.34 150.98 11.11 100.00
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 67.40 100.10 46.30 60.20 936.80 1089.60 19.03 17.31 162.82 9.39 100.00
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 73.70 97.70 47.40 58.60 1003.20 1063.20 19.49 17.83 167.96 9.43 100.00
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 22.90 91.00 30.50 67.20 436.00 1054.40 15.25 11.96 125.33 10.24 100.00
– Ihned je vidět, že disk je 100% utilizován.
DISKSTATS
[oracle@oracle ~]$ cat /proc/diskstats 1 0 ram0 0 0 0 0 0 0 0 0 0 0 0 1 1 ram1 0 0 0 0 0 0 0 0 0 0 0 1 2 ram2 0 0 0 0 0 0 0 0 0 0 0 1 3 ram3 0 0 0 0 0 0 0 0 0 0 0 1 4 ram4 0 0 0 0 0 0 0 0 0 0 0 1 5 ram5 0 0 0 0 0 0 0 0 0 0 0 1 6 ram6 0 0 0 0 0 0 0 0 0 0 0 1 7 ram7 0 0 0 0 0 0 0 0 0 0 0 1 8 ram8 0 0 0 0 0 0 0 0 0 0 0 1 9 ram9 0 0 0 0 0 0 0 0 0 0 0 1 10 ram10 0 0 0 0 0 0 0 0 0 0 0 1 11 ram11 0 0 0 0 0 0 0 0 0 0 0 1 12 ram12 0 0 0 0 0 0 0 0 0 0 0 1 13 ram13 0 0 0 0 0 0 0 0 0 0 0 1 14 ram14 0 0 0 0 0 0 0 0 0 0 0 1 15 ram15 0 0 0 0 0 0 0 0 0 0 0 7 0 loop0 0 0 0 0 0 0 0 0 0 0 0 7 1 loop1 0 0 0 0 0 0 0 0 0 0 0 7 2 loop2 0 0 0 0 0 0 0 0 0 0 0 7 3 loop3 0 0 0 0 0 0 0 0 0 0 0 7 4 loop4 0 0 0 0 0 0 0 0 0 0 0 7 5 loop5 0 0 0 0 0 0 0 0 0 0 0 7 6 loop6 0 0 0 0 0 0 0 0 0 0 0 7 7 loop7 0 0 0 0 0 0 0 0 0 0 0 8 0 sda 34608389 54934715 1353144938 322972473 297590545 757173201 7465050216 3650381753 7 2679894176 3973222735 8 1 sda1 33366972 50988571 1311644450 154037435 175883738 752740252 7429188080 1972748178 7 1661368700 2126693390 8 2 sda2 1241166 3946089 41498040 168921644 49818 4432949 35862136 58972454 0 112613875 227893198 11 0 sr0 0 0 0 0 0 0 0 0 0 0 0
Popis sloupců
Field 1 — # of reads issued
Field 2 — # of reads merged
Field 3 — # of sectors read
Field 4 — # of milliseconds spent reading
Field 5 — # of writes completed
Field 6 — # of writes merged
Field 7 — # of sectors written
Field 8 — # of milliseconds spent writing
Field 9 — # of I/Os currently in progress
Field 10 — # of milliseconds spent doing I/Os
Field 11 — weighted # of milliseconds spent doing I/Os
VMSTAT
[oracle@oracle ~]$ vmstat 2 10
procs -----------memory---------- ---swap-- -----io-- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 6 2265908 165900 1244960 11337392 2 1 56 306 0 1 6 2 74 19 0
0 1 2265908 171496 1245784 11337720 0 0 446 440 1407 3841 2 1 55 42 0
1 1 2265908 172620 1246080 11337696 0 0 152 458 1161 2477 2 1 65 32 0
0 1 2265908 175116 1246424 11337956 0 0 200 826 1403 4943 2 1 66 31 0
0 2 2265908 165552 1247020 11338536 0 0 316 656 1362 3956 3 1 62 34 0
0 2 2265908 173132 1247384 11338400 0 0 208 422 1282 2532 1 0 58 40 0
0 4 2265908 174372 1247800 11338652 0 0 228 594 1305 4793 3 1 56 40 0
– Disk maká a CPU je relativně v klidu
r Number of processes waiting for runtime
b Number of processes in uninterruptible sleep
swpd Total virtual memory (swap) in use (KB)
free Total idle memory (KB)
buff Total memory used as buffers (KB)
cache Total memory used as cache (KB)
si Memory swapped in from disk (KB/s)
so Memory swapped out to disk (KB/s)
bi Blocks read in (blocks/s) from block device
bo Blocks written out (blocks/s) per second to block device
in Interrupts per second
cs Context switches per second
us User-level code time as a percentage of total CPU time
sy System-level code time as a percentage of total CPU time
id Idle time as a percentage of total CPU time
wa Time waiting for I/O completion
[oracle@oracle ~]$ vmstat Total DISK READ: 2.53 M/s | Total DISK WRITE: 241.72 K/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND 29736 be/4 oracle 2.64 M 0.00 B 0.00 % 99.42 % oracleTEST (LOCAL=NO) 26982 be/4 oracle 2.51 M 0.00 B 0.00 % 99.30 % oracleTEST (LOCAL=NO) 28770 be/4 oracle 19.00 M 0.00 B 0.00 % 97.75 % oracleTEST (LOCAL=NO) 29847 be/4 oracle 1496.00 K 0.00 B 0.00 % 95.69 % oracleTEST (LOCAL=NO) 5803 be/4 oracle 2.41 M 0.00 B 0.00 % 70.83 % ls --color=auto 22287 be/4 oracle 0.00 B 960.00 K 0.00 % 59.25 % ora_dbw0_VYVOJ1 9292 be/4 oracle 96.00 K 0.00 B 0.00 % 27.89 % -bash 4625 be/4 oracle 0.00 B 0.00 B 0.00 % 6.76 % ls --color=auto 26626 be/4 oracle 16.00 K 0.00 B 0.00 % 6.19 % oracleTEST (LOCAL=NO) 20581 be/4 oracle 0.00 B 88.00 K 0.00 % 5.52 % ora_dbw0_TEST 3357 be/4 oracle 0.00 B 640.00 K 0.00 % 5.39 % ora_dbw0_KONTROLA 4156 be/4 oracle 0.00 B 96.00 K 0.00 % 5.27 % ora_ckpt_orcl 20583 be/4 oracle 4.00 K 32.00 K 0.00 % 4.90 % ora_lgwr_TEST 3361 be/4 oracle 0.00 B 224.00 K 0.00 % 4.87 % ora_ckpt_KONTROLA 28084 be/4 oracle 0.00 B 0.00 B 0.00 % 4.07 % oracleTEST (LOCAL=NO) 20585 be/4 oracle 0.00 B 96.00 K 0.00 % 3.93 % ora_ckpt_TEST 22291 be/4 oracle 0.00 B 96.00 K 0.00 % 2.60 % ora_ckpt_VYVOJ1 383 be/3 root 0.00 B 20.00 K 0.00 % 1.67 % [jbd2/sda1-8] 27460 be/4 oracle 124.00 K 0.00 B 0.00 % 1.55 % java -server -Xmx384M -XX:MaxPermS~cle.moro.cz_TEST/config/server.xml 3359 be/4 oracle 0.00 B 8.00 K 0.00 % 0.66 % ora_lgwr_KONTROLA 2819 be/4 oracle 0.00 B 16.00 K 0.00 % 0.00 % tnslsnr LISTENER -inherit 26449 be/4 oracle 0.00 B 4.00 K 0.00 % 0.00 % java -server -Xmx384M -XX:MaxPermS~cle.moro.cz_TEST/config/server.xml
– Zde je vidět, které služby konkrétně disk zatěžují
PS
[oracle@oracle ~]$ ps -ef | grep pmon oracle 3339 1 0 Sep04 ? 00:07:44 ora_pmon_KONTROLA oracle 4134 1 0 Sep04 ? 00:04:56 ora_pmon_orcl oracle 16521 16109 0 20:58 pts/5 00:00:00 grep pmon oracle 20563 1 0 Oct07 ? 00:00:26 ora_pmon_TEST oracle 22269 1 0 Oct07 ? 00:01:16 ora_pmon_VYVOJ1
– Na serveru běží 4 databáze a on to evidentně nestihá
MPSTAT
[oracle@oracle ~]$ mpstat -P ALL Linux 2.6.32-358.el6.x86_64 (oracle.moro.cz) 10/09/2014 _x86_64_ (4 CPU)
09:01:00 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle 09:01:00 PM all 5.49 0.00 1.46 18.81 0.00 0.13 0.00 0.00 74.12 09:01:00 PM 0 9.36 0.00 2.11 40.07 0.00 0.30 0.00 0.00 48.16 09:01:00 PM 1 4.93 0.00 1.42 19.49 0.00 0.07 0.00 0.00 74.08 09:01:00 PM 2 4.20 0.00 1.25 9.83 0.00 0.07 0.00 0.00 84.65 09:01:00 PM 3 3.51 0.00 1.05 6.00 0.00 0.06 0.00 0.00 89.38
– Pamět je v pořádku, většinou IDLE
SAR
[oracle@oracle ~]$ sar -u 2 5 Linux 2.6.32-358.el6.x86_64 (oracle.moro.cz) 10/09/2014 _x86_64_ (4 CPU)
09:02:13 PM CPU %user %nice %system %iowait %steal %idle 09:02:15 PM all 0.75 0.00 0.25 79.82 0.00 19.17 09:02:17 PM all 1.25 0.00 0.38 72.75 0.00 25.62 09:02:19 PM all 1.51 0.00 0.38 79.80 0.00 18.32 09:02:21 PM all 1.38 0.00 0.50 88.22 0.00 9.90 09:02:23 PM all 1.25 0.00 0.62 77.88 0.00 20.25 Average: all 1.23 0.00 0.43 79.69 0.00 18.66
Active / Total Objects (% used) : 1643171 / 2374642 (69.2%) Active / Total Slabs (% used) : 78609 / 78622 (100.0%) Active / Total Caches (% used) : 111 / 191 (58.1%) Active / Total Size (% used) : 213449.43K / 309924.98K (68.9%) Minimum / Average / Maximum Object : 0.02K / 0.13K / 4096.00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 548064 547979 99% 0.02K 3806 144 15224K avtab_node 411264 387423 94% 0.03K 3672 112 14688K size-32 379953 231730 60% 0.10K 10269 37 41076K buffer_head 327063 63135 19% 0.07K 6171 53 24684K selinux_inode_security 169802 34789 20% 0.06K 2878 59 11512K size-64 99280 34761 35% 0.19K 4964 20 19856K dentry 93961 77704 82% 0.55K 13423 7 53692K radix_tree_node 83182 66413 79% 0.20K 4378 19 17512K vm_area_struct 76392 52295 68% 1.00K 19098 4 76392K ext4_inode_cache 45640 33902 74% 0.19K 2282 20 9128K filp 37884 27965 73% 0.05K 492 77 1968K anon_vma_chain 31740 22802 71% 0.04K 345 92 1380K anon_vma 12940 12892 99% 0.19K 647 20 2588K size-192 11583 11551 99% 0.14K 429 27 1716K sysfs_dir_cache 6300 5351 84% 0.12K 210 30 840K size-128 6216 6128 98% 0.58K 1036 6 4144K inode_cache
FREE
[oracle@oracle ~]$ free total used free shared buffers cached Mem: 16246024 16073056 172968 0 453964 12747760 -/+ buffers/cache: 2871332 13374692 Swap: 19249144 2550120 16699024 - Paměti taky není nazbyt a hodilo by se přidat nebo upravit nastavení databází.
MEMINFO
[oracle@oracle ~]$ cat /proc/meminfo MemTotal: 16246024 kB MemFree: 167960 kB Buffers: 449764 kB Cached: 12822512 kB SwapCached: 82024 kB Active: 9841304 kB Inactive: 5299560 kB Active(anon): 5630812 kB Inactive(anon): 1089864 kB Active(file): 4210492 kB Inactive(file): 4209696 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 19249144 kB SwapFree: 16692352 kB Dirty: 300 kB Writeback: 32 kB AnonPages: 1799356 kB Mapped: 3216376 kB Shmem: 4851928 kB Slab: 330996 kB SReclaimable: 196172 kB SUnreclaim: 134824 kB KernelStack: 5640 kB PageTables: 449900 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 27372156 kB Committed_AS: 14631204 kB VmallocTotal: 34359738367 kB VmallocUsed: 389556 kB VmallocChunk: 34359315532 kB HardwareCorrupted: 0 kB AnonHugePages: 428032 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 8192 kB DirectMap2M: 16685056 kB
TOP
top - 21:07:06 up 35 days, 12:45, 5 users, load average: 18.20, 17.95, 17.89 Tasks: 505 total, 1 running, 504 sleeping, 0 stopped, 0 zombie Cpu(s): 3.7%us, 0.5%sy, 0.0%ni, 12.8%id, 83.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 16246024k total, 16078872k used, 167152k free, 450712k buffers Swap: 19249144k total, 2558500k used, 16690644k free, 12893752k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16009 oracle 20 0 1751m 337m 335m S 5.0 2.1 0:03.08 oracle 17207 oracle 20 0 1751m 178m 175m S 4.7 1.1 0:00.64 oracle
OS WATCHER
Oracle provides a collection of Linux/Unix scripts that gather and store metrics for CPU, memory, disk, and
network usage. The OS Watcher tool suite automates the gathering of statistics using tools such as top,
vmstat, iostat, mpstat, netstat, and traceroute. If you don’t have these utilities installed, see
Chapter 10 for details on installing the sysstat package.
You can obtain OS Watcher from Oracle’s MetaLink web site. Search for document ID 301137.1 or for the
document titled “OS Watcher User Guide.” Navigate to the Contents page, and search for the Download link.
This utility also has an optional graphical component for visually displaying performance metrics. The
OS Watcher utility is currently supported on the following platforms: Linux, Solaris, AIX, Tru64, and HP-UX.
OSWatcher (Includes: [Video]) (Doc ID 301137.1)