Monitoring Server Load

SYSCTL

# Controls the default maxmimum size of a mesage queue

kernel.msgmnb = 65536

# Controls the maximum size of a message, in bytes

kernel.msgmax = 65536

# Controls the maximum shared segment size, in bytes

kernel.shmmax = 68719476736

# Controls the maximum number of shared memory segments, in pages

kernel.shmall = 4294967296

fs.file-max = 2097152

kernel.pid_max = 2097152

net.nf_conntrack_max = 131072

net.core.rmem_max = 16777216

net.core.wmem_max = 16777216

net.core.rmem_default = 1048576

net.core.wmem_default = 1048576

net.core.netdev_max_backlog = 65536

net.core.somaxconn = 131072

net.core.optmem_max = 25165824

net.ipv4.tcp_rmem = 65536 1048576 16777216

net.ipv4.tcp_wmem = 64436 1048576 16777216

net.ipv4.tcp_max_syn_backlog = 65536

vm.max_map_count = 262144

IOSTAT / SYSSTAT

• Whenever a Linux system CPU is occupied by a process, it is unavailable for processing other requests. The rest of the pending requests must wait until the CPU is free. This becomes a bottleneck in the system.

• Finding CPU utilization is one of the most important tasks. Linux comes with various utilities to report CPU utilization.

TOP

the top command provides a dynamic real-time view of a running system it will display system summary information as well as a list of tasks currently being managed by the Linux kernel. The top command monitors cpu utilization, process statistics and memory utilization. It will show you information such as :

• uptime

• load average

• process counts

• CPU status

• utilization statistics for both memory and swap space

MPSTAT

• if you are using a multiple core system you can use mpstat to display utilization of each core individually.

SAR

# sar -u 2 5

Output (for each 2 seconds. 5 lines are displayed):

Linux 2.6.9-42.0.3.ELsmp (www1lab2.xyz.ac.in) 01/13/2007

05:33:24 AM CPU %user %nice %system %iowait %idle

05:33:26 AM all 9.50 0.00 49.00 0.00 41.50

05:33:28 AM all 16.79 0.00 74.69 0.00 8.52

05:33:30 AM all 17.21 0.00 80.30 0.00 2.49

05:33:32 AM all 16.75 0.00 81.00 0.00 2.25

05:33:34 AM all 14.29 0.00 72.43 0.00 13.28

Average: all 14.91 0.00 71.49 0.00 13.61

%user: Percentage of CPU utilization that occurred while executing at the user level (application).

%nice: Percentage of CPU utilization that occurred while executing at the user level with nice priority.

%system: Percentage of CPU utilization that occurred while executing at the system level (kernel).

%iowait: Percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request.

%idle: Percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request.

To get multiple samples and multiple reports set an output file for the sar command. Run the sar command as a background process using.

# sar -o output.file 12 8 >/dev/null 2>&1 &

Better use nohup command so that you can logout and check back report later on:

# nohup sar -o output.file 12 8 >/dev/null 2>&1 &

All data is captured in binary form and saved to a file (data.file). The data can then be selectively displayed ith the sar command using the -f option.

# sar -f data.file

Finally, you need to determine which process is monopolizing or eating the CPUs. Following command will displays the top 10 CPU users on the Linux system.

# ps -eo pcpu,pid,user,args | sort -k 1 -r | head -10

OR

# ps -eo pcpu,pid,user,args | sort -r -k1 | less

Common Log files

/var/log/messages : General message and system related stuff

/var/log/auth.log : Authentication logs

/var/log/kern.log : Kernel logs

/var/log/cron.log : Crond logs (cron job)

/var/log/maillog : Mail server logs

/var/log/qmail/ : Qmail log directory (more files inside this directory)

/var/log/httpd/ : Apache access and error logs directory

/var/log/lighttpd/ : Lighttpd access and error logs directory

/var/log/boot.log : System boot log

/var/log/mysqld.log : MySQL database server log file /var/log/secure or /var/log/auth.log : Authentication log

/var/log/utmp or /var/log/wtmp : Login records file

/var/log/yum.log : Yum command log file.

/var/log/debug : Debugging log messages

/var/log/daemon.log : Running services such as squid, ntpd and others log message to this file

/var/log/dmesg : Linux kernel ring buffer log

/var/log/dpkg.log : All binary package log includes package installation and other information

/var/log/faillog : User failed login log file

/var/log/user.log : All userlevel logs

/var/log/xorg.0.log : X.org log file

/var/log/apache2/* : Apache web server log files directory

/var/log/fsck/* : fsck command log