How do I find out Linux Resource utilization to detect system bottlenecks?

Q. How can I find out Linux Resource utilization using vmstat command? How do I get information about high disk I/O and memory usage?

A. vmstat command reports information about processes, memory, paging, block IO, traps, and cpu activity. However, a real advantage of vmstat command output - is to the point and (concise) easy to read/understand. The output of vmstat command use to help identify system bottlenecks. Please note that Linux vmstat does not count itself as a running process.

Here is an output of vmstat command from my enterprise grade system:
$ vmstat -S M
Output:

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
3  0      0   1963    607   2359    0    0     0     0    0     1 32  0 68  0

Where,

• The fist line is nothing but six different categories. The second line gives more information about each category. This second line gives all data you need.
• -S M: vmstat lets you choose units (k, K, m, M) default is K (1024 bytes) in the default mode. I am using M since this system has over 4 GB memory. Without -M option it will use K as unit

$ vmstat
Output:

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
3  0      0 2485120 621952 2415368  0    0     0     0    0     1 32  0 68  0

Field Description For Vm Mode

(a) procs is the process-related fields are:

• r: The number of processes waiting for run time.
• b: The number of processes in uninterruptible sleep.

(b) memory is the memory-related fields are:

• swpd: the amount of virtual memory used.
• free: the amount of idle memory.
• buff: the amount of memory used as buffers.
• cache: the amount of memory used as cache.

(c) swap is swap-related fields are:

• si: Amount of memory swapped in from disk (/s).
• so: Amount of memory swapped to disk (/s).

(d) io is the I/O-related fields are:

• bi: Blocks received from a block device (blocks/s).
• bo: Blocks sent to a block device (blocks/s).

(e) system is the system-related fields are:

• in: The number of interrupts per second, including the clock.
• cs: The number of context switches per second.

(f) cpu is the CPU-related fields are:

These are percentages of total CPU time.

• us: Time spent running non-kernel code. (user time, including nice time)
• sy: Time spent running kernel code. (system time)
• id: Time spent idle. Prior to Linux 2.5.41, this includes IO-wait time.
• wa: Time spent waiting for IO. Prior to Linux 2.5.41, shown as zero.

As you see the first output produced gives averages data since the last reboot. Additional reports give information on a sampling period of length delay. You need to sample data using delays i.e. collect data by setting intervals. For example collect data every 2 seconds (or collect data every 2 second 5 times only):
$ vmstat -S M 2
OR
$ vmstat -S M 2 5
Output:

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
3  0      0   1756    607   2359    0    0     0     0    0     1 32  0 68  0
3  0      0   1756    607   2359    0    0     0     0 1018    65 38  0 62  0
3  0      0   1756    607   2359    0    0     0     0 1011    64 37  0 63  0
3  0      0   1756    607   2359    0    0     0    20 1018    72 37  0 63  0
3  0      0   1756    607   2359    0    0     0     0 1012    64 37  0 62  0
3  0      0   1756    607   2359    0    0     0     0 1011    65 38  0 63  0
3  0      0   1995    607   2359    0    0     0     0 1012    62 35  2 63  0
3  0      0   1731    607   2359    0    0     0     0 1012    64 34  3 62  0
3  0      0   1731    607   2359    0    0     0     0 1013    72 38  0 62  0
3  0      0   1731    607   2359    0    0     0     0 1013    63 37  0 63  0

This is what most system administrators do to identify system bottlenecks. I hope all of you find vmstat data is concise and easy to read.

Scalr errors after install

After installing Scalr and adding a client I've tried to add an application to test out my setup. But at the second step I've got an alert saying:
-----------------------------
Error Type: LoadXML
Description: Incorrect XML
-----------------------------

A quick look at the apache log revealed the problem:
-----------------------------
File does not exist: /var/scalr/app/www/farm_amis.xml
-----------------------------

I thought that maybe I've missed a file so I did a svn checkout of scalr repository and tried to find the specified file:

#apt-get install subversion
#svn checkout http://scalr.googlecode.com/svn/trunk/ scalr-read-only
#find scalr-read-only -name farm_amis.xml -type f

Nothing came up. Weird. After a bit of looking around I've found a file called farm_amis_xml.php. So in fact farm_amis.xml was only a mod_rewrite directive.

Time to fix it, enable mod_rewrite and .htaccess files for apache2:

Edit /etc/apache2/sites-available/000-default and change the lines containing AllowOverride None to AllowOverride All

Go to /etc/apache2/mods-enabled and execute the following command:
# ln -s ../mods-available/rewrite.load

Restart apache2 server and everything should be ok:
#/etc/init.d/apache2 restart

Text manipulation 'sed'

1. #sed 's/string1/string2/g'
(Replace string1 with string2)

2. #sed 's/\(.*\)1/\12/g'
(Modify anystring1 to anystring2)

3. #sed '/ *#/d; /^ *$/d'
(Remove comments and blank lines)

4. #sed ':a; /\\$/N; s/\\\n//; ta'
(Concatenate lines with trailing \)

5. #sed 's/[ \t]*$//'
(Remove trailing spaces from lines)

6.# sed 's/\([`"$\]\)/\\\1/g'
(Escape shell metacharacters active within double quotes)

7.#seq 10 | sed "s/^/ /; s/ *\(.\{7,\}\)/\1/"
(Right align numbers)

8.#sed -n '1000p;1000q'
(Print 1000th line)

9.#sed -n '10,20p;20q'
(Print lines 10 to 20)

10. #sed -n 's/.*