CIT 470 Advanced Network and System Administration: Lab #13: Monit

Lab #13: Monit Lab
Date: March 31, 2008

In this lab, we will use the open source tool monit to monitor some of our system processes. Monit can alert the administrator to problems via log files or e-mail, and it can also restart servers that have failed. Monit can monitor both the local host and remote hosts using ping and can check TCP/IP port connections.

Monit is the most widely deployed open source monitoring tool, but god, a tool written in Ruby, has become popular for more complex tasks. God is configured using the Ruby programming language, providing unlimited flexibility in monitoring tasks at the cost of being considerably more complex to use than monit.

Installing Monit
1. Download the monit source code from its web site.
2. Unpack and build monit. You will discover that you are missing some software that monit depends on, that you will need to install using yum.
```
tar zxvf monit-VERSION.tar.gz
cd monit-VERSION
./configure
make
```
3. Once monit is build, become root and install it.
```
make install
```
4. Check the install by attempting to run monit.
```
monit -h
```

Configuring Monit

Create a configuration file /etc/monitrc with the following contents. This will configure monit to check every 60 seconds and send alerts both to syslog and to the e-mail address root@localhost. The check stanza describes how to monitor and restart syslog.

#
# CIT 470 Lab XX
# Your Name
#
set daemon 60
set logfile syslog facility log_daemon
set alert root@localhost

# Run monit web server so "monit status" and other commands work
set httpd port 2812 and use address localhost
    allow localhost   # Allow only localhost to connect
    allow admin:monit # Allow Basic Auth

# Monit syslog process and restart if it goes down
check process syslogd with pidfile /var/run/syslogd.pid
    start program = "/etc/init.d/syslog start"
    stop program = "/etc/init.d/syslog stop"
    if 5 restarts within 5 cycles then timeout

Start monit as root. We use the verbose option so we can see what monit is doing.
```
monit -v
```
Check status. If anything is in the initializing state, wait 30 seconds, then check again.
```
monit summary
```
Once status is normal, check the detailed status information.
```
monit status
```
Kill the running syslog process.
```
kill SYSLOG_PID
```

Verify that you have killed syslog.

ps auxw | grep syslog
service syslog status

Check if monit has noticed that syslog has been killed.
```
monit status
watch -n 1 monit summary
```
Since we killed syslog, we won't see a syslog entry from monit, but we should receive an e-mail message.
```
tail -f /var/mail/root
```

Verify that monit has restart syslog.

ps auxw | grep syslog
service syslog status
monit status

Monitoring Mail
1. Configure monit to monitor sendmail.
2. Run monit again so that it re-reads its configuration file.
```
monit -v
```
3. Kill sendmail.
```
kill SENDMAIL_PID
```
4. Watch the log file to see monit restart the mail server. Notice the log messages that show that monit realizes that it cannot notify you by mail in this situation.
```
tail -f /var/log/messages
```
System Monitoring
1. Configure monit to alert if the root partition is more than 80% full.
2. Run monit again so that it re-reads its configuration file.
```
monit -v
```
3. Use dd to fill more than 80% of the disk.
```
dd if=/dev/zero of=/diskhog bs=1M count=NUMBER_OF_BLOCKS_TO_FILL_DISK
```
4. Watch the log file for an alert. Watch the monitoring status in another window.
```
tail -f /var/log/messages
watch -n 1 monit summary
```
5. Fix the space issue.
```
rm /diskhog
```
6. Configure monit to alert if the load average is greater than 2 for one minute.
7. Run monit again so that it re-reads its configuration file.
```
monit -v
```
8. Start lots of time wasting processes to drive up the load average. Use uptime to watch the load average in one window to see if it's high enough for a full minute. Watch the monit status in another window at the same time.
```
watch -n 1 uptime
watch -n 1 monit summary
```
9. Watch the log file for an alert.
```
tail -f /var/log/messages
```
Remote System Monitoring
1. Start your virtual machine.
2. Configure monit to check whether the VM is up once every 15 seconds.
3. Check status. If anything is in the initializing state, wait 30 seconds, then check again.
```
monit summary
```
4. Shutdown the virtual machine.
5. Watch the log file until you receive an alert. Watch monit status at the same time in another window.
```
tail -f /var/log/messages
watch -n 1 monit summary
```

Submit your /etc/monitrc file with your name in the comments so that I can identify you.