Monitoring virtual ethernet interfaces with Cacti and SNMP

Prerequisites

Some familiarity with standard unix tools and networking, snmp and cacti ;)

Introduction

• SNMP does not monitor virtual interfaces (ethx:y) out of the box?
• running an Apache as root is no solution so that scripts may issue every system command
• same thing but always in the need to configure sudo or suexec so that only what we want[tm] can be executed

Now I hope you will enjoy my solution: monitoring IP traffic regardless of the interfaces. We will setup iptables, configure snmpd, and then configure cacti.

Note: This tutorial is based on Debian 3.1 (Sarge).

1. IPTABLES

First, we need to configure iptables.

We will create new chains and reference them accordingly:

root@bla:~# iptables -N traffic_in
root@bla:~# iptables -N traffic_out
root@bla:~# iptables -I INPUT 1 -j traffic_in
root@bla:~# iptables -I OUTPUT 1 -j traffic_out

We will need a script which lists all IP addresses present on the system. Usually, it will be run with the -g (generate) parameter, -i is just there to generate an index for the SNMP lookup.

#!/bin/bash
case "$1" in
  -g)
        ifconfig | grep "inet addr" | awk '{print $2}' | awk -F: '{print $2}'
  ;;

  -i)
        ipcount=`ifconfig | grep "inet addr" | awk '{print $2}' | awk -F: '{print $2}' | wc -l`
        for i in `seq 1 $ipcount`; do
                echo $i;
        done
        exit $ipcount # this is the value at OID .1.3.6.1.4.1.2021.49.42.100.1
  ;;

  *)
    exit 254
  ;;
esac

For this, I will use a small script which puts all IP addresses present on the system into the appropriate tables that we created before. You can guess what the -f parameter is good for, and usually it will not be needed. Subsequent runs of the script will add any new IP address to both in/out chains. The items in the chains are merely there to count, so it is correct that they do not have any target (-j).

#!/bin/bash

case "$1" in
  -f)
        iptables -F traffic_in;
        iptables -F traffic_out;
        ;;
esac

for ip in `/usr/local/sbin/snmp_ips/iptables_traffic_iplist.sh -g`; do
        iptables -vnL traffic_out | grep $ip >/dev/null || iptables -A traffic_out -i ! lo -s $ip;
        iptables -vnL traffic_in | grep $ip >/dev/null || iptables -A traffic_in -o ! lo -d $ip;
done

2. SNMPD

The idea is that snmpd should export the newly available data read-only. For this purpose we use snmpd's exec feature (thanks ck). (One could also be 1337 and use the pass feature, but exec will do just fine.)

[ /usr/local/sbin/snmp_ips/iptables_traffic_eval.sh ]

#!/bin/bash
[ $1 ] || exit 42;

iptables -vxnL traffic_$1 | grep -v Chain | grep -v bytes | awk {'print $2'}

Edit snmpd.conf and make sure (at least) the following lines are present and uncommented. I cannot produce an SNMP tutorial here, but make sure that readonly access to your system is available via snmp using your very own community string (in this case public).

com2sec readonly default public
group MyROGroup v1 readonly
group MyROGroup v2c readonly
view all included .1 80
access MyROGroup ""      any       noauth    exact  all    none   none

exec .1.3.6.1.4.1.2021.49 iplist /usr/local/sbin/snmp_ips/iptables_traffic_iplist.sh -g
exec .1.3.6.1.4.1.2021.49.42 ipindex /usr/local/sbin/snmp_ips/iptables_traffic_iplist.sh -i
exec .1.3.6.1.4.1.2021.50 traffic_in /usr/local/sbin/snmp_ips/iptables_traffic_eval.sh in
exec .1.3.6.1.4.1.2021.51 traffic_out /usr/local/sbin/snmp_ips/iptables_traffic_eval.sh out

But first, reload snmpd:

 root@bla:~# /etc/init.d/snmpd restart

root@bla:~# pgrep snmpd
30678

If it did not, examine the logs.

snmpwalk -On -v 2c -c public localhost .1.3.6.1.4.1.2021.49

snmpwalk -On -v 2c -c public localhost .1.3.6.1.4.1.2021.50

snmpwalk -On -v 2c -c public localhost .1.3.6.1.4.1.2021.51

(-On is just an output option and can be left out)

3. Cacti

We need to create a new one. However, we can just duplicate the existing Template called "Interface - Traffic".

I will call the new template "IP Traffic". Edit it and give it as data source name: "IP Traffic - |query_ipIP|".

Note: We could as well just use the Interface Traffic template, but just to leave it customizable I create a new template here.

We also need a new graph template. Again, duplicate one of the graphs called "Interface - Traffic (...)".

Also, edit it, give it the name "IP Traffic - |host_hostname| - |query_ipIP|" and for each of the ten items at the top of the screen, you will have to edit it and select the appropriate new data source (in this case "IP Traffic - (traffic_in)" for the top five and "IP Traffic - (traffic_out)" for the lower five.

Note: same as before

Note: now, this step really is required

Create a new data query. Name it "SNMP - IP Traffic", for instance. The data input method will be "Get SNMP data (indexed)". As you will notice, we need an XML Data Query file for cacti. We do it the Debian way[tm], so put the file, which I will call ips.xml, into /usr/local/share/cacti/resources/snmp_queries/ and give the file with its full path to cacti.

<interface>
        <name>Get SNMP IPs</name>
        <description>Queries a host for a list of IPs with traffic monitored by iptables</description>
        <oid_index>.1.3.6.1.4.1.2021.49.42.101</oid_index>
        <oid_num_indexes>.1.3.6.1.4.1.2021.49.42.100.1</oid_num_indexes>
        <index_order>ipIP:ipIndex</index_order>
        <index_order_type>numeric</index_order_type>
        <index_title_format>|chosen_order_field|</index_title_format>

        <fields>
                <ipIndex>
                        <name>Index</name>
                        <method>walk</method>
                        <source>value</source>
                        <direction>input</direction>
                        <oid>.1.3.6.1.4.1.2021.49.42.101</oid>
                </ipIndex>
                <ipIP>
                        <name>IP Address</name>
                        <method>walk</method>
                        <source>value</source>
                        <direction>input</direction>
                        <oid>.1.3.6.1.4.1.2021.49.101</oid>
                </ipIP>
                <ipInBytes>
                        <name>Incoming Traffic</name>
                        <method>walk</method>
                        <source>value</source>
                        <direction>output</direction>
                        <oid>.1.3.6.1.4.1.2021.50.101</oid>
                </ipInBytes>
                <ipOutBytes>
                        <name>Outgoing Traffic</name>
                        <method>walk</method>
                        <source>value</source>
                        <direction>output</direction>
                        <oid>.1.3.6.1.4.1.2021.51.101</oid>
                </ipOutBytes>
        </fields>
</interface>

Make sure to associate the data query with the graph template, on the Data Queries Edit screen (the list at the bottom) using "Add". Select ipInBytes as the data source for traffic_in and ipOutBytes for the traffic_out item, activating both checkboxes

Note: to troubleshoot data collection, you may want to look at the rrd files directly using rrdtool. Usually the files will be in /var/lib/cacti/rra/

Almost there... now, goto the Devices screen, by selecting "Devices" in the left hand menu. Create a new device, again you can just duplicate localhost, but set it to be a Generic SNMP-enabled Host. Now select the device from the devices list that you would like to use the new feature on, and add the Data Query "SNMP - IP Traffic" to its "Associated Data Queries".

If all is fine, you will see a status message like "Success [32 Items, 16 Rows]". In this case there are sixteen IP addresses with an input and output traffic value each.

Now it is so cool: goto "New Graphs", select (one of) the host(s) you just associated the data query to, and immediately you will see a list of IP addresses on the system in a list. Select those which you wish to monitor and click "create" at the bottom of the screen twice. Also add the newly created graphs in the Graph Trees screen. Done!