Configuration

Netflow-indexer uses a small configuration file that setups the type of indexer to use and the location of the files on disk. It has the following settings:

• indexer - the type of indexer to use. nfdump, nfdump_full, flowtools, or flowtools_full
• dbpath - the path to save the indexes to
• fileglob - the shell glob that will expand to the flow data files for the current hour
• allfileglob - the shell glob that will expand to all flow data files
• pathregex - a regular expression or simple string used to extract metadata from flow file paths.

Example Configuration files

nfdump using full indexing(recommended)

[nfi]
indexer = nfdump_full
dbpath = /data/nfdump_xap
flowpath = /data/nfsen/profiles/live/podium
fileglob = %(flowpath)s/nfcapd.%(year)s%(month)s%(day)s*
allfileglob = %(flowpath)s/nfcapd.*
pathregex = /profiles/:profile/:source/nfcapd

nfdump

[nfi]
indexer = nfdump
dbpath = /data/nfdump_xap
flowpath = /data/nfsen/profiles/live/podium
fileglob = %(flowpath)s/nfcapd.%(year)s%(month)s%(day)s*
allfileglob = %(flowpath)s/nfcapd.*
pathregex = /profiles/:profile/:source/nfcapd

flow-tools

[nfi]
indexer = flowtools
dbpath = /usr/local/var/db/flows/nfi
flowpath = /usr/local/var/db/flows/packeteer
fileglob= %(flowpath)s/%(year)s/%(year)s-%(month)s/%(year)s-%(month)s-%(day)s/ft-v05.%(year)s-%(month)s-%(day)s.%(hour)s*
allfileglob = %(flowpath)s/*/*/*/ft-v05.*

Cron

Netflow-indexer should be run from cron 5 minutes after every hour when using the nfdump indexer and every 5 minutes when using the nfdump_full indexer:

MAILTO=root
PATH=/usr/local/python_env/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
45 0 * * *   cd /data/nfdump_xap/ && ./daily_compact > /dev/null
*/5  * * * * sleep 30;netflow-index-update  /data/nfdump_xap/nfdump.ini
55 0 * * *   netflow-index-cleanup /data/nfdump_xap/nfdump.ini -d

Daily compaction

xapian allows you to compact an index for read-only use. Compaction yields disk usage and speed improvements. daily compaction is a work in progress

examples/daily_compact.sh

#!/bin/sh
DAY=`date +"%Y%m%d" -d "60 minutes ago"`

./xap_compact ${DAY}.db

examples/xap_compact.sh

#!/bin/sh
orig="$1"
tmp=tmp_$$.db
tmp2=tmp2_$$.db

if [ -e $orig/.compacted ] ; then
    exit 0
fi

xapian-compact -F $orig $tmp && mv $orig $tmp2 && mv $tmp $orig && rm -rf $tmp2 && touch $orig/.compacted