Lots of Logs

Linux/Debian:

This entry is one of those notes I talked about in my intro.

I was trying to forward my logs from a firewall that I setup using IPTables to a splunk server, and I ended up learning a good bit about the logging system. I'm sure I have not done things in the most efficient manner but for my purposes it worked well.

First I started logging some things using IPTables. I logged at the info level. As soon as I restarted the firewall the log messages started flooding my console. I logged in with putty and continued from there.

First to forward the logs I edited /etc/syslog.conf
I added:
kern.* @192.168.1.5 (Replace IPAddress with that of the server you are forwarding to)

This will forward all alerts coming from the kernal. You could be much more specific or do a *.* to forward EVERYTHING. See man syslog.conf for more information.

Next I restarted sysklogd:
/etc/init.d/sysklogd restart

That did it. The logs were forwarding just fine to the splunk server.

At this point I was getting all of my logs from the firewall showing up on my console, which made the console basically useless. First I checked my syslog.conf to see if there was an entry that was sending those logs to the console, but there weren't any. So I did some digging and found a few sites that talked about printk. It seems that printk controls what gets sent to your console along with any entries in syslog.conf that may point to your console.

So to fix it I typed:
echo "4 1 1 7" > /proc/sys/kernel/printk

That fixed the info level logs showing up on my console and to fix it for future reboots I edited /etc/sysctl.conf and added:
kernel.printk = 4 4 1 7

I then restarted sysklogd again:
/etc/init.d/sysklogd restart

A strange thing happened though. The alerts weren't showing up on the console or in the log files. I messed around for quite a while and finally fixed the problem by restarting klogd:
/etc/init.d/klogd restart

Update: It looks like depending on the version of klogd, and some other things I'm not yet aware of, when you restart klogd it might reset printk = 7 1 1 7 (Happened on one server but not the other). To fix this run "/etc/init.d/klogd stop" then run "/sbin/klogd -c 4" then run "/etc/init.d/klogd start"

That fixed things. Everything seems to work well now. Check out these web sites for further information and reference.

Information about changing printk
http://old.nabble.com/Suppressing-kernel-%27printk%27s.-td15280888.html

More information about printk
http://www.de-brauwer.be/wiki/wikka.php?wakka=printk

Great information about IPTables
http://www.faqs.org/docs/iptables