Taking control of system logs -- How to install Logger

By Mar Matthias Darin

This is a HOWTO for installing Logger, a Linux-based logging utility, and taking control of your system logs. System logs can be overwhelming and are most often overlooked; many users simply do not know what to do with them or how to maintain them. Log maintenance can be a daunting task, especially if Web or mail server applications are also involved in the maintenance schedule. My goal here is to provide a simple methodology that anyone running Linux can use.

The classic programs like syslogd and klogd are, at the very least, old and outdated for most modern system requirements. In extreme situations, they can be flat outright hostile. I wrote Logger to address these and other issues.

Let's begin by describing these classic programs and exactly what they do. klogd, the kernel logging daemon, is responsible for collecting all messages from the kernel and passing them to syslogd, the system logging daemon. One question comes to mind immediately: why two different processes?

This is a good question indeed. The current school of thought is that it is easier and more accurate to have a dedicated program for logging kernel messages. This does make sense - but at the same time is completely irrational, since klogd simply passes the messages to syslogd. To me, this seems like double the work and wasted resources. With that being said, let's have a look at syslogd.

As mentioned earlier, this is the system logging daemon. It is responsible for logging all message from programs like bind, wuftp, proftp, ident, telnet, most user space drivers, many antispam tools, and so on. A very diverse list of software relies on syslogd.

With such important software, one wants to ask, why re-invent the wheel or mess with something that isn't broken? The short and simple answer is there are many modern programs that can break (sys|k)logd - and every ounce of system performance really matters. Two-gigabyte file limits and sending log data as plain text are just a couple of very good reasons why you may want to replace these venerable programs.

I run a small server with seven domains. Each domain has its own Web, email, FTP, and domain name server along with the base kernel and system logs. That's six logs per machine for a total of 42 different logs I have to maintain on a daily basis. I did not include proxy, cache, email scanning logs, and so forth for simplicity - but these are typical to any basic server. With all this log data to be handled, where would I get time for my other administrator responsibilities, like answering my email?

There is one other issue I haven't touched on yet: excessive log writing can thrash even the most high-end servers, causing anything from massive slowdowns to down time. Excessive logging can also cause hard drives to heat up and wear out faster. I know this from personal experience. My logging drive was running at a whopping 125F before I started using Logger; now, it's at a comfortable 95F and I have no more server crashes caused by overheating. Laptops and desktops are also affected, though on a smaller scale.

This is the very reason why I wrote Logger: to take control of this mess and save my hardware from such thrashing as well as heat problems. So, with all that being said, let's move on to the installation of Logger.

The Logger package must be installed on the hard drive. It can be downloaded from here. I recommend putting the Logger archive in the '/usr/src' area for convenience and security (remember, writing to this directory requires root permissions.)

[ There are a number of schemes for installing software that is not a part of the distribution tree; generally, these involve using some flavor of '/usr/local' or '/opt', which are designated for those purposes. There are also programs such as stow, which are designed to support easy installation and removal of such "alien" programs; installing it and learning to use it may be a worthwhile investment of time if you're going to do this on a regular basis.

The question of whether to install software that requires root access and does not come from the standard distribution tree should also be considered at length. If you are a programmer, are expert in the language used to write the program, and are certain that it is neither malicious nor contains any vulnerabilities - or can have the above sort of verification from someone you trust implicitly - then you may be in the clear. If the above is not the case, you are, in my opinion, taking an unwarranted risk. -- Ben ]

The Logger tarball should be unpacked into a directory called 'Logger' and the path should be changed to that folder:

             mkdir /usr/src/Logger
             mv Logger.tgz /usr/src/Logger
             cd /usr/src/Logger
             tar xvzf Logger.tgz

Logger must then be compiled by typing:

             ./COMPILE

[ The COMPILE script contains flags for the C compiler affecting the code generation. The script assumes that it will run on IA32 systems and uses the "-march=i686 -mtune=i686" flags. This won't work on x86_64 (AMD64/EM64T) systems. Use "-mtune=generic" or pick suitable flags from the Gentoo Safe CFLAGS Wiki. -- René ]

If no errors occurred, then the files need to be installed:

             mv Logger /sbin
             mv LogPipe /usr/local/bin

Now, it's time to write the Logger configuration file. This is simply a flat text file called 'Logger.conf' and it's saved in the '/etc' directory. The instruction set is rather simple and easy to use; here is a very basic 'Logger.conf' file that will work with any machine (an example is included in the Logger tarball):

          1. ###
          2. ### Logger configuration file
          3. ###
          4.
          5. ### Queues needed
          6.
          7. Queue Klog       0 0 0600 1 /var/log/Klog.log
          8. Queue Slog       0 0 0600 25 /var/log/Syslog.log
          9.
          10.### Sys/KLog Entries
          11.
         12.Kernel MyCPU Klog
         13.SysLog MyCPU Slog

Lines 1-3, 5, and 10 are comments. Any line that starts with a pound sign (#) is a comment and is ignored by Logger. Blank Lines (4, 6, 9, and 11) are also ignored. The first actual command to Logger is line 7.

The first command Logger encounters is Queue. Here is a detailed breakdown of line 7:

1. Queue lines must always start with the word Queue. To Logger, a Queue is a section of memory that is used to keep all the log data before it is written to disk. These are created and maintained by Logger automatically when Logger sees the Queue command.
2. This is the Queue Name or Identity and must be unique to each Queue. This is how Logger knows where to put the log data from a specific log source.
3. Owner - this is the numeric UID of the owner and can be gotten from the Linux 'stat' command or '/etc/passwd'. For example, root has a numeric value of 0.
4. Group - this is the numeric value of the group and can be gotten from the Linux stat command. For example, root has a numeric value of 0.
5. Permissions, leading ZERO (0) must be present - this sets the access right to the file and can be gotten from the Linux stat command. For example, owner access only has a numeric value of 0600.
6. This is the number of lines of log data you want to Queue to hold before the information gets saved. The minimum is one line. There is no maximum limit, but the number should be practical.
7. Prefix to entry (* = no prefix).
8. Output File(s)/TCP Connections (Logs can be sent via TCP only as well, just omit the file).

With the above example as a starting point, we'll now add a laptop. Add the following two lines to the 'Logger.conf' on the desktop (the above example):

         1. TCP 3900 * Klog
         2. TCP 3901 * Slog

Logger will now need to be restarted. The above two lines tell Logger to set up TCP servers on ports 3900 and 3901 whereby kernel and system data will be received accordingly. It should be noted that any number of computers can now use this log server. Be sure to set your firewall rules appropriately.

Now for the 'Logger.conf' settings for the laptop (remember that Logger has to be installed first):

         1.  ###
         2.  ### Logger configuration file
         3.  ###
         4.
         5.  ### ALL ENTRIES PER LINE ARE REQUIRED.
         6.
         7.  ### Output Queues - MUST be listed first
         8.
         9.  Queue Klog       0 0 0600 1 @10.0.100.0:3900
         10. Queue Slog      0 0 0600 25 @10.0.100.0:3901
         11. 
         12. ### Sys/KLog Entries
         13. 
         14. Kernel Laptop Klog
         15. SysLog Laptop Slog

The 10.0.100.0 is an example used to identify the IP addresses of the desktop computer. It needs to be the actual desktop computer IP address. Also, notice that the ports in the desktop example match the order and sequence on the laptop computer.

Below is a slightly more complex example:

For this example, the above diagram will be the basis on which a small network will be established with these assumptions:

         1. Alpha, Beta, Delta, Gamma, and Omega are servers.
         2. IP address will be assumed as follows:
            (a) Alpha - 10.0.100.0
            (b) Beta - 10.0.100.1
            (c) Delta - 10.0.100.2
            (d) Gamma - 10.0.100.3
            (e) Omega - 10.0.200.0
         3. The server layout will be as follows:
            (a) Alpha, Beta, and Delta are Web servers running Apache.
            (b) Delta is an email server.
            (c) Omega is the dedicated log server.
         4. All servers have Logger installed.
         5. Apache and Exim have been configured as follows:
            (a) Apache will save log data to:
                i. /tmp/logger/ApacheLog for general log data
                ii. /tmp/logger/ApacheErr for log errors
            (b) Exim will save log data to:
                i. /tmp/logger/EximMain
                ii. /tmp/logger/EximReject
                iii. /tmp/logger/EximPanic

With the above layout, here are the various 'Logger.conf' files for each machine:

For Alpha, Beta, and Delta:

            ###
            ### Logger configuration file for Alpha, Beta, and Delta
            ###
            ### The prefix will need to change for each machine.

            ### Output Queues - MUST be listed first

            Queue   Klog            0   0   0600   1      @10.0.200.0:3900
            Queue   Slog            0   0   0600   25     @10.0.200.0:3901
            Queue   ApacheLog       0   0   0644   1000   @10.0.200.0:3902
            Queue   ApacheErr       0   0   0644   1      @10.0.200.0:3903

            ### Sys/KLog Entries

            Kernel Alpha Klog
            SysLog Alpha Slog

            ### Apache entries

            Pipe /tmp/logger/ApacheLog Alpha ApacheLog
            Pipe /tmp/logger/ApacheErr Alpha ApacheErr

Notice that Exim is not listed here at all and doesn't need to be.

For Gamma:

            ###
            ### Logger configuration file for Gamma
            ###

            ### Output Queues - MUST be listed first

            Queue Klog              0 0 0600 1            @10.0.200.0:3900
            Queue Slog              0 0 0600 25           @10.0.200.0:3901
            Queue EximLog           0 0 0640 500          @10.0.200.0:3904

            ### Sys/KLog Entries

            Kernel Gamma Klog
            SysLog Gamma Slog

            ### Exim Entries

            Pipe /tmp/logger/EximMain   Main   EximLog
            Pipe /tmp/logger/EximPanic Panic EximLog
            Pipe /tmp/logger/EximReject Reject EximLog

Notice that Apache is not listed here and doesn't need to be. Also notice that the three Exim logs are consolidated into one single file. Each Exim file can be saved separately by simply providing a Queue for each.

Finally, Omega:

            ###
            ### Logger configuration file for Omega
            ###

            ### Output Queues - MUST be listed first

            Queue   Klog         0   0   0600   1      /var/log/Klog
            Queue   Slog         0   0   0600   25     /var/log/SysLog
            Queue   ApacheLog    0   0   0644   1000   /var/log/ApacheLog
            Queue   ApacheErr    0   0   0644   1      /var/log/ApacheErr
            Queue   EximLog      0   0   0640   500    /var/log/EximLog

            ### Sys/KLog Entries

            ### Local Logs

            Kernel Omega Klog
            SysLog Omega Slog

            ### Logs from the other machines

            TCP 3900 * Klog
            TCP 3901 * Slog

            ### Apache entries

            TCP 3902 * ApacheLog
            TCP 3903 * ApacheErr

            ### Exim Entries

            TCP 3904 * EximLog

Restart Logger, Exim, and Apache on each of the machines, and all log data will be written to the Omega machine. An important point to note: Logger must be started first on Omega, then the other machines. Restart applications, such as Apache and Exim, last.

The examples above are just the beginning of where Logger can be used. Applications/Servers like Apache, Exim, Sendmail, Postfix, Maildrop, Squid, Snort, and such (including any piece of Linux software that writes log data in plain text) can benefit from Logger. Implementations can range from a simple single computer to a large multi-layered corporation consolidating all log data to a central security point. Logger's versatility and flexibility can dramatically open the doors to new possibilities and performance improvements on any system.

Talkback: Discuss this article with The Answer Gang

---

I am a 28 year veteran programmer and a Bishop. My specialities in Computer Sciences are database and security. I have been a professional college level instructor for Computer Programming and Office Administration. I have extensive training in learning disabilities.