---Xjack.org--- --

Log Tool version 1.2.x

What is it?

Logtool is a command line program that will parse ASCII logfiles into a more palatable format. It will take anything resembling a syslog or multilog file (this includes syslog-ng, and probably most of the other variants out there), and crunch it into one of the following formats for your viewing pleasure:

  • ANSI (colorized for easy "at a glance" viewing)
  • ASCII (for e-mail'ed reports, and term's that don't support color)
  • CSV (for importing into your favorite spreadsheet/database)
  • HTML (for generating web pages)
  • RAW (for no good reason)

It can be configured to parse the data any one of several ways, including stripping the host, and/or program fields, and modifying the time display format of the log entry's.

It's additional features include support for config file based regular expressions to do everything from excluding/including certain log entry's (using both in conjunction you can create boolean searches), as well as color coding the ANSI and HTML output. And of course there's a config file so you can define your defaults once, and not have to remember those pesky command line switches.

New in 1.2.0:

Changes are WAY too much to list in total, but a few highlights:

  • Now speaks/converts TAI64 dates (IE: D.J.Bernstien's multilog and friends), so you can | a multilog file into logtool and get the same results you would expect from a syslog file.
  • Now can optionally suppress duplicate messages.
  • Now can support and colorize unknown logfile formats
  • Now can do really cool things with snort and iptables syslog entries (ip->hostname resolution, special colors, and more!)
  • Now has special options for a syslog-ng formated log.
  • Code architecture is VERY VERY VERY changed and better.
    • Light years better at error handling.
    • Nice facility to write more modules like snort and iptables.
    • Way more efficient (but not faster; too much new processing :)
  • And much much more!!! Try it, you'll like it!! (I hope :)

Sample output from the HTML module in 1.2.0, demonstrating the colorization for most of the modules is below (note: IP addresses/hostnames munged to protect the guilty):

Nov 17 17:29:12 minideb snort: WEB-IIS cmd.exe access [1:1002:5] [Web Application Attack] [Pri: 1]: {TCP} dummyxjack.org(192.168.1.9):3992 -> dummy.xjack.org(192.168.1.2):80
Nov 21 18:39:06 minideb snort: WARNING: Not IPv4 datagram! (snort_decoder) [116:1:1] {TCP} dummy.xjack.org(192.168.1.1):0 -> dummy.xjack.org(192.168.1.2):0
Nov 10 15:15:39 max snort: Portscan detected from 192.168.1.2: 6 targets 6 ports in 95 seconds (spp_portscan2) [117:1:1] {ICMP} dummy.xjack.org(192.168.1.2) -> dummy.xjack.org(192.168.1.7)
Nov 7 12:00:00 max snort: Possible RETRANSMISSION detection (spp_stream4) [111:3:1] {TCP} 192.168.9.10:49905 -> dummy.xjack.org(192.168.1.2):80
Mar 5 03:00:01 192.168.1.9 PAM_unix[16545]: (cron) session opened for user root by (uid=0)
Mar 5 03:00:01 192.168.1.9 logger: Hourly Check In!!!
Mar 5 02:48:09 192.168.1.9 iptables: -j LOG: Bad packet on pub int: {UDP} dummy.xjack.org(192.168.1.1):1028 -> dummy.xjack.org(192.168.1.2):137

Uses:

I expect this program to have primarily three uses (although you can use it for whatever you want, of course :)

  1. As an engine for use by shell scripts to generate nice little reports to e-mail people
  2. As a tool to use in conjunction with simple shell scripts for automated webpage generation.
  3. As a logfile monitoring tool for network operations center like environments, where it's nice to have logfile's scrolling across the screen in easy to read color. I've included a "redbeep" option so that in such cases, "red" events will generate a Ctrl-G to catch the attention of NOC personnel.

You can check out the documentation for more information on the things you can/can't do with logtool (and how to do them :).

Downloads:

You can download logtool by clicking one of the links below. It is HIGHLY recommended that you upgrade to 1.2.x if possible, though some ultra-conservative enviornments may choose to stay with the 1.0.x. branch.

Notes: as of version 1.0.5 GNU autoconf and friends are used to configure for build. Also note, that the 'logtail' program is no longer bundled with logtool. It is recommended that if you need such functionality that you donwnload retail, another of my projects, which duplicates the same functionality with (hopefully) better error handling. (Or of course, you may also continue to use logtail if you wish).

Version Notes: Versions 1.2.0 through 1.2.3 had several bugs, the worst of which were related to the handling of unformatted input. Version 1.2.4 addresses these, and a enough other non-trivial bugs, that upgrading to 1.2.4 is HIGHLY recommended for production environments. Version 1.2.5 introduces the escaping of <, >, and & for HTML output, and a few minor internal changes. Version 1.2.6 fixes the serious bug introduced with 1.2.5 regarding the escaping mentioned above.
Version 1.2.7 adds a patch to do diropen() if config/regex files happen to be directories instead of files, as well as fixes a Makefile issue so that $(MAKE) is called, which should ease building on systems which require use of gmake to get GNU functionality.

Public Beta: Versions 1.3.x are intended as a public beta of the next major release of logtool. It is hoped that if you have a bit of time to contribute, you can use these versions and provide feedback to the author, but it is NOT recommended that these versions be used in a mission-critical environment. This release is only for people who are willing to experience problems and report them to the author. Version 1.3.1 simply tracks the updates to 1.2.5. Otherwise it is identical to 1.3.0.

RedHat RPM's: RedHat recently made me angry enough with their new licensing and support policy that I have migrated to Debian for both my private use, and my corporate ifrastructure. As such, I no longer have reliable access to a current RedHat system to build RPM's on. However, you can type 'rpm -ta logtool-$VERSION.tar.gz' to build your own RPM's. If someone wants the job of maintaining binary and source RPM's, e-mail me.

Notes/Platforms:

This program _should_ compile/run on almost anything resembling UNIX. It was written in plain old C library function call's (no snprintf() or other things that are known to be pesky on some UNIX'es (Solaris anyone?)), and it compiles clean with -Wall -Werror passed to gcc, so I assume it is something like decently written code (though I'm a self-taught programmer, so all bets are off if the compiler lied to me :).

I have various reports of version 1.0.x compiling/running properly on the following:

  • Linux (all known distributions and architectures (including at least one embedded system variant)). The following have been verified by Wouter Verhelst: Linux/Alpha, Linux/ARM, Linux for HP PA-RISC, The Hurd on ia32, Linux/ia32, Linux/ia64, Linux/m68k Linux/MIPS (both on Big Endian MIPS-processors (mips) and Little Endian MIPS-processors (mipsel)) Linux/PPC, Linux for S/390 (which IBM now markets as zSeries) Linux/SH4 (more commonly known as the processor in Dreamcast game consoles) and last but not least Linux/SPARC.
  • OS X Server 10.2 on a PowerMac G4 dual 800 (must install devtools!).
  • HP-UX (B.10.10 A) on a 9000/712.
  • FreeBSD (see the ports section of the -current tree)
  • Solaris 2.6 with the original CC compiler (logtool-1.0.2 or newer).
  • SunOS 5.7 with gcc version 2.95.2 (logtool-1.0.2 or newer).
  • Solaris 8.0 with gcc 2.95.2 (logtool-1.0.2 or newer).
  • OpenBSD 2.8 and subsequent with gcc 2.95.3 (logtool-1.0.3 or newer).
  • SCO UnixWare 7.1.1 with gcc-2.95.2 (logtool-1.0.3 or newer).
  • AIX 4.3.3 with gcc-2.95.2 (logtool-1.0.4 or newer).
  • Windows NT 4.0 with the Cyg-GNU-Win32 toolkit, and one minor Makefile tweak.
  • IRIX
  • All UNIX platforms using the egcs/gcc compiler, a reasonable fascimilie of a proper ANSI C library, and something vaguely compatible with GNU make. (NOTE: this does not mean that all possible platforms have been tested; it means that all tested platforms with this configuration have worked fine, and that based on this, there should be no problems with other platforms using the same configuration).

It is assumed that 1.2.0 will compile on many if not all of the platforms listed above. anyone who runs into any issues, platform specific or otherwise, please contact the author so we can get the bugs ironed out.

Anyone who successfully compiles/installs Log Tool on a platform not mentioned above, please let me know, so I can include your platform in the list.

And of course, any feature requests/bug reports, e-mail me, and I'll see what I can do for you.

--A.L.Lambert

Sponsored by: ManISec Inc..

This page was last updated: 04/08/2010


( Copyright 2001-2002 A.L.Lambert <al@xjack.org>, all rights reserved. )