[logs] High performance syslog aggregation Nov 30 2007 08:36PM
Steve Bernacki (loganalysis f copacetic net)
First of all, thank you to those who responded to my last message
regarding syslog load balancing.

I'm currently researching how to best implement a high-peformance, high
volume syslog aggregation. In our current environment, we have many
devices logging to a small set of "front end" syslog aggregators which
run syslog-ng. Currently, these front-end aggregators have a number of
filters enabled, which negatively impacts thruput. What I'm looking to
do is place new systems in front of the existing systems that simply
capture, queue, and forward messages based on a very limit set of
searchable criteria (no regexes needed!). These systems should also
have the ability to queue incoming messages onto disk and replay them in
the event that a receiver goes down or becomes temporarily overburdened.

My first thought was to implement an architecture similar the following:

Hosts --(UDP)--> (front end) --(TCP)-->(multiple receivers)

In researching my "free" and "nearly free" options for doing this,
syslog-ng community edition comes the closest, however only the
commercial version supports "store and forward" for TCP syslog streams.
rsyslog looks like a promising alternative option, although I haven't
been able to confirm through its documentation whether it supports any
type of "store and forward" mechanism.

What other tools and/or solutions have I missed? I've considered using
syslog-ng to log to a program which ultimately stores and forwards
messages, although it seems like there must be a better way of doing

Thanks once again for your guidance,
Steve Bernacki, Jr
To date, the Pan-Massachusetts Challenge has raised 204 million
dollars for cancer research. Get involved! http://www.pmc.org/
LogAnalysis mailing list
LogAnalysis (at) loganalysis (dot) org [email concealed]

[ reply ]


Privacy Statement
Copyright 2010, SecurityFocus