On Tue, 2007-12-04 at 21:19 -0800, Tom Le wrote:
> On Nov 30, 2007 12:36 PM, Steve Bernacki <loganalysis (at) f.copacetic (dot) net [email concealed]>
> wrote:
> > My first thought was to implement an architecture similar the
> following:
> >
> > Hosts --(UDP)--> (front end) --(TCP)-->(multiple receivers)
> >
> > In researching my "free" and "nearly free" options for doing this,
> > syslog-ng community edition comes the closest, however only the
> > commercial version supports "store and forward" for TCP syslog
> streams.
> > rsyslog looks like a promising alternative option, although I
> haven't
> > been able to confirm through its documentation whether it supports
> any
> > type of "store and forward" mechanism.
>
> Have you considered building a Perl-based read & forward application?
>
> The front-end servers just write syslog to a file (which you probably
> are already doing anyways). The Perl application lives on each
> front-end server and does a "stateful tail" that handles log rotation
> (or use Marcus's retail and pipe to Perl if you don't want to build
> your own). The Perl app sends tcp or udp syslog to the "multiple
> receivers". Make sure syslog-ng front-end servers is configured to
> capture the originating IP address. You can add custom forwarding
> rules based on whatever you need: source IP, facility/priority,
> service name, or even regex-based identification of specific messages
> types.
>
> In other words, you use the native syslog functionality of writing to
> a file as your "store" mechanism, and then you just intelligently read
> and forward. You can forward in real-time or at whatever desired
> interval. You can even add code for throttling, depending on how fast
> your multiple receivers can receive syslog message.
>
> The beauty of this method is you can also perform replays whenever
> needed (i.e. say you are testing a SIM or SIEM tool) and at whatever
> speeds desired.
>
> Shouldn't take anymore than a few hours to build for a typical Perl
> developer.

Please note that I'm affiliated with BalaBit, so my view on syslog-ng is
biased. Don't read this if you don't want to read product specific
details.

What syslog-ng does here is a bit more sophisticated: disk-buffering is
not a standard logfile that you have to poll in order to send to the
target server.

Polling is inherently slow, and introduces latencies (how often do you
poll the log-file).

In syslog-ng a message only touches the spool file if the destination is
not fast enough, most of the messages are forwarded without ever
touching the disk.

A relay may not have the disk capacity to store all messages locally,
the spool file is a circular buffer, the maximum size can be specified.
Whenever the spool file would overflow, flow-control kicks in and slows
down the sending application to avoid draining the central log
collector.

So if either performance, latency or disk capacity is an issue the
solution in syslog-ng is way superior. And disk buffering works in
tandem with any of the well known syslog-ng destinations: tcp(), pipe(),
program(), sql(), each with an independent spool file.

More information:

http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/ch07s04.htm
l

--
Bazsi
_______________________________________________
LogAnalysis mailing list
LogAnalysis (at) loganalysis (dot) org [email concealed]
http://www.loganalysis.org/mailman/listinfo/loganalysis

[ reply ]