LogAnalysis
[logs] naming multiple output files with syslog-ng Dec 17 2007 02:20PM
Christian Folini (christian folini post ch) (2 replies)
RE: [logs] naming multiple output files with syslog-ng Dec 20 2007 03:55PM
Jan Monsch (jan monsch csnc ch) (1 replies)
RE: [logs] naming multiple output files with syslog-ng Dec 20 2007 04:25PM
Marcus J. Ranum (mjr ranum com) (3 replies)
Re: [logs] naming multiple output files with syslog-ng Dec 27 2007 01:14AM
Mordechai T. Abzug (morty frakir org) (1 replies)
Re: [logs] naming multiple output files with syslog-ng Jan 01 2008 01:05PM
Chris Brenton (cbrenton chrisbrenton org)
RE: [logs] naming multiple output files with syslog-ng Dec 20 2007 06:22PM
Paul Melson (pmelson gmail com) (1 replies)
Re: [logs] naming multiple output files with syslog-ng Dec 21 2007 06:55AM
Christian Folini (christian folini post ch) (2 replies)
Re: [logs] naming multiple output files with syslog-ng Dec 21 2007 04:16PM
Chris Wee (chris wee loglogic com) (1 replies)
Re: [logs] naming multiple output files with syslog-ng Dec 24 2007 10:49AM
Christian Folini (christian folini post ch)
RE: [logs] naming multiple output files with syslog-ng Dec 21 2007 10:19AM
Jan Monsch (jan monsch csnc ch) (1 replies)
RE: [logs] naming multiple output files with syslog-ng Dec 23 2007 08:06PM
Marcus J. Ranum (mjr ranum com)
Re: [logs] naming multiple output files with syslog-ng Dec 20 2007 05:23PM
Bill Burge (bill burge com)
Re: [logs] naming multiple output files with syslog-ng Dec 18 2007 06:22AM
Tom Le (dottom gmail com)
On Dec 17, 2007 6:20 AM, Christian Folini <christian.folini (at) post (dot) ch [email concealed]> wrote:
> I managed to get quite close by configuring ErrorLog
> in apache as follows:
> ErrorLog "| /usr/bin/logger -t www.example.com_port80 -u
/tmp/logger.socket"
> ErrorLog "| /usr/bin/logger -t www.example.com_port443 -u
/tmp/logger.socket"
>
> And then configure syslog-ng along the lines of
> destination df_file { file(".../$PROGRAM/$HOST/apache-error.log"); };

Logger is really expensive used in this way unless your log files are
relatively small. Can you use native syslog forwarding capability by
defining a custom facility/priority in Apache config? (Also, technically,
the service name should be 32 or fewer characters per RFC3164... so it's
possible using this format could violate that for long hostnames.)

You may want to consider not using the service name in this way as it can
limit your ability to do other types of analysis later using tools that rely
on the service name.
If you want to perform any customized parsing of the error logs, you can
pipe to Perl (it's not as bad as people perceive) or build a small compiled
application to do the custom formatting for you. You don't have to build it
-- say a C application -- from scratch (though not that bad given a small
scope). You can use the traditional Lex/Yac/Bison approach and just add
syslog forwarding (lots of source code floating around for syslog
forwarding).

There is also Snare for Apache (
http://www.intersectalliance.com/projects/SnareApache/index.html) which you
can modify source code for your custom format and add parsing rules. The
Snare Text agent is pretty straightforward and you can use that as an idea
for how to build your own parser + forwarder.

Finally, I'd add one final question to consider. How critical are your
logs? Most centralized syslog logging solutions, even those using TCP only
delivery, endure some small amount of message loss (for a variety of
differnet reasons). So does your logging have to ensure delivery with
queueing/store & forward mechanisms?
<p>On Dec 17, 2007 6:20 AM, Christian Folini <<a href="mailto:christian.folini (at) post (dot) ch [email concealed]">christian.folini (at) post (dot) ch [email concealed]</a>> wrote:<br>> I managed to get quite close by configuring ErrorLog<br>> in apache as follows:<br>
> ErrorLog "| /usr/bin/logger -t <a href="http://www.example.com_port80/">www.example.com_port80</a> -u /tmp/logger.socket"<br>> ErrorLog "| /usr/bin/logger -t <a href="http://www.example.com_port443/">
www.example.com_port443</a> -u /tmp/logger.socket"<br>><br>> And then configure syslog-ng along the lines of<br>> destination df_file { file(".../$PROGRAM/$HOST/apache-error.log"); };</p>
<p>Logger is really expensive used in this way unless your log files are relatively small.  Can you use native syslog forwarding capability by defining a custom facility/priority in Apache config?  (Also, technically, the service name should be 32 or fewer characters per RFC3164... so it's possible using this format could violate that for long hostnames.)
</p>
<p>You may want to consider not using the service name in this way as it can limit your ability to do other types of analysis later using tools that rely on the service name.</p>
<div>If you want to perform any customized parsing of the error logs, you can pipe to Perl (it's not as bad as people perceive) or build a small compiled application to do the custom formatting for you.  You don't have to build it -- say a C application -- from scratch (though not that bad given a small scope).  You can use the traditional Lex/Yac/Bison approach and just add syslog forwarding (lots of source code floating around for syslog forwarding). 
</div>
<div> </div>
<div>There is also Snare for Apache (<a href="http://www.intersectalliance.com/projects/SnareApache/index.html">
http://www.intersectalliance.com/projects/SnareApache/index.html</a>) which you can modify source code for your custom format and add parsing rules.  The Snare Text agent is pretty straightforward and you can use that as an idea for how to build your own parser + forwarder. 
</div>
<div> </div>
<div>Finally, I'd add one final question to consider.  How critical are your logs?  Most centralized syslog logging solutions, even those using TCP only delivery, endure some small amount of message loss (for a variety of differnet reasons).  So does your logging have to ensure delivery with queueing/store & forward mechanisms? 
</div>
_______________________________________________
LogAnalysis mailing list
LogAnalysis (at) loganalysis (dot) org [email concealed]
http://www.loganalysis.org/mailman/listinfo/loganalysis

[ reply ]


 

Privacy Statement
Copyright 2010, SecurityFocus