LogAnalysis
[logs] naming multiple output files with syslog-ng Dec 17 2007 02:20PM
Christian Folini (christian folini post ch) (2 replies)
RE: [logs] naming multiple output files with syslog-ng Dec 20 2007 03:55PM
Jan Monsch (jan monsch csnc ch) (1 replies)
Hi Christian

I also thought about using the pipe to an executable in Apache log
configuration. But I personally do not like the solution, because the
logger gets invoked by the Apache process and this most likely produces
overhead in the Apache process.

On the net I found an interesting solution by using FIFO files under
Unix. This allows seperating the Apache log writing process from the
process which handles post-processing of the log file. The FIFO files
behave like a queue where a process can read from the top of the queue
while the writing process adds stuff to the end of the queue. FIFOs are
created using the Unix command mkfifo.

This concept allows you to have more complex post-processing without
interfering with the Apache web server. I use this to convert the log
files to CSV format which can then be efficently imported into a
database using the database loader.

Merry Chistmas
Jan

-----Original Message-----
From: loganalysis-bounces (at) loganalysis (dot) org [email concealed]
[mailto:loganalysis-bounces (at) loganalysis (dot) org [email concealed]] On Behalf Of Christian
Folini
Sent: Montag, 17. Dezember 2007 15:21
To: loganalysis (at) loganalysis (dot) org [email concealed]
Subject: [logs] naming multiple output files with syslog-ng

Hello everyone,

I am new to this list, after having visited the loganalysis website many
times in the last few weeks. I am working on a fairly big logfile
centralisation project. We are evaluating syslog-ng and I am trying to
configure it to meet our present standards. It basically works, but I
have not yet found an elegant solution to get the files to the right
destination. So I thought maybe you guys have a hint or two for me.

We have a central loghost, but so far the centralisation is not
concurrent. The logfiles I am talking about are apache error logs for a
start (and then ModSecurity audit logs, guardian logs, apache access
logs, cgi-logs, you name it.)

Now there are n hosts with p apache servers serving p virtual domains.
Most apaches serve http and https. We are used to keeping seperate error
logfiles for apache-server, virtualhost-port80 and virtualhost-port 443.

On the central host, they reside in a hierarchy as follows:
.../apache/<virtualhost>/<physicalhost>/port<port>-error.log

->

...
.../apache/www.example.com/h00341/port80-error.log
../apache/www.example.com/h00341/port443-error.log
.../apache/www.example.com/h00445/port80-error.log
.../apache/www.example.com/h00445/port443-error.log
...

I managed to get quite close by configuring ErrorLog in apache as
follows:
ErrorLog "| /usr/bin/logger -t www.example.com_port80 -u
/tmp/logger.socket"
ErrorLog "| /usr/bin/logger -t www.example.com_port443 -u
/tmp/logger.socket"

And then configure syslog-ng along the lines of destination df_file {
file(".../$PROGRAM/$HOST/apache-error.log"); };

This brings me files along the lines of:
.../apache/www.example.com_port80/h00341/apache-error.log
.../apache/www.example.com_port443/h00341/apache-error.log

That's close, but not quite.

I could use a program(regex) filter, but this is performance relevant
and highly unwelcome, and it would still not help me to get rid of the
"_port<portnum>" in the program variable.

I found one or two alternatives, but they are equally inelegant and I
would rather not use them.

To sum it up: I do have a working solution, but it's not good looking.

Any thoughts?

regs,

Christian Folini

_______________________________________________
LogAnalysis mailing list
LogAnalysis (at) loganalysis (dot) org [email concealed]
http://www.loganalysis.org/mailman/listinfo/loganalysis

_______________________________________________
LogAnalysis mailing list
LogAnalysis (at) loganalysis (dot) org [email concealed]
http://www.loganalysis.org/mailman/listinfo/loganalysis

[ reply ]
RE: [logs] naming multiple output files with syslog-ng Dec 20 2007 04:25PM
Marcus J. Ranum (mjr ranum com) (3 replies)
Re: [logs] naming multiple output files with syslog-ng Dec 27 2007 01:14AM
Mordechai T. Abzug (morty frakir org) (1 replies)
Re: [logs] naming multiple output files with syslog-ng Jan 01 2008 01:05PM
Chris Brenton (cbrenton chrisbrenton org)
RE: [logs] naming multiple output files with syslog-ng Dec 20 2007 06:22PM
Paul Melson (pmelson gmail com) (1 replies)
Re: [logs] naming multiple output files with syslog-ng Dec 21 2007 06:55AM
Christian Folini (christian folini post ch) (2 replies)
Re: [logs] naming multiple output files with syslog-ng Dec 21 2007 04:16PM
Chris Wee (chris wee loglogic com) (1 replies)
Re: [logs] naming multiple output files with syslog-ng Dec 24 2007 10:49AM
Christian Folini (christian folini post ch)
RE: [logs] naming multiple output files with syslog-ng Dec 21 2007 10:19AM
Jan Monsch (jan monsch csnc ch) (1 replies)
RE: [logs] naming multiple output files with syslog-ng Dec 23 2007 08:06PM
Marcus J. Ranum (mjr ranum com)
Re: [logs] naming multiple output files with syslog-ng Dec 20 2007 05:23PM
Bill Burge (bill burge com)
Re: [logs] naming multiple output files with syslog-ng Dec 18 2007 06:22AM
Tom Le (dottom gmail com)


 

Privacy Statement
Copyright 2010, SecurityFocus