LogAnalysis
[logs] naming multiple output files with syslog-ng Dec 17 2007 02:20PM
Christian Folini (christian folini post ch) (2 replies)
RE: [logs] naming multiple output files with syslog-ng Dec 20 2007 03:55PM
Jan Monsch (jan monsch csnc ch) (1 replies)
RE: [logs] naming multiple output files with syslog-ng Dec 20 2007 04:25PM
Marcus J. Ranum (mjr ranum com) (3 replies)
Re: [logs] naming multiple output files with syslog-ng Dec 27 2007 01:14AM
Mordechai T. Abzug (morty frakir org) (1 replies)
Re: [logs] naming multiple output files with syslog-ng Jan 01 2008 01:05PM
Chris Brenton (cbrenton chrisbrenton org)
RE: [logs] naming multiple output files with syslog-ng Dec 20 2007 06:22PM
Paul Melson (pmelson gmail com) (1 replies)
Re: [logs] naming multiple output files with syslog-ng Dec 21 2007 06:55AM
Christian Folini (christian folini post ch) (2 replies)
Hey guys,

I thought my configuration question was almost innocent and now we
are talking about the fundamentals.

Well, I do not like the additional logger binary running next to apache
in my setup. Apache supports writing the error log to syslog directly,
but that's not possible with the 15 other kinds of logfiles
my project is dealing with. For reasons of consistency, we will
therefore stick with a pipe.

Now as for a performance overhead, this reminds me of a University
lecture. The professor talked about storage and performance.
He grouped storage systems in 5 categories, grouped by their
relative distance to the CPU: cache, ram, disk, net, tape.
(Maybe there were more, but that's the ones I remember.)
The main difference between them is the access time, according to his
lecture.
He recommended everyone should try to use the storage close
to the CPU for performance reasons. Unless you really have to use
a different storage. But once you are accessing storage further away
from the CPU, all access to storage closer to the CPU becomes
practically irrelevant.

For a webserver this means, that the latency of network traffic makes
disk access for logging almost non-existing. Besides Marcus pointed out,
that Apache is logging after having dealt with the client's request.

To be more blunt: The bottleneck here are the majority of users, that
access our webservers with slow modems. It's not the additional pipe
that sits between the server and the logging facility.

I did some tests to be on safe grounds, when I started our project. The
actual numbers back my assumptions laid out above. This is especially
true for SSL, as the handshake makes this "storage access" even
more expensive. A couple of thousand times more expensive than
the logging process, no matter how much logfiles I would write.

Binary formats and csv logging. A great deal faster and a lot more
storage efficient. However, this is not always a priority.
My priority is, that the logfiles I am writing today be readable
tomorrow. Possibly with standard tools. I am quite sure that any
binary format would be the wrong format tomorrow. If these guys are
smart, they will be able to get the data out of the DB again, tomorrow.
But it's likely to cause headaches and meetings and stuff.
Ascii logging is the safe path; and we can always import the data
into a db later on or transform them into csv format. Actually I
did some work on that lately. All that is bugging me is the time it
takes to parse the ascii logfiles. It takes too long for my
taste, but then it's also a lot of logfiles. ;)

In case you are still following my long post: I am still looking for
an answer for a syslog-ng configuration.

On the central host, I want to get a filetree as follows:
.../apache/<virtualhost>/<physicalhost>/port<port>-error.log

And just can't make up my mind how to achieve this in an elegant
way.
ErrorLog "| /usr/bin/logger -t www.example.com_port443 -u /tmp/logger.socket"
and
destination df_file {
file(".../$PROGRAM/$HOST/apache-error.log"); };
get me the following:
.../apache/www.example.com_port80/h00341/apache-error.log

That's close. But really what I wanted. More thoughts?

regs,

Christian

P.S. Marcus: Cool to read your message in this thread. I just
wrote about artificial ignorance in one of the conceptual papers
for this project. :)
_______________________________________________
LogAnalysis mailing list
LogAnalysis (at) loganalysis (dot) org [email concealed]
http://www.loganalysis.org/mailman/listinfo/loganalysis

[ reply ]
Re: [logs] naming multiple output files with syslog-ng Dec 21 2007 04:16PM
Chris Wee (chris wee loglogic com) (1 replies)
Re: [logs] naming multiple output files with syslog-ng Dec 24 2007 10:49AM
Christian Folini (christian folini post ch)
RE: [logs] naming multiple output files with syslog-ng Dec 21 2007 10:19AM
Jan Monsch (jan monsch csnc ch) (1 replies)
RE: [logs] naming multiple output files with syslog-ng Dec 23 2007 08:06PM
Marcus J. Ranum (mjr ranum com)
Re: [logs] naming multiple output files with syslog-ng Dec 20 2007 05:23PM
Bill Burge (bill burge com)
Re: [logs] naming multiple output files with syslog-ng Dec 18 2007 06:22AM
Tom Le (dottom gmail com)


 

Privacy Statement
Copyright 2010, SecurityFocus