LogAnalysis
[logs] naming multiple output files with syslog-ng Dec 17 2007 02:20PM
Christian Folini (christian folini post ch) (2 replies)
RE: [logs] naming multiple output files with syslog-ng Dec 20 2007 03:55PM
Jan Monsch (jan monsch csnc ch) (1 replies)
RE: [logs] naming multiple output files with syslog-ng Dec 20 2007 04:25PM
Marcus J. Ranum (mjr ranum com) (3 replies)
Re: [logs] naming multiple output files with syslog-ng Dec 27 2007 01:14AM
Mordechai T. Abzug (morty frakir org) (1 replies)
Re: [logs] naming multiple output files with syslog-ng Jan 01 2008 01:05PM
Chris Brenton (cbrenton chrisbrenton org)
RE: [logs] naming multiple output files with syslog-ng Dec 20 2007 06:22PM
Paul Melson (pmelson gmail com) (1 replies)
Re: [logs] naming multiple output files with syslog-ng Dec 21 2007 06:55AM
Christian Folini (christian folini post ch) (2 replies)
Re: [logs] naming multiple output files with syslog-ng Dec 21 2007 04:16PM
Chris Wee (chris wee loglogic com) (1 replies)
Christian,

On a previous project in 2003, we used Mysql4 MYISAM tables as the binary
logging format. It stood (and still stands) a
good chance of being readable up to 2010. The overhead of the DB engine was
neglible w.r.t. to the volume of the log (high).

Today, I would use sqlite3 because it is endian-neutral, and I think given
the support it is receiving, it has a good chance
of being supported for 5-10 years. Your ascii-parser is written in some
language (perl, python, ruby, java) and that has
to be ported to 128-bit architectures in the future which is about the same
chance as sqlite3/4/5...

Note, the performance gains we saw from binary versus ascii was due to the
nature of the data -- non-string data that required
conversion from text <-> binary representations. Your mileage may vary.

-chris

On Dec 20, 2007 10:55 PM, Christian Folini <christian.folini (at) post (dot) ch [email concealed]> wrote:

> <snip>
>
> Binary formats and csv logging. A great deal faster and a lot more
> storage efficient. However, this is not always a priority.
> My priority is, that the logfiles I am writing today be readable
> tomorrow. Possibly with standard tools. I am quite sure that any
> binary format would be the wrong format tomorrow. If these guys are
> smart, they will be able to get the data out of the DB again, tomorrow.
> But it's likely to cause headaches and meetings and stuff.
> Ascii logging is the safe path; and we can always import the data
> into a db later on or transform them into csv format. Actually I
> did some work on that lately. All that is bugging me is the time it
> takes to parse the ascii logfiles. It takes too long for my
> taste, but then it's also a lot of logfiles. ;)
> <snip>
> Christian
>
> P.S. Marcus: Cool to read your message in this thread. I just
> wrote about artificial ignorance in one of the conceptual papers
> for this project. :)
> _______________________________________________
> LogAnalysis mailing list
> LogAnalysis (at) loganalysis (dot) org [email concealed]
> http://www.loganalysis.org/mailman/listinfo/loganalysis
>
Christian,<br><br>On a previous project in 2003, we used Mysql4 MYISAM tables as the binary logging format. It stood (and still stands) a<br>good chance of being readable up to 2010. The overhead of the DB engine was neglible
w.r.t. to the volume of the log (high).<br><br>Today, I would use sqlite3 because it is endian-neutral, and I think given the support it is receiving, it has a good chance<br>of being supported for 5-10 years. Your ascii-parser is written in some language (perl, python, ruby, java) and that has
<br>to be ported to 128-bit architectures in the future which is about the same chance as sqlite3/4/5...<br><br>Note, the performance gains we saw from binary versus ascii was due to the nature of the data -- non-string data that required
<br>conversion from text <-> binary representations. Your mileage may vary.<br><br>-chris<br><br><div class="gmail_quote">On Dec 20, 2007 10:55 PM, Christian Folini <<a href="mailto:christian.folini (at) post (dot) ch [email concealed]">christian.folini (at) post (dot) ch [email concealed]
</a>> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><snip><br><br>Binary formats and csv logging. A great deal faster and a lot more
<br>storage efficient. However, this is not always a priority.<br>My priority is, that the logfiles I am writing today be readable<br>tomorrow. Possibly with standard tools. I am quite sure that any<br>binary format would be the wrong format tomorrow. If these guys are
<br>smart, they will be able to get the data out of the DB again, tomorrow.<br>But it's likely to cause headaches and meetings and stuff.<br>Ascii logging is the safe path; and we can always import the data<br>into a db later on or transform them into csv format. Actually I
<br>did some work on that lately. All that is bugging me is the time it<br>takes to parse the ascii logfiles. It takes too long for my<br>taste, but then it's also a lot of logfiles. ;)<br> <snip><font color="#888888">
<br>Christian<br></font><br>P.S. Marcus: Cool to read your message in this thread. I just<br>wrote about artificial ignorance in one of the conceptual papers<br>for this project. :)<br><div><div></div><div class="Wj3C7c">
_______________________________________________<br>LogAnalysis mailing list<br><a href="mailto:LogAnalysis (at) loganalysis (dot) org [email concealed]">LogAnalysis (at) loganalysis (dot) org [email concealed]</a
><br><a href="http://www.loganalysis.org/mailman/listinfo/loganalysis" target="_blank">
http://www.loganalysis.org/mailman/listinfo/loganalysis</a><br></div></d
iv></blockquote></div><br>
_______________________________________________
LogAnalysis mailing list
LogAnalysis (at) loganalysis (dot) org [email concealed]
http://www.loganalysis.org/mailman/listinfo/loganalysis

[ reply ]
Re: [logs] naming multiple output files with syslog-ng Dec 24 2007 10:49AM
Christian Folini (christian folini post ch)
RE: [logs] naming multiple output files with syslog-ng Dec 21 2007 10:19AM
Jan Monsch (jan monsch csnc ch) (1 replies)
RE: [logs] naming multiple output files with syslog-ng Dec 23 2007 08:06PM
Marcus J. Ranum (mjr ranum com)
Re: [logs] naming multiple output files with syslog-ng Dec 20 2007 05:23PM
Bill Burge (bill burge com)
Re: [logs] naming multiple output files with syslog-ng Dec 18 2007 06:22AM
Tom Le (dottom gmail com)


 

Privacy Statement
Copyright 2010, SecurityFocus