LogAnalysis
[logs] naming multiple output files with syslog-ng Dec 17 2007 02:20PM
Christian Folini (christian folini post ch) (2 replies)
RE: [logs] naming multiple output files with syslog-ng Dec 20 2007 03:55PM
Jan Monsch (jan monsch csnc ch) (1 replies)
RE: [logs] naming multiple output files with syslog-ng Dec 20 2007 04:25PM
Marcus J. Ranum (mjr ranum com) (3 replies)
Re: [logs] naming multiple output files with syslog-ng Dec 27 2007 01:14AM
Mordechai T. Abzug (morty frakir org) (1 replies)
Re: [logs] naming multiple output files with syslog-ng Jan 01 2008 01:05PM
Chris Brenton (cbrenton chrisbrenton org)
RE: [logs] naming multiple output files with syslog-ng Dec 20 2007 06:22PM
Paul Melson (pmelson gmail com) (1 replies)
Re: [logs] naming multiple output files with syslog-ng Dec 21 2007 06:55AM
Christian Folini (christian folini post ch) (2 replies)
Re: [logs] naming multiple output files with syslog-ng Dec 21 2007 04:16PM
Chris Wee (chris wee loglogic com) (1 replies)
Re: [logs] naming multiple output files with syslog-ng Dec 24 2007 10:49AM
Christian Folini (christian folini post ch)
Hey Chris,

This discussion is getting closer to a fundamental discussion.
I have no worries with the format of mysql or sqlite for this
project. But it's going to be a nonstandard format and standard
tools won't be able to read it. One can always convert the data
back to an ascii format, but I prefer to do it the other way
around: I leave the data in the standard format and I import
it into a DB when I have to. Actually I am going to prepare
a simple import method for sqlite with predefined scripts.
Sury my scripts will be outdated, they will cease to work
a few years from now. But I guess grep, awk and sed will still be
around and if it's not, then some other tool able to read it.
(The original sed tutorial of 1978 still works fine, btw.)

But let's go back too the fundamental discussion. Beneath
a thin layer of sysadmin camouflage I am a learned medievalist.
I spent ten years at University studying medieval documents.
I had diplomas of Charlemagne in my hands. And believe it
or not: You can read them without any additional tools.
Your eyes is all you need. Sure, the documents will look Latin
to you, but they have a very clear encoding and it proved to
be stable for 1200 years. And now you try to tell me about
a database format being readable for another 5-10 years. You
must be bloody joking.
(You can also read documents dating 1800 years, but this
was before they invented the "space" in writing and this
makesreadinglatindocumentssomewhatannoying.

Electronical storage is so volatile, it makes my historian's
heart cry. Even the encoding is volatile. The best I can think
of is ascii. It's been stable for something around 50 years
and hopefully my grand children will still find import filters
to read it one way or the other. However, I have friends, who
struggle to import data from custom databases from the 80ties
(Declined such a job offer from the national archives, actually).

Under the line: If you want to be sure your data is readable
tomorrow, then go with a standard format. Custom binary formats
are fun and certainly helpful. But rather as an addition then
a replacement. Pray you have the storage capacity to keep data
in both formats. If you don't, well then you'll have to make
choice...

cheers,

Christian

On Fri, Dec 21, 2007 at 08:16:19AM -0800, Chris Wee wrote:
> Christian,
>
> On a previous project in 2003, we used Mysql4 MYISAM tables as the binary
> logging format. It stood (and still stands) a
> good chance of being readable up to 2010. The overhead of the DB engine was
> neglible w.r.t. to the volume of the log (high).
>
> Today, I would use sqlite3 because it is endian-neutral, and I think given
> the support it is receiving, it has a good chance
> of being supported for 5-10 years. Your ascii-parser is written in some
> language (perl, python, ruby, java) and that has
> to be ported to 128-bit architectures in the future which is about the same
> chance as sqlite3/4/5...
>
> Note, the performance gains we saw from binary versus ascii was due to the
> nature of the data -- non-string data that required
> conversion from text <-> binary representations. Your mileage may vary.
>
> -chris
>
> On Dec 20, 2007 10:55 PM, Christian Folini <christian.folini (at) post (dot) ch [email concealed]> wrote:
>
> > <snip>
> >
> > Binary formats and csv logging. A great deal faster and a lot more
> > storage efficient. However, this is not always a priority.
> > My priority is, that the logfiles I am writing today be readable
> > tomorrow. Possibly with standard tools. I am quite sure that any
> > binary format would be the wrong format tomorrow. If these guys are
> > smart, they will be able to get the data out of the DB again, tomorrow.
> > But it's likely to cause headaches and meetings and stuff.
> > Ascii logging is the safe path; and we can always import the data
> > into a db later on or transform them into csv format. Actually I
> > did some work on that lately. All that is bugging me is the time it
> > takes to parse the ascii logfiles. It takes too long for my
> > taste, but then it's also a lot of logfiles. ;)
> > <snip>
> > Christian
> >
> > P.S. Marcus: Cool to read your message in this thread. I just
> > wrote about artificial ignorance in one of the conceptual papers
> > for this project. :)
> > _______________________________________________
> > LogAnalysis mailing list
> > LogAnalysis (at) loganalysis (dot) org [email concealed]
> > http://www.loganalysis.org/mailman/listinfo/loganalysis
> >
_______________________________________________
LogAnalysis mailing list
LogAnalysis (at) loganalysis (dot) org [email concealed]
http://www.loganalysis.org/mailman/listinfo/loganalysis

[ reply ]
RE: [logs] naming multiple output files with syslog-ng Dec 21 2007 10:19AM
Jan Monsch (jan monsch csnc ch) (1 replies)
RE: [logs] naming multiple output files with syslog-ng Dec 23 2007 08:06PM
Marcus J. Ranum (mjr ranum com)
Re: [logs] naming multiple output files with syslog-ng Dec 20 2007 05:23PM
Bill Burge (bill burge com)
Re: [logs] naming multiple output files with syslog-ng Dec 18 2007 06:22AM
Tom Le (dottom gmail com)


 

Privacy Statement
Copyright 2010, SecurityFocus