On 9/17/07, Daniel Cid <danielcid (at) yahoo.com (dot) br [email concealed]> wrote:
| First of all, I think most projects do log analysis
| wrong. They confuse log decoding with rule matching
| and end up with hundred of regexes that are checked on
| every log. Regexes can be used to extract some bits of
| patterns from the logs, but not as the main method to
| do the log analysis...

I think scale is an important factor to discuss when we talk about
performance. With advances in CPU speeds, even the worse case scenario of
looping through N regex rules is not that bad for most applications. You
mentioned OSSEC with 500 rules, which isn't that bad a problem even if you
had to try all of them.

Now if you had 5k, 50k, or 500k rules... then scale becomes a factor.
Similarly, you have to look at throughput as well... do you need to parse
100 messages per second or 10,000?
On 9/17/07, <b class="gmail_sendername">Daniel Cid</b> <<a href="mailto:danielcid (at) yahoo.com (dot) br [email concealed]">danielcid (at) yahoo.com (dot) br [email concealed]</a>> wrote:<br><div><span class="gmail_quote"></span>| First of all, I think most projects do log analysis
<br>| wrong. They confuse log decoding with rule matching<br>| and end up with hundred of regexes that are checked on<br>| every log. Regexes can be used to extract some bits of<br>| patterns from the logs, but not as the main method to
<br>| do the log analysis...<br>
<br>
I think scale is an important factor to discuss when we talk about
performance.  With advances in CPU speeds, even the worse case scenario
of looping through N regex rules is not that bad for most
applications.  You mentioned OSSEC with 500 rules, which isn't that bad
a problem even if you had to try all of them.<br>
<br>
Now if you had 5k, 50k, or 500k rules... then scale becomes a factor.  Similarly, you have to look at throughput as well... do you need to parse 100 messages per second or 10,000?<br>
<br>
</div>
_______________________________________________
LogAnalysis mailing list
LogAnalysis (at) loganalysis (dot) org [email concealed]
http://www.loganalysis.org/mailman/listinfo/loganalysis

[ reply ]