I love Perl and its powerful regular expression engine, but whenever
trying to use multiple regexps in big datasets remember to make sure
your expressions are as tight as possible. In particular:

http://search.cpan.org/~rgarcia/perl-5.10.0/pod/perlre.pod#Backtracking_
_

Even though the map {..} is linear regarding the size of the regular
expression array, applying a regular expression to a string can degrade
to exponential time (which can be nasty for long log lines) -- from the
above URL:

"WARNING: Particularly complicated regular expressions can take
exponential time to solve because of the immense number of possible ways
they can use backtracking to try for a match. For example, without
internal optimizations done by the regular expression engine, this will
take a painfully long time to run:

'aaaaaaaaaaaa' =~ /((a{0,5}){0,5})*[c]/

And if you used *'s in the internal groups instead of limiting them to 0
through 5 matches, then it would take forever--or until you ran out of
stack space. Moreover, these internal optimizations are not always
applicable. For example, if you put {0,5} instead of * on the external
group, no current optimization is applicable, and the match takes a long
time to finish.

A powerful tool for optimizing such beasts is what is known as an
"independent group", which does not backtrack (see (?>pattern)). Note
also that zero-length look-ahead/look-behind assertions will not
backtrack to make the tail match, since they are in "logical" context:
only whether they match is considered relevant. For an example where
side-effects of look-ahead might have influenced the following match,
see (?>pattern)."

On Fri, 2008-03-07 at 20:51 -0800, Michael Kinsley wrote:
> You can use perl's IO::Multiplex module to watch all those files with
> non-blocking IO.
>
>
> Courtesy of the Perl Cookbook and a little extra map{} from me:
>
>
> use IO::Multiplex;
>
>
> my $regex_array = [
> qr/Pattern1/,
>
> qr/Pattern2/,
>
> qr/Pattern..n/
>
>
>
> ];
>
>
> $mux = IO::Multiplex->new( );
> $mux->add($FH1);
> $mux->add($FH2); # ... and so on for all the filehandles to manage
> $mux->set_callback_object(_ _PACKAGE_ _); # or an object
> $mux->Loop( );
>
>
> sub mux_input {
> my ($package, $mux, $fh, $input) = @_;
>
> map { $line =~ m/$_/ and print "Matched Line of Interest: $line
> \n" } @{$regex_array};
> }
>
>
>
> Michael Kinsley
>
>
> Consulting Engineer
>
>
> SenSage, Inc.
> 55 Hawthorne Street Ste. 700
> San Francisco, CA 94105 USA
>
>
> email: michael.kinsley (at) sensage (dot) com [email concealed]
> mobile: +1.415.465.0106
> fax: +1.415.371.1385
>
>
>
>
> On Mar 7, 2008, at 1:38 PM, Zonker Harris wrote:
>
> > I'm using Conserver, which makes reverse-TCP connections to console
> > server serial ports, so I can manage my hosts and net gear. This
> > results in an ASCII text file for each device.
> >
> >
> > What tool(s) can I use to watch all (500+) files for 'interesting'
> > strings, like malloc errors, failed logins, net connection/port/link
> > failures?
> >
> >
> > I've found log watcher, but it is a one- file-at-a-time deal. I'd
> > rather not reinvent the wheel if there is a good answer out there.
> > I'm hoping to get to RSA con this year, and perhaps find other
> > pointers to share.
> >
> >
> > Thank you for any tips,
> >
> >
> > =Z=
> >
> >
> > http://www.conserved.com/consoles/
> > http://consoleteam.blogspot.com/
> >
> >
> > _______________________________________________
> > LogAnalysis mailing list
> > LogAnalysis (at) loganalysis (dot) org [email concealed]
> > http://www.loganalysis.org/mailman/listinfo/loganalysis
>
>
> _______________________________________________
> LogAnalysis mailing list
> LogAnalysis (at) loganalysis (dot) org [email concealed]
> http://www.loganalysis.org/mailman/listinfo/loganalysis
--
Ulisses Reina Montenegro de Albuquerque
AB34 A154 28C1 84A8 676E 202F 264B 117F 88BA B1A5
Matrix Development/Tempest Security Intelligence
http://www.tempest.com.br/

_______________________________________________
LogAnalysis mailing list
LogAnalysis (at) loganalysis (dot) org [email concealed]
http://www.loganalysis.org/mailman/listinfo/loganalysis

[ reply ]