LogAnalysis
RE: [logs] How to define Log, Event, and Alert? Jul 24 2008 04:43PM
Tina Bird (tbird precision-guesswork com) (1 replies)
RE: [logs] How to define Log, Event, and Alert? Jul 25 2008 02:23AM
Bill Scherr IV (bschnzl cotse net) (1 replies)
Re: [logs] How to define Log, Event, and Alert? Jul 25 2008 04:14AM
Anton Chuvakin (anton chuvakin org) (1 replies)
RE: [logs] How to define Log, Event, and Alert? Jul 25 2008 09:23AM
Rainer Gerhards (rgerhards hq adiscon com) (1 replies)
If I may generalize things a bit...

I'd replace TIME by "sequence identifier", with sequence identifier
defined as being something that is monotonically increasing. A timestamp
is an object that we think to be a natural example of a monotonically
increasing function. HOWEVER, if we look at existing technology, this is
not always the case. In fact, it is more often NOT the case than it
is... If we have two systems a and b and these systems do not have time
synchronized, and have a system c which is the event collector (and
collects only events from a and b), then c may record time stamps inside
its log that do not monotonically increase. For example, it may record:

02:00:00 event a1
01:00:00 event b1
02:01:00 event a2
01:01:00 event b2

Of course, this is still a TIMED record of occurrences. However, in this
sense this is used, "TIMED" includes a sense of temporal order (at least
to me). In the above log, we may not have the correct temporal order.
We may be able to reconstruct it by sorting on the timestamp. That would
be a valid approach if the timestamps are indeed correct (compared to
universal time). But if a and/or b has incorrect time, we would create a
wrong temporal order. Indeed, in this sense the monotonically increasing
identity of the log in question would actually not be the timestamp but
rather the *sequence of recording*, kind of a meta-property not directly
contained in the property set of the individual event record (but rather
obtained by its relationship to its predecessor in the log file).

Now let's assume a log without a timestamp. These things happens, e.g.
in debug logs (and all too often in others I have seen).

If we define

> Log = a TIMED record of the above occurence.

such a "log" would obviously not be a log, because it does not fulfill
the requirement to be timed.

If a log instead is "a record of events with a sequence identifier",
that problem does not exist. The sequence identifier in that case would
be the derived property I mentioned above.

The question remains if such a definition is actually useful. The
sequence identifier is obviously something with very vague semantics.
They depend on the observer as well as the correctness of the "sequence
identifier generating function" on all systems in question.

Let's get back to the simple case of timestamps: as outlined above, the
semantics of timestamps depend on time sync. Even if there is ntp
timesync, timestamps (with reasonable precision) are always
questionable. They are approximate, even on the same system. With
standard syslog timestamps (second precision!) the problem is easy to
see: one may receive hundreds of events within the same second. So even
if time is correct, an observer is unable to detect any order of events.
If looking just at the timestamps, one must conclude that all events
happened at once. If looking at the semantics of the messages, one most
often also can conclude this is impossible (e.g. how to delete a file
before it is created?). Obviously, the timestamp alone is never
sufficient to detect order of events, even on a single system. Granted,
for practical purposes a high resolution timestamp (with good time
synchronization) is most often a sufficiently well approximation of the
time an event happened. But do you really trust it? ...always? Have a
look at your own correlation engines: do they work on pure timestamps -
or do they include some other properties, like the order of event log
records inside the log?

Now let me try to define what I think a log actually is:

An EVENT is a set of properties that describe a state change (in the
sense I have described state change yesterday). The contents of this set
is depending on the entity who's state changes as well as on the
observer. [so it may actually be a set of two sets: entity-related
properties and observer-related properties]

An event is generated when a state changes.

An EVENT RECORD is the physical representation of an event inside an
event log. It is a set of (at least) two sets: the event's set itself as
well as a set of meta-properties which describe the event record (e.g.
it's size, it's order inside an event log record, it's format, ...). I
think this has a tendency to contain observer-related properties, too.
Some Meta-properties are obviously event log related (like sequence of
recording).

Finally, an (EVENT) LOG is a set of event records. Please note that
according to this definition, the stream of syslog messages flowing from
system a to system b is also an event log - not just log files or
database tables. So an event log does not necessarily need to be
persistend.

There is no inherent order inside event logs. In practice we experience
a "natural order" (one record begins before another), but that actually
is a meta-property of the event record. So we can order event records
based on their meta-properties. It just happens that a text log is
physically ordered by the sequence meta property.

[side-note: if we look at a UDP based transmission of an event log, we
notice that the sender's perception of the event log is different from
the receiver's: the physical order of event log records may be
different, as well as their absolute number (if UDP discards messages)]

Finally, different log files, database tables, log stores in general are
just partitions of the event log set. This also explains why we need to
merge different log files if we would like to have a broader view of
events that happened inside our system: We need to re-unit the
partitions that contain things of interest for our needs so that we
build a sufficiently large set for our analysis algorithm (that set
being a complete partition of the log set in regard to what we are
interested in).

[side-note: if we need to unite such sets, we often experience the
problem that we do not have a truly identifying function for event log
records, which makes it often really hard to create real supersets,
where no two elements are the same]

Again, I see that my definitions are not really one-liners, but I think
if we try to use too-simple definitions, we end up incorrectly
describing the objects we are talking about. That will most probably
lead us to wrong conclusions, and, in a very practical sense, to a)
wrong implementations and b) wrong actions taken on the conclusions we
think we have drawn (but were invalid).

Rainer

> -----Original Message-----
> From: loganalysis-bounces (at) loganalysis (dot) org [email concealed] [mailto:loganalysis-
> bounces (at) loganalysis (dot) org [email concealed]] On Behalf Of Anton Chuvakin
> Sent: Friday, July 25, 2008 6:14 AM
> To: bschnzl (at) cotse (dot) net [email concealed]
> Cc: loganalysis (at) loganalysis (dot) org [email concealed]
> Subject: Re: [logs] How to define Log, Event, and Alert?
>
> Good point. So:
>
> Event = something that happened on a system.
> Log = a TIMED record of the above occurence.
>
>
> On Thu, Jul 24, 2008 at 7:23 PM, Bill Scherr IV <bschnzl (at) cotse (dot) net [email concealed]>
> wrote:
> > So...
> >
> > I gather a temporal mention to be appropriate beyond the definition
> of the Log. Also, most systems break off their logs by
> > size, not time. Although there is a definite time to each log, they
> are not consistent, even with the same log gatherer. Right or
> > wrong, that is how I find them. Suggestions below (if I may be so
> bold):
> >
> > Circa 11:26, 23 Jul 2008, a note, claiming source Heinbockel, Bill
> <heinbockel (at) mitre (dot) org [email concealed]>, was sent to me:
> >
> > From: "Heinbockel, Bill" <heinbockel (at) mitre (dot) org [email concealed]>
> > To: <loganalysis (at) loganalysis (dot) org [email concealed]>
> > Subject: [logs] How to define Log, Event, and Alert?
> >
> >>
> >>
> >> Here is our initial shot at defining these terms:
> >>
> >>
> >> Event:
> >> A discrete, distinct, and discernible state change in an
> >> environment.
> >
> > A discrete, distinct, and discernible state change in an environment
> at a recorded (or given) time.
> >>
> >> Alert (n):
> >> A warning or notification generated in response to an event.
> >>
> >> Alert (v):
> >> The act of generating, transport, or displaying a warning or
> >> notification in response to an event.
> >>
> >> Log Entry:
> >> The record of an event in a log. Event log, event record, log
> >> message, log record, and audit record are all synonyms that have
> been
> >> used to refer to log entries.
> >
> > The record of an event in a log, in sequence, usually with a
> timestamp. <thesaurus reference to follow>
> >>
> >> Log (n):
> >> The record comprising one or more log entries accumulated
over
> >> a given period. This may be electronic (e.g. stored in memory,
disk,
> >> software, database, text file, etc), physical (e.g. on paper), or
> even
> >> verbal (e.g., "Between 10:00 and 10:01 we received a series of
> several
> >> thousand SYN packets that we acknowledged, but full TCP connections
> >> were not completed. At 10:02, our server resources exceeded the
> >> maximum tolerable level and crashed.").
> >>
> >> Log (v):
> >> The act of recording or storing one or more events.
> >>
> >>
> >>
> >> What do you think?
> >> Can these definitions be changed/improved in anyway?
> >> Is there any examples, synonyms, or clarifications that should be
> >> added?
> >>
> >
> > Event: The same state change may occur repeatedly.
> > Log Entry: No entry happens without context.
> >
> >
> >
> >
> > Bill Scherr IV, GSEC, GCIA
> > Principal Security Engineer
> > EWA Information and Infrastructure Technologies
> > bscherr (at) iit-tek (dot) com [email concealed]
> > bscherr (at) ewa (dot) com [email concealed]
> > 703-478-7608
> > _______________________________________________
> > LogAnalysis mailing list
> > LogAnalysis (at) loganalysis (dot) org [email concealed]
> > http://www.loganalysis.org/mailman/listinfo/loganalysis
> >
>
>
>
> --
> Anton Chuvakin, Ph.D., GCIA, GCIH, GCFA
> http://www.chuvakin.org
> http://chuvakin.blogspot.com
> http://www.info-secure.org
> _______________________________________________
> LogAnalysis mailing list
> LogAnalysis (at) loganalysis (dot) org [email concealed]
> http://www.loganalysis.org/mailman/listinfo/loganalysis

_______________________________________________
LogAnalysis mailing list
LogAnalysis (at) loganalysis (dot) org [email concealed]
http://www.loganalysis.org/mailman/listinfo/loganalysis

[ reply ]
RE: [logs] How to define Log, Event, and Alert? Jul 26 2008 02:48AM
Bill Scherr IV (bschnzl cotse net) (1 replies)
RE: [logs] How to define Log, Event, and Alert? Jul 29 2008 03:31PM
Rainer Gerhards (rgerhards hq adiscon com) (1 replies)
RE: [logs] How to define Log, Event, and Alert? Jul 30 2008 07:41AM
Eric Fitzgerald (Eric Fitzgerald microsoft com) (1 replies)
RE: [logs] How to define Log, Event, and Alert? Jul 30 2008 09:56AM
Rainer Gerhards (rgerhards hq adiscon com)


 

Privacy Statement
Copyright 2010, SecurityFocus