LogAnalysis
RE: [logs] How to define Log, Event, and Alert? Jul 24 2008 04:43PM
Tina Bird (tbird precision-guesswork com) (1 replies)
RE: [logs] How to define Log, Event, and Alert? Jul 25 2008 02:23AM
Bill Scherr IV (bschnzl cotse net) (1 replies)
Re: [logs] How to define Log, Event, and Alert? Jul 25 2008 04:14AM
Anton Chuvakin (anton chuvakin org) (1 replies)
RE: [logs] How to define Log, Event, and Alert? Jul 25 2008 09:23AM
Rainer Gerhards (rgerhards hq adiscon com) (1 replies)
RE: [logs] How to define Log, Event, and Alert? Jul 26 2008 02:48AM
Bill Scherr IV (bschnzl cotse net) (1 replies)
Wow! Comments interlaced!

Circa 11:23, 25 Jul 2008, a note, claiming source Rainer Gerhards <rgerhards (at) hq.adiscon (dot) com [email concealed]>, was sent to me:

Subject: RE: [logs] How to define Log, Event, and Alert?
Date sent: Fri, 25 Jul 2008 11:23:40 +0200
From: "Rainer Gerhards" <rgerhards (at) hq.adiscon (dot) com [email concealed]>
To: "Anton Chuvakin" <anton (at) chuvakin (dot) org [email concealed]>, <bschnzl (at) cotse (dot) net [email concealed]>
Copies to: <loganalysis (at) loganalysis (dot) org [email concealed]>

> If I may generalize things a bit...
>
> I'd replace TIME by "sequence identifier", with sequence identifier
> defined as being something that is monotonically increasing. A timestamp
> is an object that we think to be a natural example of a monotonically
> increasing function. HOWEVER, if we look at existing technology, this is
> not always the case. In fact, it is more often NOT the case than it
> is... If we have two systems a and b and these systems do not have time
> synchronized, and have a system c which is the event collector (and
> collects only events from a and b), then c may record time stamps inside
> its log that do not monotonically increase. For example, it may record:
>
> 02:00:00 event a1
> 01:00:00 event b1
> 02:01:00 event a2
> 01:01:00 event b2
>
If we keep the original log intact, the out of sync timestamps are information in themselves.

> Of course, this is still a TIMED record of occurrences. However, in this
> sense this is used, "TIMED" includes a sense of temporal order (at least
> to me). In the above log, we may not have the correct temporal order.
> We may be able to reconstruct it by sorting on the timestamp. That would
> be a valid approach if the timestamps are indeed correct (compared to
> universal time). But if a and/or b has incorrect time, we would create a
> wrong temporal order. Indeed, in this sense the monotonically increasing
> identity of the log in question would actually not be the timestamp but
> rather the *sequence of recording*, kind of a meta-property not directly
> contained in the property set of the individual event record (but rather
> obtained by its relationship to its predecessor in the log file).
>
If a file is deleted before it is created, then we locate the shift in timestamps. Different context has different meaning. For
instance, a timestamp shift coincident with the file deletion says real badness, while a timeshift occuring an odd time before
(3hrs, 24mins, 5 secs) may just indicate that an admin noticed that ntpd had died. Rather than try to sync the time in the
logs, what can we do with variations? Automatically?

> Now let's assume a log without a timestamp. These things happens, e.g.
> in debug logs (and all too often in others I have seen).
>
> If we define
>
> > Log = a TIMED record of the above occurence.
>
> such a "log" would obviously not be a log, because it does not fulfill
> the requirement to be timed.
>
> If a log instead is "a record of events with a sequence identifier",
> that problem does not exist. The sequence identifier in that case would
> be the derived property I mentioned above.
>
The sequence is the order in which it appears in the log, plus or minus network latency. (wait for it)

> The question remains if such a definition is actually useful. The
> sequence identifier is obviously something with very vague semantics.
> They depend on the observer as well as the correctness of the "sequence
> identifier generating function" on all systems in question.
>
> Let's get back to the simple case of timestamps: as outlined above, the
> semantics of timestamps depend on time sync. Even if there is ntp
> timesync, timestamps (with reasonable precision) are always
> questionable. They are approximate, even on the same system. With
> standard syslog timestamps (second precision!) the problem is easy to
> see: one may receive hundreds of events within the same second. So even
> if time is correct, an observer is unable to detect any order of events.
> If looking just at the timestamps, one must conclude that all events
> happened at once. If looking at the semantics of the messages, one most
> often also can conclude this is impossible (e.g. how to delete a file
> before it is created?). Obviously, the timestamp alone is never
> sufficient to detect order of events, even on a single system. Granted,
> for practical purposes a high resolution timestamp (with good time
> synchronization) is most often a sufficiently well approximation of the
> time an event happened. But do you really trust it? ...always? Have a
> look at your own correlation engines: do they work on pure timestamps -
> or do they include some other properties, like the order of event log
> records inside the log?
>
Why are you looking at logs? Is the machine broke? There should be plenty of context to render a correct determination. Is
the machine compromised? Then you must find an independent log that says roughly the same thing.

> Now let me try to define what I think a log actually is:
>
> An EVENT is a set of properties that describe a state change (in the
> sense I have described state change yesterday). The contents of this set
> is depending on the entity who's state changes as well as on the
> observer. [so it may actually be a set of two sets: entity-related
> properties and observer-related properties]
>
> An event is generated when a state changes.
>
"when" being the temporal aspect, recorded from the view of the program sensing the state change.

> An EVENT RECORD is the physical representation of an event inside an
> event log. It is a set of (at least) two sets: the event's set itself as
> well as a set of meta-properties which describe the event record (e.g.
> it's size, it's order inside an event log record, it's format, ...). I
> think this has a tendency to contain observer-related properties, too.
> Some Meta-properties are obviously event log related (like sequence of
> recording).
>
can we keep this term at "log entry"? All of these extra attributes can be determined from a working copy of the log file.

> Finally, an (EVENT) LOG is a set of event records. Please note that
> according to this definition, the stream of syslog messages flowing from
> system a to system b is also an event log - not just log files or
> database tables. So an event log does not necessarily need to be
> persistend.
>
And here we can separate by machine with filters.

> There is no inherent order inside event logs. In practice we experience
> a "natural order" (one record begins before another), but that actually
> is a meta-property of the event record. So we can order event records
> based on their meta-properties. It just happens that a text log is
> physically ordered by the sequence meta property.
>
> [side-note: if we look at a UDP based transmission of an event log, we
> notice that the sender's perception of the event log is different from
> the receiver's: the physical order of event log records may be
> different, as well as their absolute number (if UDP discards messages)]
>
> Finally, different log files, database tables, log stores in general are
> just partitions of the event log set. This also explains why we need to
> merge different log files if we would like to have a broader view of
> events that happened inside our system: We need to re-unit the
> partitions that contain things of interest for our needs so that we
> build a sufficiently large set for our analysis algorithm (that set
> being a complete partition of the log set in regard to what we are
> interested in).
>
> [side-note: if we need to unite such sets, we often experience the
> problem that we do not have a truly identifying function for event log
> records, which makes it often really hard to create real supersets,
> where no two elements are the same]
>
> Again, I see that my definitions are not really one-liners, but I think
> if we try to use too-simple definitions, we end up incorrectly
> describing the objects we are talking about. That will most probably
> lead us to wrong conclusions, and, in a very practical sense, to a)
> wrong implementations and b) wrong actions taken on the conclusions we
> think we have drawn (but were invalid).
>
> Rainer
>
But, if we define simple definitions, analysts can filter, assign meaning to meta data, and perform actions with proper forensic
practices to arrive at common conclusions. These conclusions can be automated, or at least automatically correlated for
later human examination. Thus we define the conclusions later, and simply. All of this can be packaged by interns (coders)
at a keyboard. Or are we trying to eliminate the need for human intelligence?

I don't think Bill's definitions were that far off...

B.

> > -----Original Message-----
> > From: loganalysis-bounces (at) loganalysis (dot) org [email concealed] [mailto:loganalysis-
> > bounces (at) loganalysis (dot) org [email concealed]] On Behalf Of Anton Chuvakin
> > Sent: Friday, July 25, 2008 6:14 AM
> > To: bschnzl (at) cotse (dot) net [email concealed]
> > Cc: loganalysis (at) loganalysis (dot) org [email concealed]
> > Subject: Re: [logs] How to define Log, Event, and Alert?
> >
> > Good point. So:
> >
> > Event = something that happened on a system.
> > Log = a TIMED record of the above occurence.
> >
> >
> > On Thu, Jul 24, 2008 at 7:23 PM, Bill Scherr IV <bschnzl (at) cotse (dot) net [email concealed]>
> > wrote:
> > > So...
> > >
> > > I gather a temporal mention to be appropriate beyond the definition
> > of the Log. Also, most systems break off their logs by
> > > size, not time. Although there is a definite time to each log, they
> > are not consistent, even with the same log gatherer. Right or
> > > wrong, that is how I find them. Suggestions below (if I may be so
> > bold):
> > >
> > > Circa 11:26, 23 Jul 2008, a note, claiming source Heinbockel, Bill
> > <heinbockel (at) mitre (dot) org [email concealed]>, was sent to me:
> > >
> > > From: "Heinbockel, Bill" <heinbockel (at) mitre (dot) org [email concealed]>
> > > To: <loganalysis (at) loganalysis (dot) org [email concealed]>
> > > Subject: [logs] How to define Log, Event, and Alert?
> > >
> > >>
> > >>
> > >> Here is our initial shot at defining these terms:
> > >>
> > >>
> > >> Event:
> > >> A discrete, distinct, and discernible state change in an
> > >> environment.
> > >
> > > A discrete, distinct, and discernible state change in an environment
> > at a recorded (or given) time.
> > >>
> > >> Alert (n):
> > >> A warning or notification generated in response to an event.
> > >>
> > >> Alert (v):
> > >> The act of generating, transport, or displaying a warning or
> > >> notification in response to an event.
> > >>
> > >> Log Entry:
> > >> The record of an event in a log. Event log, event record, log
> > >> message, log record, and audit record are all synonyms that have
> > been
> > >> used to refer to log entries.
> > >
> > > The record of an event in a log, in sequence, usually with a
> > timestamp. <thesaurus reference to follow>
> > >>
> > >> Log (n):
> > >> The record comprising one or more log entries accumulated
> over
> > >> a given period. This may be electronic (e.g. stored in memory,
> disk,
> > >> software, database, text file, etc), physical (e.g. on paper), or
> > even
> > >> verbal (e.g., "Between 10:00 and 10:01 we received a series of
> > several
> > >> thousand SYN packets that we acknowledged, but full TCP connections
> > >> were not completed. At 10:02, our server resources exceeded the
> > >> maximum tolerable level and crashed.").
> > >>
> > >> Log (v):
> > >> The act of recording or storing one or more events.
> > >>
> > >>
> > >>
> > >> What do you think?
> > >> Can these definitions be changed/improved in anyway?
> > >> Is there any examples, synonyms, or clarifications that should be
> > >> added?
> > >>
> > >
> > > Event: The same state change may occur repeatedly.
> > > Log Entry: No entry happens without context.
> > >
> > >
> > >
> > >
> > > Bill Scherr IV, GSEC, GCIA
> > > Principal Security Engineer
> > > EWA Information and Infrastructure Technologies
> > > bscherr (at) iit-tek (dot) com [email concealed]
> > > bscherr (at) ewa (dot) com [email concealed]
> > > 703-478-7608
> > > _______________________________________________
> > > LogAnalysis mailing list
> > > LogAnalysis (at) loganalysis (dot) org [email concealed]
> > > http://www.loganalysis.org/mailman/listinfo/loganalysis
> > >
> >
> >
> >
> > --
> > Anton Chuvakin, Ph.D., GCIA, GCIH, GCFA
> > http://www.chuvakin.org
> > http://chuvakin.blogspot.com
> > http://www.info-secure.org
> > _______________________________________________
> > LogAnalysis mailing list
> > LogAnalysis (at) loganalysis (dot) org [email concealed]
> > http://www.loganalysis.org/mailman/listinfo/loganalysis

Bill Scherr IV, GSEC, GCIA
Principal Security Engineer
EWA Information and Infrastructure Technologies
bscherr (at) iit-tek (dot) com [email concealed]
bscherr (at) ewa (dot) com [email concealed]
703-478-7608
_______________________________________________
LogAnalysis mailing list
LogAnalysis (at) loganalysis (dot) org [email concealed]
http://www.loganalysis.org/mailman/listinfo/loganalysis

[ reply ]
RE: [logs] How to define Log, Event, and Alert? Jul 29 2008 03:31PM
Rainer Gerhards (rgerhards hq adiscon com) (1 replies)
RE: [logs] How to define Log, Event, and Alert? Jul 30 2008 07:41AM
Eric Fitzgerald (Eric Fitzgerald microsoft com) (1 replies)
RE: [logs] How to define Log, Event, and Alert? Jul 30 2008 09:56AM
Rainer Gerhards (rgerhards hq adiscon com)


 

Privacy Statement
Copyright 2010, SecurityFocus