LogAnalysis
[logs] announcing rsyslog 3.11.0 - first open source disk bufferingsyslogd Jan 31 2008 09:08PM
Rainer Gerhards (rgerhards gmail com)
[I am the developer of rsyslog and as such have a certain bias. I
still think this mail is technically important and thus I am posting
to this list (pending moderator decision, of course).]

Hi all,

it is a great pleasure for me to do a fully buzzword-compliant ;)
announcement for rsyslog 3.11.0, which is available immediately. This
release is a very important milestone:

Now rsyslog is the world's first-ever purely open-sourced syslogd
supporting disk based buffering individually for each log
destination[*] (called "action" in rsyslog speak). Besides, each
action can now be executed on its own thread. Together with the
multi-threaded main worker pool, rsyslog is more than ready for
tomorrow's massively multi-core machines. I think another "world's
first" is the ability to automatically discard low priority syslog
messages under congestion conditions.

The new engine provides, among others:

* reliable storage of messages while a destination is down
* persistence of message even when rsyslogd is shut down
* automatic retry of failed actions until they succeed,
of course without any message loss
* unparalleled ability to handle message burst
* fully automatic worker thread pools
* slow actions (e.g. database writes) are de-coupled from
quick local actions (e.g. file writes)
* rate-limited action processing
* flow control slows senders down when queues get full
* ability to discard less important messages in favor
of higher important ones when the queue runs out of
space

All of this while using minimal system resources.

The system can be set up with few configuration commands, but provides
very in-depth commands to manipulate aspects like worker thread
startup and shutdown parameters, queue sizes, disk quotas, watermark
algorithm settings and many more (sometimes exotic) tweaking knobs.

Full details about the new system and its parameters can be found at:
http://www.rsyslog.com/doc-queues.html

An actual use case, doing massive database inserts, is documented here:
http://www.rsyslog.com/doc-rsyslog_high_database_rate.html

Rsyslog 3.11.0 is fresh off the "development press". While it is well
tested in lab, I expect that there are bugs, especially given the
magnitude of changes and the massively multithreaded processing flow.
As such, I do not yet advise to use it on important production
machines.

I would, however, be very interested in feedback from the field,
including bug reports. My focus in the next weeks will be on making
the new engine rock-solid while at the same time introducing many new
features which are quite easy to build on top of the new engine (like
more advanced rate limiting algorithms, action execution-time windows
and more advanced ways to flow-control inputs). My hope is that within
two month or so we will have a very stable version suitable for
production use in the most demanding environments.

I hope that this announcement will motivate some people to try rsyslog
out and provide feedback to the project. Just be warned that I will
not be at my desk next week: I am taking my now hopefully deserved
break after a long time of hard coding. I thought to delay the
announcement, but have decided it is probably better to get some head
start. So please do not be disappointed if I do not reply immediately
on comments posted. For comments and questions, I recommend using
either the mailing list or the web forum at

http://www.rsyslog.com

You may download the tarball either from the rsyslog site above or
from its Freshmeat info page:

http://freshmeat.net/projects/rsyslog/

Rsyslog is released under GPLv3. Those interested in new release
notifications are advised to subscribe to rsyslog on Freshmeat. That
service provides low-volume update notifications (in contrast to the
mailing list, which can be quite bursty at times).

Finally, I would like to thank everybody who contributed to the
project. Without contributions (of any type), we would definitely not
be able to achieve this important milestone. Please keep the good
stuff coming!

I hope rsyslog will be useful. All feedback is highly appreciated.

Best regards,
Rainer Gerhards
Adiscon

[*] I know that syslog-ng does disk buffering, but it does not support
it in the open source edition.
_______________________________________________
LogAnalysis mailing list
LogAnalysis (at) loganalysis (dot) org [email concealed]
http://www.loganalysis.org/mailman/listinfo/loganalysis

[ reply ]


 

Privacy Statement
Copyright 2010, SecurityFocus