Introduction

Forensic analysts and incident response engineers are armed with a slew of open source and commercial forensic toolsets to attempt to understand and analyze break-ins they did not witness. The most critical component of forensic analysis is system log files. In particular, the analyst must be able to understand and recognize footprints that exploits leave on system logfiles. Identifying these signatures, and their impact on the application within the log files, is the key to understanding what took place during a security incident.

This paper will focus on the identification of the footprints that exploits leave on system logfiles and what they mean, as well as the most common traces that some recent exploits leave. It is hoped that this discussion will help to create a set of methodologies for readers to follow when conducting incident response and forensic analysis, thereby introducing readers to the world of forensic analysis using system and application log files as an evidentiary resource in place of intrusion detection systems.

Log Notification

This section will discuss some tools that provide log notification to the end user. All tools work in conjunction with Syslog-ng (Syslog Next Generation) and can send log alerts off site via e-mail or other means of communication. With these tools, users can recognise the signatures of exploits from Web applications such as Apache and IIS, which will help develop a footprint that can be used to gain an understanding of the incident. Some of the footprints discussed will include the latest Apache Chunked Encoding exploit and PHP overflows. This section will also provide a walkthrough for installing some of the more robust end-user logging tools. These tools will help to provide better administration of system logs and critical event notification.

LogSentry

LogSentry (formerly dubbed Logcheck) was designed to automatically monitor system logs, e-mailing the administrator when security violations occur. The tool is based on a program that is shipped with the TIS Gauntlet firewall; however, it has been improved upon in many ways to make it work well for end-user systems.

LogSentry provides numerous alert notification options, which are specified by the system administrator. The most common method of notification is e-mail: LogSentry interacts with Sendmail to notify a list of recipients when critical events are logged.

Installing LogSentry

Prior to discussing implementation of the tool, it will be helpful to go over the installation and configuration procedures for LogSentry and how to configure it to work with Syslog-ng. The following example uses a program called wget. You can install wget via the ports collection distributed with FreeBSD by issuing:

Alternatively, you can also download it from ftp://ftp.dl.ac.uk/acc14/ftp-mirror/wget/pub/unix/itul/wget/ in the following manner:

You will notice you have several files located in the logcheck-1.1.1 directory, all of which are detailed in the Install file. We can now discuss configuring and preparing LogSentry for the purposes of this article. This will cover several main set-ups, which includes editing the Syslog-ng configuration file to enable a more verbose logging capability.

Configuring Syslog-ng: Step One

On most systems, I would recommend that users configure Syslog-ng to send all alert messages to one file for LogSentry to parse through. This configuration ensures that messages will not be missed. To do this we will now start configuring Syslog-ng. This involves editing the /usr/local/etc/syslog-ng/syslog-ng.conf file. This file contains various parameters for Syslog-ng. If readers need more information on these parameters, they should check the manual pages of Syslog-ng for more information. (Another good resource for setting up secure remote log servers is Eric Hines' article, Flying Pigs: Snorting Next Generation Secure Remote Log Servers over TCP.)

Create the Syslog-ng configuration file after downloading and installing Syslog-ng. It can be found at http://www.balabit.hu/en/downloads/syslog-ng/.

################################
# Create your source           #
# log device file. This will   #
# be different for each        #
# OS type                      #
################################


source fw {              
               unix-dgram("/dev/log");              
               internal();
};


#################################
# Assign the destination        #
# (TCP or local file)           #
#################################


destination localhost {
              file("/var/log/syslog-ng.all");
};


################################
# Tie your SOURCE and DST      #
################################


log {
            source(fw); destination(localhost);


};

This will log all data to the "syslog-ng.all" file located in /var/log. If you wish, you can just edit the /var/log/syslog-ng.all line to another location of your choice. Remember, this will act as your central logging file for Syslog-ng. Once you have configured your syslog-ng.conf file, restart Syslog-ng by sending the HUP signal to it. This can be done by issuing: killall "HUP syslog-ng as root.

Once you have restarted the Syslog-ng server, you must now go to your log directory, and change the log file to owner root, group wheel and mode 600 on file permissions. First check if the file exists; if it doesn't, you should create it.

I would also suggest changing the owner to root, group wheel, and mode 600 to any similar log files. Log files contain very sensitive information, and without proper permissions they may reveal system errors, passwords, and vulnerabilities to unprivileged users.

Tip: BSD users may go to the /etc directory and edit the /etc/daily, /etc/weekly, and /etc/monthly scripts and configure the "rotate()" script function to change log permissions on rotation since they are reset automatically by cron. Simply change the line:

cp /dev/null "$file"; chmod 644 "$file"

cp /dev/null "$file"; chmod 600 "$file".

Configuring LogSentry, Step Two

This section will discuss how to configure the logtail and logcheck files that are distributed with the LogSentry package. However, before going further, it is necessary to edit logcheck.sh that, by default, is located in the /usr/local/etc directory. Open it up in your favourite text editor and find the line entitled: "Configuration Section".

You will want to change the "Sysadmin" variable to a name of your liking, which can be a local user of the system or an e-mail address if using remote logging. Scroll further down in the logcheck.sh file until you some to a line entitled: "Log File Configuration Section" and uncomment the log files that apply to you. Since we are configuring LogSentry to run with Syslog-ng, we'll want to add the following line under our system type:

Once complete, you will want to compile LogSentry for your specific operating system. The syntax for this would be:

Once LogSentry has been installed, I would suggest editing the local crontab to run it every hour. You can easily edit this to run more frequently, or less frequently, depending on your preference.

Note: If you changed the default location of the logcheck files during configuration, you'll obviously have to edit the above lines to your custom location for the crontab to work correctly.

Upon completion of modifying the crontab, you will want to send it a HUP to restart crond, allowing it to restart and reload its configuration.

To test it out, run the logcheck.sh script by hand and cause several alerts. Something as simple as a bad su works great. Make sure that it's being picked up in the log (/var/log/syslog-ng.all). If you are getting repeat messages when running the logcheck.sh script, something is wrong with the logtail binary. Remember to check the permissions of the logcheck files.

Application Log Analysis (Apache)

We will now begin the attack section of this paper. (Please do not e-mail SecurityFocus or Fate Labs requesting a copy of these exploits.)

Case Study One: Gobbles (apache-nosejob.c)

This attack exploits the "chunking" vulnerability in versions of Apache that fail to properly calculate the required buffer sizes when processing requests coded with the "Chunked Encoding" mechanism (see SecurityFocus BID:5033)

Gobbles Research has released two exploits for this vulnerability (apache-scalp.c and apache-nosejob.c). In this exercise, we will be exploiting an OpenBSD box running a vulnerable version of the Apache Web Server. The only differences between these two exploits are that Apache-scalp only provides target shellcode for the OpenBSD Operating System (3.0, 3.1) and Apache-nosejob targets for FreeBSD, OpenBSD, and NetBSD.

In our exercise we will be using the Apache-nosejob exploit for attacking an OpenBSD 3.1 system running a vulnerable Apache Server v1.3.24. First, let's look at the exploit itself.

Now, what if we were responding to this attack but we had no idea how the user originally broke in. Let's go ahead and check the Apache logfile now. (I've talked to several admins who didn't even know where the Apache logfiles were kept. This is critical, as anything and everything that Apache logs is stored in these crucial files. They should be located in $prefix/logs/. The file we are concerned about right now is /path/to/apache/logs/error_log.)

Readers might ask, "Does this exploit leave any fingerprint at all in the Apache logfiles?" Well, don't expect it to say "you are being hit by the Gobbles Apache Scalp exploit" (grin). Rather, we will need to take a close look at the error messages it generated. I will only be pasting a piece of the complete log as the same following error is repeated for several pages.

Take a look at that. The child processes for Apache kept "Segfaulting" as a result of the exploit. Shortly thereafter we executed a bad su attempt that, despite what you might think, still shows up in Syslog even though we aren't actually logged into an interactive shell. We are actually logged in through port 80 under Apache, executing commands as the Apache process. See below:

System Log Analysis (SSHD)

Some time ago a vulnerability was discovered in the CRC32 Compensation Attack Detector within versions of OpenSSH and SSH. Several exploits surfaced from Teso that exploited this vulnerability. In this section we will provide you several fingerprints that the x3.bin exploit leaves on system log files.

The target (victim) system set-up for this lab was running a vulnerable version of SSH-1.2.27

This was the attack launched against the vulnerable system. We will now take a look at what showed up in our Syslog-ng logfile. After reading this section, we invite you to read a very well written writeup on this vulnerability by David Dittrich after a machine was compromised on the University of Washington network. To read the CERT advisory on this vulnerability, refer to http://www.kb.cert.org/vuls/id/945216

What do you notice about these logs? What sticks out shouldn't occur in any other circumstance unless an attack was taking place? The first thing is clearly that message from the SSHD process itself (fatal: Local: crc32 compensation attack: network attack detected) is warning that it is under some form of CRC32 attack. This compensation attack detector is exactly the vulnerability the exploit targets.

Another notable fingerprint this exploit leaves is the multiple connection attempts from the same IP address multiple times within 6-7 seconds apart. The presence of hundreds of connection attempts would likely indicate an automated script rather than someone who forgot their password. Another major fingerprint would be the "corrupted check bytes on input" error in the log file. This is the only time I've ever seen this error from SSH.

The following is a Snort packetdump of the last packet from the attack. The complete packet dump from the attack can be utilized for making additional IDS signatures and can be downloaded from the Log Forensics Team page at the Fate Labs Web site.

Conclusion

The most important point of this discussion is to emphasize that security administrators must monitor system logfiles just as closely as they monitor IDS alerts. Remember that intrusion detection systems can only detect attacks they know about (IDS signatures) and that new zero-day exploits will not be detected unless the IDS is configured to listen for that specific attack. The Apache "Chunk" exploit discussed above is a perfect example of this is. Once it surfaced, we had to put together our own signatures for it until Snort signatures were made available. We were only able to see this attack being used because we closely monitor our Apache logfiles.

Multiple open source tools that make log notification and central logging possible are currently available for end-users. We recommend that security administrators replace Syslog with Syslog-ng, as well as installing a log notification tool such as Logsentry to have critical events e-mailed to them. Another good idea would be to set up a secure central logging server for storing logs off-site. This is critical when a system becomes compromised, as all logs on that system can no longer be trusted.

Finally, since making the network unassailable is impossible, the only other option we have is to get it as close as we can. True security isn't achieved with expensive firewall appliances, VPNs, or even intrusion detection systems, it's achieved through monitoring log files, watching for the attacks that intrusion detection systems may have missed. As security solutions become more refined, so will the tools and methods of the Blackhat hacker. This can be demonstrated through tools such as ADMmutate and IDS evasion methods such as fragmentation attacks. If all security administrators only monitor IDS logs, they are leaving themselves vulnerable to attacks that may go undetected.

Eric Hines has worked in the Information Security Industry for over 10 years. He has been an advisor to the Federal Bureau of Investigation/NIPC, and state Police Departments and used as an expert witness in the conviction of hackers. He has also been a speaker at Blackhat Briefings, Defcon, Homeland Defense Symposiums and has presented to the Technology Insertion Panel for the Defense Information Systems Agency (DISA).

Alan Neville has been involved in information security for the past three years. Director of Fate Research Labs, Europe, Alan heads all European operations as well as manages the Ireland honeynet for Fate Research Labs in Dublin. Alan has authored several whitepapers on securing Linux, configuring firewalls, and is also author of the recent advisory on Nullsoft SHOUTcast.