1. Introduction

Sometimes the live procedure described here is the only way to acquire incident data because certain types of malicious code, such as LKM based rootkits, are loaded only to memory and don't modify any file or directory. A similar situation exists in Windows operating systems -- the Code Red worm is a good example of this, where the malicious code was not saved as a file, but was inserted into and then run directory from memory.

On the other hand, methods presented below also have serious limitations and violate the primary requirement of the collection procedure for digital investigation -- a requirement which can not be easily fulfilled. That is: every user and kernel space tool used to collect data by nature changes the state of the target system. By running any tools on a live system we load them into memory and create at least one process which can overwrite possible evidence. By creating a new process, the memory management system of the operating system allocates data in main memory and then can overwrite other unallocated data in main memory or in the swap file system.

Other problems arise when we plan to take legal actions and need to comply with local laws. The signs of intrusions found in images of main memory can be untrusted, because they could be created by our acquisition tools. So before taking any action we must decide whether to acquire some data from a live compromised system or not. It is very often worth it to collect such information. In the main memory image we can find passwords or decrypted files. Using /proc pseudo file system we can also recover programs that have been deleted but are still allocated in memory.

In an ideal world, I could imagine a kind of hardware based solution for Intel-based computers, which would allow us to dump the whole memory to an external storage device without assistance of operating system. Such a solution exits on Sparc machines, whereby we can dump the whole physical memory by using the OpenBoot firmware. Unfortunately, no similar solution exists for Intel- or AMD-based computers.

Despite the above problem, software based methods also have advantages for forensic purposes, and I'll try to show them in this paper. The main goal of this article is a presentation of methods used during an evidence collection procedure. All collected data can be used later to perform offline forensic analysis. Some of presented tasks can be also be performed in the preparation and identification phases of the incident response cycle -- these are two of the six phases defined in a guide called "Incident Handling Step by step", published by the SANS Institute.

2. Forensic Analysis

• 2.1 Fitting to the environment
• 2.2 Preparing the forensic toolkit media
• 2.3 Data collecting from a live system - step by step procedure
• 2.4 Initial data analysis and keyword searching

Sections 2.1, 2.2, and part of 2.3 will be discussed in this article; the remaining steps and some offline procedures will be discussed next month in part two of this article series.

2.1 Fitting to the environment

Before taking any activities on the compromised system we have to create a paper copy of our data collection procedure. An example procedure can be found in chapter three of this article. This procedure helps us to avoid any mistakes during the forensics of an incident. We must make additional notes after every finished step as well as if something goes wrong. Documentation is important, and is something to keep in mind if we plan to take our forensic case to court.

Our next step is to record the results of commands run during our phase of data gathering. From there, we connect a destination host to the same local area network on which we will be sending information from the compromised host. Remember, we are not allowed to write any results on the compromised system. Recording data locally on the compromised host can delete signs of an intrusion. To make less of an impact on a compromised system we have to send all our digital data to a remote, or destination, host. This is one of the most important rules in the forensic analysis process. And once again, as described earlier this is a requirement that is not always easily to fulfill.

If we don't have a forensic toolkit available for install on removable media, now is a good time to prepare it for our compromised system. Using tools from this toolkit we will collect all important data, beginning from the volatile to the less volatile.

The following methods describe a method how to prepare our media into a forensic toolkit.

2.2 Preparing the forensic toolkit media

• Try not to run programs on a compromised system. Why? An intruder could modify system commands (such as a netstat) or system libraries (such as a libproc), rendering the results unreliable. To fulfill this criteria we have to prepare versions of the tools which are compiled statically.
• Try not to run programs which can modify the meta-data of files and directories.
• All results from the investigation must be written to a remote location. To fulfill this criteria we will use the remote host as our destination location. The netcat tool will be used to transfer digital data.
• You have to use tools to calculate the hash values of the digital data. This is a kind of assurance that the digital data has not been altered. A best practice is to make sure that data is not altered and is properly saved on the destination host, so we also will compare hash values calculated on both the source and the destination. Sometimes it's impossible to calculate a hash value on the compromised host -- a good example of this is with main memory. When we try to use md5sum on the /dev/mem device twice in a row, every time the hash value will be different. This happens because every time we load that program into memory (and thus create a new process which needs memory to operate) we change the state of the memory. In our procedure we calculate hash values of digital data immediately after collection is completed, as well as (when possible) on both the source and destination host. To maintain the integrity of all results we will use md5sum tool.
• The required criteria about preventing our tools from writing data to the memory and even the swap space of the compromised system cannot be fulfilled for some steps. This will be discussed in greater detail in section 2.3. For now, let's ensure we have a proper forensic toolkit on removable media, as showin in Table 1.

When we build all above tools successfully, we can copy all of them to our removable media (such as a CD-RW disc).

2.3 Data collecting from a live system - a step by step procedure

Step 1: Take a photograph of a compromised system's screen

This is a kind of screenshot, and of course we have to use a digital camera to do this task. This is a simple step.

Before moving on to step two, mounting our media, let's first think about the impact this next step will have on a compromised system. What will be an effect of our activity? For the moment let's ignore the impact it will have on the compromised system's memory.

It is clear that we have to mount external media into the compromised system. We must use the untrusted mount command to perform this task. This will be probably the sole situation when an untrusted system command is used. If everything will go according to the plan, we will run the rest of the command from the mounted media using tools that we trust.

We also have to check to see what the impact of the mount command will have on the system. I have done some research on a computer, and Table 2 lists the relevant files and directories that are modified.

# strace /bin/mount /mnt/cdrom

Table 2: Files accessed by the mount command.

File	Modified Meta-data by the mount command
/etc/ld.so.cache	atime
/lib/tls/libc.so.6	atime
/usr/lib/locale/locale-archive	atime
/etc/fstab	atime
/etc/mtab*	atime, mtime, ctime
/dev/cdrom	atime
/bin/mount	atime

*We can avoid access to this file by using a "-n" switch.

We can imagine a situation when an intruder modifies the mount command. When someone tries to run this command perhaps a special process, which removes all evidence from the compromised system, is initiated instead of allowing the media to be mounted. Such a process is called a "deadman switch". But let's assume this is not the case, and now go back to the process of data collection.

I suggest that one verify every command that is going to be put on the forensic toolkit media, which later will be used on the compromised system to collect evidence.

We also have to stop and think about potential problems met during the mounting process:

• After putting the media into a drive, the Volume Manager process will mount the media automatically. Which files and directories will be modified? Are these files listed in the table 1?
• Suppose an unknown media is currently mounted on a compromised system. Then the first task is to unmount that media. How should we safely unmount it? I can suggest two solutions. We can use the untrusted unmount command or we can put the trusted unmount command (statically linked) on a floppy disc. Next, we use the untrusted mount command to mount the floppy and then run the trusted unmount command. It is a little bit complicated but effective. We still use only one untrusted command.
• An administrator is logged off or even worse an administrator password is changed by an intruder. When the administrator is logged off we have to login into the system. What files will be accessed or modified during the login process? How many additional processes will be created? If the administrator password was changed what are the other accounts on the system? What volatile data can be collected without access to a shell? Open TCP/UDP ports, current connections, what else?
• Are there other unpredictable problems?

Step 2: Media mounting

Let's go ahead and mount our media, in this case a CD-ROM with our toolkit.

# mount -n /mnt/cdrom

If the mounting process is successful we can start with the most important phase of data collection. Remember, all results generated by trusted commands have to be sent to the remote host. I use the netcat tool and the pipe method to do this. To better differentiate which tasks are performed on which host, all commands run on the compromised host will be prefixed with a (compromised) word in brackets. Commands run on the remote host will be prefixed with a (remote) word in brackets. Consider the following example.

To send information about an actual date of the compromised system into the remote location (the IP address of remote host in this case is 192.168.1.100) we have to open TCP port on the remote host as it follows:

(remote host)# nc -l -p 8888 > date_compromised

Next, on the compromised host we do the following:

(compromised host)# /mnt/cdrom/date | /mnt/cdrom/nc 192.168.1.100 8888 -w 3

To maintain the integrity of digital evidence we calculate the hash value of the collected file and clearly document every step on our paper copy, to document this procedure.

(remote host)# md5sum date_compromised > date_compromised.md5

Sometimes we can generate checksums on the compromised system and send the result to the remote host. A bit more about some of the problems this can cause has been discussed elsewhere in this article.

(compromised host)# /mnt/cdrom/md5sum /etc/fstab | /mnt/cdrom/nc 192.168.1.100 8888 -w 3

Step 3: Current date

The result is presented in the UTC format (Coordinated Universal Time)

(remote)# nc -l -p port > date_compromised
(compromised)# /mnt/cdrom/date -u | /mnt/cdrom/nc (remote) port
(remote)# md5sum date_compromised > date_compromised.md5


Step 4: Cache tables

First, we have to collect information from cache tables because the lifetime of this data, placed in the tables, is very short. I will collect data from the arp and routing tables.

Mac address cache table:
(remote)# nc -l -p port > arp_compromised
(compromised)# /mnt/cdrom/arp -an | /mnt/cdrom/nc (remote) port
(remote)# md5sum arp_compromised > arp_compromised.md5


Kernel route cache table:
(remote)# nc -l -p port > route_compromised
(compromised) # /mnt/cdrom/route -Cn | /mnt/cdrom/nc (remote) port
(remote)#md5sum route_compromised > route_compromised.md5


Step 5: Current, pending connections and open TCP/UDP ports.

Now, we start collecting information about current connections and open TCP/UDP ports. Information about all active raw sockets will be gathered in step eight.

(remote)#nc -l -p port > connections_compromised
(compromised)# /mnt/cdrom/netstat -an | /mnt/cdrom/nc (remote) port
(remote)#md5sum connections_compromised > connections_compromised.md5


We can use the cat command instead of the netstat one in this case. Information about open ports is kept in the /proc pseudo file system (/proc/net/tcp and /proc/net/udp files). Information about current connections is placed in the /proc/net/netstat file. All data in those files are represented in the hex format.

For example: 0100007F:0401 in decimal is 127.0.0.1:1025.

As mentioned before, current connections can be detected by analyzing of the recorded traffic. It is important to note: an easy method of detecting a rootkit, loaded into kernel memory, is when one of its tasks is hiding an open port. We have to scan the compromised host remotely and compare the detected open ports with our result from the netstat command. But this causes a lot of harm and we once again change the state of the compromised system, in step seven I will present an alternate method of detecting hidden LKM based rootkits.

Concluding part one