- Consult your security policy
- If you do not have a security policy
- Consult with management
Depending on how your organization is structured, it may be
important to notify management in order to facilitate
internal coordination of your recovery effort. Also be
aware that intrusions may get the attention of the media.
- Consult with your legal counsel
Before you get started in your recovery, your organization
needs to decide if pursuing a legal investigation is an
option.
Note that the CERT Coordination Center and AusCERT
(Australian Computer Emergency Response Team) are involved
in providing technical assistance and facilitating
communications in response to computer security incidents
involving hosts on the Internet. We do not have legal
expertise and cannot offer legal advice or opinions. For
legal advice, we recommend that you consult with your legal
counsel. Your legal counsel can provide you with legal
options (both civil and criminal) and courses of action
based on you or your organization's needs.
It is up to you how you wish to pursue this incident. You
may wish to secure your systems or to contact law
enforcement to investigate the case.
If you are interested in determining the identity of or
pursuing action against the intruder, we suggest that you
consult your management and legal counsel to see if any
local, state, or federal laws have been violated. Based on
that, you could then choose to contact a law enforcement
agency and see if they wish to pursue an investigation.
We encourage you to discuss the root compromise activity
with your management and legal counsel to answer the
following questions:
- What is your legal status in terms of your ability
to trap intruders or trace connections (i.e., do you have a
login banner stating that connections can be tracked or
traced? See CERT
Advisory CA-92:19, "Keystroke Logging Banner").
- What are your legal responsibilities if your site is
aware of the activity and does not take steps to prevent
it?
- Have any local, state, or federal laws been
violated?
- Should an investigation be pursued?
- Should you report the activity to local, state, or
national) law enforcement?
- Contact law enforcement agencies
In general, if you are interested in pursuing any type of
investigation or legal prosecution, we'd encourage you to
first discuss the activity with your organization's
management and legal counsel and to notify any appropriate
law enforcement agencies (in accordance with any policies
or guidelines at your site).
Keep in mind that unless one of the parties involved
contacts law enforcement, any efforts to trap or trace the
intruder may be to no avail. We suggest you contact law
enforcement before attempting to set a trap or tracing an
intruder.
U.S. sites interested in an investigation can contact their
local Federal Bureau of Investigation (FBI) field office. To
find contact information for your local FBI field office, please
consult your local telephone directory or see the FBI's field
offices web page available at:
http://www.fbi.gov/contact/fo/fo.htm
You may wish to contact the U.S. Secret Service for incidents
involving the following:
- theft or abuse of credit card information (e.g., credit
card fraud, the exchange of credit card information)
- threats to the President of the United States
(e.g., threatening email messages)
- impersonation of the President of the United States
(e.g., the creation of forged email appearing to come from
the President)
To contact the Secret Service:
Secret Service main phone number: +1 202 435-7700
Financial Crimes Division - Electronic Crimes Section
Phone: +1 202 435-5850
Fax: +1 202 435 7607
Non-U.S. sites may want to discuss the activity with their
local law enforcement agency to determine the appropriate steps
that should be taken with regard to pursuing an investigation.
For information on contacting Australian Law Enforcement agencies, please see the Australian High Tech Crime Centre (AHTCC) web site:
http://www.ahtcc.gov.au/how_to_contact.htm
- Notify others within your organization
In addition to notifying management and legal counsel at
your site, you may also need to notify others within
your organization who may be directly affected by your
recovery process (e.g., other administrators or users).
- Document all of the steps you take in
recovering
The importance of documenting every step you take in
recovery can not be overstated. Recovering from a system
compromise can be a hectic and time-consuming process and
hasty decisions are often made. Documenting the steps you
take in recovery will help prevent hasty decisions and give
you a record of all the steps you took to recover, which
you can reference in the future. Documenting the steps you
take in recovery also may be useful if there is a legal
investigation.
- Disconnect compromised system(s) from the
network
To regain control, you will need to disconnect all
compromised machines from your network including dial in
connections. After that you may wish to operate in single
user mode in UNIX or as the local administrator in NT to
ensure that you have complete control of the machine;
however, by rebooting or changing to single user/local
administrator mode, you may lose some useful information
because all processes executing at the time of discovery
will be killed.
Therefore, you may wish to work through steps in section C.5. Look for signs of a network sniffer to
determine if the compromised system is currently running a
network sniffer.
Operating in single user mode on UNIX systems will prevent
users, intruders, and intruder processes from accessing or
changing state on the compromised machine while you are
going through the recovery process.
If you do not disconnect the compromised machine from the
network, you run the risk that the intruder may be
connected to your machine and may be undoing your steps as
you try to recover the machine.
- Copy an image of the compromised
system(s)
Before analyzing the intrusion we encourage you to create a
backup of your system. This will provide a "snapshot" of
the file system at the time that the root compromise was
first discovered. You may need to refer back to this backup
in the future.
If you have an available disk which is the same size and
model as the disk in the compromised system, you can use
the dd command in UNIX to make an exact copy
of the compromised system.
For example, on a Linux system with two SCSI disks, the
following command would make an exact replica of the
compromised system (/dev/sda) to the disk of the same size
and model (/dev/sdb).
# dd if=/dev/sda of=/dev/sdb
Please read the dd man page for more information.
There are many other ways to create a backup of your
system. On NT systems there is no built in command like
dd, but there are a number of third party
applications that will make an image copy of an entire hard
drive.
Creating a low level backup is important in case you ever
need to restore the state of the compromised machine when
it was first discovered. Also, files may be needed for a legal
investigation. Label, sign, and date the backup and keep
the backup in a secure location to maintain integrity of
the data.
- Look for modifications made to system
software and configuration files
Verify all system binaries and configuration files.
When looking for modifications of system software and
configuration files, keep in mind that any tool you are
using on the compromised system to verify the integrity of
binaries and configuration files could itself be
modified. Also keep in mind that the kernel (operating
system) itself could be modified. Because of this, we
encourage you to boot from a trusted kernel and obtain a
known clean copy of any tool you intend to use in analyzing
the intrusion. On UNIX systems you can create a boot disk
and make it write protected to obtain a trustworth kernel.
We urge you to check all of your system binaries thoroughly
against distribution media. We have seen an extensive range
of Trojan horse binaries that have been installed by
intruders.
Some of the binaries which are commonly replaced by Trojan
horses on UNIX systems are: telnet, in.telnetd, login, su,
ftp, ls, ps, netstat, ifconfig, find, du, df, libc, sync,
inetd, and syslogd. Also check any binaries referenced in
/etc/inetd.conf, critical network and system programs, and
shared object libraries.
On NT systems, Trojan horses commonly introduce computer
viruses or "remote administration" programs such as Back
Orifice and NetBus. There have been cases where the system
file that handles internet connectivity was replaced with a
Trojan horse.
Because some Trojan horse programs could have the same
timestamps as the original binaries and give the correct
sum values, we recommend you use
cmp on UNIX systems to make a direct
comparison of the binaries and the original distribution
media.
Alternatively, you can check the MD5 results for either
UNIX or NT on suspect binaries against a list of MD5
checksums from known good binaries. Ask your vendor if they
make MD5 checksums available for their distribution
binaries.
Additionally, verify your configuration files against
copies that you know to be unchanged.
When inspecting your configuration files on UNIX systems,
you may want to
When inspecting NT systems, you may want to
- check for odd users or group memberships.
- check for changes to registry entries that start
programs at logon or services. (see LISTING 1)
- check for unauthorized hidden shares with the 'net
share' command or Server Manager tool.
- check for processes that you do not identify using
the pulist.exe tool from the NT resource kit or the NT
Task Manager.
- Look for modifications to data
Data on compromised systems is often modified by intruders.
We encourage you to verify the integrity of web pages, ftp
archives, files in users' home directories, and any other
data files on your system.
- Look for tools and data left behind by the
intruder
Intruders will commonly install custom-made tools for
continued monitoring or for access to a compromised
system.
The common classes of files left behind by intruders are
- Network Sniffers
A network sniffer is a utility which will monitor
and log network activity to a file. Intruders
commonly use network sniffers to capture username
and password data that is passed in cleartext over
the network. (see section C.5 below)
Sniffers are more common on UNIX systems, but on NT
systems check for key logging programs.
- Trojan Horse Programs
Trojan horse programs are programs that appear to
perform one function while actually performing a
different function. Intruders use Trojan horse
programs to hide their activity, capture username
and password data, and create backdoors for future
access to a compromised system. (see section C.1 above)
- Backdoors
Backdoor programs are designed to hide itself
inside a target host. The backdoor allows the user
that installed it to access the system without
using normal authorization or vulnerability
exploitation.
- Vulnerability Exploits
A majority of compromises are a result of machines
running vulnerable versions of software. Intruders
often use tools to exploit known vulnerabilities
and gain unauthorized access. These tools are often
left behind on the system in "hidden"
directories.
- Other Intruder Tools
The intruder tools listed above are not intended to
be a conclusive or comprehensive list. There may be
other tools left behind by an intruder. Some of the
other types of tools you may find are tools to
- probe systems for vulnerabilities
- launch widespread probes of many other sites
- launch denial of service attacks
- use your computing and networking resources
- Intruder Tool Output
You may find log files from any number of intruder
tools. These log files may contain information
about other sites involved, vulnerabilities of your
compromised machine(s), and vulnerabilities at other
sites.
We encourage you to search thoroughly for such tools and
output files. Be sure to use a known clean copy of any tool
that you use to search for intruder tools.
When searching for intruder tools on a compromised system
- Look for unexpected ASCII files in the /dev
directory on UNIX systems. Some of the Trojan
binaries rely on configuration files which are
often found in /dev.
- Look very carefully for hidden files or
directories. If an intruder has created a new
account and home directory then there may be hidden
files or directories.
- Look for files or directories with strange names
such as "..." (three dots) or ".. " (two dots and
some whitespace) [UNIX]. Intruders often try and
hide files within such directories. On NT systems,
look for files and directories that closely match
what may appear as a system file (EXPLORE.EXE,
UMGR32.EXE, etc).
- Review log files
Reviewing your log files will help you get a better idea of
how your machine was compromised, what happened during the
compromise, and what remote hosts accessed your machine.
Keep in mind when reviewing any log files from a
compromised machine that any of the logs could have been
modified by the intruder.
On UNIX systems, you may need to look in your
/etc/syslog.conf file to find where syslog is logging
messages. NT systems generally log everything to one of
three logs for NT events, all of which are viewed through
the Event Viewer. Other NT applications such as IIS server
may log to other locations. IIS by default writes logs to
the c:winntsystem32logfiles directory.
Below is a list of some of the more common UNIX log file
names, their function, and what to look for in those
files. Depending on how your system is configured, you may
or may not have the following log files.
- messages
The messages log will contain a wide variety
of information. Look for anomalies in this file.
Anything out of the ordinary should be
inspected. Also, look for events that occurred
around the known time of the intrusion.
- xferlog
If the compromised system has a functioning ftp
server, xferlog will contain log files for
all of the ftp transfers. This may help you
discover what intruder tools have been uploaded
to your system, as well as what information has
been downloaded from your system.
- utmp
This file contains binary information for every
user currently logged in. This file is only useful
to determine who is currently logged in. One way to
access this data is the who command.
- wtmp
Every time a user successfully logs in, logs out, or
your machine reboots, the wtmp file is
modified. This is a binary file; thus, you need to
use a tool to obtain useful information from this
file. One such tool is last. The output from
last will contain a table which
associates user names with login times and the host
name where the connection originated. Checking this
file for suspicious connections (e.g., from
unauthorized hosts) may be
useful in determining other hosts that may have
been involved and
finding what accounts on your system may have been
compromised.
- secure
Some versions of UNIX (RedHat Linux for example)
log tcp wrapper messages to the secure log
file. Every time a connection is established with
one of the services running out of inetd that uses
tcp wrappers, a log message is appended to this log
file. When looking through this log file, look for
anomalies such as services that were accessed that
are not commonly used, or for connections from
unfamiliar hosts.
The common item to look for when reviewing log files is
anything that appears out of the ordinary.
- Look for signs of a network
sniffer
When a system compromise occurs, intruders could
potentially install a network monitoring program on UNIX
systems, commonly called a sniffer (or packet sniffer), to
capture user account and password information. For NT
systems, remote administration programs would be more
commonly used for the same purpose.
The first step to take in determining if a sniffer is installed
on your system is to see if any process currently has any of
your network interfaces in promiscuous mode. If any interface is
in promiscuous mode, then a sniffer could be installed on your
system. Note that detecting promiscuous interfaces will not be
possible if you have rebooted your machine or are operating in
single user mode since your discovery of this intrusion.
There are a couple of tools designed for this purpose.
Keep in mind that some legitimate network monitors and
protocol analyzers will set a network interface in
promiscuous mode. Detecting an interface in promiscuous
mode does not necessarily mean that an intruder's sniffer is
running on a system.
Another issue to consider is that sniffer log files tend to
grow quickly in size. You may want to use utilities such as
df to determine if part of the filesystem is
larger than expected. Remember that df is
often replaced by a Trojan horse program when sniffers are
installed; therefore, be sure to obtain a known clean copy
of that utility if you do use it.
If you find that a packet sniffer has been installed on
your systems, we strongly urge you to examine the output
file from the sniffer to determine what other machines are
at risk. Machines at risk are those that appear in the
destination field of a captured packet, but if passwords
across systems are common or if the source and destination
machines trust each other the source machine will also be
at further risk.
Many common sniffers will log each connection as follows:
-- TCP/IP LOG -- TM: Tue Nov 15 15:12:29 --
PATH: not_at_risk.domain.com(1567) => at_risk.domain.com(telnet)
For sniffer logs of this particular format, you can obtain a list of
affected machines by executing the following command:
% grep PATH: $sniffer_log_file | awk '{print $4}' |
awk -F( '{print $1}'| sort -u
You may need to adjust the command for your particular
case. Also, some sniffers encrypt their logs so they may
not be obvious. Because of this check for files that grow
quickly.
You should be aware that there may be other machines at risk in
addition to the ones that appear in the sniffer log. This may be
because the intruder has obtained previous sniffer logs from your
systems or through other attack methods.
For more information, we encourage you to review CERT
Advisory CA-94:01, available from:
- http://www.cert.org/advisories/CA-1994-01.html
The advisory describes of sniffer activity and suggests
approaches for addressing this problem.
Please send us a list of all hosts you know to be
affected. This will help us determine the scope of the
problem.
If Australian or New Zealand hosts have been involved,
please inform auscert@auscert.org.au.
- Check other systems on your
network
We encourage you to check all of your systems, not just
those that you know to be compromised. In your check
include any systems associated with the compromised system
through shared network-based services (such as NIS and NFS)
or through any method of trust (such as systems in
hosts.equiv or .rhosts files, or a Kerberos server).
In examining other systems on your network, we encourage
you to use our Intruder Detection Checklists:
- http://www.cert.org/tech_tips/intruder_detection_checklist.html
- Windows NT Intruder Detection Checklist
http://www.auscert.org.au/render.html?it=1972
- Check for systems involved or affected at
remote sites
While examining log files, intruder output files,
and any files modified or created during and since the time
of the intrusion, look for information that leads you to
suspect that another site may be linked with the
compromise. We often find that other sites linked to a
compromise (whether upstream or downstream of the
compromise) have often themselves been victims of a
compromise. Therefore it is important that any other
potential victim sites are identified and notified as soon
as possible.