Introduction to Intrusion Analysis

Analyzing a Simple Multi-Host Intrusion

I once wrote a one-week course on Linux/UNIX security. Each student had three systems — Red Hat Linux, OpenBSD, and Solaris. A few of the exercises had to be done on a specific one, like one using iptables that obviously had to run on Linux, or one on Solaris's RBAC solution that had to run on Solaris. However, most of the exercises could be done on any operating system, and some used a combination — use OpenBSD to hijack a connection between Linux and Solaris. It ran on-site at a couple of organizations that used macOS desktops and laptops, and many of the exercises could run there as well.

In one exercise they set up syslog to centrally collect log data. Then, in a following one, they analyzed the result of collecting syslog data from multiple hosts during a sequence of intrusions.

Here is that exercise.

The Log Data

This is the result of using a syslog derivative like Rsyslog to centrally collect messages from several UNIX-family systems. This includes messages from the SSH daemon as well as local authentication services like login. These send slightly different messages on the different platforms. PAM can add its own log messages. Also, they may use different device names — console versus ttyC0 for the physical console, pts/0 versus pty/0 for remote terminals, and so on.

Download the log file

The entries are all classic syslog messages. Date, time, reporting hostname, the name of the reporting process, and then its specifically formatted message content.

The first message, at 08:02:08 on August 25, comes from a host named earth. Its proftpd process, an FTP daemon running as process ID #5352, reports a connection from a host named betelgeuse at IP address 128.46.144.12, successful connection as the anonymous FTP user.

See if you can answer these questions:

Which accounts were involved, on which systems?
Was the perpetrator the person who owned the most abused account, or was it probably someone else?
Where would we need to check for data the intruder has deleted, modified, or created?

What story does the following log extract tell?

Aug 25 08:02:08 earth proftpd[5352]: earth (betelgeuse[128.46.144.12]) - ANON anonymous: Login successful.
Aug 25 08:12:30 earth proftpd[5359]: earth (betelgeuse[128.46.144.12]) - ANON anonymous: Login successful.
Aug 25 08:18:42 earth proftpd[5362]: earth (betelgeuse[128.46.144.12]) - ANON anonymous: Login successful.
Aug 25 08:32:40 mercury login(pam_unix)[2840]: session opened for user mccoy by LOGIN(uid=0)
Aug 25 08:32:40 mercury login -- root[2840]: LOGIN ON /dev/console BY mccoy
Aug 25 08:37:12 neptune proftpd[6680]: neptune (rigel[128.46.144.18]) - ANON anonymous: Login successful.
Aug 25 08:40:44 mars login: LOGIN (spock) ON /dev/ttyC0
Aug 25 08:44:48 earth proftpd[5394]: earth (rigel[128.46.144.18]) - ANON anonymous: Login successful.
Aug 25 08:47:14 mars login: [ID 658745 auth.notice] LOGIN ON /dev/pts/5 FROM mercury,mccoy
Aug 25 08:49:02 jupiter login: LOGIN (scotty) ON /dev/ttyC0
Aug 25 08:50:12 neptune proftpd[6682]: neptune (12-202-90-251.comcast.net.[12.202.90.251]) - ANON anonymous: Login successful.
Aug 25 08:52:45 neptune proftpd[6685]: neptune (chello213047120178.9.11.vie.surfer.at[213.47.120.178]) - ANON anonymous: Login successful.
Aug 25 08:55:12 mercury sshd[7156]: Accepted password for scotty from jupiter port 23200 ssh2
Aug 25 08:55:21 neptune proftpd[6688]: neptune (p7051.tokyo.ocn.ne.jp[222.146.167.51]) - ANON anonymous: Login successful.
Aug 25 08:56:28 venus sshd[9914]: Accepted password for scotty from jupiter port 23208 ssh2
Aug 25 08:56:50 pluto login: LOGIN (sulu) ON /dev/ttyC0
Aug 25 08:57:01 earth sshd[6860]: Accepted password for scotty from jupiter port 23216 ssh2
Aug 25 08:57:31 mars sshd[4829]: Accepted password for scotty from jupiter port 23222 ssh2
Aug 25 08:58:08 mars sshd[4355]: Accepted password for scotty from jupiter port 23248 ssh2
Aug 25 08:58:11 neptune proftpd[6690]: neptune (d216-232-217-253.bchsia.telus.net[216.232.217.253]) - ANON anonymous: Login successful.
Aug 25 08:58:19 mars sshd[4355]: Accepted password for scotty from jupiter port 23251 ssh2
Aug 25 08:58:33 saturn sshd[4355]: Accepted password for scotty from jupiter port 23259 ssh2
Aug 25 08:58:55 uranus sshd[5491]: Accepted password for scotty from jupiter port 23261 ssh2
Aug 25 08:59:18 neptune sshd[5491]: Accepted password for scotty from jupiter port 23267 ssh2
Aug 25 08:59:30 pluto sshd[5491]: Accepted password for scotty from jupiter port 23271 ssh2
Aug 25 09:02:22 earth proftpd[5398]: earth (24-205-176-91.wc-eres.charterpipeline.net[24.205.176.91]) - ANON anonymous: Login successful.
Aug 25 09:12:21 earth sshd[6528]: Accepted password for root from mercury port 32784 ssh2
Aug 25 09:30:12 neptune proftpd[6693]: neptune (d150-42-147.home.cgocable.net[24.150.42.147]) - ANON anonymous: Login successful.
Aug 25 09:33:40 neptune proftpd[6697]: neptune (cpe-066-057-154-036.nc.rr.com[66.57.154.36]) - ANON anonymous: Login successful.
Aug 25 09:34:38 earth proftpd[5400]: earth (p4003-ipad01kamokounan.kagoshima.ocn.ne.jp[218.224.62.3]) - ANON anonymous: Login successful.
Aug 25 09:47:19 neptune login: [ID 658745 auth.crit] REPEATED LOGIN FAILURES ON /dev/console, chekov
Aug 25 09:47:21 earth proftpd[5401]: earth (cp479471-a.schoo1.lb.home.nl[217.123.92.217]) - ANON anonymous: Login successful.
Aug 25 09:48:04 neptune login: LOGIN (chekov) ON /dev/console
Aug 25 09:48:51 neptune proftpd[6699]: neptune (61-222-148-235.HINET-IP.hinet.net[61.222.148.235]) - ANON anonymous: Login successful.
Aug 25 09:49:00 neptune proftpd[6701]: neptune (adsl-68-22-2-35.dsl.sfldmi.ameritech.net[68.22.2.35]) - ANON anonymous: Login successful.
Aug 25 09:49:09 neptune su: [auth.crit] 'su root' failed for chekov on /dev/console
Aug 25 09:49:18 neptune su: [auth.crit] 'su root' failed for chekov on /dev/console
Aug 25 09:49:24 earth proftpd[5412]: earth (AAmiens-107-1-18-247.w81-48.abo.wanadoo.fr[81.48.124.247]) - ANON anonymous: Login successful.
Aug 25 09:49:29 neptune su: [auth.crit] 'su root' failed for chekov on /dev/console
Aug 25 09:51:57 earth proftpd[5448]: earth (mail.demarcation.org.za[196.26.55.132]) - ANON anonymous: Login successful.
Aug 25 09:52:21 saturn sshd[4355]: Accepted password for chekov from neptune port 32934 ssh2
Aug 25 09:52:39 neptune proftpd[6723]: neptune (65-86-133-123.client.dsl.net[65.86.133.123]) - ANON anonymous: Login successful.
Aug 25 09:52:40 saturn su(pam_unix)[5552]: authentication failure; logname=chekov uid=1072 euid=0 tty= ruser=chekov rhost=  user=root
Aug 25 09:52:44 saturn su(pam_unix)[5553]: authentication failure; logname=chekov uid=1072 euid=0 tty= ruser=chekov rhost=  user=root
Aug 25 09:52:45 earth proftpd[5453]: earth (cmstore.cellmania.com[207.215.202.38]) - ANON anonymous: Login successful.
Aug 25 09:52:48 earth sshd[5423]: Accepted password for sulu from pluto port 32984 ssh2
Aug 25 09:52:51 saturn su(pam_unix)[5554]: authentication failure; logname=chekov uid=1072 euid=0 tty= ruser=chekov rhost=  user=root
Aug 25 09:52:59 saturn su(pam_unix)[5554]: authentication failure; logname=chekov uid=1072 euid=0 tty= ruser=chekov rhost=  user=root
Aug 25 09:54:11 earth proftpd[5485]: earth (03-035.195.popsite.net[64.24.43.35]) - ANON anonymous: Login successful.
Aug 25 09:57:04 neptune proftpd[6732]: neptune (i-195-137-73-104.freedom2surf.net[195.137.73.104]) - ANON anonymous: Login successful.
Aug 25 09:58:48 venus sshd[4355]: Accepted password for chekov from neptune port 32942 ssh2
Aug 25 09:58:51 neptune proftpd[6744]: neptune (pool-141-158-99-222.pitt.east.verizon.net[141.158.99.222]) - ANON anonymous: Login successful.
Aug 25 09:58:53 venus su(pam_unix)[6423]: authentication failure; logname=chekov uid=1072 euid=0 tty= ruser=chekov rhost=  user=root
Aug 25 09:59:00 neptune proftpd[6750]: neptune (ool-44c4c154.dyn.optonline.net[68.196.193.84]) - ANON anonymous: Login successful.
Aug 25 09:59:04 venus su(pam_unix)[6424]: authentication failure; logname=chekov uid=1072 euid=0 tty= ruser=chekov rhost=  user=root
Aug 25 09:59:15 venus su(pam_unix)[6425]: authentication failure; logname=chekov uid=1072 euid=0 tty= ruser=chekov rhost=  user=root
Aug 25 09:59:17 earth proftpd[5491]: earth (ramonstaal.demon.nl[82.161.2.126]) - ANON anonymous: Login successful.
Aug 25 09:59:24 earth proftpd[5499]: earth (pcp08153519pcs.benslm01.pa.comcast.net[68.42.11.204]) - ANON anonymous: Login successful.
Aug 25 09:59:25 venus su(pam_unix)[6426]: session opened for user mccoy by chekov (uid=1072)
Aug 25 10:02:44 earth login(pam_unix)[2950]: session opened for user kirk by LOGIN(uid=0)
Aug 25 10:02:44 earth login -- root[2950]: LOGIN ON /dev/console BY kirk
Aug 25 10:04:09 neptune proftpd[6756]: neptune (211-232-104-75.intertns.com[211.232.104.75]) - ANON anonymous: Login successful.
Aug 25 10:18:03 venus login: LOGIN (uhura) ON /dev/ttyC0
Aug 25 10:19:43 neptune proftpd[6758]: neptune (115-122-37-213.libre.auna.net[213.37.122.115]) - ANON anonymous: Login successful.
Aug 25 10:21:28 earth proftpd[5503]: earth (modemcable051.173-201-24.mc.videotron.ca[24.201.173.51]) - ANON anonymous: Login successful.
Aug 25 10:27:48 earth proftpd[5509]: earth (host2-179.pool8019.interbusiness.it[80.19.179.2]) - ANON anonymous: Login successful.
Aug 25 10:29:21 mercury sshd[8488]: Accepted password for uhura from venus port 24325 ssh2
Aug 25 10:33:01 earth sshd[5672]: Accepted password for sulu from pluto port 33440 ssh2
Aug 25 10:39:17 neptune proftpd[6780]: neptune (cr200716242.cable.net.co[200.71.62.42]) - ANON anonymous: Login successful.
Aug 25 10:41:54 jupiter sshd[8945]: Accepted password for uhura from venus port 24354 ssh2
Aug 25 10:48:05 jupiter sshd[9024]: Accepted password for kirk from earth port 43449 ssh2
Aug 25 11:05:48 saturn sshd[9024]: Accepted password for scotty from jupiter port 49453 ssh2
Aug 25 11:14:13 neptune proftpd[6789]: neptune (MTL-HSE-ppp185384.qc.sympatico.ca[65.94.163.164]) - ANON anonymous: Login successful.
Aug 25 11:17:42 earth proftpd[5514]: earth (crl7.library.miskatonic.edu[207.155.245.196]) - ANON anonymous: Login successful.
Aug 25 11:20:58 neptune proftpd[6794]: neptune (dialup-4.227.106.210.Dial1.Dallas1.Level3.net[4.227.106.210]) - ANON anonymous: Login successful.
Aug 25 11:28:13 earth sshd[9891]: Accepted password for spock from mars port 43584 ssh2
Aug 25 11:30:29 earth proftpd[5518]: earth (crl7.library.miskatonic.edu[207.155.245.196]) - ANON anonymous: Login successful.
Aug 25 11:41:50 earth proftpd[5518]: earth (crl7.library.miskatonic.edu[207.155.245.196]) - ANON anonymous: Login successful.
Aug 25 11:54:19 earth sshd[5688]: Accepted password for sulu from pluto port 33707 ssh2
Aug 25 11:58:02 neptune proftpd[6802]: neptune (i220-99-232-195.s02.a035.ap.plala.or.jp[220.99.232.195]) - ANON anonymous: Login successful.

Continue scrolling down to see an analysis.

Spoiler alert: Answers appear below.

Getting Started

The log starts with some anonymous FTP events, which continue on down through the file. Oh no, anonymous, they didn't really log in! Don't panic, this isn't really a problem.

Anonymous FTP can be a very reasonable way to provide browser access to a large archive. You connect via FTP, give anonymous or just ftp as a user name, and anything containing an "@" character as a password. This is what browsers do with an ftp:// URL. Yes, it's now somewhat old-fashioned, but there's nothing inherently insecure about a public FTP archive.

$ grep -i anonymous syslog.txt
Aug 25 08:02:08 earth proftpd[5352]: earth (betelgeuse[128.46.144.12]) - ANON anonymous: Login successful.
Aug 25 08:12:30 earth proftpd[5359]: earth (betelgeuse[128.46.144.12]) - ANON anonymous: Login successful.
Aug 25 08:18:42 earth proftpd[5362]: earth (betelgeuse[128.46.144.12]) - ANON anonymous: Login successful.
Aug 25 08:37:12 neptune proftpd[6680]: neptune (rigel[128.46.144.18]) - ANON anonymous: Login successful.
Aug 25 08:44:48 earth proftpd[5394]: earth (rigel[128.46.144.18]) - ANON anonymous: Login successful.
Aug 25 08:50:12 neptune proftpd[6682]: neptune (12-202-90-251.comcast.net.[12.202.90.251]) - ANON anonymous: Login successful.
Aug 25 08:52:45 neptune proftpd[6685]: neptune (chello213047120178.9.11.vie.surfer.at[213.47.120.178]) - ANON anonymous: Login successful.
Aug 25 08:55:21 neptune proftpd[6688]: neptune (p7051.tokyo.ocn.ne.jp[222.146.167.51]) - ANON anonymous: Login successful.
Aug 25 08:58:11 neptune proftpd[6690]: neptune (d216-232-217-253.bchsia.telus.net[216.232.217.253]) - ANON anonymous: Login successful.
Aug 25 09:02:22 earth proftpd[5398]: earth (24-205-176-91.wc-eres.charterpipeline.net[24.205.176.91]) - ANON anonymous: Login successful.
Aug 25 09:30:12 neptune proftpd[6693]: neptune (d150-42-147.home.cgocable.net[24.150.42.147]) - ANON anonymous: Login successful.
Aug 25 09:33:40 neptune proftpd[6697]: neptune (cpe-066-057-154-036.nc.rr.com[66.57.154.36]) - ANON anonymous: Login successful.
Aug 25 09:34:38 earth proftpd[5400]: earth (p4003-ipad01kamokounan.kagoshima.ocn.ne.jp[218.224.62.3]) - ANON anonymous: Login successful.
Aug 25 09:47:21 earth proftpd[5401]: earth (cp479471-a.schoo1.lb.home.nl[217.123.92.217]) - ANON anonymous: Login successful.
Aug 25 09:48:51 neptune proftpd[6699]: neptune (61-222-148-235.HINET-IP.hinet.net[61.222.148.235]) - ANON anonymous: Login successful.
Aug 25 09:49:00 neptune proftpd[6701]: neptune (adsl-68-22-2-35.dsl.sfldmi.ameritech.net[68.22.2.35]) - ANON anonymous: Login successful.
Aug 25 09:49:24 earth proftpd[5412]: earth (AAmiens-107-1-18-247.w81-48.abo.wanadoo.fr[81.48.124.247]) - ANON anonymous: Login successful.
Aug 25 09:51:57 earth proftpd[5448]: earth (mail.demarcation.org.za[196.26.55.132]) - ANON anonymous: Login successful.
Aug 25 09:52:39 neptune proftpd[6723]: neptune (65-86-133-123.client.dsl.net[65.86.133.123]) - ANON anonymous: Login successful.
Aug 25 09:52:45 earth proftpd[5453]: earth (cmstore.cellmania.com[207.215.202.38]) - ANON anonymous: Login successful.
Aug 25 09:54:11 earth proftpd[5485]: earth (03-035.195.popsite.net[64.24.43.35]) - ANON anonymous: Login successful.
Aug 25 09:57:04 neptune proftpd[6732]: neptune (i-195-137-73-104.freedom2surf.net[195.137.73.104]) - ANON anonymous: Login successful.
Aug 25 09:58:51 neptune proftpd[6744]: neptune (pool-141-158-99-222.pitt.east.verizon.net[141.158.99.222]) - ANON anonymous: Login successful.
Aug 25 09:59:00 neptune proftpd[6750]: neptune (ool-44c4c154.dyn.optonline.net[68.196.193.84]) - ANON anonymous: Login successful.
Aug 25 09:59:17 earth proftpd[5491]: earth (ramonstaal.demon.nl[82.161.2.126]) - ANON anonymous: Login successful.
Aug 25 09:59:24 earth proftpd[5499]: earth (pcp08153519pcs.benslm01.pa.comcast.net[68.42.11.204]) - ANON anonymous: Login successful.
Aug 25 10:04:09 neptune proftpd[6756]: neptune (211-232-104-75.intertns.com[211.232.104.75]) - ANON anonymous: Login successful.
Aug 25 10:19:43 neptune proftpd[6758]: neptune (115-122-37-213.libre.auna.net[213.37.122.115]) - ANON anonymous: Login successful.
Aug 25 10:21:28 earth proftpd[5503]: earth (modemcable051.173-201-24.mc.videotron.ca[24.201.173.51]) - ANON anonymous: Login successful.
Aug 25 10:27:48 earth proftpd[5509]: earth (host2-179.pool8019.interbusiness.it[80.19.179.2]) - ANON anonymous: Login successful.
Aug 25 10:39:17 neptune proftpd[6780]: neptune (cr200716242.cable.net.co[200.71.62.42]) - ANON anonymous: Login successful.
Aug 25 11:14:13 neptune proftpd[6789]: neptune (MTL-HSE-ppp185384.qc.sympatico.ca[65.94.163.164]) - ANON anonymous: Login successful.
Aug 25 11:17:42 earth proftpd[5514]: earth (crl7.library.miskatonic.edu[207.155.245.196]) - ANON anonymous: Login successful.
Aug 25 11:20:58 neptune proftpd[6794]: neptune (dialup-4.227.106.210.Dial1.Dallas1.Level3.net[4.227.106.210]) - ANON anonymous: Login successful.
Aug 25 11:30:29 earth proftpd[5518]: earth (crl7.library.miskatonic.edu[207.155.245.196]) - ANON anonymous: Login successful.
Aug 25 11:41:50 earth proftpd[5518]: earth (crl7.library.miskatonic.edu[207.155.245.196]) - ANON anonymous: Login successful.
Aug 25 11:58:02 neptune proftpd[6802]: neptune (i220-99-232-195.s02.a035.ap.plala.or.jp[220.99.232.195]) - ANON anonymous: Login successful.

Soon after that, there is a series of remote logins by user scotty who first logs in on the console of jupiter at 08:49:02 and soon makes a series of SSH connections to other hosts. This alarms some people, but it seems reasonable. He sits at the jupiter workstation, and he's checking or updating data on several other machines.

$ grep scotty syslog.txt
Aug 25 08:49:02 jupiter login: LOGIN (scotty) ON /dev/ttyC0
Aug 25 08:55:12 mercury sshd[7156]: Accepted password for scotty from jupiter port 23200 ssh2
Aug 25 08:56:28 venus sshd[9914]: Accepted password for scotty from jupiter port 23208 ssh2
Aug 25 08:57:01 earth sshd[6860]: Accepted password for scotty from jupiter port 23216 ssh2
Aug 25 08:57:31 mars sshd[4829]: Accepted password for scotty from jupiter port 23222 ssh2
Aug 25 08:58:08 mars sshd[4355]: Accepted password for scotty from jupiter port 23248 ssh2
Aug 25 08:58:19 mars sshd[4355]: Accepted password for scotty from jupiter port 23251 ssh2
Aug 25 08:58:33 saturn sshd[4355]: Accepted password for scotty from jupiter port 23259 ssh2
Aug 25 08:58:55 uranus sshd[5491]: Accepted password for scotty from jupiter port 23261 ssh2
Aug 25 08:59:18 neptune sshd[5491]: Accepted password for scotty from jupiter port 23267 ssh2
Aug 25 08:59:30 pluto sshd[5491]: Accepted password for scotty from jupiter port 23271 ssh2
Aug 25 11:05:48 saturn sshd[9024]: Accepted password for scotty from jupiter port 49453 ssh2

All but one happen within a 4 minute and 18 second period starting at 08:55:12, just over six minutes after the initial login, and ending at 08:59:30.

Again, there is nothing alarming about that pattern.

Suspicious Activity Begins

The possibly suspicious activity starts at 09:47:19 on host neptune. Let's extract just one user's activity on that host.

$ grep 'neptune.*chekov' syslog.txt
Aug 25 09:47:19 neptune login: [ID 658745 auth.crit] REPEATED LOGIN FAILURES ON /dev/console, chekov
Aug 25 09:48:04 neptune login: LOGIN (chekov) ON /dev/console
[... more lines omitted ...]

Assuming the login program on that system works as usual, it sends a "REPEATED LOGIN FAILURES" message after three consecutive failures. That message and the successful report following at 09:48:04 indicate that it took user chekov from four to six attempts within 45 seconds to log in. Maybe that was caused by typing errors, it's hard to enter your password when you can't see what you're typing. But see what followed that between 09:49:09 and 09:49:29:

$ grep 'neptune.*chekov' syslog.txt
[... initial lines omitted ...]
Aug 25 09:49:09 neptune su: [auth.crit] 'su root' failed for chekov on /dev/console
Aug 25 09:49:18 neptune su: [auth.crit] 'su root' failed for chekov on /dev/console
Aug 25 09:49:29 neptune su: [auth.crit] 'su root' failed for chekov on /dev/console

Look at your keyboard, "D" is next to "S". The command du is very useful and frequently used, and it's very easy to accidentally type su when you meant du. However, three times in a row is suspicious. Let's see what else they got up to.

Tracking That User

The chekov user account is behaving strangely. Let's extract all of that user's activity. We're already looked at the chekov entries up through 09:49:29. Here are those plus all those following it in the log. That user:

Connects via SSH from neptune to saturn at 09:52:21.
Makes 4 unsuccessful guesses at the root password on saturn, starting at 09:52:40. Notice that saturn is running a different flavor of UNIX, as its su program produces a different message format for the same type of event.
Connects via SSH from neptune to venus at 09:58:48.
Makes 4 unsuccessful guesses at the root password on venus, starting at 09:58:53.
Still within that remote connection to venus, user chekov successfully transitions to the mccoy user account at 09:59:25.

$ grep 'chekov' syslog.txt
Aug 25 09:47:19 neptune login: [ID 658745 auth.crit] REPEATED LOGIN FAILURES ON /dev/console, chekov
Aug 25 09:48:04 neptune login: LOGIN (chekov) ON /dev/console
Aug 25 09:49:09 neptune su: [auth.crit] 'su root' failed for chekov on /dev/console
Aug 25 09:49:18 neptune su: [auth.crit] 'su root' failed for chekov on /dev/console
Aug 25 09:49:29 neptune su: [auth.crit] 'su root' failed for chekov on /dev/console
Aug 25 09:52:21 saturn sshd[4355]: Accepted password for chekov from neptune port 32934 ssh2
Aug 25 09:52:40 saturn su(pam_unix)[5552]: authentication failure; logname=chekov uid=1072 euid=0 tty= ruser=chekov rhost=  user=root
Aug 25 09:52:44 saturn su(pam_unix)[5553]: authentication failure; logname=chekov uid=1072 euid=0 tty= ruser=chekov rhost=  user=root
Aug 25 09:52:51 saturn su(pam_unix)[5554]: authentication failure; logname=chekov uid=1072 euid=0 tty= ruser=chekov rhost=  user=root
Aug 25 09:52:59 saturn su(pam_unix)[5554]: authentication failure; logname=chekov uid=1072 euid=0 tty= ruser=chekov rhost=  user=root
Aug 25 09:58:48 venus sshd[4355]: Accepted password for chekov from neptune port 32942 ssh2
Aug 25 09:58:53 venus su(pam_unix)[6423]: authentication failure; logname=chekov uid=1072 euid=0 tty= ruser=chekov rhost=  user=root
Aug 25 09:59:04 venus su(pam_unix)[6424]: authentication failure; logname=chekov uid=1072 euid=0 tty= ruser=chekov rhost=  user=root
Aug 25 09:59:15 venus su(pam_unix)[6425]: authentication failure; logname=chekov uid=1072 euid=0 tty= ruser=chekov rhost=  user=root
Aug 25 09:59:25 venus su(pam_unix)[6426]: session opened for user mccoy by chekov (uid=1072)

We Probably Know Who Is Not Behind This

It initially took chekov from four to six attempts to log in on the console of neptune. But after that, there was no trouble authenticating with a password over SSH.

If I had to guess, I would say that this isn't being done by the person associated with the chekov account. That user had an obvious password, guessable within four to six guesses, and so some other person is probably doing this.

We Probably Know Who Is Behind This

At 09:59:25 on host venus, user chekov easily transitioned to the user mccoy. No multiple guesses.

My best explanation for what we've seen is that this is being done either by the actual person associated with user mccoy, or by an intruder who subverted the mccoy account some time ago. Frustrated by repeated failures to guess root passwords, user mccoy may have had a "how to hack" file stored in their home directory where chekov couldn't access it. They changed identities with su to read that file, leaving tracks in the log.

How Could We Prove Or Disprove That Theory?

Let's find all the logged activity of the mccoy user, and review the start of the chekov activity.

$ grep 'mccoy' syslog.txt
Aug 25 08:32:40 mercury login(pam_unix)[2840]: session opened for user mccoy by LOGIN(uid=0)
Aug 25 08:32:40 mercury login -- root[2840]: LOGIN ON /dev/console BY mccoy
Aug 25 08:47:14 earth sshd[6860]: Accepted password for mccoy from mercury port 27732 ssh2
Aug 25 09:59:25 venus su(pam_unix)[6426]: session opened for user mccoy by chekov (uid=1072)
$ grep 'chekov' syslog.txt | head -4
Aug 25 09:47:19 neptune login: [ID 658745 auth.crit] REPEATED LOGIN FAILURES ON /dev/console, chekov
Aug 25 09:48:04 neptune login: LOGIN (chekov) ON /dev/console
Aug 25 09:49:09 neptune su: [auth.crit] 'su root' failed for chekov on /dev/console
Aug 25 09:49:18 neptune su: [auth.crit] 'su root' failed for chekov on /dev/console

Someone using the mccoy account logged in on the console of mercury at 08:32:40. Then at 08:47:14, they were still there as they connected via SSH to earth.

Just over one hour later, at 09:47:19, the suspicious activity by chekov started on the console of neptune.

So... How long would it take someone to travel from the physical console of mercury to that of neptune? If it would probably take more than an hour, then the two sets of activity by mccoy were probably done by two different people.

If those two computers were closer together, then we don't know. Maybe the activity by mccoy between 08:32:40 and 08:47:14 was done by one person, and the abuse of the chekov account was done by someone else. Or maybe they were done by the same person. We just can't tell.

What Do We Still Not Know?

We don't see any information about the ends of sessions in this log snippet. Apparently none of the authentication services on any of these hosts log those events. So, we don't know if any of these highly suspicious sessions are still active:
chekov on neptune
chekov on saturn
chekov on venus
chekov as mccoy on venus
Since the first is the origin of remote connections to the other machines, if the first has ended, so have the others.

We also have no information about what the intruder may have done in the way of file access — reading, creating, modifying, or deleting data on any of the above combinations of host and user.

Ready to go Further?

The above syslog file is an artificial one I built. I collected some example success and failure messages for multiple services on Linux, Solaris, and BSD, and used those to create the above sequence. The remote clients of the FTP service are simply copied from the real files.

Analyzing a Real Intrusion

Go deeper with this analysis of a real intrusion. It includes actual data from a real intrusion. A graduate student in a laboratory at a major university installed Linux, created an account with an obvious password for the professor, and then neglected to tell the professor. I've changed the IP addresses and full host name, and the exploited user name. Otherwise, it's real data.