Linux / UNIX keyboard.

Intrusion Analysis

Analyzing a Hacked Linux System

This is an analysis of multiple intrusions into a poorly configured Linux system. A graduate student added a system to a laboratory network and created a user with a very weak password. An attacker broke into that system and started using it to attack a network of systems at General Motors, who immediately detected the attack and reported it to the university. Tracing things back, we will see that the attacker had earlier broken into a system at a German web-hosting company. This is typical: systems are used to attack and exploit other systems.

Background and very brief summary

The host slim was added to a laboratory network at a major U.S. university.

No, the account name was not really julie, the IP address of slim was not really 1.2.3.4, and yes, the full domain name has been removed from this report.

Out of the accounts added, two had very weak passwords: julie and mysql. The account julie was a "Joe Account" — its password was the same as the login name.

The lab was soon contacted by the university's network administrators — other organizations were reporting that slim had been attacking their servers. Sure enough, Dshield had collected seven reports from six targets during a three-day period! Each report was for multiple attacks. Dshield data is based on collecting the world's Snort intrusion detection logs. Some organization directly contacted the university to complain about attacks from slim and the university's network group called the lab. The lab immediately unplugged slim's Ethernet cable but left everything running.

To use Dshield to see if a host has been attacking others:
  http://www.dshield.org/ipinfo.html?ip=1.2.3.4
For more details, including the times and target ports:
  http://www.dshield.org/ipdetails.html?ip=1.2.3.4
Of course, change the IP address to one of interest!

To summarize what had happened:

Both accounts julie and mysql had been compromised, by independent hackers possibly unaware of each other. In at least the first case, the attack came from another exploited system overseas.

The user julie was running a port scanner to find SSH servers, and then running an automated SSH password guesser to try to break accounts on those servers. This included a stealthy backdoor so the hacker could use this system with a lower probability of detection.

The user mysql was running an IRC chat node for a group of hackers led by a Romanian and including some members elsewhere in Europe and at the University of Maine — or at least using or abusing hosts in those locations to connect via IRC!

In the following I walk through the analysis, showing how the story was derived from the evidence.


What is the compromised host doing?

The lab created an account for me on slim, and I immediately ran top. Not knowing what had happened, I was prepared to be very skeptical about what I saw. If they had gotten root privileges, the installation of a root kit would mean that I really couldn't trust anything I saw. I would have to reboot the system from media, like a Knoppix DVD, or maybe a specially-designed forensics DVD, so I could use tools that I trusted.

A simple root kit might replace crucial utilities like top, ps, netstat, and so on. Worse yet, it might replace the shared libraries, subverting any dynamically-linked programs (and most everything on Linux is dynamically linked!). Or worst of all, it might replace kernel modules or even those plus the monolithic kernel. Once you're running the bad guy's kernel, you really can't trust anything.

However, top showed me that a process named pscan2 owned by the user julie was using 100% of the CPU time! Here is what ps showed me:

% ps axuwww | egrep 'PID|julie '
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
julie     5157  0.0  0.0   4056  1192 ?        Ss   Aug10   0:01 SCREEN
julie     5158  0.0  0.1   5572  3172 pts/1    Ss   Aug10   0:00 /bin/bash
julie     5177  0.0  0.0   5636  2572 pts/1    S+   Aug10   0:00 /bin/bash
julie     3901  0.0  0.0   1452    96 ?        S    Aug13   0:00 bash
julie    24890  0.0  0.0   3604  1276 pts/1    S+   10:52   0:00 /bin/bash ./a 130.172
julie    24891  100  0.0   1596   492 pts/1    R+   10:52   2:19 ./pscan2 130.172 22 

Let's use lsof to see what that pscan2 process is doing. Important information in the below output includes:

Here's the output (and no, the IP address for slim is not really 1.2.3.4!). As you can see, it was rather busy on the network....

% lsof -n -p 24891
COMMAND   PID   USER   FD   TYPE     DEVICE      SIZE       NODE NAME
pscan2  24891  julie  cwd    DIR        8,1      4096   26345487 /tmp/.X11-fonts/.. 
pscan2  24891  julie  rtd    DIR        8,1      4096          2 /
pscan2  24891  julie  txt    REG        8,1     21407   26345492 /tmp/.X11-fonts/.. /pscan2
pscan2  24891  julie  mem    REG        0,0                    0 [heap] (stat: No such file or directory)
pscan2  24891  julie  mem    REG        8,1   1248904    1114479 /lib/tls/i686/cmov/libc-2.4.so
pscan2  24891  julie  mem    REG        8,1    105112    1081348 /lib/ld-2.4.so
pscan2  24891  julie    0u   CHR      136,1                    3 /dev/pts/1
pscan2  24891  julie    1u   CHR      136,1                    3 /dev/pts/1
pscan2  24891  julie    2u   CHR      136,1                    3 /dev/pts/1
pscan2  24891  julie    3w   REG        8,1         0   26345493 /tmp/.X11-fonts/.. /130.172.pscan.22
pscan2  24891  julie    4u  IPv4   53754278                  TCP 1.2.3.4:60448->130.172.47.25:ssh (SYN_SENT)
pscan2  24891  julie    5u  IPv4   53754279                  TCP 1.2.3.4:39799->130.172.47.26:ssh (SYN_SENT)
pscan2  24891  julie    6u  IPv4   53754280                  TCP 1.2.3.4:53838->130.172.47.27:ssh (SYN_SENT)
pscan2  24891  julie    7u  IPv4   53754281                  TCP 1.2.3.4:37173->130.172.47.28:ssh (SYN_SENT)
pscan2  24891  julie    8u  IPv4   53754282                  TCP 1.2.3.4:43614->130.172.47.29:ssh (SYN_SENT)
pscan2  24891  julie    9u  IPv4   53754283                  TCP 1.2.3.4:38012->130.172.47.30:ssh (SYN_SENT)
pscan2  24891  julie   10u  IPv4   53754284                  TCP 1.2.3.4:50746->130.172.47.31:ssh (SYN_SENT)
pscan2  24891  julie   11u  IPv4   53754285                  TCP 1.2.3.4:39671->130.172.47.32:ssh (SYN_SENT)
pscan2  24891  julie   12u  IPv4   53754286                  TCP 1.2.3.4:47444->130.172.47.33:ssh (SYN_SENT)
pscan2  24891  julie   13u  IPv4   53754287                  TCP 1.2.3.4:42614->130.172.47.34:ssh (SYN_SENT)
pscan2  24891  julie   14u  IPv4   53754288                  TCP 1.2.3.4:33253->130.172.47.35:ssh (SYN_SENT)
pscan2  24891  julie   15u  IPv4   53754289                  TCP 1.2.3.4:58916->130.172.47.36:ssh (SYN_SENT)
pscan2  24891  julie   16u  IPv4   53754290                  TCP 1.2.3.4:49819->130.172.47.37:ssh (SYN_SENT)
pscan2  24891  julie   17u  IPv4   53754291                  TCP 1.2.3.4:43781->130.172.47.38:ssh (SYN_SENT)
pscan2  24891  julie   18u  IPv4   53754292                  TCP 1.2.3.4:44462->130.172.47.39:ssh (SYN_SENT)
pscan2  24891  julie   19u  IPv4   53754293                  TCP 1.2.3.4:53078->130.172.47.40:ssh (SYN_SENT)
pscan2  24891  julie   20u  IPv4   53754294                  TCP 1.2.3.4:51166->130.172.47.41:ssh (SYN_SENT)
pscan2  24891  julie   21u  IPv4   53754295                  TCP 1.2.3.4:49919->130.172.47.42:ssh (SYN_SENT)
pscan2  24891  julie   22u  IPv4   53754296                  TCP 1.2.3.4:33049->130.172.47.43:ssh (SYN_SENT)
pscan2  24891  julie   23u  IPv4   53754297                  TCP 1.2.3.4:44201->130.172.47.44:ssh (SYN_SENT)
pscan2  24891  julie   24u  IPv4   53754298                  TCP 1.2.3.4:60096->130.172.47.45:ssh (SYN_SENT)
pscan2  24891  julie   25u  IPv4   53754299                  TCP 1.2.3.4:57085->130.172.47.46:ssh (SYN_SENT)
pscan2  24891  julie   26u  IPv4   53754300                  TCP 1.2.3.4:34920->130.172.47.47:ssh (SYN_SENT)
pscan2  24891  julie   27u  IPv4   53754301                  TCP 1.2.3.4:47956->130.172.47.48:ssh (SYN_SENT)
pscan2  24891  julie   28u  IPv4   53754302                  TCP 1.2.3.4:53559->130.172.47.49:ssh (SYN_SENT)
pscan2  24891  julie   29u  IPv4   53754303                  TCP 1.2.3.4:57522->130.172.47.50:ssh (SYN_SENT)
pscan2  24891  julie   30u  IPv4   53754304                  TCP 1.2.3.4:41672->130.172.47.51:ssh (SYN_SENT)
pscan2  24891  julie   31u  IPv4   53754305                  TCP 1.2.3.4:43809->130.172.47.52:ssh (SYN_SENT)
pscan2  24891  julie   32u  IPv4   53754306                  TCP 1.2.3.4:51029->130.172.47.53:ssh (SYN_SENT)
pscan2  24891  julie   33u  IPv4   53754307                  TCP 1.2.3.4:40608->130.172.47.54:ssh (SYN_SENT)
pscan2  24891  julie   34u  IPv4   53754308                  TCP 1.2.3.4:37479->130.172.47.55:ssh (SYN_SENT)
pscan2  24891  julie   35u  IPv4   53754309                  TCP 1.2.3.4:57940->130.172.47.56:ssh (SYN_SENT)
pscan2  24891  julie   36u  IPv4   53754310                  TCP 1.2.3.4:49230->130.172.47.57:ssh (SYN_SENT)
pscan2  24891  julie   37u  IPv4   53754311                  TCP 1.2.3.4:58717->130.172.47.58:ssh (SYN_SENT)
pscan2  24891  julie   38u  IPv4   53754312                  TCP 1.2.3.4:59216->130.172.47.59:ssh (SYN_SENT)
pscan2  24891  julie   39u  IPv4   53754313                  TCP 1.2.3.4:55537->130.172.47.60:ssh (SYN_SENT)
pscan2  24891  julie   40u  IPv4   53754314                  TCP 1.2.3.4:59840->130.172.47.61:ssh (SYN_SENT)
pscan2  24891  julie   41u  IPv4   53754315                  TCP 1.2.3.4:38460->130.172.47.62:ssh (SYN_SENT)
pscan2  24891  julie   42u  IPv4   53754316                  TCP 1.2.3.4:54235->130.172.47.63:ssh (SYN_SENT)
pscan2  24891  julie   43u  IPv4   53754317                  TCP 1.2.3.4:49512->130.172.47.64:ssh (SYN_SENT)
pscan2  24891  julie   44u  IPv4   53754318                  TCP 1.2.3.4:40106->130.172.47.65:ssh (SYN_SENT)
pscan2  24891  julie   45u  IPv4   53754319                  TCP 1.2.3.4:51525->130.172.47.66:ssh (SYN_SENT)
pscan2  24891  julie   46u  IPv4   53754320                  TCP 1.2.3.4:38483->130.172.47.67:ssh (SYN_SENT)
pscan2  24891  julie   47u  IPv4   53754321                  TCP 1.2.3.4:45578->130.172.47.68:ssh (SYN_SENT)
pscan2  24891  julie   48u  IPv4   53754322                  TCP 1.2.3.4:46124->130.172.47.69:ssh (SYN_SENT)
pscan2  24891  julie   49u  IPv4   53754323                  TCP 1.2.3.4:52372->130.172.47.70:ssh (SYN_SENT)
pscan2  24891  julie   50u  IPv4   53754324                  TCP 1.2.3.4:47683->130.172.47.71:ssh (SYN_SENT)
pscan2  24891  julie   51u  IPv4   53754325                  TCP 1.2.3.4:53668->130.172.47.72:ssh (SYN_SENT)
pscan2  24891  julie   52u  IPv4   53753929                  TCP 1.2.3.4:33430->130.172.45.186:ssh (SYN_SENT)
pscan2  24891  julie   53u  IPv4   53753930                  TCP 1.2.3.4:39803->130.172.45.187:ssh (SYN_SENT)
pscan2  24891  julie   54u  IPv4   53753931                  TCP 1.2.3.4:53224->130.172.45.188:ssh (SYN_SENT)
pscan2  24891  julie   55u  IPv4   53753932                  TCP 1.2.3.4:46532->130.172.45.189:ssh (SYN_SENT)
pscan2  24891  julie   56u  IPv4   53753933                  TCP 1.2.3.4:57163->130.172.45.190:ssh (SYN_SENT)
pscan2  24891  julie   57u  IPv4   53753934                  TCP 1.2.3.4:51463->130.172.45.191:ssh (SYN_SENT)
pscan2  24891  julie   58u  IPv4   53753935                  TCP 1.2.3.4:49014->130.172.45.192:ssh (SYN_SENT)
pscan2  24891  julie   59u  IPv4   53753936                  TCP 1.2.3.4:35785->130.172.45.193:ssh (SYN_SENT)
pscan2  24891  julie   60u  IPv4   53753937                  TCP 1.2.3.4:47371->130.172.45.194:ssh (SYN_SENT)
pscan2  24891  julie   61u  IPv4   53753938                  TCP 1.2.3.4:44310->130.172.45.195:ssh (SYN_SENT)
pscan2  24891  julie   62u  IPv4   53753939                  TCP 1.2.3.4:37952->130.172.45.196:ssh (SYN_SENT)
pscan2  24891  julie   63u  IPv4   53753940                  TCP 1.2.3.4:44075->130.172.45.197:ssh (SYN_SENT)
pscan2  24891  julie   64u  IPv4   53753941                  TCP 1.2.3.4:41698->130.172.45.198:ssh (SYN_SENT)
pscan2  24891  julie   65u  IPv4   53753942                  TCP 1.2.3.4:45007->130.172.45.199:ssh (SYN_SENT)
pscan2  24891  julie   66u  IPv4   53753943                  TCP 1.2.3.4:34097->130.172.45.200:ssh (SYN_SENT)
pscan2  24891  julie   67u  IPv4   53753944                  TCP 1.2.3.4:55748->130.172.45.201:ssh (SYN_SENT)
pscan2  24891  julie   68u  IPv4   53753945                  TCP 1.2.3.4:35486->130.172.45.202:ssh (SYN_SENT)
pscan2  24891  julie   69u  IPv4   53753946                  TCP 1.2.3.4:44978->130.172.45.203:ssh (SYN_SENT)
pscan2  24891  julie   70u  IPv4   53753947                  TCP 1.2.3.4:58275->130.172.45.204:ssh (SYN_SENT)
pscan2  24891  julie   71u  IPv4   53753948                  TCP 1.2.3.4:33134->130.172.45.205:ssh (SYN_SENT)
pscan2  24891  julie   72u  IPv4   53753949                  TCP 1.2.3.4:42515->130.172.45.206:ssh (SYN_SENT)
pscan2  24891  julie   73u  IPv4   53753950                  TCP 1.2.3.4:43391->130.172.45.207:ssh (SYN_SENT)
pscan2  24891  julie   74u  IPv4   53753951                  TCP 1.2.3.4:46745->130.172.45.208:ssh (SYN_SENT)
pscan2  24891  julie   75u  IPv4   53753952                  TCP 1.2.3.4:57498->130.172.45.209:ssh (SYN_SENT)
pscan2  24891  julie   76u  IPv4   53753953                  TCP 1.2.3.4:43963->130.172.45.210:ssh (SYN_SENT)
pscan2  24891  julie   77u  IPv4   53753954                  TCP 1.2.3.4:41439->130.172.45.211:ssh (SYN_SENT)
pscan2  24891  julie   78u  IPv4   53753955                  TCP 1.2.3.4:48143->130.172.45.212:ssh (SYN_SENT)
pscan2  24891  julie   79u  IPv4   53753956                  TCP 1.2.3.4:39298->130.172.45.213:ssh (SYN_SENT)
pscan2  24891  julie   80u  IPv4   53753957                  TCP 1.2.3.4:42772->130.172.45.214:ssh (SYN_SENT)
pscan2  24891  julie   81u  IPv4   53753958                  TCP 1.2.3.4:53126->130.172.45.215:ssh (SYN_SENT)
pscan2  24891  julie   82u  IPv4   53753959                  TCP 1.2.3.4:48424->130.172.45.216:ssh (SYN_SENT)
pscan2  24891  julie   83u  IPv4   53753960                  TCP 1.2.3.4:53871->130.172.45.217:ssh (SYN_SENT)
pscan2  24891  julie   84u  IPv4   53753961                  TCP 1.2.3.4:53953->130.172.45.218:ssh (SYN_SENT)
pscan2  24891  julie   85u  IPv4   53753962                  TCP 1.2.3.4:60488->130.172.45.219:ssh (SYN_SENT)
pscan2  24891  julie   86u  IPv4   53753963                  TCP 1.2.3.4:54874->130.172.45.220:ssh (SYN_SENT)
pscan2  24891  julie   87u  IPv4   53753964                  TCP 1.2.3.4:41433->130.172.45.221:ssh (SYN_SENT)
pscan2  24891  julie   88u  IPv4   53753965                  TCP 1.2.3.4:58304->130.172.45.222:ssh (SYN_SENT)
pscan2  24891  julie   89u  IPv4   53753966                  TCP 1.2.3.4:57391->130.172.45.223:ssh (SYN_SENT)
pscan2  24891  julie   90u  IPv4   53753967                  TCP 1.2.3.4:33557->130.172.45.224:ssh (SYN_SENT)
pscan2  24891  julie   91u  IPv4   53753968                  TCP 1.2.3.4:49447->130.172.45.225:ssh (SYN_SENT)
pscan2  24891  julie   92u  IPv4   53753969                  TCP 1.2.3.4:49000->130.172.45.226:ssh (SYN_SENT)
pscan2  24891  julie   93u  IPv4   53753970                  TCP 1.2.3.4:40296->130.172.45.227:ssh (SYN_SENT)
pscan2  24891  julie   94u  IPv4   53753971                  TCP 1.2.3.4:46191->130.172.45.228:ssh (SYN_SENT)
pscan2  24891  julie   95u  IPv4   53753972                  TCP 1.2.3.4:47531->130.172.45.229:ssh (SYN_SENT)
pscan2  24891  julie   96u  IPv4   53753973                  TCP 1.2.3.4:50223->130.172.45.230:ssh (SYN_SENT)
pscan2  24891  julie   97u  IPv4   53753974                  TCP 1.2.3.4:37735->130.172.45.231:ssh (SYN_SENT)
pscan2  24891  julie   98u  IPv4   53753975                  TCP 1.2.3.4:33381->130.172.45.232:ssh (SYN_SENT)
pscan2  24891  julie   99u  IPv4   53753976                  TCP 1.2.3.4:39546->130.172.45.233:ssh (SYN_SENT)
pscan2  24891  julie  100u  IPv4   53753977                  TCP 1.2.3.4:41713->130.172.45.234:ssh (SYN_SENT)
pscan2  24891  julie  101u  IPv4   53753978                  TCP 1.2.3.4:42231->130.172.45.235:ssh (SYN_SENT)
pscan2  24891  julie  102u  IPv4   53753979                  TCP 1.2.3.4:42339->130.172.45.236:ssh (SYN_SENT)
pscan2  24891  julie  103u  IPv4   53753980                  TCP 1.2.3.4:36467->130.172.45.237:ssh (SYN_SENT)
pscan2  24891  julie  104u  IPv4   53753981                  TCP 1.2.3.4:41212->130.172.45.238:ssh (SYN_SENT)
pscan2  24891  julie  105u  IPv4   53753982                  TCP 1.2.3.4:47971->130.172.45.239:ssh (SYN_SENT)
pscan2  24891  julie  106u  IPv4   53753983                  TCP 1.2.3.4:54026->130.172.45.240:ssh (SYN_SENT)
pscan2  24891  julie  107u  IPv4   53753984                  TCP 1.2.3.4:36043->130.172.45.241:ssh (SYN_SENT)
pscan2  24891  julie  108u  IPv4   53753985                  TCP 1.2.3.4:56844->130.172.45.242:ssh (SYN_SENT)
pscan2  24891  julie  109u  IPv4   53753986                  TCP 1.2.3.4:51402->130.172.45.243:ssh (SYN_SENT)
pscan2  24891  julie  110u  IPv4   53753987                  TCP 1.2.3.4:37137->130.172.45.244:ssh (SYN_SENT)
pscan2  24891  julie  111u  IPv4   53753988                  TCP 1.2.3.4:60103->130.172.45.245:ssh (SYN_SENT)
pscan2  24891  julie  112u  IPv4   53753989                  TCP 1.2.3.4:57909->130.172.45.246:ssh (SYN_SENT)
pscan2  24891  julie  113u  IPv4   53753990                  TCP 1.2.3.4:40779->130.172.45.247:ssh (SYN_SENT)
pscan2  24891  julie  114u  IPv4   53753991                  TCP 1.2.3.4:53799->130.172.45.248:ssh (SYN_SENT)
pscan2  24891  julie  115u  IPv4   53753992                  TCP 1.2.3.4:59629->130.172.45.249:ssh (SYN_SENT)
pscan2  24891  julie  116u  IPv4   53753993                  TCP 1.2.3.4:56370->130.172.45.250:ssh (SYN_SENT)
pscan2  24891  julie  117u  IPv4   53753994                  TCP 1.2.3.4:41563->130.172.45.251:ssh (SYN_SENT)
pscan2  24891  julie  118u  IPv4   53753995                  TCP 1.2.3.4:38592->130.172.45.252:ssh (SYN_SENT)
pscan2  24891  julie  119u  IPv4   53753996                  TCP 1.2.3.4:44563->130.172.45.253:ssh (SYN_SENT)
pscan2  24891  julie  120u  IPv4   53753997                  TCP 1.2.3.4:53463->130.172.45.254:ssh (SYN_SENT)
pscan2  24891  julie  121u  IPv4   53753998                  TCP 1.2.3.4:48402->130.172.46.0:ssh (SYN_SENT)
pscan2  24891  julie  122u  IPv4   53753999                  TCP 1.2.3.4:44459->130.172.46.1:ssh (SYN_SENT)
pscan2  24891  julie  123u  IPv4   53754000                  TCP 1.2.3.4:37235->130.172.46.2:ssh (SYN_SENT)
pscan2  24891  julie  124u  IPv4   53754001                  TCP 1.2.3.4:35592->130.172.46.3:ssh (SYN_SENT)
pscan2  24891  julie  125u  IPv4   53754002                  TCP 1.2.3.4:41735->130.172.46.4:ssh (SYN_SENT)
pscan2  24891  julie  126u  IPv4   53754003                  TCP 1.2.3.4:55467->130.172.46.5:ssh (SYN_SENT)
pscan2  24891  julie  127u  IPv4   53754004                  TCP 1.2.3.4:45688->130.172.46.6:ssh (SYN_SENT)
pscan2  24891  julie  128u  IPv4   53754005                  TCP 1.2.3.4:50191->130.172.46.7:ssh (SYN_SENT)
pscan2  24891  julie  129u  IPv4   53754006                  TCP 1.2.3.4:34441->130.172.46.8:ssh (SYN_SENT)
pscan2  24891  julie  130u  IPv4   53754007                  TCP 1.2.3.4:37898->130.172.46.9:ssh (SYN_SENT)
pscan2  24891  julie  131u  IPv4   53754008                  TCP 1.2.3.4:57892->130.172.46.10:ssh (SYN_SENT)
pscan2  24891  julie  132u  IPv4   53754009                  TCP 1.2.3.4:46474->130.172.46.11:ssh (SYN_SENT)
pscan2  24891  julie  133u  IPv4   53754010                  TCP 1.2.3.4:33528->130.172.46.12:ssh (SYN_SENT)
pscan2  24891  julie  134u  IPv4   53754011                  TCP 1.2.3.4:46437->130.172.46.13:ssh (SYN_SENT)
pscan2  24891  julie  135u  IPv4   53754012                  TCP 1.2.3.4:40632->130.172.46.14:ssh (SYN_SENT)
pscan2  24891  julie  136u  IPv4   53754013                  TCP 1.2.3.4:53968->130.172.46.15:ssh (SYN_SENT)
pscan2  24891  julie  137u  IPv4   53754014                  TCP 1.2.3.4:43605->130.172.46.16:ssh (SYN_SENT)
pscan2  24891  julie  138u  IPv4   53754015                  TCP 1.2.3.4:60385->130.172.46.17:ssh (SYN_SENT)
pscan2  24891  julie  139u  IPv4   53754016                  TCP 1.2.3.4:47944->130.172.46.18:ssh (SYN_SENT)
pscan2  24891  julie  140u  IPv4   53754017                  TCP 1.2.3.4:58519->130.172.46.19:ssh (SYN_SENT)
pscan2  24891  julie  141u  IPv4   53754018                  TCP 1.2.3.4:52132->130.172.46.20:ssh (SYN_SENT)
pscan2  24891  julie  142u  IPv4   53754019                  TCP 1.2.3.4:34172->130.172.46.21:ssh (SYN_SENT)
pscan2  24891  julie  143u  IPv4   53754020                  TCP 1.2.3.4:47490->130.172.46.22:ssh (SYN_SENT)
pscan2  24891  julie  144u  IPv4   53754021                  TCP 1.2.3.4:56722->130.172.46.23:ssh (SYN_SENT)
pscan2  24891  julie  145u  IPv4   53754022                  TCP 1.2.3.4:49665->130.172.46.24:ssh (SYN_SENT)
pscan2  24891  julie  146u  IPv4   53754023                  TCP 1.2.3.4:46085->130.172.46.25:ssh (SYN_SENT)
pscan2  24891  julie  147u  IPv4   53754024                  TCP 1.2.3.4:43503->130.172.46.26:ssh (SYN_SENT)
pscan2  24891  julie  148u  IPv4   53754025                  TCP 1.2.3.4:47800->130.172.46.27:ssh (SYN_SENT)
pscan2  24891  julie  149u  IPv4   53754026                  TCP 1.2.3.4:35991->130.172.46.28:ssh (SYN_SENT)
pscan2  24891  julie  150u  IPv4   53754027                  TCP 1.2.3.4:54309->130.172.46.29:ssh (SYN_SENT)
pscan2  24891  julie  151u  IPv4   53754028                  TCP 1.2.3.4:52180->130.172.46.30:ssh (SYN_SENT)
pscan2  24891  julie  152u  IPv4   53754029                  TCP 1.2.3.4:54812->130.172.46.31:ssh (SYN_SENT)
pscan2  24891  julie  153u  IPv4   53754030                  TCP 1.2.3.4:45554->130.172.46.32:ssh (SYN_SENT)
pscan2  24891  julie  154u  IPv4   53754031                  TCP 1.2.3.4:56001->130.172.46.33:ssh (SYN_SENT)
pscan2  24891  julie  155u  IPv4   53754032                  TCP 1.2.3.4:57242->130.172.46.34:ssh (SYN_SENT)
pscan2  24891  julie  156u  IPv4   53754033                  TCP 1.2.3.4:46722->130.172.46.35:ssh (SYN_SENT)
pscan2  24891  julie  157u  IPv4   53754034                  TCP 1.2.3.4:38650->130.172.46.36:ssh (SYN_SENT)
pscan2  24891  julie  158u  IPv4   53754035                  TCP 1.2.3.4:54045->130.172.46.37:ssh (SYN_SENT)
pscan2  24891  julie  159u  IPv4   53754036                  TCP 1.2.3.4:48415->130.172.46.38:ssh (SYN_SENT)
pscan2  24891  julie  160u  IPv4   53754037                  TCP 1.2.3.4:36644->130.172.46.39:ssh (SYN_SENT)
pscan2  24891  julie  161u  IPv4   53754038                  TCP 1.2.3.4:43431->130.172.46.40:ssh (SYN_SENT)
pscan2  24891  julie  162u  IPv4   53754039                  TCP 1.2.3.4:33752->130.172.46.41:ssh (SYN_SENT)
pscan2  24891  julie  163u  IPv4   53754040                  TCP 1.2.3.4:41099->130.172.46.42:ssh (SYN_SENT)
pscan2  24891  julie  164u  IPv4   53754041                  TCP 1.2.3.4:47371->130.172.46.43:ssh (SYN_SENT)
pscan2  24891  julie  165u  IPv4   53754042                  TCP 1.2.3.4:49324->130.172.46.44:ssh (SYN_SENT)
pscan2  24891  julie  166u  IPv4   53754043                  TCP 1.2.3.4:37386->130.172.46.45:ssh (SYN_SENT)
pscan2  24891  julie  167u  IPv4   53754044                  TCP 1.2.3.4:42302->130.172.46.46:ssh (SYN_SENT)
pscan2  24891  julie  168u  IPv4   53754045                  TCP 1.2.3.4:45391->130.172.46.47:ssh (SYN_SENT)
pscan2  24891  julie  169u  IPv4   53754046                  TCP 1.2.3.4:45158->130.172.46.48:ssh (SYN_SENT)
pscan2  24891  julie  170u  IPv4   53754047                  TCP 1.2.3.4:47037->130.172.46.49:ssh (SYN_SENT)
pscan2  24891  julie  171u  IPv4   53754048                  TCP 1.2.3.4:44191->130.172.46.50:ssh (SYN_SENT)
pscan2  24891  julie  172u  IPv4   53754049                  TCP 1.2.3.4:56457->130.172.46.51:ssh (SYN_SENT)
pscan2  24891  julie  173u  IPv4   53754050                  TCP 1.2.3.4:38784->130.172.46.52:ssh (SYN_SENT)
pscan2  24891  julie  174u  IPv4   53754051                  TCP 1.2.3.4:53302->130.172.46.53:ssh (SYN_SENT)
pscan2  24891  julie  175u  IPv4   53754052                  TCP 1.2.3.4:53474->130.172.46.54:ssh (SYN_SENT)
pscan2  24891  julie  176u  IPv4   53754053                  TCP 1.2.3.4:38427->130.172.46.55:ssh (SYN_SENT)
pscan2  24891  julie  177u  IPv4   53754054                  TCP 1.2.3.4:33204->130.172.46.56:ssh (SYN_SENT)
pscan2  24891  julie  178u  IPv4   53754055                  TCP 1.2.3.4:43975->130.172.46.57:ssh (SYN_SENT)
pscan2  24891  julie  179u  IPv4   53754056                  TCP 1.2.3.4:43859->130.172.46.58:ssh (SYN_SENT)
pscan2  24891  julie  180u  IPv4   53754057                  TCP 1.2.3.4:41574->130.172.46.59:ssh (SYN_SENT)
pscan2  24891  julie  181u  IPv4   53754058                  TCP 1.2.3.4:39617->130.172.46.60:ssh (SYN_SENT)
pscan2  24891  julie  182u  IPv4   53754059                  TCP 1.2.3.4:57293->130.172.46.61:ssh (SYN_SENT)
pscan2  24891  julie  183u  IPv4   53754060                  TCP 1.2.3.4:50834->130.172.46.62:ssh (SYN_SENT)
pscan2  24891  julie  184u  IPv4   53754061                  TCP 1.2.3.4:51658->130.172.46.63:ssh (SYN_SENT)
pscan2  24891  julie  185u  IPv4   53754062                  TCP 1.2.3.4:40389->130.172.46.64:ssh (SYN_SENT)
pscan2  24891  julie  186u  IPv4   53754063                  TCP 1.2.3.4:49077->130.172.46.65:ssh (SYN_SENT)
pscan2  24891  julie  187u  IPv4   53754064                  TCP 1.2.3.4:48412->130.172.46.66:ssh (SYN_SENT)
pscan2  24891  julie  188u  IPv4   53754065                  TCP 1.2.3.4:53878->130.172.46.67:ssh (SYN_SENT)
pscan2  24891  julie  189u  IPv4   53754066                  TCP 1.2.3.4:51157->130.172.46.68:ssh (SYN_SENT)
pscan2  24891  julie  190u  IPv4   53754067                  TCP 1.2.3.4:60214->130.172.46.69:ssh (SYN_SENT)
pscan2  24891  julie  191u  IPv4   53754068                  TCP 1.2.3.4:56819->130.172.46.70:ssh (SYN_SENT)
pscan2  24891  julie  192u  IPv4   53754069                  TCP 1.2.3.4:38137->130.172.46.71:ssh (SYN_SENT)
pscan2  24891  julie  193u  IPv4   53754070                  TCP 1.2.3.4:50263->130.172.46.72:ssh (SYN_SENT)
pscan2  24891  julie  194u  IPv4   53754071                  TCP 1.2.3.4:52100->130.172.46.73:ssh (SYN_SENT)
pscan2  24891  julie  195u  IPv4   53754072                  TCP 1.2.3.4:48060->130.172.46.74:ssh (SYN_SENT)
pscan2  24891  julie  196u  IPv4   53754073                  TCP 1.2.3.4:52012->130.172.46.75:ssh (SYN_SENT)
pscan2  24891  julie  197u  IPv4   53754074                  TCP 1.2.3.4:46179->130.172.46.76:ssh (SYN_SENT)
pscan2  24891  julie  198u  IPv4   53754075                  TCP 1.2.3.4:33859->130.172.46.77:ssh (SYN_SENT)
pscan2  24891  julie  199u  IPv4   53754076                  TCP 1.2.3.4:50042->130.172.46.78:ssh (SYN_SENT)
pscan2  24891  julie  200u  IPv4   53754077                  TCP 1.2.3.4:36225->130.172.46.79:ssh (SYN_SENT)
pscan2  24891  julie  201u  IPv4   53754078                  TCP 1.2.3.4:33870->130.172.46.80:ssh (SYN_SENT)
pscan2  24891  julie  202u  IPv4   53754079                  TCP 1.2.3.4:33844->130.172.46.81:ssh (SYN_SENT)
pscan2  24891  julie  203u  IPv4   53754080                  TCP 1.2.3.4:35432->130.172.46.82:ssh (SYN_SENT)
pscan2  24891  julie  204u  IPv4   53754081                  TCP 1.2.3.4:54318->130.172.46.83:ssh (SYN_SENT)
pscan2  24891  julie  205u  IPv4   53754082                  TCP 1.2.3.4:43414->130.172.46.84:ssh (SYN_SENT)
pscan2  24891  julie  206u  IPv4   53754083                  TCP 1.2.3.4:54438->130.172.46.85:ssh (SYN_SENT)
pscan2  24891  julie  207u  IPv4   53754084                  TCP 1.2.3.4:48497->130.172.46.86:ssh (SYN_SENT)
pscan2  24891  julie  208u  IPv4   53754085                  TCP 1.2.3.4:58182->130.172.46.87:ssh (SYN_SENT)
pscan2  24891  julie  209u  IPv4   53754086                  TCP 1.2.3.4:54435->130.172.46.88:ssh (SYN_SENT)
pscan2  24891  julie  210u  IPv4   53754087                  TCP 1.2.3.4:51929->130.172.46.89:ssh (SYN_SENT)
pscan2  24891  julie  211u  IPv4   53754088                  TCP 1.2.3.4:36468->130.172.46.90:ssh (SYN_SENT)
pscan2  24891  julie  212u  IPv4   53754089                  TCP 1.2.3.4:34169->130.172.46.91:ssh (SYN_SENT)
pscan2  24891  julie  213u  IPv4   53754090                  TCP 1.2.3.4:39415->130.172.46.92:ssh (SYN_SENT)
pscan2  24891  julie  214u  IPv4   53754091                  TCP 1.2.3.4:37447->130.172.46.93:ssh (SYN_SENT)
pscan2  24891  julie  215u  IPv4   53754092                  TCP 1.2.3.4:37943->130.172.46.94:ssh (SYN_SENT)
pscan2  24891  julie  216u  IPv4   53754093                  TCP 1.2.3.4:37007->130.172.46.95:ssh (SYN_SENT)
pscan2  24891  julie  217u  IPv4   53754094                  TCP 1.2.3.4:42069->130.172.46.96:ssh (SYN_SENT)
pscan2  24891  julie  218u  IPv4   53754095                  TCP 1.2.3.4:52857->130.172.46.97:ssh (SYN_SENT)
pscan2  24891  julie  219u  IPv4   53754096                  TCP 1.2.3.4:33068->130.172.46.98:ssh (SYN_SENT)
pscan2  24891  julie  220u  IPv4   53754097                  TCP 1.2.3.4:37000->130.172.46.99:ssh (SYN_SENT)
pscan2  24891  julie  221u  IPv4   53754098                  TCP 1.2.3.4:58233->130.172.46.100:ssh (SYN_SENT)
pscan2  24891  julie  222u  IPv4   53754099                  TCP 1.2.3.4:52885->130.172.46.101:ssh (SYN_SENT)
pscan2  24891  julie  223u  IPv4   53754100                  TCP 1.2.3.4:53393->130.172.46.102:ssh (SYN_SENT)
pscan2  24891  julie  224u  IPv4   53754101                  TCP 1.2.3.4:52375->130.172.46.103:ssh (SYN_SENT)
pscan2  24891  julie  225u  IPv4   53754102                  TCP 1.2.3.4:40894->130.172.46.104:ssh (SYN_SENT)
pscan2  24891  julie  226u  IPv4   53754103                  TCP 1.2.3.4:59808->130.172.46.105:ssh (SYN_SENT)
pscan2  24891  julie  227u  IPv4   53754104                  TCP 1.2.3.4:39880->130.172.46.106:ssh (SYN_SENT)
pscan2  24891  julie  228u  IPv4   53754105                  TCP 1.2.3.4:55644->130.172.46.107:ssh (SYN_SENT)
pscan2  24891  julie  229u  IPv4   53754106                  TCP 1.2.3.4:52018->130.172.46.108:ssh (SYN_SENT)
pscan2  24891  julie  230u  IPv4   53754107                  TCP 1.2.3.4:53972->130.172.46.109:ssh (SYN_SENT)
pscan2  24891  julie  231u  IPv4   53754108                  TCP 1.2.3.4:57115->130.172.46.110:ssh (SYN_SENT)
pscan2  24891  julie  232u  IPv4   53754109                  TCP 1.2.3.4:55765->130.172.46.111:ssh (SYN_SENT)
pscan2  24891  julie  233u  IPv4   53754110                  TCP 1.2.3.4:50150->130.172.46.112:ssh (SYN_SENT)
pscan2  24891  julie  234u  IPv4   53754111                  TCP 1.2.3.4:49269->130.172.46.113:ssh (SYN_SENT)
pscan2  24891  julie  235u  IPv4   53754112                  TCP 1.2.3.4:43691->130.172.46.114:ssh (SYN_SENT)
pscan2  24891  julie  236u  IPv4   53754113                  TCP 1.2.3.4:48080->130.172.46.115:ssh (SYN_SENT)
pscan2  24891  julie  237u  IPv4   53754114                  TCP 1.2.3.4:44977->130.172.46.116:ssh (SYN_SENT)
pscan2  24891  julie  238u  IPv4   53754115                  TCP 1.2.3.4:56764->130.172.46.117:ssh (SYN_SENT)
pscan2  24891  julie  239u  IPv4   53754116                  TCP 1.2.3.4:52785->130.172.46.118:ssh (SYN_SENT)
pscan2  24891  julie  240u  IPv4   53754117                  TCP 1.2.3.4:38921->130.172.46.119:ssh (SYN_SENT)
pscan2  24891  julie  241u  IPv4   53754118                  TCP 1.2.3.4:56102->130.172.46.120:ssh (SYN_SENT)
pscan2  24891  julie  242u  IPv4   53754119                  TCP 1.2.3.4:45924->130.172.46.121:ssh (SYN_SENT)
pscan2  24891  julie  243u  IPv4   53754120                  TCP 1.2.3.4:35162->130.172.46.122:ssh (SYN_SENT)
pscan2  24891  julie  244u  IPv4   53754121                  TCP 1.2.3.4:34603->130.172.46.123:ssh (SYN_SENT)
pscan2  24891  julie  245u  IPv4   53754122                  TCP 1.2.3.4:51920->130.172.46.124:ssh (SYN_SENT)
pscan2  24891  julie  246u  IPv4   53754123                  TCP 1.2.3.4:51150->130.172.46.125:ssh (SYN_SENT)
pscan2  24891  julie  247u  IPv4   53754124                  TCP 1.2.3.4:46978->130.172.46.126:ssh (SYN_SENT)
pscan2  24891  julie  248u  IPv4   53754125                  TCP 1.2.3.4:41940->130.172.46.127:ssh (SYN_SENT)
pscan2  24891  julie  249u  IPv4   53754126                  TCP 1.2.3.4:39125->130.172.46.128:ssh (SYN_SENT)
pscan2  24891  julie  250u  IPv4   53754127                  TCP 1.2.3.4:55114->130.172.46.129:ssh (SYN_SENT)
pscan2  24891  julie  251u  IPv4   53754128                  TCP 1.2.3.4:41870->130.172.46.130:ssh (SYN_SENT)
pscan2  24891  julie  252u  IPv4   53754129                  TCP 1.2.3.4:51672->130.172.46.131:ssh (SYN_SENT)
pscan2  24891  julie  253u  IPv4   53754130                  TCP 1.2.3.4:58716->130.172.46.132:ssh (SYN_SENT)
pscan2  24891  julie  254u  IPv4   53754131                  TCP 1.2.3.4:36253->130.172.46.133:ssh (SYN_SENT)
pscan2  24891  julie  255u  IPv4   53754132                  TCP 1.2.3.4:50498->130.172.46.134:ssh (SYN_SENT)
pscan2  24891  julie  256u  IPv4   53754133                  TCP 1.2.3.4:55268->130.172.46.135:ssh (SYN_SENT)
pscan2  24891  julie  257u  IPv4   53754134                  TCP 1.2.3.4:50377->130.172.46.136:ssh (SYN_SENT)
pscan2  24891  julie  258u  IPv4   53754135                  TCP 1.2.3.4:43997->130.172.46.137:ssh (SYN_SENT)
pscan2  24891  julie  259u  IPv4   53754136                  TCP 1.2.3.4:37257->130.172.46.138:ssh (SYN_SENT)
pscan2  24891  julie  260u  IPv4   53754137                  TCP 1.2.3.4:44547->130.172.46.139:ssh (SYN_SENT)
pscan2  24891  julie  261u  IPv4   53754138                  TCP 1.2.3.4:34848->130.172.46.140:ssh (SYN_SENT)
pscan2  24891  julie  262u  IPv4   53754139                  TCP 1.2.3.4:60159->130.172.46.141:ssh (SYN_SENT)
pscan2  24891  julie  263u  IPv4   53754140                  TCP 1.2.3.4:60170->130.172.46.142:ssh (SYN_SENT)
pscan2  24891  julie  264u  IPv4   53754141                  TCP 1.2.3.4:50127->130.172.46.143:ssh (SYN_SENT)
pscan2  24891  julie  265u  IPv4   53754142                  TCP 1.2.3.4:43398->130.172.46.144:ssh (SYN_SENT)
pscan2  24891  julie  266u  IPv4   53754143                  TCP 1.2.3.4:44819->130.172.46.145:ssh (SYN_SENT)
pscan2  24891  julie  267u  IPv4   53754144                  TCP 1.2.3.4:40111->130.172.46.146:ssh (SYN_SENT)
pscan2  24891  julie  268u  IPv4   53754145                  TCP 1.2.3.4:52893->130.172.46.147:ssh (SYN_SENT)
pscan2  24891  julie  269u  IPv4   53754146                  TCP 1.2.3.4:46016->130.172.46.148:ssh (SYN_SENT)
pscan2  24891  julie  270u  IPv4   53754147                  TCP 1.2.3.4:59231->130.172.46.149:ssh (SYN_SENT)
pscan2  24891  julie  271u  IPv4   53754148                  TCP 1.2.3.4:55291->130.172.46.150:ssh (SYN_SENT)
pscan2  24891  julie  272u  IPv4   53754149                  TCP 1.2.3.4:41768->130.172.46.151:ssh (SYN_SENT)
pscan2  24891  julie  273u  IPv4   53754150                  TCP 1.2.3.4:34703->130.172.46.152:ssh (SYN_SENT)
pscan2  24891  julie  274u  IPv4   53754151                  TCP 1.2.3.4:54302->130.172.46.153:ssh (SYN_SENT)
pscan2  24891  julie  275u  IPv4   53754152                  TCP 1.2.3.4:35922->130.172.46.154:ssh (SYN_SENT)
pscan2  24891  julie  276u  IPv4   53754153                  TCP 1.2.3.4:48727->130.172.46.155:ssh (SYN_SENT)
pscan2  24891  julie  277u  IPv4   53754154                  TCP 1.2.3.4:34171->130.172.46.156:ssh (SYN_SENT)
pscan2  24891  julie  278u  IPv4   53754155                  TCP 1.2.3.4:33009->130.172.46.157:ssh (SYN_SENT)
pscan2  24891  julie  279u  IPv4   53754156                  TCP 1.2.3.4:41413->130.172.46.158:ssh (SYN_SENT)
pscan2  24891  julie  280u  IPv4   53754157                  TCP 1.2.3.4:53888->130.172.46.159:ssh (SYN_SENT)
pscan2  24891  julie  281u  IPv4   53754158                  TCP 1.2.3.4:53604->130.172.46.160:ssh (SYN_SENT)
pscan2  24891  julie  282u  IPv4   53754159                  TCP 1.2.3.4:39026->130.172.46.161:ssh (SYN_SENT)
pscan2  24891  julie  283u  IPv4   53754160                  TCP 1.2.3.4:52677->130.172.46.162:ssh (SYN_SENT)
pscan2  24891  julie  284u  IPv4   53754161                  TCP 1.2.3.4:43888->130.172.46.163:ssh (SYN_SENT)
pscan2  24891  julie  285u  IPv4   53754162                  TCP 1.2.3.4:56697->130.172.46.164:ssh (SYN_SENT)
pscan2  24891  julie  286u  IPv4   53754163                  TCP 1.2.3.4:36846->130.172.46.165:ssh (SYN_SENT)
pscan2  24891  julie  287u  IPv4   53754164                  TCP 1.2.3.4:34166->130.172.46.166:ssh (SYN_SENT)
pscan2  24891  julie  288u  IPv4   53754165                  TCP 1.2.3.4:34959->130.172.46.167:ssh (SYN_SENT)
pscan2  24891  julie  289u  IPv4   53754166                  TCP 1.2.3.4:32928->130.172.46.168:ssh (SYN_SENT)
pscan2  24891  julie  290u  IPv4   53754167                  TCP 1.2.3.4:58302->130.172.46.169:ssh (SYN_SENT)
pscan2  24891  julie  291u  IPv4   53754168                  TCP 1.2.3.4:41929->130.172.46.170:ssh (SYN_SENT)
pscan2  24891  julie  292u  IPv4   53754169                  TCP 1.2.3.4:52624->130.172.46.171:ssh (SYN_SENT)
pscan2  24891  julie  293u  IPv4   53754170                  TCP 1.2.3.4:42406->130.172.46.172:ssh (SYN_SENT)
pscan2  24891  julie  294u  IPv4   53754171                  TCP 1.2.3.4:45145->130.172.46.173:ssh (SYN_SENT)
pscan2  24891  julie  295u  IPv4   53754172                  TCP 1.2.3.4:55858->130.172.46.174:ssh (SYN_SENT)
pscan2  24891  julie  296u  IPv4   53754173                  TCP 1.2.3.4:43078->130.172.46.175:ssh (SYN_SENT)
pscan2  24891  julie  297u  IPv4   53754174                  TCP 1.2.3.4:60338->130.172.46.176:ssh (SYN_SENT)
pscan2  24891  julie  298u  IPv4   53754175                  TCP 1.2.3.4:51955->130.172.46.177:ssh (SYN_SENT)
pscan2  24891  julie  299u  IPv4   53754176                  TCP 1.2.3.4:36965->130.172.46.178:ssh (SYN_SENT)
pscan2  24891  julie  300u  IPv4   53754177                  TCP 1.2.3.4:37328->130.172.46.179:ssh (SYN_SENT)
pscan2  24891  julie  301u  IPv4   53754178                  TCP 1.2.3.4:52466->130.172.46.180:ssh (SYN_SENT)
pscan2  24891  julie  302u  IPv4   53754179                  TCP 1.2.3.4:47474->130.172.46.181:ssh (SYN_SENT)
pscan2  24891  julie  303u  IPv4   53754180                  TCP 1.2.3.4:45081->130.172.46.182:ssh (SYN_SENT)
pscan2  24891  julie  304u  IPv4   53754181                  TCP 1.2.3.4:54442->130.172.46.183:ssh (SYN_SENT)
pscan2  24891  julie  305u  IPv4   53754182                  TCP 1.2.3.4:48874->130.172.46.184:ssh (SYN_SENT)
pscan2  24891  julie  306u  IPv4   53754183                  TCP 1.2.3.4:40780->130.172.46.185:ssh (SYN_SENT)
pscan2  24891  julie  307u  IPv4   53754184                  TCP 1.2.3.4:57972->130.172.46.186:ssh (SYN_SENT)
pscan2  24891  julie  308u  IPv4   53754185                  TCP 1.2.3.4:35467->130.172.46.187:ssh (SYN_SENT)
pscan2  24891  julie  309u  IPv4   53754186                  TCP 1.2.3.4:35736->130.172.46.188:ssh (SYN_SENT)
pscan2  24891  julie  310u  IPv4   53754187                  TCP 1.2.3.4:39229->130.172.46.189:ssh (SYN_SENT)
pscan2  24891  julie  311u  IPv4   53754188                  TCP 1.2.3.4:45278->130.172.46.190:ssh (SYN_SENT)
pscan2  24891  julie  312u  IPv4   53754189                  TCP 1.2.3.4:60249->130.172.46.191:ssh (SYN_SENT)
pscan2  24891  julie  313u  IPv4   53754190                  TCP 1.2.3.4:50456->130.172.46.192:ssh (SYN_SENT)
pscan2  24891  julie  314u  IPv4   53754191                  TCP 1.2.3.4:48295->130.172.46.193:ssh (SYN_SENT)
pscan2  24891  julie  315u  IPv4   53754192                  TCP 1.2.3.4:34045->130.172.46.194:ssh (SYN_SENT)
pscan2  24891  julie  316u  IPv4   53754193                  TCP 1.2.3.4:42462->130.172.46.195:ssh (SYN_SENT)
pscan2  24891  julie  317u  IPv4   53754194                  TCP 1.2.3.4:34303->130.172.46.196:ssh (SYN_SENT)
pscan2  24891  julie  318u  IPv4   53754195                  TCP 1.2.3.4:60534->130.172.46.197:ssh (SYN_SENT)
pscan2  24891  julie  319u  IPv4   53754196                  TCP 1.2.3.4:37171->130.172.46.198:ssh (SYN_SENT)
pscan2  24891  julie  320u  IPv4   53754197                  TCP 1.2.3.4:52029->130.172.46.199:ssh (SYN_SENT)
pscan2  24891  julie  321u  IPv4   53754198                  TCP 1.2.3.4:50123->130.172.46.200:ssh (SYN_SENT)
pscan2  24891  julie  322u  IPv4   53754199                  TCP 1.2.3.4:53467->130.172.46.201:ssh (SYN_SENT)
pscan2  24891  julie  323u  IPv4   53754326                  TCP 1.2.3.4:47489->130.172.47.73:ssh (SYN_SENT)
pscan2  24891  julie  324u  IPv4   53754327                  TCP 1.2.3.4:56282->130.172.47.74:ssh (SYN_SENT)
pscan2  24891  julie  325u  IPv4   53754328                  TCP 1.2.3.4:38961->130.172.47.75:ssh (SYN_SENT)
pscan2  24891  julie  326u  IPv4   53754200                  TCP 1.2.3.4:59121->130.172.46.202:ssh (SYN_SENT)
pscan2  24891  julie  327u  IPv4   53754201                  TCP 1.2.3.4:39879->130.172.46.203:ssh (SYN_SENT)
pscan2  24891  julie  328u  IPv4   53754202                  TCP 1.2.3.4:35346->130.172.46.204:ssh (SYN_SENT)
pscan2  24891  julie  329u  IPv4   53754203                  TCP 1.2.3.4:56256->130.172.46.205:ssh (SYN_SENT)
pscan2  24891  julie  330u  IPv4   53754204                  TCP 1.2.3.4:41189->130.172.46.206:ssh (SYN_SENT)
pscan2  24891  julie  331u  IPv4   53754205                  TCP 1.2.3.4:51392->130.172.46.207:ssh (SYN_SENT)
pscan2  24891  julie  332u  IPv4   53754206                  TCP 1.2.3.4:43464->130.172.46.208:ssh (SYN_SENT)
pscan2  24891  julie  333u  IPv4   53754207                  TCP 1.2.3.4:36370->130.172.46.209:ssh (SYN_SENT)
pscan2  24891  julie  334u  IPv4   53754208                  TCP 1.2.3.4:41357->130.172.46.210:ssh (SYN_SENT)
pscan2  24891  julie  335u  IPv4   53754209                  TCP 1.2.3.4:53765->130.172.46.211:ssh (SYN_SENT)
pscan2  24891  julie  336u  IPv4   53754210                  TCP 1.2.3.4:46289->130.172.46.212:ssh (SYN_SENT)
pscan2  24891  julie  337u  IPv4   53754211                  TCP 1.2.3.4:45903->130.172.46.213:ssh (SYN_SENT)
pscan2  24891  julie  338u  IPv4   53754212                  TCP 1.2.3.4:60710->130.172.46.214:ssh (SYN_SENT)
pscan2  24891  julie  339u  IPv4   53754213                  TCP 1.2.3.4:41585->130.172.46.215:ssh (SYN_SENT)
pscan2  24891  julie  340u  IPv4   53754214                  TCP 1.2.3.4:57765->130.172.46.216:ssh (SYN_SENT)
pscan2  24891  julie  341u  IPv4   53754215                  TCP 1.2.3.4:44541->130.172.46.217:ssh (SYN_SENT)
pscan2  24891  julie  342u  IPv4   53754216                  TCP 1.2.3.4:53749->130.172.46.218:ssh (SYN_SENT)
pscan2  24891  julie  343u  IPv4   53754217                  TCP 1.2.3.4:52824->130.172.46.219:ssh (SYN_SENT)
pscan2  24891  julie  344u  IPv4   53754218                  TCP 1.2.3.4:47099->130.172.46.220:ssh (SYN_SENT)
pscan2  24891  julie  345u  IPv4   53754219                  TCP 1.2.3.4:46320->130.172.46.221:ssh (SYN_SENT)
pscan2  24891  julie  346u  IPv4   53754220                  TCP 1.2.3.4:60889->130.172.46.222:ssh (SYN_SENT)
pscan2  24891  julie  347u  IPv4   53754221                  TCP 1.2.3.4:42054->130.172.46.223:ssh (SYN_SENT)
pscan2  24891  julie  348u  IPv4   53754222                  TCP 1.2.3.4:60412->130.172.46.224:ssh (SYN_SENT)
pscan2  24891  julie  349u  IPv4   53754223                  TCP 1.2.3.4:37266->130.172.46.225:ssh (SYN_SENT)
pscan2  24891  julie  350u  IPv4   53754224                  TCP 1.2.3.4:52086->130.172.46.226:ssh (SYN_SENT)
pscan2  24891  julie  351u  IPv4   53754225                  TCP 1.2.3.4:43091->130.172.46.227:ssh (SYN_SENT)
pscan2  24891  julie  352u  IPv4   53754226                  TCP 1.2.3.4:57044->130.172.46.228:ssh (SYN_SENT)
pscan2  24891  julie  353u  IPv4   53754227                  TCP 1.2.3.4:42529->130.172.46.229:ssh (SYN_SENT)
pscan2  24891  julie  354u  IPv4   53754228                  TCP 1.2.3.4:44553->130.172.46.230:ssh (SYN_SENT)
pscan2  24891  julie  355u  IPv4   53754229                  TCP 1.2.3.4:44380->130.172.46.231:ssh (SYN_SENT)
pscan2  24891  julie  356u  IPv4   53754230                  TCP 1.2.3.4:58800->130.172.46.232:ssh (SYN_SENT)
pscan2  24891  julie  357u  IPv4   53754231                  TCP 1.2.3.4:34210->130.172.46.233:ssh (SYN_SENT)
pscan2  24891  julie  358u  IPv4   53754232                  TCP 1.2.3.4:37923->130.172.46.234:ssh (SYN_SENT)
pscan2  24891  julie  359u  IPv4   53754233                  TCP 1.2.3.4:41876->130.172.46.235:ssh (SYN_SENT)
pscan2  24891  julie  360u  IPv4   53754234                  TCP 1.2.3.4:47700->130.172.46.236:ssh (SYN_SENT)
pscan2  24891  julie  361u  IPv4   53754235                  TCP 1.2.3.4:44968->130.172.46.237:ssh (SYN_SENT)
pscan2  24891  julie  362u  IPv4   53754236                  TCP 1.2.3.4:60061->130.172.46.238:ssh (SYN_SENT)
pscan2  24891  julie  363u  IPv4   53754237                  TCP 1.2.3.4:54535->130.172.46.239:ssh (SYN_SENT)
pscan2  24891  julie  364u  IPv4   53754238                  TCP 1.2.3.4:42449->130.172.46.240:ssh (SYN_SENT)
pscan2  24891  julie  365u  IPv4   53754239                  TCP 1.2.3.4:58802->130.172.46.241:ssh (SYN_SENT)
pscan2  24891  julie  366u  IPv4   53754240                  TCP 1.2.3.4:50352->130.172.46.242:ssh (SYN_SENT)
pscan2  24891  julie  367u  IPv4   53754241                  TCP 1.2.3.4:35278->130.172.46.243:ssh (SYN_SENT)
pscan2  24891  julie  368u  IPv4   53754242                  TCP 1.2.3.4:48165->130.172.46.244:ssh (SYN_SENT)
pscan2  24891  julie  369u  IPv4   53754243                  TCP 1.2.3.4:54040->130.172.46.245:ssh (SYN_SENT)
pscan2  24891  julie  370u  IPv4   53754244                  TCP 1.2.3.4:36795->130.172.46.246:ssh (SYN_SENT)
pscan2  24891  julie  371u  IPv4   53754245                  TCP 1.2.3.4:50270->130.172.46.247:ssh (SYN_SENT)
pscan2  24891  julie  372u  IPv4   53754246                  TCP 1.2.3.4:53716->130.172.46.248:ssh (SYN_SENT)
pscan2  24891  julie  373u  IPv4   53754247                  TCP 1.2.3.4:57835->130.172.46.249:ssh (SYN_SENT)
pscan2  24891  julie  374u  IPv4   53754248                  TCP 1.2.3.4:34915->130.172.46.250:ssh (SYN_SENT)
pscan2  24891  julie  375u  IPv4   53754249                  TCP 1.2.3.4:37605->130.172.46.251:ssh (SYN_SENT)
pscan2  24891  julie  376u  IPv4   53754250                  TCP 1.2.3.4:57715->130.172.46.252:ssh (SYN_SENT)
pscan2  24891  julie  377u  IPv4   53754251                  TCP 1.2.3.4:34432->130.172.46.253:ssh (SYN_SENT)
pscan2  24891  julie  378u  IPv4   53754252                  TCP 1.2.3.4:55234->130.172.46.254:ssh (SYN_SENT)
pscan2  24891  julie  379u  IPv4   53754253                  TCP 1.2.3.4:51303->130.172.47.0:ssh (SYN_SENT)
pscan2  24891  julie  380u  IPv4   53754254                  TCP 1.2.3.4:34857->130.172.47.1:ssh (SYN_SENT)
pscan2  24891  julie  381u  IPv4   53754255                  TCP 1.2.3.4:53421->130.172.47.2:ssh (SYN_SENT)
pscan2  24891  julie  382u  IPv4   53754256                  TCP 1.2.3.4:59955->130.172.47.3:ssh (SYN_SENT)
pscan2  24891  julie  383u  IPv4   53754257                  TCP 1.2.3.4:43803->130.172.47.4:ssh (SYN_SENT)
pscan2  24891  julie  384u  IPv4   53754258                  TCP 1.2.3.4:52454->130.172.47.5:ssh (SYN_SENT)
pscan2  24891  julie  385u  IPv4   53754259                  TCP 1.2.3.4:43674->130.172.47.6:ssh (SYN_SENT)
pscan2  24891  julie  386u  IPv4   53754260                  TCP 1.2.3.4:56845->130.172.47.7:ssh (SYN_SENT)
pscan2  24891  julie  387u  IPv4   53754261                  TCP 1.2.3.4:39312->130.172.47.8:ssh (SYN_SENT)
pscan2  24891  julie  388u  IPv4   53754262                  TCP 1.2.3.4:53155->130.172.47.9:ssh (SYN_SENT)
pscan2  24891  julie  389u  IPv4   53754263                  TCP 1.2.3.4:37639->130.172.47.10:ssh (SYN_SENT)
pscan2  24891  julie  390u  IPv4   53754264                  TCP 1.2.3.4:42868->130.172.47.11:ssh (SYN_SENT)
pscan2  24891  julie  391u  IPv4   53754265                  TCP 1.2.3.4:52302->130.172.47.12:ssh (SYN_SENT)
pscan2  24891  julie  392u  IPv4   53754266                  TCP 1.2.3.4:45744->130.172.47.13:ssh (SYN_SENT)
pscan2  24891  julie  393u  IPv4   53754267                  TCP 1.2.3.4:50228->130.172.47.14:ssh (SYN_SENT)
pscan2  24891  julie  394u  IPv4   53754268                  TCP 1.2.3.4:46106->130.172.47.15:ssh (SYN_SENT)
pscan2  24891  julie  395u  IPv4   53754269                  TCP 1.2.3.4:52244->130.172.47.16:ssh (SYN_SENT)
pscan2  24891  julie  396u  IPv4   53754270                  TCP 1.2.3.4:52519->130.172.47.17:ssh (SYN_SENT)
pscan2  24891  julie  397u  IPv4   53754271                  TCP 1.2.3.4:40709->130.172.47.18:ssh (SYN_SENT)
pscan2  24891  julie  398u  IPv4   53754272                  TCP 1.2.3.4:35796->130.172.47.19:ssh (SYN_SENT)
pscan2  24891  julie  399u  IPv4   53754273                  TCP 1.2.3.4:43581->130.172.47.20:ssh (SYN_SENT)
pscan2  24891  julie  400u  IPv4   53754274                  TCP 1.2.3.4:51817->130.172.47.21:ssh (SYN_SENT)
pscan2  24891  julie  401u  IPv4   53754275                  TCP 1.2.3.4:58156->130.172.47.22:ssh (SYN_SENT)
pscan2  24891  julie  402u  IPv4   53754276                  TCP 1.2.3.4:39417->130.172.47.23:ssh (SYN_SENT)
pscan2  24891  julie  403u  IPv4   53754277                  TCP 1.2.3.4:33922->130.172.47.24:ssh (SYN_SENT)

Each of those TCP sockets in SYN_SENT state represents one TCP connection that slim was in the process of opening. Each is a connection from slim to the SSH socket on some remote host.


OK, what's really going on?

Look at the hacker's program itself — the program text and shared libraries, and the process's current working directory. We know it's PID 24891:

% lsof -n -p 24891 | less
COMMAND   PID   USER   FD   TYPE     DEVICE      SIZE       NODE NAME
pscan2  24891  julie  cwd    DIR        8,1      4096   26345487 /tmp/.X11-fonts/.. 
pscan2  24891  julie  rtd    DIR        8,1      4096          2 /
pscan2  24891  julie  txt    REG        8,1     21407   26345492 /tmp/.X11-fonts/.. /pscan2
pscan2  24891  julie  mem    REG        0,0                    0 [heap] (stat: No such file or directory)
pscan2  24891  julie  mem    REG        8,1   1248904    1114479 /lib/tls/i686/cmov/libc-2.4.so
pscan2  24891  julie  mem    REG        8,1    105112    1081348 /lib/ld-2.4.so
pscan2  24891  julie    0u   CHR      136,1                    3 /dev/pts/1
pscan2  24891  julie    1u   CHR      136,1                    3 /dev/pts/1
pscan2  24891  julie    2u   CHR      136,1                    3 /dev/pts/1
pscan2  24891  julie    3w   REG        8,1         0   26345493 /tmp/.X11-fonts/.. /130.172.pscan.22
pscan2  24891  julie    4u   IPv4   53754278                  TCP 1.2.3.4:60448->130.172.47.25:ssh (SYN_SENT)
pscan2  24891  julie    5u   IPv4   53754279                  TCP 1.2.3.4:39799->130.172.47.26:ssh (SYN_SENT)
pscan2  24891  julie    6u   IPv4   53754280                  TCP 1.2.3.4:53838->130.172.47.27:ssh (SYN_SENT)
^C 

First, let's go find the hacker's program. But wait a minute, /tmp/.X11-fonts/.. is just a roundabout way of expressing /tmp, right? Maybe. Or maybe not. Let's investigate:

% cd /tmp/.X11-fonts
% ls -la
total 24
drwxr-xr-x  3 julie    julie    4096 Aug 16 10:22 .
drwxrwxrwt 12 root     root     4096 Aug 16 10:22 ..
drwxr-xr-x  2 julie    julie    4096 Aug 16 10:22 .. 

So what's wrong with that picture? Nothing, you say? Well, it looks familiar, and this is exactly why hackers do this trick. But you can't have two files with the same name, even if they are the sort of boring clutter like ".." that you usually overlook! Let's see what the names really are:

% cd /tmp/.X11-fonts
% ls -la | cat -A
total 24
drwxr-xr-x  3 julie    julie    4096 Aug 16 10:22 .$
drwxrwxrwt 12 root     root     4096 Aug 16 10:22 ..$
drwxr-xr-x  2 julie    julie    4096 Aug 16 10:22 .. $
% ls -laF
total 24
drwxr-xr-x  3 julie    julie    4096 Aug 16 10:22 ./
drwxrwxrwt 12 root     root     4096 Aug 16 10:22 ../
drwxr-xr-x  2 julie    julie    4096 Aug 16 10:22 .. / 

Ahah! The hacker has done this:

  1. Created a new directory named /tmp/.X11-fonts. You can do that since /tmp is world-writable and no such directory already existed. And you can get away with it because it looks like the normal clutter in /tmp.
  2. Changed to that new directory and typed:
    mkdir ".. "
  3. Hidden all his stuff in "/tmp/.X11-fonts/.. "

Let's see what's there:

% cd "/tmp/.X11-fonts/.. "
% ls -laF
total 2384
drwx------ 2 julie    julie       4096 Aug 14 11:38 ./
drwxr-xr-x 3 julie    julie       4096 Aug 10 12:33 ../
-rw-r--r-- 1 julie    julie          0 Aug 14 11:33 130.175.pscan.22
-rw-r--r-- 1 julie    julie          0 Mar 23 11:23 203.12.pscan.22
-rw-r--r-- 1 julie    julie          0 Mar 23 08:38 242.0.pscan.22
-rwx------ 1 julie    julie        366 Oct 24  2005 a*
-rwxr-xr-x 1 julie    julie      12336 Nov 11  2005 a1*
-rwxr-xr-x 1 julie    julie       4734 Nov 24  2005 a2*
-rwxr-xr-x 1 julie    julie        832 Nov 24  2005 a3*
-rw-r--r-- 1 julie    julie    1457413 Aug 10 14:00 pass_file
-rwx------ 1 julie    julie      21407 Jul 21  2004 pscan2*
-rwxr-xr-x 1 julie    julie     842736 Nov 24  2004 ssh-scan*
-rwxr-xr-x 1 julie    julie       3957 Nov 24  2005 start*
-rw-r--r-- 1 julie    julie         74 Aug 13 16:15 vuln.txt
% file *
130.175.pscan.22: empty
203.12.pscan.22:  empty
242.0.pscan.22:   empty
a:                Bourne-Again shell script text executable
a1:               ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.2.5, dynamically linked (uses shared libs), not stripped
a2:               ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.2.5, dynamically linked (uses shared libs), not stripped
a3:               ASCII English text
pass_file:        ASCII C++ program text
pscan2:           ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.2.5, dynamically linked (uses shared libs), not stripped
ssh-scan:         ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.0.0, statically linked, stripped
start:            ASCII English text
vuln.txt:         ASCII text 

The program pscan2 is a very simple port scanner. Not at all sophisticated — if you want a full-featured port scanner get Nmap. This one is awfully simple and is run like this:
  % pscan2 targethost 22
That would attempt to connect to every TCP port from 1 through 22 on targethost and report those ports where it could connect. If you're curious but unwilling to ask Google for pscan2.c on your own, here it is:

/* 
 * internet port scanner 
 *
 * This program will scan a hosts TCP ports printing all ports that accept
 * connections, and if known, the service name.
 * This program can be trivially altered to do UDP ports also.
 *
 * Kopywrong (K) Aug. 25, '94 pluvius@io.org
 *
 * Hey kiddies, this is a C program, to run it do this:
 * $ cc -o pscan pscan.c
 * $ pscan <host> [max port]
 *
 * No, this will not get you root.
 * 
 * Changes:
 * Changed fprintf to printf in line 34 to work with my Linux 1.1.18 box
 * Netrunner 1/18/95 11:30pm
 * 
*/

static char sccsid[] = "@(#)pscan.c     1.0     (KRAD) 08/25/94";
#include <stdio.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <netdb.h>

#define MAX_PORT 1024 /* scan up to this port */
int s;
struct sockaddr_in addr;
char rmt_host[100];

int skan(port)
int port;
{
 int r;
    s = socket(PF_INET, SOCK_STREAM, IPPROTO_TCP);
    if (s < 0) {
       /* fprintf("ERROR: socket() failed\n"); */
       /* Changed to printf for my Linux 1.1.18 box */
       printf("ERROR: socket() failed\n");
       exit(0);
    }

    addr.sin_family = PF_INET;
    addr.sin_port = port;
    addr.sin_addr.s_addr = inet_addr(rmt_host);

    r = connect(s,(struct sockaddr *) &addr, sizeof(addr));

    close(s);

    if (r < 0) {
       return (1 == 0);
    }

    return (1 == 1);
}

main(argc,argv) 
int argc;
char *argv[];
{
 int a,b,c,d,e,f;
 struct hostent *foo;
 struct servent *bar;

   if (argc < 2) {
      fprintf(stderr,"usage: %s <host> [highest port]\n",argv[0]);
      exit(0);
   }

   if (sscanf(argv[1],"%d.%d.%d.%d",&a,&b,&c,&d) != 4) {
      foo = gethostbyname(argv[1]);
      if (foo == NULL) {
         fprintf(stderr,"error: cannot resolve host %s\n",argv[1]);
         exit(0);
      }
      sprintf(rmt_host,"%d.%d.%d.%d",(unsigned char )foo->h_addr_list[0][0],
              (unsigned char ) foo->h_addr_list[0][1], 
              (unsigned char ) foo->h_addr_list[0][2], 
              (unsigned char ) foo->h_addr_list[0][3]);
   } else {
      strncpy(rmt_host,argv[1],99);
   }


   if (argc > 2) {
      f = atoi(argv[2]);
   } else
      f = MAX_PORT;

   fprintf(stdout,"Scanning host %s - TCP ports 1 through %d\n",rmt_host,f);

   for (e =1;e<=f;e++) {
    char serv[100];
      if (skan(e)) {
         bar = getservbyport(e,"tcp");
         printf("%d (%s) is running.\n",e,(bar == NULL) ? "UNKNOWN" :
                bar->s_name);
      }
   }
}

Let's run one last test before pausing for some analysis. What else is the compromised account doing?

% lsof -in | egrep -w 'PID|julie ' | grep -v '^pscan2'
COMMAND PID    USER   FD   TYPE  DEVICE      SIZE       NODE NAME
a1     3901   julie  cwd    DIR     8,1      4096          2 /
a1     3901   julie  rtd    DIR     8,1      4096          2 /
a1     3901   julie  txt    REG     8,1     12336   26345489 /tmp/.X11-fonts/.. /a1
a1     3901   julie  mem    REG     8,1   1248904    1114479 /lib/tls/i686/cmov/libc-2.4.so
a1     3901   julie  mem    REG     8,1    105112    1081348 /lib/ld-2.4.so
a1     3901   julie  mem    REG     0,0                    0 [stack] (stat: No such file or directory)
a1     3901   julie    0u   CHR     1,3                 4898 /dev/null
a1     3901   julie    1u   CHR     1,3                 4898 /dev/null
a1     3901   julie    2u   CHR     1,3                 4898 /dev/null
a1     3901   julie    3u  IPv4   12644                  TCP *:2345 (LISTEN)
screen 5157   julie  cwd    DIR     8,1      4096   26345487 /tmp/.X11-fonts/.. 
screen 5157   julie  rtd    DIR     8,1      4096          2 /
screen 5157   julie  txt    REG     8,1    301368   22708956 /usr/bin/screen
screen 5157   julie  mem    REG     0,0                    0 [heap] (stat: No such file or directory)
screen 5157   julie  mem    REG     8,1     38372    1114488 /lib/tls/i686/cmov/libnss_files-2.4.so
screen 5157   julie  mem    REG     8,1     34320    1114490 /lib/tls/i686/cmov/libnss_nis-2.4.so
screen 5157   julie  mem    REG     8,1     75488    1114485 /lib/tls/i686/cmov/libnsl-2.4.so
screen 5157   julie  mem    REG     8,1     30428    1114486 /lib/tls/i686/cmov/libnss_compat-2.4.so
screen 5157   julie  mem    REG     8,1    208336   22758941 /usr/lib/locale/en_US.utf8/LC_CTYPE
screen 5157   julie  mem    REG     8,1        54   22758946 /usr/lib/locale/en_US.utf8/LC_NUMERIC
screen 5157   julie  mem    REG     8,1      2451   22758949 /usr/lib/locale/en_US.utf8/LC_TIME
screen 5157   julie  mem    REG     8,1    880094   22758940 /usr/lib/locale/en_US.utf8/LC_COLLATE
screen 5157   julie  mem    REG     8,1       286   22758944 /usr/lib/locale/en_US.utf8/LC_MONETARY
screen 5157   julie  mem    REG     8,1      9640    1114482 /lib/tls/i686/cmov/libdl-2.4.so
screen 5157   julie  mem    REG     8,1   1248904    1114479 /lib/tls/i686/cmov/libc-2.4.so
screen 5157   julie  mem    REG     8,1     29184    1081439 /lib/libpam.so.0.79
screen 5157   julie  mem    REG     8,1     21864    1114481 /lib/tls/i686/cmov/libcrypt-2.4.so
screen 5157   julie  mem    REG     8,1      9652    1114498 /lib/tls/i686/cmov/libutil-2.4.so
screen 5157   julie  mem    REG     8,1    300352    1081423 /lib/libncursesw.so.5.5
screen 5157   julie  mem    REG     8,1        52   22773854 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES
screen 5157   julie  mem    REG     8,1        34   22758947 /usr/lib/locale/en_US.utf8/LC_PAPER
screen 5157   julie  mem    REG     8,1        77   22758945 /usr/lib/locale/en_US.utf8/LC_NAME
screen 5157   julie  mem    REG     8,1       155   22758939 /usr/lib/locale/en_US.utf8/LC_ADDRESS
screen 5157   julie  mem    REG     8,1        59   22758948 /usr/lib/locale/en_US.utf8/LC_TELEPHONE
screen 5157   julie  mem    REG     8,1        23   22758943 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT
screen 5157   julie  mem    REG     8,1     25460   22725254 /usr/lib/gconv/gconv-modules.cache
screen 5157   julie  mem    REG     8,1       391   22758942 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION
screen 5157   julie  mem    REG     8,1    105112    1081348 /lib/ld-2.4.so
screen 5157   julie    0r   CHR     1,3                 4898 /dev/null
screen 5157   julie    1w   CHR     1,3                 4898 /dev/null
screen 5157   julie    2w   CHR     1,3                 4898 /dev/null
screen 5157   julie    3r  FIFO    0,15                12621 /var/run/screen/S-julie /5157.pts-0.slim
screen 5157   julie    5u   REG    0,15      6144       8879 /var/run/utmp
screen 5157   julie    6u   CHR     5,2                 2433 /dev/ptmx
bash   5158   julie  cwd    DIR     8,1      4096   26345487 /tmp/.X11-fonts/.. 
bash   5158   julie  rtd    DIR     8,1      4096          2 /
bash   5158   julie  txt    REG     8,1    676836   30277637 /bin/bash
bash   5158   julie  mem    REG     0,0                    0 [heap] (stat: No such file or directory)
bash   5158   julie  mem    REG     8,1     38372    1114488 /lib/tls/i686/cmov/libnss_files-2.4.so
bash   5158   julie  mem    REG     8,1     34320    1114490 /lib/tls/i686/cmov/libnss_nis-2.4.so
bash   5158   julie  mem    REG     8,1     75488    1114485 /lib/tls/i686/cmov/libnsl-2.4.so
bash   5158   julie  mem    REG     8,1     30428    1114486 /lib/tls/i686/cmov/libnss_compat-2.4.so
bash   5158   julie  mem    REG     8,1    208336   22758941 /usr/lib/locale/en_US.utf8/LC_CTYPE
bash   5158   julie  mem    REG     8,1        54   22758946 /usr/lib/locale/en_US.utf8/LC_NUMERIC
bash   5158   julie  mem    REG     8,1      2451   22758949 /usr/lib/locale/en_US.utf8/LC_TIME
bash   5158   julie  mem    REG     8,1    880094   22758940 /usr/lib/locale/en_US.utf8/LC_COLLATE
bash   5158   julie  mem    REG     8,1       286   22758944 /usr/lib/locale/en_US.utf8/LC_MONETARY
bash   5158   julie  mem    REG     8,1   1248904    1114479 /lib/tls/i686/cmov/libc-2.4.so
bash   5158   julie  mem    REG     8,1      9640    1114482 /lib/tls/i686/cmov/libdl-2.4.so
bash   5158   julie  mem    REG     8,1    258944    1081421 /lib/libncurses.so.5.5
bash   5158   julie  mem    REG     8,1        52   22773854 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES
bash   5158   julie  mem    REG     8,1        34   22758947 /usr/lib/locale/en_US.utf8/LC_PAPER
bash   5158   julie  mem    REG     8,1        77   22758945 /usr/lib/locale/en_US.utf8/LC_NAME
bash   5158   julie  mem    REG     8,1       155   22758939 /usr/lib/locale/en_US.utf8/LC_ADDRESS
bash   5158   julie  mem    REG     8,1        59   22758948 /usr/lib/locale/en_US.utf8/LC_TELEPHONE
bash   5158   julie  mem    REG     8,1        23   22758943 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT
bash   5158   julie  mem    REG     8,1     25460   22725254 /usr/lib/gconv/gconv-modules.cache
bash   5158   julie  mem    REG     8,1       391   22758942 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION
bash   5158   julie  mem    REG     8,1    105112    1081348 /lib/ld-2.4.so
bash   5158   julie    0u   CHR   136,1                    3 /dev/pts/1
bash   5158   julie    1u   CHR   136,1                    3 /dev/pts/1
bash   5158   julie    2u   CHR   136,1                    3 /dev/pts/1
bash   5158   julie  255u   CHR   136,1                    3 /dev/pts/1
bash   5177   julie  cwd    DIR     8,1      4096   26345487 /tmp/.X11-fonts/.. 
bash   5177   julie  rtd    DIR     8,1      4096          2 /
bash   5177   julie  txt    REG     8,1    676836   30277637 /bin/bash
bash   5177   julie  mem    REG     0,0                    0 [heap] (stat: No such file or directory)
bash   5177   julie  mem    REG     8,1     38372    1114488 /lib/tls/i686/cmov/libnss_files-2.4.so
bash   5177   julie  mem    REG     8,1     34320    1114490 /lib/tls/i686/cmov/libnss_nis-2.4.so
bash   5177   julie  mem    REG     8,1     75488    1114485 /lib/tls/i686/cmov/libnsl-2.4.so
bash   5177   julie  mem    REG     8,1     30428    1114486 /lib/tls/i686/cmov/libnss_compat-2.4.so
bash   5177   julie  mem    REG     8,1    208336   22758941 /usr/lib/locale/en_US.utf8/LC_CTYPE
bash   5177   julie  mem    REG     8,1        54   22758946 /usr/lib/locale/en_US.utf8/LC_NUMERIC
bash   5177   julie  mem    REG     8,1      2451   22758949 /usr/lib/locale/en_US.utf8/LC_TIME
bash   5177   julie  mem    REG     8,1    880094   22758940 /usr/lib/locale/en_US.utf8/LC_COLLATE
bash   5177   julie  mem    REG     8,1       286   22758944 /usr/lib/locale/en_US.utf8/LC_MONETARY
bash   5177   julie  mem    REG     8,1   1248904    1114479 /lib/tls/i686/cmov/libc-2.4.so
bash   5177   julie  mem    REG     8,1      9640    1114482 /lib/tls/i686/cmov/libdl-2.4.so
bash   5177   julie  mem    REG     8,1    258944    1081421 /lib/libncurses.so.5.5
bash   5177   julie  mem    REG     8,1        52   22773854 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES
bash   5177   julie  mem    REG     8,1        34   22758947 /usr/lib/locale/en_US.utf8/LC_PAPER
bash   5177   julie  mem    REG     8,1        77   22758945 /usr/lib/locale/en_US.utf8/LC_NAME
bash   5177   julie  mem    REG     8,1       155   22758939 /usr/lib/locale/en_US.utf8/LC_ADDRESS
bash   5177   julie  mem    REG     8,1        59   22758948 /usr/lib/locale/en_US.utf8/LC_TELEPHONE
bash   5177   julie  mem    REG     8,1        23   22758943 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT
bash   5177   julie  mem    REG     8,1     25460   22725254 /usr/lib/gconv/gconv-modules.cache
bash   5177   julie  mem    REG     8,1       391   22758942 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION
bash   5177   julie  mem    REG     8,1    105112    1081348 /lib/ld-2.4.so
bash   5177   julie    0u   CHR   136,1                    3 /dev/pts/1
bash   5177   julie    1u   CHR   136,1                    3 /dev/pts/1
bash   5177   julie    2u   CHR   136,1                    3 /dev/pts/1
bash   5177   julie  255r   REG     8,1      3957   26345497 /tmp/.X11-fonts/.. /start
a     24890   julie  cwd    DIR     8,1      4096   26345487 /tmp/.X11-fonts/.. 
a     24890   julie  rtd    DIR     8,1      4096          2 /
a     24890   julie  txt    REG     8,1    676836   30277637 /bin/bash
a     24890   julie  mem    REG     0,0                    0 [heap] (stat: No such file or directory)
a     24890   julie  mem    REG     8,1    208336   22758941 /usr/lib/locale/en_US.utf8/LC_CTYPE
a     24890   julie  mem    REG     8,1        54   22758946 /usr/lib/locale/en_US.utf8/LC_NUMERIC
a     24890   julie  mem    REG     8,1      2451   22758949 /usr/lib/locale/en_US.utf8/LC_TIME
a     24890   julie  mem    REG     8,1    880094   22758940 /usr/lib/locale/en_US.utf8/LC_COLLATE
a     24890   julie  mem    REG     8,1       286   22758944 /usr/lib/locale/en_US.utf8/LC_MONETARY
a     24890   julie  mem    REG     8,1   1248904    1114479 /lib/tls/i686/cmov/libc-2.4.so
a     24890   julie  mem    REG     8,1      9640    1114482 /lib/tls/i686/cmov/libdl-2.4.so
a     24890   julie  mem    REG     8,1    258944    1081421 /lib/libncurses.so.5.5
a     24890   julie  mem    REG     8,1        52   22773854 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES
a     24890   julie  mem    REG     8,1        34   22758947 /usr/lib/locale/en_US.utf8/LC_PAPER
a     24890   julie  mem    REG     8,1        77   22758945 /usr/lib/locale/en_US.utf8/LC_NAME
a     24890   julie  mem    REG     8,1       155   22758939 /usr/lib/locale/en_US.utf8/LC_ADDRESS
a     24890   julie  mem    REG     8,1        59   22758948 /usr/lib/locale/en_US.utf8/LC_TELEPHONE
a     24890   julie  mem    REG     8,1        23   22758943 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT
a     24890   julie  mem    REG     8,1     25460   22725254 /usr/lib/gconv/gconv-modules.cache
a     24890   julie  mem    REG     8,1       391   22758942 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION
a     24890   julie  mem    REG     8,1    105112    1081348 /lib/ld-2.4.so
a     24890   julie    0u   CHR   136,1                    3 /dev/pts/1
a     24890   julie    1u   CHR   136,1                    3 /dev/pts/1
a     24890   julie    2u   CHR   136,1                    3 /dev/pts/1
a     24890   julie  255r   REG     8,1       366   26345488 /tmp/.X11-fonts/.. /a

What has been the sequence of events so far?

The attacker compromised the account julie. We will look into how that might have happened below.

Given that account, a variety of software was installed in an obscure location under /tmp.

One component was a1, a daemon process that listens for connections on TCP port 2345. See PID 3901 in the above output, with its stdin, stdout, and stderr streams connected to /dev/null while it is listening for connections on TCP port 2345. The attacker connects to that port, gets a shell, and can then run his programs. The connection will not be logged because there is no call to login or sshd. The listener is shown in the above output but it is obscured by volume. See the following output for a list of what is using that one virtual terminal:

% lsof /dev/pts/1
COMMAND  PID    USER   FD  TYPE  DEVICE   SIZE    NODE NAME
bash    5158   julie    0u  CHR   136,1              3 /dev/pts/1
bash    5158   julie    1u  CHR   136,1              3 /dev/pts/1
bash    5158   julie    2u  CHR   136,1              3 /dev/pts/1
bash    5158   julie  255u  CHR   136,1              3 /dev/pts/1
bash    5177   julie    0u  CHR   136,1              3 /dev/pts/1
bash    5177   julie    1u  CHR   136,1              3 /dev/pts/1
bash    5177   julie    2u  CHR   136,1              3 /dev/pts/1
a      24890   julie    0u  CHR   136,1              3 /dev/pts/1
a      24890   julie    1u  CHR   136,1              3 /dev/pts/1
a      24890   julie    2u  CHR   136,1              3 /dev/pts/1
pscan2 24891   julie    0u  CHR   136,1              3 /dev/pts/1
pscan2 24891   julie    1u  CHR   136,1              3 /dev/pts/1
pscan2 24891   julie    2u  CHR   136,1              3 /dev/pts/1 

We interrupted the hacker in the process of port-scanning a number of hosts. After unplugging the Ethernet cable, there were 400 TCP connections in state SYN_SENT from slim to these hosts:
130.172.45.186 through 130.172.45.254
130.172.46.0   through 130.172.46.254
130.172.47.0   through 130.172.47.72
According to whois:

% whois 130.172.44.186
[Querying whois.arin.net]
[whois.arin.net]

OrgName:    General Motors Corporation
OrgID:      GMC-20
Address:    200 Renaissance Center
City:       Detroit
StateProv:  MI
PostalCode: 48265
Country:    US

NetRange:   130.172.0.0 - 130.172.255.255
CIDR:       130.172.0.0/16
NetName:    GM-130-172-B
NetHandle:  NET-130-172-0-0-1
Parent:     NET-130-0-0-0-0
NetType:    Direct Assignment
NameServer: NS1.EDS.COM
NameServer: NS2.EDS.COM
Comment:
RegDate:
Updated:    2006-10-11 

After the initial reconnaissance with pscan2 had found a list of SSH servers, the hacker would use ssh-scan to try the list of login/password guesses stored in pass_file. Look again at the process table and some of the file names and timestamps:

% ps axuwww | egrep 'PID|pscan2'
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
julie    24891  100  0.0   1596   492 pts/1    R+   10:52   2:19 ./pscan2 130.172 22
% ls -lF *pscan*
-rw-r--r-- 1 julie    julie          0 Aug 14 11:33 130.175.pscan.22
-rw-r--r-- 1 julie    julie          0 Mar 23 11:23 203.12.pscan.22
-rw-r--r-- 1 julie    julie          0 Mar 23 08:38 242.0.pscan.22
-rwx------ 1 julie    julie      21407 Jul 21  2004 pscan2* 

He was scanning the entire 130.172.0.0/16 block when we interrupted him. And it would appear that back in late March, five months before, he scanned the 203.12.0.0/16 and 242.0.0.0/16 blocks! The first is an ISP in Western Australia and the second is part of the multicast block. However, when we investigate the breaking of the account in the following section, we will see that those two empty output files must have been installed with the hacker software, complete with timestamps from the past. It's a mystery why some hacker (this one, or another who created the archive) tried to scan a range of multicast addresses for SSH connections. This hacker only broke the julie account on August 8th.


The initial break-in

Looking at the log file /var/log/authlog and the older versions /var/log/authlog.*.gz, we can find that the user julie was compromised during a password-guessing attack. I wouldn't call it a "brute-force" attack because it only tries one password for most user accounts. Here is the log entry:

  [... MANY lines deleted ...]
Aug  8 07:34:13 slim sshd[19651]: Invalid user joneweb from 213.131.231.147
Aug  8 07:34:13 slim sshd[19651]: (pam_unix) check pass; user unknown
Aug  8 07:34:13 slim sshd[19651]: (pam_unix) authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=server012.boxfix.de
Aug  8 07:34:15 slim sshd[19651]: Failed password for invalid user joneweb from 213.131.231.147 port 39345 ssh2
Aug  8 07:34:16 slim sshd[19653]: Invalid user jopling from 213.131.231.147
Aug  8 07:34:16 slim sshd[19653]: (pam_unix) check pass; user unknown
Aug  8 07:34:16 slim sshd[19653]: (pam_unix) authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=server012.boxfix.de
Aug  8 07:34:18 slim sshd[19653]: Failed password for invalid user jopling from 213.131.231.147 port 54419 ssh2
Aug  8 07:34:19 slim sshd[19655]: Invalid user jostling from 213.131.231.147
Aug  8 07:34:19 slim sshd[19655]: (pam_unix) check pass; user unknown
Aug  8 07:34:19 slim sshd[19655]: (pam_unix) authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=server012.boxfix.de
Aug  8 07:34:21 slim sshd[19655]: Failed password for invalid user jostling from 213.131.231.147 port 53038 ssh2
Aug  8 07:34:22 slim sshd[19657]: Invalid user jot from 213.131.231.147
Aug  8 07:34:22 slim sshd[19657]: (pam_unix) check pass; user unknown
Aug  8 07:34:22 slim sshd[19657]: (pam_unix) authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=server012.boxfix.de
Aug  8 07:34:24 slim sshd[19657]: Failed password for invalid user jot from 213.131.231.147 port 54470 ssh2
Aug  8 07:34:25 slim sshd[19659]: Accepted password for julie from 213.131.231.147 port 46402 ssh2
Aug  8 07:34:25 slim sshd[19661]: (pam_unix) session opened for user julie by (uid=0)
Aug  8 07:34:33 slim sshd[19681]: Invalid user kallkwik from 213.131.231.147
Aug  8 07:34:33 slim sshd[19681]: (pam_unix) check pass; user unknown
Aug  8 07:34:33 slim sshd[19681]: (pam_unix) authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=server012.boxfix.de
Aug  8 07:34:35 slim sshd[19681]: Failed password for invalid user kallkwik from 213.131.231.147 port 48407 ssh2
Aug  8 07:34:36 slim sshd[19683]: Invalid user kandshom from 213.131.231.147
Aug  8 07:34:36 slim sshd[19683]: (pam_unix) check pass; user unknown
Aug  8 07:34:36 slim sshd[19683]: (pam_unix) authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=server012.boxfix.de
Aug  8 07:34:38 slim sshd[19683]: Failed password for invalid user kandshom from 213.131.231.147 port 36984 ssh2
Aug  8 07:34:39 slim sshd[19685]: Invalid user keenelec from 213.131.231.147
Aug  8 07:34:39 slim sshd[19685]: (pam_unix) check pass; user unknown
Aug  8 07:34:39 slim sshd[19685]: (pam_unix) authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=server012.boxfix.de
Aug  8 07:34:41 slim sshd[19685]: Failed password for invalid user keenelec from 213.131.231.147 port 57969 ssh2
Aug  8 07:34:42 slim sshd[19687]: Invalid user kelvin from 213.131.231.147
Aug  8 07:34:42 slim sshd[19687]: (pam_unix) check pass; user unknown
Aug  8 07:34:42 slim sshd[19687]: (pam_unix) authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=server012.boxfix.de
  [... more lines deleted ...]
Aug  8 07:41:21 slim sshd[19661]: (pam_unix) session closed for user julie 
  [... MANY more lines deleted ...] 

A note on reading this log data — sshd uses privilege separation, so you will see two sshd PIDs per successful connection. 19659 and 19661 for the connection where julie was broken.

Also, sshd runs as root so you expect to see pam_unix report that the session was opened by UID 0. This is always the case, including legitimate authentication events. It's not a sign of an attack against root, it's the fact that user authentication must run with root privileges.

The attack started at 06:27:09 and continued until 07:41:19. During that time it made 1,463 password guesses:

Presumably it did the sensible thing and assumed that "Joe Accounts" are very common. I would assume that all of the one-password-guess logins tried using the login as the password (why should it have used different logic only for julie?). The three most common weak passwords are: the literal string password, the login itself (it's a Joe Account), and the literal string admin.


When and how was the compromised account abused?

What happened after the initial exploit, and when did it happen? Here are the entries from /var/log/authlog with blank lines manually inserted to divide the sessions.

Aug  8 07:34:25 slim sshd[19659]: Accepted password for julie from 213.131.231.147 port 46402 ssh2
Aug  8 07:34:25 slim sshd[19661]: (pam_unix) session opened for user julie by (uid=0)
Aug  8 07:41:21 slim sshd[19661]: (pam_unix) session closed for user julie 

Aug  8 19:11:15 slim sshd[20240]: Accepted password for julie from 172.183.24.159 port 7008 ssh2
Aug  8 19:11:15 slim sshd[20242]: (pam_unix) session opened for user julie by (uid=0)
Aug  8 19:15:09 slim passwd[20274]: (pam_unix) authentication failure; logname=julie uid=1007 euid=0 tty= ruser= rhost=  user=julie 
Aug  8 19:15:23 slim passwd[20275]: (pam_unix) password changed for julie 
Aug  8 19:15:23 slim passwd[20275]: (pam_unix) Password for julie was changed

Aug  8 19:16:45 slim sshd[20279]: Accepted password for julie from 172.183.24.159 port 7032 ssh2
Aug  8 19:16:45 slim sshd[20281]: (pam_unix) session opened for user julie by (uid=0)
Aug  8 19:24:28 slim sshd[20281]: (pam_unix) session closed for user julie 

Aug 10 12:32:10 slim sshd[5113]: (pam_unix) authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=ac9eaf55.ipt.aol.com  user=julie 
Aug 10 12:32:12 slim sshd[5113]: Failed password for julie from 172.158.175.85 port 1344 ssh2
Aug 10 12:32:15 slim sshd[5113]: Accepted password for julie from 172.158.175.85 port 1344 ssh2
Aug 10 12:32:15 slim sshd[5115]: (pam_unix) session opened for user julie by (uid=0)
Aug 11 09:35:23 slim sshd[5115]: (pam_unix) session closed for user julie 

Aug 12 03:02:34 slim sshd[29448]: Accepted password for julie from 172.158.63.17 port 1179 ssh2
Aug 12 03:02:34 slim sshd[29451]: (pam_unix) session opened for user julie by (uid=0)
Aug 12 03:05:00 slim sshd[29451]: (pam_unix) session closed for user julie 

Aug 12 11:02:10 slim sshd[1712]: Accepted password for julie from 172.158.63.17 port 2281 ssh2
Aug 12 11:02:10 slim sshd[1742]: (pam_unix) session opened for user julie by (uid=0)
Aug 12 18:54:00 slim sshd[1742]: (pam_unix) session closed for user julie 

Aug 13 06:45:54 slim sshd[3945]: Accepted password for julie from 172.173.119.144 port 1194 ssh2
Aug 13 06:45:54 slim sshd[3953]: (pam_unix) session opened for user julie by (uid=0)
Aug 13 08:31:00 slim sshd[3953]: (pam_unix) session closed for user julie 

Aug 13 13:30:44 slim sshd[3274]: Accepted password for julie from 172.173.1.192 port 2045 ssh2
Aug 13 13:30:44 slim sshd[3276]: (pam_unix) session opened for user julie by (uid=0)
Aug 13 20:06:37 slim sshd[3276]: (pam_unix) session closed for user julie 

Aug 14 03:10:47 slim sshd[9439]: Accepted password for julie from 172.180.161.180 port 43661 ssh2
Aug 14 03:10:47 slim sshd[9441]: (pam_unix) session opened for user julie by (uid=0)
Aug 14 08:46:05 slim sshd[9441]: (pam_unix) session closed for user julie 

Aug 8 07:34:25 — The initial automated exploit guessed the password.

This came from 213.131.231.147 / server012.boxfix.de, a web server apparently in Düsseldorf, Germany, based on traceroute and whois. The Düsseldorf server had already been hacked and was just used by the hacker to attack others — just as this host slim was in turn used to attack General Motors.

Aug 8 19:11:15 — The first active use by hacker. Tried to change the password at 19:15:09, but mis-typed the current password! Successfully changed the password at 19:15:23.

This came from 172.183.24.159 / acb7189f.ipt.aol.com. The hacker is somewhere in the US using an America Online connection. You can't really tell where AOL hosts are located. There is effectively a tunnel from the client to the AOL server, so traceroute can only show you the routing back to where they pop out of the AOL tunnel.

Aug 8 19:16:45 — Tested that changed password, coming back in from that AOL tunnel exit at 172.183.24.159 / acb7189f.ipt.aol.com.

Aug 10 12:32:10 — Initially mis-typed (or forgot?) that new password! Then got it right and stayed logged in for about 17 hours. During this session the hacker installed or modified the file pass_file, used for SSH password brute-force guessing.

This connection was from a different AOL exit point, 172.158.175.85 / ac9eaf55.ipt.aol.com. But this is simply the way that AOL works.

Aug 12 03:02:34 — Logged in for just 2 min 26 sec from 172.158.63.17 / ac9e3f11.ipt.aol.com.

Aug 12 11:02:10 — Logged in for almost 8 hours from 172.158.63.17 / ac9e3f11.ipt.aol.com. Presumably the hacker had left his computer turned on and connected to AOL since the connection 8 hours before, because it is again coming from the same IP address.

Aug 13 06:45:54 — Logged in for about an hour and 45 minutes from 172.173.119.144 / acad7790.ipt.aol.com.

Aug 13 13:30:44 — Logged in for just over 6.5 hours from 172.173.1.192 / acad01c0.ipt.aol.com.

Aug 14 03:10:47 — Logged in for about 5.5 hours from 172.173.161.180 / acada1b4.ipt.aol.com.

Aug 14 10:20:00 (approximate) — The compromise was noticed and the host was unplugged.

How was the hacker running this?

The file a is a shell script wrapper that does this:

  1. Accept a single argument, the prefix of an IP block. For example, 130.172
  2. Use pscan2 to collect a list of SSH servers in that IP block.
  3. Use ssh-scan to try to guess passwords on those SSH servers. The program ssh-scan will then assume:

It is very interesting that while the attack against slim attacked just 1,399 unique logins, the planned attack with that pass_file would have tried 76,940 unique logins and made 96,774 total password guesses. The planned attack was going to be more than 50 times as ambitious. The plan might be to use the weak systems like slim to attack the stronger ones.

Here is the wrapper script a:

#!/bin/bash
if [ $# != 1 ]; then
        echo " usage: $0 <b class>"
        exit;
fi



echo "# Go planet..!"
./pscan2 $1 22 

sleep 10
cat $1.pscan.22 |sort |uniq > mfu.txt
oopsnr2=`grep -c . mfu.txt`
echo "# found $oopsnr2 servers"
echo "------------------------"
echo "# Good Luck!"
./ssh-scan 100
rm -rf $1.pscan.22 mfu.txt
echo "thats all.. wanna play again?"

And, an associated script a3 to create a banner:

clear
echo "****************************Glu presents:**************************"
echo "*.____    .__                      ___________                    *"
echo "*|    |   |__| ____  __ _____  ___ \__    ___/___ _____    _____  *"
echo "*|    |   |  |/    \|  |  \  \/  /   |    |_/ __ \\__  \  /     \  *"
echo "*|    |___|  |   |  \  |  />    <    |    |\  ___/ / __ \|  Y Y  \*"
echo "*|_______ \__|___|  /____//__/\_ \   |____| \___  >____  /__|_|  /*"
echo "*        \/       \/            \/              \/     \/      \/ *"
echo "*         Lastest version of brute force password checker         *"
echo "*                      Enter the dark side!                       *"
echo "*                      Descover the unknown                       *"
echo "*****************Greets to all #Linux-Team members*****************" 

Finally, in the interest of completeness, an outer wrapper script start to automate the scanning of a large block of IP addresses by repeated calls to the script a:

clear
echo "****************************Glu presents:**************************"
echo "*.____    .__                      ___________                    *"
echo "*|    |   |__| ____  __ _____  ___ \__    ___/___ _____    _____  *"
echo "*|    |   |  |/    \|  |  \  \/  /   |    |_/ __ \\__  \  /     \  *"
echo "*|    |___|  |   |  \  |  />    <    |    |\  ___/ / __ \|  Y Y  \*"
echo "*|_______ \__|___|  /____//__/\_ \   |____| \___  >____  /__|_|  /*"
echo "*        \/       \/            \/              \/     \/      \/ *"
echo "*         Lastest version of brute force password checker         *"
echo "*                      Enter the dark side!                       *"
echo "*                      Descover the unknown                       *"
echo "*****************Greets to all #Linux-Team members*****************"
if [ -f a ]; then
./a1
./a2
./a3
./a $1.0
./a $1.1
./a $1.2
./a $1.3
./a $1.4
./a $1.5
./a $1.6
./a $1.7
./a $1.8
./a $1.9
./a $1.10
./a3
./a $1.11
./a $1.12
./a $1.13
./a $1.14
./a $1.15
./a $1.16
./a $1.17
./a $1.18
./a $1.19
./a $1.20
./a2
./a3
./a $1.21
./a $1.22
./a $1.23
./a $1.24
./a $1.25
./a $1.26
./a $1.27
./a $1.28
./a $1.29
./a $1.30
./a3
./a $1.31
./a $1.32
./a $1.33
./a $1.34
./a $1.35
./a $1.36
./a $1.37
./a $1.38
./a $1.39
./a $1.40
./a2
./a3
./a $1.41
./a $1.42
./a $1.43
./a $1.44
./a $1.45
./a $1.46
./a $1.47
./a $1.48
./a $1.49
./a $1.50
./a3
./a $1.51
./a $1.52
./a $1.53
./a $1.54
./a $1.55
./a $1.56
./a $1.57
./a $1.58
./a $1.59
./a $1.60
./a2
./a3
./a $1.61
./a $1.62
./a $1.63
./a $1.64
./a $1.65
./a $1.66
./a $1.67
./a $1.68
./a $1.69
./a $1.70
./a3
./a $1.71
./a $1.72
./a $1.73
./a $1.74
./a $1.75
./a $1.76
./a $1.77
./a $1.78
./a $1.79
./a $1.80
./a2
./a3
./a $1.81
./a $1.82
./a $1.83
./a $1.84
./a $1.85
./a $1.86
./a $1.87
./a $1.88
./a $1.89
./a $1.90
./a3
./a $1.91
./a $1.92
./a $1.93
./a $1.94
./a $1.95
./a $1.96
./a $1.97
./a $1.98
./a $1.99
./a $1.100
./a2
./a3
./a $1.101
./a $1.102
./a $1.103
./a $1.104
./a $1.105
./a $1.106
./a $1.107
./a $1.108
./a $1.109
./a $1.110
./a3
./a $1.111
./a $1.112
./a $1.113
./a $1.114
./a $1.115
./a $1.116
./a $1.117
./a $1.118
./a $1.119
./a $1.120
./a2
./a3
./a $1.121
./a $1.122
./a $1.123
./a $1.124
./a $1.125
./a $1.126
./a $1.127
./a $1.128
./a $1.129
./a $1.130
./a3
./a $1.131
./a $1.132
./a $1.133
./a $1.134
./a $1.135
./a $1.136
./a $1.137
./a $1.138
./a $1.139
./a $1.140
./a2
./a3
./a $1.141
./a $1.142
./a $1.143
./a $1.144
./a $1.145
./a $1.146
./a $1.147
./a $1.148
./a $1.149
./a $1.150
./a3
./a $1.151
./a $1.152
./a $1.153
./a $1.154
./a $1.155
./a $1.156
./a $1.157
./a $1.158
./a $1.159
./a $1.160
./a2
./a3
./a $1.161
./a $1.162
./a $1.163
./a $1.164
./a $1.165
./a $1.166
./a $1.167
./a $1.168
./a $1.169
./a $1.170
./a3
./a $1.171
./a $1.172
./a $1.173
./a $1.174
./a $1.175
./a $1.176
./a $1.177
./a $1.178
./a $1.179
./a $1.180
./a2
./a3
./a $1.181
./a $1.182
./a $1.183
./a $1.184
./a $1.185
./a $1.186
./a $1.187
./a $1.188
./a $1.189
./a $1.190
./a3
./a $1.191
./a $1.192
./a $1.193
./a $1.194
./a $1.195
./a $1.196
./a $1.197
./a $1.198
./a $1.199
./a $1.200
./a2
./a3
./a $1.201
./a $1.202
./a $1.203
./a $1.204
./a $1.205
./a $1.206
./a $1.207
./a $1.208
./a $1.209
./a $1.210
./a3
./a $1.211
./a $1.212
./a $1.213
./a $1.214
./a $1.215
./a $1.216
./a $1.217
./a $1.218
./a $1.219
./a $1.220
./a2
./a3
./a $1.221
./a $1.222
./a $1.223
./a $1.224
./a $1.225
./a $1.226
./a $1.227
./a $1.228
./a $1.229
./a3
./a $1.230
./a $1.231
./a $1.232
./a $1.233
./a $1.234
./a $1.235
./a $1.236
./a $1.237
./a $1.238
./a $1.239
./a2
./a3
./a $1.240
./a $1.241
./a $1.242
./a $1.243
./a $1.244
./a $1.245
./a $1.246
./a $1.247
./a $1.248
./a $1.249
./a3
./a $1.250
./a $1.251
./a $1.252
./a $1.253
./a $1.254
./a2
./a3
./a $1.255
killall -9 a
else 
echo # Ciudat ..Nu Ai Urmat Instructiunile  #
echo # trebui dat mv assh a sau mv scan a   #
echo # orice ai avea tu ... dohh ..         #
killall -9 a
killall -9 pscan2
fi 

The other simultaneous hack

As mentioned in the initial background section, mysql was another account that had a weak password. The network activity makes it pretty obvious that mysql has been taken over by a hacker. Here we use lsof to look at network activity and grep to ignore all that SSH scanning being done by the first hack:

% lsof -n | grep -v '^pscan2 '
COMMAND    PID      USER  FD   TYPE    DEVICE   SIZE    NODE NAME
a1        3901     julie   3u  IPv4     12644            TCP *:2345 (LISTEN)
hpiod     4302      root   0u  IPv4      9591            TCP 127.0.0.1:2208 (LISTEN)
python    4308     hplip   4u  IPv4      9624            TCP 127.0.0.1:45178 (LISTEN)
sshd      4484      root   3u  IPv6     11733            TCP *:ssh (LISTEN)
mysqld    4598     mysql  16u  IPv4     11887            TCP *:mysql (LISTEN)
cupsd     4909    cupsys   1u  IPv4     12231            TCP 127.0.0.1:ipp (LISTEN)
bash     21212     mysql   3u  IPv4   9319772            TCP *:ircd (LISTEN)
apache2  32318      root   4u  IPv6  22302295            TCP *:www (LISTEN)
apache2  32320  www-data   4u  IPv6  22302295            TCP *:www (LISTEN)
apache2  32321  www-data   4u  IPv6  22302295            TCP *:www (LISTEN)
apache2  32323  www-data   4u  IPv6  22302295            TCP *:www (LISTEN) 

Did you spot it?

Process 3901 is from our first hacker — it is a daemon listening on TCP port 2345. You would connect to it with something like this:
  % telnet slim.example.edu 2345
You would be rewarded with that banner shown above and a command prompt.

The hackers that took over the mysql account did something similar. They had a program listening for connections on TCP port 6667, generally used for Internet Relay Chat and listed as ircd in /etc/services. Look at the line for process 21212 above.

The program name is bash, but that's just its name. It isn't really a Bash shell, it's an IRC relay program. It is named bash so it looks less suspicious in the process table. Although what might appear to be a Bash shell listening to a TCP port looks awfully suspicious when you see it in the lsof output!

% lsof -p 21212
COMMAND  PID   USER  FD  TYPE  DEVICE   SIZE       NODE NAME
bash   21212  mysql cwd   DIR     8,1   4096   26345502 /tmp/.ICE-unix/.bash
bash   21212  mysql rtd   DIR     8,1   4096          2 /
bash   21212  mysql txt   REG     8,1 444848   26345750 /tmp/.ICE-unix/.bash/bash
bash   21212  mysql mem   REG     0,0                 0 [heap] (stat: No such file or directory)
bash   21212  mysql   0u  CHR   136,0                 2 /dev/pts/0 (deleted)
bash   21212  mysql   1u  CHR   136,0                 2 /dev/pts/0 (deleted)
bash   21212  mysql   2u  CHR   136,0                 2 /dev/pts/0 (deleted)
bash   21212  mysql   3u IPv4 9319772               TCP *:ircd (LISTEN)
bash   21212  mysql   4w  REG     8,1 148375   26345500 /tmp/.ICE-unix/.bash/log/psybnc.log
bash   21212  mysql   5w  REG     8,1      6   26345835 /tmp/.ICE-unix/.bash/psybnc.pid
bash   21212  mysql   7w  REG     8,1      0   26345837 /tmp/.ICE-unix/.bash/log/USER1.TRL 

Deja vu! These hackers have also hidden their stuff in an obscure place under /tmp. It's the psybnc IRC chatbot. Learn about it at:
   Wikipedia
and
   psybnc.at.

Here is how they set it up. Don't overlook alex's password while you're reading this:

PSYBNC.SYSTEM.PORT1=6667
PSYBNC.SYSTEM.HOST1=*
PSYBNC.HOSTALLOWS.ENTRY0=*;*
USER1.USER.LOGIN=alex
USER1.USER.USER=Chelu de la Resita
USER1.USER.PASS==1l0R`F'J`91z`v`0`o
USER1.USER.RIGHTS=1
USER1.USER.VLINK=0
USER1.USER.PPORT=0
USER1.USER.PARENT=0
USER1.USER.QUITTED=0
USER1.USER.DCCENABLED=1
USER1.USER.AUTOGETDCC=0
USER1.USER.AIDLE=0
USER1.USER.LEAVEQUIT=0
USER1.USER.AUTOREJOIN=1
USER1.USER.SYSMSG=1
USER1.USER.LASTLOG=0
USER1.USER.NICK=Alex\ 

How did they get into the mysql account?

More password guessing, although there is something very odd about how it happened. Here is a list of password guessing attacks against the mysql account. Date, time(s), and number of SSH password authentication failures in the first column, attack source in the second column:

Date / Time(s) / Number Attacker — IP and whois output
Jun 25
22:42:56

1
58.66.176.215
World Crossing Telecom(GuangZhou) Ltd.
17/FL,International Bank Center,
191# DongFengXi Rd. Guang Zhou
China
Jun 27
18:13:34-18:24:37

13
202.43.226.3
TSN Internet
Port Macquarie
New South Wales
Australia
Jun 27
19:02:49

1
85.17.19.6
LeaseWeb B.V.
Amsterdam
Netherlands
Jun 27
23:03:15-07:16:05

36
201.226.233.10
Cable & Wireless Panama
Panama City
Panama
Jul 6
14:30:48-15:22:17

7
59.124.57.147
Xin Nuo Information Co., Ltd.
Taipei
Taiwan
Jul 7
21:23:20-21:31:15

2
88.191.40.16
ProXad Network / Free SAS
Paris
France
Jul 8
14:01:11-14:09:11

2
217.199.186.105
PIPEX Communications
Carlton House
London
U.K.
Jul 8
15:56:30-16:04:40


2
82.109.33.229
Round Foundry Media Centre
Leeds
U.K.
Jul 9
12:44:23-12:47:04

68
66.129.94.149
Venture Internet
Daytona Beach
Florida
U.S.A.
Jul 9
14:55:20

1
219.143.71.21
Chinanet Beijing Province Network
China Telecom
No.31, Jingrong Street
Beijing 100032
China
Jul 10
13:55:39

1
220.66.100.237
Koje College
Jangseungpo-dong, Geoje-si, Gyeongnam
Korea
Jul 31
23:20:08-23:48:29

4
218.4.72.186
Guangfu Middle School Wuzhong District
Suzhou City
Jiangsu Province
China

OK, that's just the normal background noise on the Internet. Hackers are always rattling your doorknobs. But something strange happens just two days after the attack from the Guangfu Middle School:

Aug  2 09:26:10 slim sshd[9581]: Invalid user myaql from 79.114.95.231
Aug  2 09:26:10 slim sshd[9581]: reverse mapping checking getaddrinfo for 79-114-95-231.rdsnet.ro failed - POSSIBLE BREAK-IN ATTEMPT!
Aug  2 09:26:10 slim sshd[9581]: Failed none for invalid user myaql from 79.114.95.231 port 2358 ssh2
Aug  2 09:26:17 slim sshd[9583]: reverse mapping checking getaddrinfo for 79-114-95-231.rdsnet.ro failed - POSSIBLE BREAK-IN ATTEMPT!
Aug  2 09:26:20 slim sshd[9583]: (pam_unix) authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=79.114.95.231  user=mysql
Aug  2 09:26:22 slim sshd[9583]: Failed password for mysql from 79.114.95.231 port 2359 ssh2
Aug  2 09:26:25 slim sshd[9583]: Accepted password for mysql from 79.114.95.231 port 2359 ssh2
Aug  2 09:26:25 slim sshd[9585]: (pam_unix) session opened for user mysql by (uid=0)
Aug  2 09:28:06 slim sshd[9581]: fatal: Timeout before authentication for 79.114.95.231
Aug  2 09:41:30 slim sshd[9585]: (pam_unix) session closed for user mysql 

The IP block 79.114.80.0 - 79.114.95.255, which is the CIDR block 79.114.80.0/20, is assigned to:
RCS & RDS S.A.
FiberLink Customers
Resita City
Romania.

The somewhat panicked error messages about "reverse mapping checking getaddrinfo failed" are because 231.95.114.79.in-addr.arpa resolves to a PTR record of 79-114-95-231.rdsnet.ro, but the fully-qualified hostname 79-114-95-231.rdsnet.ro does not resolve to any A record. Yes, I am convinced that this is a successful attack of some sort, but I don't think that sloppy DNS maintenance by a Romanian ISP had anything to do with it. Usually these messages are false alarms about security, and just point out the sloppiness of many ISPs' DNS servers.

Look at those log entries — this seems to have been done by a human rather than any automated system. First, the remote user mis-typed mysql as myaql (09:26:10). That's just a one-character error on the keyboard. Then they spelled mysql correctly but got the password wrong once (09:26:20). After the 3-second timeout of sshd plus another two seconds of typing, they got the password correct (09:26:25). They were connected for almost exactly 15 minutes before logging out.

There are more password guessing attacks against mysql from all over the place. However, there is no sign that the mysql password was changed! As we saw above, passwd uses PAM, and we would have seen a message when mysql authenticated to the passwd program to change that password. How does this hacker know an obscure mysql password when other hackers made up to 68 guesses without finding it?

Further down the log we see the following:

Aug 11 14:28:12 slim sshd[21192]: Accepted password for mysql from 79.114.94.196 port 1598 ssh2
Aug 11 14:28:12 slim sshd[21194]: (pam_unix) session opened for user mysql by (uid=0)
Aug 11 19:59:11 slim sshd[21194]: (pam_unix) session closed for user mysql
[....]
Aug 12 04:18:29 slim sshd[30110]: Accepted password for mysql from 79.114.95.103 port 1080 ssh2
Aug 12 04:18:29 slim sshd[30122]: (pam_unix) session opened for user mysql by (uid=0)
Aug 12 20:51:54 slim sshd[30122]: (pam_unix) session closed for user mysql
[....]
Aug 13 04:33:26 slim sshd[466]: Accepted password for mysql from 79.114.94.111 port 1127 ssh2
Aug 13 04:33:26 slim sshd[468]: (pam_unix) session opened for user mysql by (uid=0)
[....]
Aug 13 09:45:54 slim sshd[32046]: Accepted password for mysql from 79.114.95.210 port 1081 ssh2
Aug 13 09:45:54 slim sshd[32048]: (pam_unix) session opened for user mysql by (uid=0)
Aug 13 13:57:23 slim sshd[32048]: (pam_unix) session closed for user mysql
[....]
Aug 14 04:28:32 slim sshd[10432]: Accepted password for mysql from 79.114.95.205 port 1083 ssh2
Aug 14 04:28:32 slim sshd[10435]: (pam_unix) session opened for user mysql by (uid=0)
Aug 14 04:37:44 slim sshd[11561]: Accepted password for mysql from 79.114.95.205 port 1089 ssh2
Aug 14 04:37:44 slim sshd[11569]: (pam_unix) session opened for user mysql by (uid=0)
Aug 14 04:40:18 slim sshd[11569]: (pam_unix) session closed for user mysql
Aug 14 08:47:46 slim sshd[10435]: (pam_unix) session closed for user mysql
[....] 

That's a total of six connections from different IP addresses belonging to that Romanian ISP:

Aug 11 14:28:12 - 19:59:11   from 79.114.94.196
Aug 12 04:18:29 - 20:51:54   from 79.114.95.103
Aug 13 04:33:26 - unclosed   from 79.114.94.111
Aug 13 09:45:54 - 13:57:23   from 79.114.95.210
Aug 14 04:28:32 - 08:47:46   from 79.114.95.205
Aug 14 04:37:44 - 04:40:18   from 79.114.95.205 

Look at the psybnc files' timestamps shown below. The hacker did the following:

  1. Copied a tar file or similar archive of the psybnc code to slim.
  2. Extracted the archive. That preserves the timestamps. Remember that the psybnc binary is named bash to try to evade detection. The binary was built in September 2001, a Romanian README file added the following summer, and the archive was created some time after that.
  3. Generated an initial configuration file and started the IRC server. The IRC server wrote its PID into a file. The server was reconfigured seven minutes later.

The IRC server was up and running within the first two minutes of the session on Aug 11. I have no idea why it wasn't done during the initial connection back on Aug 2.

$ ls -ltrF /tmp/.ICE-unix/.bash/
-rw-r--r-- 1 mysql    mysql       369 Aug  8  2000 psybncchk
-rw-r--r-- 1 mysql    mysql      3756 Sep 14  2000 targets.mak
-rw-r--r-- 1 mysql    mysql     17982 Mar 25  2001 COPYING
-rw-r--r-- 1 mysql    mysql      2660 Apr 28  2001 FAQ
-rw-r--r-- 1 mysql    mysql     15738 Jul 15  2001 SCRIPTING
-rw-r--r-- 1 mysql    mysql        85 Jul 22  2001 TODO
-rw-r--r-- 1 mysql    mysql     35076 Jul 23  2001 README
-rw-r--r-- 1 mysql    mysql      1441 Jul 23  2001 Makefile
-rw-r--r-- 1 mysql    mysql       821 Jul 23  2001 config.h
-rw-r--r-- 1 mysql    mysql     22676 Jul 24  2001 CHANGES
-rw-r--r-- 1 mysql    mysql      1742 Sep 18  2001 salt.h
-rw-r--r-- 1 mysql    mysql      5608 Sep 18  2001 makesalt
-rw-r--r-- 1 mysql    mysql       700 Sep 18  2001 makefile.out
-rw-r--r-- 1 mysql    mysql        41 Sep 18  2001 psybnc.md5sum
-rwxr-xr-x 1 mysql    mysql    444848 Sep 18  2001 bash*
-rw-r--r-- 1 mysql    mysql       101 Jun 29  2002 README.ro
drwxr-xr-x 2 mysql    mysql      4096 Feb 23  2003 help/                    
drwxr-xr-x 2 mysql    mysql      4096 Feb 23  2003 src/
drwxr-xr-x 3 mysql    mysql      4096 Feb 23  2003 scripts/
drwxr-xr-x 2 mysql    mysql      4096 Feb 23  2003 motd/
drwxr-xr-x 3 mysql    mysql      4096 Feb 23  2003 menuconf/
drwxr-xr-x 2 mysql    mysql      4096 Feb 23  2003 lang/
drwxr-xr-x 2 mysql    mysql      4096 Feb 23  2003 tools/
[... here is the dividing line between archive above and new below ...]
-rw------- 1 mysql    mysql         6 Aug 11 14:30 psybnc.pid
drwxr-xr-x 2 mysql    mysql      4096 Aug 11 14:30 log/
-rw------- 1 mysql    mysql       465 Aug 11 14:37 psybnc.conf.old
-rw-r--r-- 1 mysql    mysql     11349 Aug 11 14:37 missing.php
-rw------- 1 mysql    mysql       465 Aug 11 14:37 psybnc.conf
drwxr-xr-x 2 mysql    mysql      4096 Aug 14 08:29 psybnc/ 

If the IRC server really logged all activity, it didn't do very much. The log file log/psybnc.log starts like this:

Sat Aug 11 14:30:10 :Listener created :0.0.0.0 port 6667
Sat Aug 11 14:30:10 :Loading all Users..
Sat Aug 11 14:30:10 :No Users found.
Sat Aug 11 14:30:10 :psyBNC2.3BETA-cBtITLdDMSNp started (PID :21212)
Sat Aug 11 14:30:18 :connect from 79-114-94-196.rdsnet.ro
Sat Aug 11 14:30:21 :Noul User:alex (sad) a fsot adaugat de  alex
Sat Aug 11 14:30:33 :User alex () nu are nici un server adaugat
Sat Aug 11 14:31:46 :User alex () trying eu.undernet.org port 6667 ().
Sat Aug 11 14:31:51 :User alex () connected to eu.undernet.org:6667 ()
Sat Aug 11 14:32:00 :Userul alex () A fost deconectat(de la eu.undernet.org) motivul:
		Closing Link: Chsl by London.UK.Eu.Undernet.Org (K-lined)
Sat Aug 11 14:32:09 :User alex () trying us.undernet.org port 6667 ().
Sat Aug 11 14:32:11 :User alex () connected to us.undernet.org:6667 ()
Sat Aug 11 14:32:11 :Userul alex () A fost deconectat(de la us.undernet.org) motivul:
		Closing Link: Chsl by mesa.az.us.undernet.org (Sorry, your co
nnection class is full - try again later or try another server)
Sat Aug 11 14:32:29 :User alex () trying eu.undernet.org port 6667 ().
Sat Aug 11 14:35:38 :User alex: cant connect to eu.undernet.org port 6667.
Sat Aug 11 14:35:47 :User alex () trying us.undernet.org port 6667 ().
Sat Aug 11 14:37:53 :User alex quitted (from 79-114-94-196.rdsnet.ro)
Sat Aug 11 14:38:56 :User alex: cant connect to us.undernet.org port 6667.
Sat Aug 11 14:39:05 :User alex () nu are nici un server adaugat
Sat Aug 11 14:40:53 :User alex () nu are nici un server adaugat
Sat Aug 11 14:42:41 :User alex () nu are nici un server adaugat
Sat Aug 11 14:44:29 :User alex () nu are nici un server adaugat
Sat Aug 11 14:46:17 :User alex () nu are nici un server adaugat
Sat Aug 11 14:48:05 :User alex () nu are nici un server adaugat 

At that point it becomes rather repetitive.... Stripping out the lines containing "User alex () nu are nici un server adaugat" leaves us with this:

Sat Aug 11 14:30:10 :Listener created :0.0.0.0 port 6667
Sat Aug 11 14:30:10 :Loading all Users..
Sat Aug 11 14:30:10 :No Users found.
Sat Aug 11 14:30:10 :psyBNC2.3BETA-cBtITLdDMSNp started (PID :21212)
Sat Aug 11 14:30:18 :connect from 79-114-94-196.rdsnet.ro
Sat Aug 11 14:30:21 :Noul User:alex (sad) a fsot adaugat de  alex
Sat Aug 11 14:31:46 :User alex () trying eu.undernet.org port 6667 ().
Sat Aug 11 14:31:51 :User alex () connected to eu.undernet.org:6667 ()
Sat Aug 11 14:32:00 :Userul alex () A fost deconectat(de la eu.undernet.org) motivul:
		Closing Link: Chsl by London.UK.Eu.Undernet.Org (K-lined)
Sat Aug 11 14:32:09 :User alex () trying us.undernet.org port 6667 ().
Sat Aug 11 14:32:11 :User alex () connected to us.undernet.org:6667 ()
Sat Aug 11 14:32:11 :Userul alex () A fost deconectat(de la us.undernet.org) motivul:
		Closing Link: Chsl by mesa.az.us.undernet.org (Sorry, your connection class is full - try again later or try another server)
Sat Aug 11 14:32:29 :User alex () trying eu.undernet.org port 6667 ().
Sat Aug 11 14:35:38 :User alex: cant connect to eu.undernet.org port 6667.
Sat Aug 11 14:35:47 :User alex () trying us.undernet.org port 6667 ().
Sat Aug 11 14:37:53 :User alex quitted (from 79-114-94-196.rdsnet.ro)
Sat Aug 11 14:38:56 :User alex: cant connect to us.undernet.org port 6667.
Sun Aug 12 05:16:08 :connect from Hera.bergon.net
Sun Aug 12 05:16:08 :Lost Connection from Hera.bergon.net (root)
Mon Aug 13 06:39:34 :connect from fltr241.fltr.ucl.ac.be
Mon Aug 13 06:39:35 :Lost Connection from fltr241.fltr.ucl.ac.be ()
Mon Aug 13 06:39:52 :connect from fltr241.fltr.ucl.ac.be
Mon Aug 13 06:39:54 :Lost Connection from fltr241.fltr.ucl.ac.be ()
Mon Aug 13 06:40:05 :connect from fltr241.fltr.ucl.ac.be
Mon Aug 13 06:40:05 :Lost Connection from fltr241.fltr.ucl.ac.be ()
Mon Aug 13 06:40:11 :connect from fltr241.fltr.ucl.ac.be
Mon Aug 13 06:40:12 :Lost Connection from fltr241.fltr.ucl.ac.be ()
Mon Aug 13 06:40:15 :connect from fltr241.fltr.ucl.ac.be
Mon Aug 13 06:40:16 :connect from fltr241.fltr.ucl.ac.be
Mon Aug 13 06:40:19 :connect from fltr241.fltr.ucl.ac.be
Mon Aug 13 06:40:19 :Lost Connection from fltr241.fltr.ucl.ac.be ()
Mon Aug 13 06:40:19 :connect from fltr241.fltr.ucl.ac.be
Mon Aug 13 06:40:20 :Lost Connection from fltr241.fltr.ucl.ac.be ()
Mon Aug 13 06:40:20 :Lost Connection from fltr241.fltr.ucl.ac.be ()
Mon Aug 13 06:40:24 :Lost Connection from fltr241.fltr.ucl.ac.be (EEEEHEEEE)
Mon Aug 13 06:44:57 :connect from fltr241.fltr.ucl.ac.be
Mon Aug 13 06:44:57 :Lost Connection from fltr241.fltr.ucl.ac.be ()
Mon Aug 13 11:17:31 :connect from betelgeuse.umeqs.maine.edu
Mon Aug 13 11:17:31 :Lost Connection from betelgeuse.umeqs.maine.edu () 

The remaining mystery — how did they know the password on the second attempt, when other hackers made many guesses without hitting it? The best guess I can make is that they figured it out through very aggressive guessing some time before the saved log data starts (June 25). The connection on August 2 was just a test to see if it was still there. But, this seems unlikely to me....

Here are the /etc/shadow entries for julie (changed by hacker) and mysql (apparently unchanged). When I get the chance, I'll see if Crack or John the Ripper can guess the passwords.

julie:$1$8XpGtYaw$pPF41uE5j.HtJYnEgzVXZ/:13733:0:99999:7:::
mysql:$1$tbhIza5j$6Lsr0SwXeLJWeg9hplFUi0:13572:0:99999:7:::