Rack of Ethernet switches.

Visualizing Log Patterns with Color

Apache Logs in Color

Web server logs reveal patterns of activity by web crawlers. Some are indexing crawlers operated by search engines, some are mysterious. Another pattern is systematic blind searches for vulnerable server-side executables or other configuration problems. The pattern you want to see is the interested user who follows some path through the hyperlinks on your site, taking time to read the pages.

Maybe we could use color to help spot these patterns?

Maybe...

First, let's look at the result, then the explanation comes later. Here are the most recent client requests, starting most recent first. Your request for this page won't appear there as it isn't complete by the time this page was automatically generated with PHP. But if you reload the page you should see your initial request near the top.

18.207.255.49 US, United States 05/Dec/2019:15:23:39 /robots.txt
198.74.56.254 US, United States 05/Dec/2019:15:23:38 /travel/romania/bucharest/?s=tb
5.255.250.17 US, United States 05/Dec/2019:15:23:36 /travel/uk/scapa-flow/
198.74.56.254 US, United States 05/Dec/2019:15:23:36 /travel/egypt/giza.html?s=tb
93.122.146.86 RO, Romania 05/Dec/2019:15:23:35 /open-source/performance-tuning/disks.html
5.255.250.17 US, United States 05/Dec/2019:15:23:33 /robots.txt
5.255.250.17 US, United States 05/Dec/2019:15:23:04 /travel/hungary/
220.243.135.13 CN, China 05/Dec/2019:15:22:56 /robots.txt
220.243.135.228 CN, China 05/Dec/2019:15:22:56 /robots.txt
40.77.167.220 US, United States 05/Dec/2019:15:22:51 /travel/uk/glen-nevis/?s=tweetbot
1.221.236.132 KR, Korea, Republic of 05/Dec/2019:15:22:09 /open-source/rhel-centos-5-6-7-8/
207.46.13.199 US, United States 05/Dec/2019:15:22:09 /travel/france/boat/Index.html
81.47.176.43 ES, Spain 05/Dec/2019:15:22:06 /open-source/performance-tuning/tcp.html
149.154.161.12 GB, United Kingdom 05/Dec/2019:15:21:51 /open-source/performance-tuning/ethernet.html
73.239.68.246 US, United States 05/Dec/2019:15:21:16 /travel/france/school-lunch-menus/
174.202.24.43 US, United States 05/Dec/2019:15:21:15 /fun/santaism.html
5.150.103.193 GB, United Kingdom 05/Dec/2019:15:21:12 /travel/uk/skara-brae/
46.229.168.135 US, United States 05/Dec/2019:15:20:57 /travel/japan/kamakura/kencho-ji.html
46.191.233.192 RU, Russian Federation 05/Dec/2019:15:19:55 /open-source/performance-tuning/file-systems.html
36.110.147.102 CN, China 05/Dec/2019:15:19:41 /
136.243.70.68 DE, Germany 05/Dec/2019:15:19:32 /travel/russia/getting-there.html?s=tb
136.243.70.68 DE, Germany 05/Dec/2019:15:19:32 /robots.txt
80.38.7.107 ES, Spain 05/Dec/2019:15:18:47 /open-source/performance-tuning/ethernet.html
5.164.208.0 RU, Russian Federation 05/Dec/2019:15:18:40 /turkish/nouns.html
72.14.199.250 US, United States 05/Dec/2019:15:18:39 /turkish/verbs.html
5.164.208.0 RU, Russian Federation 05/Dec/2019:15:18:31 /turkish/nouns.html
23.108.46.171 US, United States 05/Dec/2019:15:18:20 /open-source/sendmail-ssl.html
66.249.79.248 US, United States 05/Dec/2019:15:18:20 /travel/athens-to-paris/
207.46.13.221 US, United States 05/Dec/2019:15:17:40 /travel/france/boat/meals.html
3.94.53.126 US, United States 05/Dec/2019:15:17:23 /fun/fingerbox/
172.58.83.251 US, United States 05/Dec/2019:15:17:19 /fun/fingerbox/
42.106.81.115 IN, India 05/Dec/2019:15:17:11 /cybersecurity/history/
52.55.217.92 US, United States 05/Dec/2019:15:17:07 /travel/russia/getting-there.html?s=tb
52.55.217.92 US, United States 05/Dec/2019:15:17:07 /
3.92.76.53 US, United States 05/Dec/2019:15:17:07 /travel/turkey/hans/
158.142.6.3 US, United States 05/Dec/2019:15:16:53 /travel/turkey/hans/
2.235.170.207 IT, Italy 05/Dec/2019:15:16:22 /open-source/performance-tuning/disks.html
66.249.79.252 US, United States 05/Dec/2019:15:16:18 /open-source/stig-compliance.html
86.236.228.5 FR, France 05/Dec/2019:15:15:42 /turkish/verbs.html
89.40.127.200 DE, Germany 05/Dec/2019:15:15:18 /travel/china/li-river.html?s=tb
199.16.157.180 US, United States 05/Dec/2019:15:15:17 /travel/usa/new-york-revolutionary/anarchists.html?s=tb
88.99.144.158 DE, Germany 05/Dec/2019:15:14:59 /travel/russia/getting-there.html?s=tb
88.99.144.158 DE, Germany 05/Dec/2019:15:14:58 /robots.txt
94.177.225.58 DE, Germany 05/Dec/2019:15:14:56 /travel/china/li-river.html?s=tb
89.40.114.195 FR, France 05/Dec/2019:15:14:56 /travel/china/li-river.html?s=tb
193.35.46.1 BG, Bulgaria 05/Dec/2019:15:14:47 /open-source/rhel-centos-multimedia.html
184.94.240.92 US, United States 05/Dec/2019:15:14:28 /
3.217.157.17 US, United States 05/Dec/2019:15:14:16 /travel/russia/getting-there.html?s=tb
37.187.162.178 FR, France 05/Dec/2019:15:14:14 /cybersecurity/availability/ddos.html
40.87.13.104 US, United States 05/Dec/2019:15:14:07 /travel/russia/getting-there.html?s=tb
54.167.36.42 US, United States 05/Dec/2019:15:14:02 /ads.txt
17.58.101.13 US, United States 05/Dec/2019:15:14:02 /travel/russia/getting-there.html?s=tb
199.59.150.180 US, United States 05/Dec/2019:15:14:01 /travel/russia/getting-there.html?s=tb
100.26.209.53 US, United States 05/Dec/2019:15:13:53 /cybersecurity/history/uss-chaumont/history.html
42.236.10.121 CN, China 05/Dec/2019:15:13:48 /radio/frequencies.html
89.187.184.194 CZ, Czech Republic 05/Dec/2019:15:13:41 /cybersecurity/history/uss-chaumont/history.html
66.102.9.2 US, United States 05/Dec/2019:15:13:39 /turkish/nouns.html
92.40.248.144 GB, United Kingdom 05/Dec/2019:15:13:39 /3d/xray/film.html
5.134.221.166 RU, Russian Federation 05/Dec/2019:15:13:15 /open-source/performance-tuning/file-systems.html
207.46.13.221 US, United States 05/Dec/2019:15:13:04 /text/reports/3snark.2.txt
143.158.254.129 US, United States 05/Dec/2019:15:12:35 /cybersecurity/basics/06-safe-email-and-web.html
74.64.162.207 US, United States 05/Dec/2019:15:12:14 /travel/belgium/bastogne-ardennes/background.html
3.92.32.166 US, United States 05/Dec/2019:15:11:57 /travel/france/flaneur-1/?s=tb
217.244.4.4 DE, Germany 05/Dec/2019:15:11:31 /open-source/performance-tuning/disks.html
172.58.230.222 US, United States 05/Dec/2019:15:11:11 /travel/usa/new-york-mcgees/
173.144.33.122 US, United States 05/Dec/2019:15:10:06 /technical/ata-ide-sata-usb-cable-pinouts.html
37.187.165.195 FR, France 05/Dec/2019:15:09:35 /travel/japan/kofun/emperor-kaika.html?s=tb
88.99.195.224 DE, Germany 05/Dec/2019:15:09:19 /travel/japan/kofun/emperor-kaika.html?s=tb
3.217.157.17 US, United States 05/Dec/2019:15:09:17 /travel/japan/kofun/emperor-kaika.html?s=tb
54.39.50.78 CA, Canada 05/Dec/2019:15:09:05 /travel/japan/kofun/emperor-kaika.html?s=tb
192.99.46.81 CA, Canada 05/Dec/2019:15:09:05 /travel/japan/kofun/emperor-kaika.html?s=tb
17.58.101.13 US, United States 05/Dec/2019:15:09:01 /travel/japan/kofun/emperor-kaika.html?s=tb
199.59.150.181 US, United States 05/Dec/2019:15:09:01 /travel/japan/kofun/emperor-kaika.html?s=tb
95.163.255.110 RU, Russian Federation 05/Dec/2019:15:08:58 /text/reports/5great.3.txt
54.198.55.229 US, United States 05/Dec/2019:15:08:54 /cybersecurity/availability/ddos.html
66.249.79.250 US, United States 05/Dec/2019:15:08:52 /turkish/Index.html
54.36.149.194 FR, France 05/Dec/2019:15:08:48 /cybersecurity/ssh-attack-study.html
94.130.167.97 DE, Germany 05/Dec/2019:15:08:30 /cybersecurity/availability/ddos.html
143.158.254.129 US, United States 05/Dec/2019:15:08:25 /cybersecurity/basics/10-road-warrior.html
34.218.79.180 US, United States 05/Dec/2019:15:08:25 /cybersecurity/availability/ddos.html
34.218.79.180 US, United States 05/Dec/2019:15:08:23 /cybersecurity/availability/ddos.html
3.217.157.17 US, United States 05/Dec/2019:15:08:15 /cybersecurity/availability/ddos.html
3.217.157.17 US, United States 05/Dec/2019:15:08:15 /robots.txt
45.79.146.81 US, United States 05/Dec/2019:15:08:11 /cybersecurity/availability/ddos.html
138.201.223.94 DE, Germany 05/Dec/2019:15:08:11 /cybersecurity/availability/ddos.html
138.201.223.94 DE, Germany 05/Dec/2019:15:08:10 /robots.txt
54.236.1.18 US, United States 05/Dec/2019:15:08:10 /open-source/raspberry-pi/sdr-ads-b-piaware-and-fr24feed.html
199.16.157.182 US, United States 05/Dec/2019:15:08:01 /cybersecurity/availability/ddos.html
17.58.101.13 US, United States 05/Dec/2019:15:08:00 /cybersecurity/availability/ddos.html
199.59.150.182 US, United States 05/Dec/2019:15:08:00 /cybersecurity/availability/ddos.html
54.236.1.18 US, United States 05/Dec/2019:15:08:00 /open-source/raspberry-pi/sdr-ads-b-piaware-and-fr24feed.html
69.131.243.177 US, United States 05/Dec/2019:15:07:52 /technical/dsl/
143.158.254.129 US, United States 05/Dec/2019:15:07:47 /cybersecurity/basics/02-smart-phone-and-tablet.html
66.249.79.252 US, United States 05/Dec/2019:15:07:44 /open-source/stig-compliance.html
199.172.169.87 US, United States 05/Dec/2019:15:07:32 /open-source/performance-tuning/ethernet.html
95.163.255.91 RU, Russian Federation 05/Dec/2019:15:07:13 /travel/turkey/hans/
95.163.255.80 RU, Russian Federation 05/Dec/2019:15:07:12 /robots.txt
143.158.254.129 US, United States 05/Dec/2019:15:06:47 /cybersecurity/basics/
143.158.254.129 US, United States 05/Dec/2019:15:06:42 /cybersecurity/
96.44.123.158 CA, Canada 05/Dec/2019:15:06:37 /radio/tv-antenna.html
132.198.90.90 US, United States 05/Dec/2019:15:06:15 /open-source/lvm-rescue-boot.html
34.207.217.38 US, United States 05/Dec/2019:15:05:23 /travel/uk/orkney-west-coast-walk/
143.158.254.129 US, United States 05/Dec/2019:15:05:23 /cybersecurity/isc2-cissp/
143.158.254.129 US, United States 05/Dec/2019:15:04:56 /cybersecurity/isc2-ccsp/
108.70.44.226 US, United States 05/Dec/2019:15:04:52 /travel/uk/orkney-west-coast-walk/
143.158.254.129 US, United States 05/Dec/2019:15:04:30 /cybersecurity/isc2-cissp/
178.197.235.166 CH, Switzerland 05/Dec/2019:15:04:14 /open-source/pdf-not-authorized.html
46.229.168.129 US, United States 05/Dec/2019:15:04:10 /technical/how-to-copy-dvds.html
46.229.168.140 US, United States 05/Dec/2019:15:04:08 /robots.txt
24.133.129.34 TR, Turkey 05/Dec/2019:15:03:19 /travel/china/li-river.html?s=tb
52.55.217.92 US, United States 05/Dec/2019:15:03:05 /travel/china/li-river.html?s=tb
52.55.217.92 US, United States 05/Dec/2019:15:03:04 /
5.255.250.17 US, United States 05/Dec/2019:15:02:15 /travel/japan/nikko/takino-shrine-path.html?s=tb
34.255.10.218 IE, Ireland 05/Dec/2019:15:01:53 /travel/usa/new-york-mcgees/
86.133.37.239 GB, United Kingdom 05/Dec/2019:15:01:39 /travel/usa/new-york-mcgees/
109.173.6.216 RU, Russian Federation 05/Dec/2019:15:01:35 /open-source/openbsd-wireless-wpa2.html
207.46.13.199 US, United States 05/Dec/2019:15:00:43 /cybersecurity/cloud-exploit.html
86.179.250.196 GB, United Kingdom 05/Dec/2019:15:00:36 /open-source/linux-break-in-howto.html
90.216.134.198 GB, United Kingdom 05/Dec/2019:14:59:58 /open-source/performance-tuning/disks.html
85.14.245.125 DE, Germany 05/Dec/2019:14:59:33 /travel/uk/orkney-neolithic/stones-of-stenness.html
50.192.196.121 US, United States 05/Dec/2019:14:59:32 /open-source/performance-tuning/ethernet.html
66.249.79.250 US, United States 05/Dec/2019:14:59:23 /open-source/samba-active-directory/slave-dns.html
71.37.212.7 US, United States 05/Dec/2019:14:59:19 /technical/dsl/
85.14.245.125 DE, Germany 05/Dec/2019:14:59:15 /travel/japan/nikko/nikko.html
66.249.79.250 US, United States 05/Dec/2019:14:59:10 /open-source/stig-compliance.html
103.24.99.234 PK, Pakistan 05/Dec/2019:14:59:07 /open-source/performance-tuning/tcp.html
85.14.245.125 DE, Germany 05/Dec/2019:14:58:55 /travel/france/boat/meals.html

Here's what's going on.

Each line above is a request from a client, extracted from Apache's /var/www/logs/access_log file. The client IP address, timestamp, and requested path were selected with awk and the client IP address converted to a country if possible with geoiplookup.

The first 3 octets or first 24 bits of the IP address are used to specify the hue, with chroma at 75% and intensity at 100%. The resulting red, green, and blue values are scaled to the range of 0-255 and printed as two-character hexadecimal in an HTML style string.

Low-numbered /8 networks appear as red, 20.0.0.0/8 through 40.0.0.0/8 are orange shifting to yellow, 50.0.0.0/8 through 110.0.0.0/8 are shades of green, the /16 networks 130.0.0.0/16 through about 180.0.0.0/16 are shades of blue, then it's shades of purple into magenta for the /24 networks 192.0.0.0/24 and up through 223.255.255.0/24.

The HTML file on the server has a line where PHP uses passthru() to call the following shell script:

#!/bin/sh

# Initial pipeline:
# tail		Just the last 200 (or slightly less after the grep)
# grep		... just the requests out of that
# cat | sort	... put into reverse order
# sed		... remove the quotes and square brackets
# awk		... print the IP address twice, timestamp, and requested path
# sed		... remove the first 3 dots to split first version of IP
#			address into octets, and remove any characters that
#			could cause trouble when inserted into this page
# I need to use the client IP address, field #5 at that point, to call
# geoiplookup.  So, send the initial pipeline into a while loop that
# assigns variables, sets a new variable, and then echoes the resulting
# collection into awk.
tail -200 /var/www/logs/access_log |
	grep 'GET.*200' |
	cat -n | sort -nr |
	sed -e 's/"/ /g' -e 's/\[//g' -e 's/\]//g' |
	awk '{print $2, $2, $5, $8}' |
	sed -e 's/\./ /' -e 's/\./ /' -e 's/\./ /' -e 's/[<>]//g' |
	while read IP1 IP2 IP3 IP4 CLIENTIP TIMESTAMP URL
	do
		COUNTRY=$( geoiplookup $CLIENTIP |
				sed 's/.*Edition: //' |
				sed 's/IP Address not found/Unknown/' )
		echo $IP1 $IP2 $IP3 $IP4 $CLIENTIP $COUNTRY $TIMESTAMP $URL |
		awk '{
			ip1 = $1;
			ip2 = $2;
			ip3 = $3;
			chroma = 0.75;
			hue = 6*(ip1*255*255 + ip2*255 + ip3)/(255*255*255);
			if (hue%2 > 1) {
				x = chroma*(1.0 - (hue%2 - 1));
			} else {
				x = chroma*(1.0 - (1 - hue%2));
			}
			if (hue < 1.0) {
				r = chroma;
				g = x;
				b = 0;
			} else if (hue < 2.0) {
				r = x;
				g = chroma;
				b = 0;
			} else if (hue < 3.0) {
				r = 0;
				g = chroma;
				b = x;
			} else if (hue < 4.0) {
				r = 0;
				g = x;
				b = chroma;
			} else if (hue < 5.0) {
				r = x;
				g = 0;
				b = chroma;
			} else {
				r = chroma;
				g = 0;
				b = x;
			}
			r = (r + 0.25)*255;
			g = (g + 0.25)*255;
			b = (b + 0.25)*255;

			printf("<div class=\"col-12 textleft\" ");
			printf("style=\"color:#000; background:#%02x%02x%02x;\"> ", r, g, b);
			for (i = 5; i <= NF; i++) {
				printf("%s ", $i);
			}
			printf("</div>\n");
		}'
	done 

Other Pages