Rack of Ethernet switches.

Visualizing Log Patterns with Color

Nginx and Apache Logs in Color

Web server logs reveal patterns of activity by web crawlers. Some are indexing crawlers operated by search engines, some are mysterious. Another pattern is systematic blind searches for vulnerable server-side executables or other configuration problems. The pattern you want to see is the interested user who follows some path through the hyperlinks on your site, taking time to read the pages.

Maybe we could use color to help spot these patterns?

Maybe...

First, let's look at the result, then the explanation comes later. Here are the most recent client requests, starting most recent first. Your request for this page won't appear there as it isn't complete by the time this page was automatically generated with PHP. But if you reload the page you should see your initial request near the top.

216.73.216.157 US, United States 30/Oct/2025:19:31:48 /technical/samsung-galaxy/sim-unlocking.html
216.73.216.157 US, United States 30/Oct/2025:19:31:48 /open-source/hp-envy-xsane.html
43.157.43.147 JP, Japan 30/Oct/2025:19:31:43 /travel/romania/food/
205.166.94.38 US, United States 30/Oct/2025:19:31:36 /travel/turkey/olimpos/?s=mb
176.92.47.15 GR, Greece 30/Oct/2025:19:31:34 /travel/greece/meteora.html
54.39.203.32 CA, Canada 30/Oct/2025:19:31:33 /open-source/?s=mb
74.7.227.173 US, United States 30/Oct/2025:19:31:32 /networking/
60.242.51.148 AU, Australia 30/Oct/2025:19:31:25 /travel/mexico/cholula-puebla.html
60.242.51.148 AU, Australia 30/Oct/2025:19:31:25 /travel/mexico/cholula-puebla.html?s=mb
60.242.51.148 AU, Australia 30/Oct/2025:19:31:24 /travel/mexico/cholula-puebla.html?s=mb
47.128.30.8 CA, Canada 30/Oct/2025:19:30:57 /travel/turkey/lycia/?s=mb
74.7.36.77 US, United States 30/Oct/2025:19:30:52 /open-source/performance-tuning/nfs.html
142.44.143.196 CA, Canada 30/Oct/2025:19:30:45 /blog/2023/01/03-whats-up-with-my-social-media.html
142.44.143.196 CA, Canada 30/Oct/2025:19:30:44 /
213.133.120.43 DE, Germany 30/Oct/2025:19:30:40 /blog/2023/01/03-whats-up-with-my-social-media.html
213.133.120.43 DE, Germany 30/Oct/2025:19:30:39 /
8.29.198.26 US, United States 30/Oct/2025:19:30:39 /rss.xml
45.150.72.132 Unknown 30/Oct/2025:19:30:21 /blog/2023/01/03-whats-up-with-my-social-media.html
216.73.216.157 US, United States 30/Oct/2025:19:30:02 /open-source/144/
216.73.216.157 US, United States 30/Oct/2025:19:30:01 /open-source/raspberry-pi/sdr-ads-b-piaware-and-fr24feed.html
216.73.216.157 US, United States 30/Oct/2025:19:30:01 /open-source/microphone.html
90.8.209.196 FR, France 30/Oct/2025:19:29:59 /rss.xml
178.128.216.242 GR, Greece 30/Oct/2025:19:29:49 /robots.txt
47.128.43.211 CA, Canada 30/Oct/2025:19:29:43 /travel/uk/scotland-pitlochry/
43.164.197.177 JP, Japan 30/Oct/2025:19:29:42 /travel/athens-to-paris/athens-thessaloniki.html
20.194.1.15 US, United States 30/Oct/2025:19:29:39 /cybersecurity/yubikey/pam_u2f.html
98.85.254.66 US, United States 30/Oct/2025:19:29:37 /open-source/linux-pam-compliance/
178.128.216.242 GR, Greece 30/Oct/2025:19:29:37 /ads.txt
185.218.195.30 FI, Finland 30/Oct/2025:19:29:33 /travel/turkey/olimpos/?s=mb
178.128.216.242 GR, Greece 30/Oct/2025:19:29:32 /robots.txt
66.249.79.5 US, United States 30/Oct/2025:19:29:29 /cybersecurity/history/Index.html
185.183.159.222 DE, Germany 30/Oct/2025:19:29:27 /travel/turkey/olimpos/?s=mb
3.86.9.227 US, United States 30/Oct/2025:19:29:23 /travel/turkey/olimpos/?s=mb
43.164.195.17 JP, Japan 30/Oct/2025:19:29:22 /travel/turkey/istanbul/Index.html
45.150.72.132 Unknown 30/Oct/2025:19:29:15 /travel/turkey/olimpos/?s=mb
137.184.191.81 US, United States 30/Oct/2025:19:29:13 /travel/turkey/olimpos/?s=mb
64.4.175.77 US, United States 30/Oct/2025:19:29:13 /travel/turkey/olimpos/?s=mb
84.247.190.244 NO, Norway 30/Oct/2025:19:29:09 /travel/turkey/olimpos/?s=mb
185.236.240.137 PL, Poland 30/Oct/2025:19:29:07 /travel/turkey/olimpos/?s=mb
65.109.5.123 US, United States 30/Oct/2025:19:29:07 /travel/turkey/olimpos/?s=mb
193.30.122.175 DE, Germany 30/Oct/2025:19:29:06 /travel/turkey/olimpos/?s=mb
65.109.144.230 US, United States 30/Oct/2025:19:29:06 /travel/turkey/olimpos/?s=mb
5.161.182.88 IR, Iran, Islamic Republic of 30/Oct/2025:19:29:06 /travel/turkey/olimpos/?s=mb
69.207.181.246 US, United States 30/Oct/2025:19:29:06 /travel/turkey/olimpos/?s=mb
147.182.159.4 US, United States 30/Oct/2025:19:29:05 /travel/turkey/olimpos/?s=mb
141.95.205.35 GB, United Kingdom 30/Oct/2025:19:29:05 /travel/turkey/olimpos/?s=mb
95.216.74.103 FI, Finland 30/Oct/2025:19:29:05 /travel/turkey/olimpos/?s=mb
132.145.173.44 US, United States 30/Oct/2025:19:29:05 /travel/turkey/olimpos/?s=mb
95.217.150.38 FI, Finland 30/Oct/2025:19:29:05 /travel/turkey/olimpos/?s=mb
51.79.78.172 FR, France 30/Oct/2025:19:29:04 /travel/turkey/olimpos/?s=mb
132.226.208.91 US, United States 30/Oct/2025:19:29:03 /travel/turkey/olimpos/?s=mb
176.28.55.227 DE, Germany 30/Oct/2025:19:29:03 /travel/turkey/olimpos/?s=mb
185.161.251.10 GB, United Kingdom 30/Oct/2025:19:29:03 /travel/turkey/olimpos/?s=mb
5.161.210.238 IR, Iran, Islamic Republic of 30/Oct/2025:19:29:02 /travel/turkey/olimpos/?s=mb
45.79.180.50 US, United States 30/Oct/2025:19:29:02 /travel/turkey/olimpos/?s=mb
140.238.157.69 US, United States 30/Oct/2025:19:29:02 /travel/turkey/olimpos/?s=mb
49.13.224.169 IN, India 30/Oct/2025:19:29:02 /travel/turkey/olimpos/?s=mb
185.233.107.67 DE, Germany 30/Oct/2025:19:29:02 /travel/turkey/olimpos/?s=mb
64.23.175.205 US, United States 30/Oct/2025:19:29:02 /travel/turkey/olimpos/?s=mb
65.21.135.163 US, United States 30/Oct/2025:19:29:02 /travel/turkey/olimpos/?s=mb
2.201.196.8 DE, Germany 30/Oct/2025:19:29:01 /travel/turkey/olimpos/?s=mb
77.174.198.11 NL, Netherlands 30/Oct/2025:19:29:01 /travel/turkey/olimpos/?s=mb
34.162.186.242 US, United States 30/Oct/2025:19:29:01 /travel/turkey/olimpos/?s=mb
142.132.159.116 CA, Canada 30/Oct/2025:19:29:01 /travel/turkey/olimpos/?s=mb
138.201.19.143 DE, Germany 30/Oct/2025:19:29:01 /travel/turkey/olimpos/?s=mb
188.166.88.71 NL, Netherlands 30/Oct/2025:19:29:01 /travel/turkey/olimpos/?s=mb
144.76.90.220 DE, Germany 30/Oct/2025:19:29:00 /travel/turkey/olimpos/?s=mb
173.67.13.106 US, United States 30/Oct/2025:19:29:00 /travel/turkey/olimpos/?s=mb
138.197.36.33 US, United States 30/Oct/2025:19:29:00 /travel/turkey/olimpos/?s=mb
51.210.99.95 FR, France 30/Oct/2025:19:29:00 /travel/turkey/olimpos/?s=mb
43.230.161.184 IN, India 30/Oct/2025:19:29:00 /travel/turkey/olimpos/?s=mb
61.64.0.23 TW, Taiwan 30/Oct/2025:19:29:00 /travel/turkey/olimpos/?s=mb
5.75.251.148 IR, Iran, Islamic Republic of 30/Oct/2025:19:29:00 /travel/turkey/olimpos/?s=mb
142.132.174.6 CA, Canada 30/Oct/2025:19:29:00 /travel/turkey/olimpos/?s=mb
209.38.111.248 US, United States 30/Oct/2025:19:29:00 /travel/turkey/olimpos/?s=mb
54.80.1.9 US, United States 30/Oct/2025:19:29:00 /travel/turkey/olimpos/?s=mb
164.92.127.252 US, United States 30/Oct/2025:19:28:59 /travel/turkey/olimpos/?s=mb
162.19.29.212 US, United States 30/Oct/2025:19:28:59 /travel/turkey/olimpos/?s=mb
65.21.211.243 US, United States 30/Oct/2025:19:28:59 /travel/turkey/olimpos/?s=mb
69.172.146.69 CA, Canada 30/Oct/2025:19:28:59 /travel/turkey/olimpos/?s=mb
85.114.140.116 DE, Germany 30/Oct/2025:19:28:59 /travel/turkey/olimpos/?s=mb
104.28.211.31 US, United States 30/Oct/2025:19:28:59 /travel/turkey/olimpos/?s=mb
142.44.143.196 CA, Canada 30/Oct/2025:19:28:58 /travel/turkey/olimpos/?s=mb
79.112.69.13 RO, Romania 30/Oct/2025:19:28:58 /travel/turkey/olimpos/?s=mb
94.234.67.156 SE, Sweden 30/Oct/2025:19:28:58 /travel/turkey/olimpos/?s=mb
86.20.105.221 GB, United Kingdom 30/Oct/2025:19:28:58 /travel/turkey/olimpos/?s=mb
109.181.40.73 GB, United Kingdom 30/Oct/2025:19:28:58 /travel/turkey/olimpos/?s=mb
134.122.101.11 US, United States 30/Oct/2025:19:28:57 /travel/turkey/olimpos/?s=mb
199.189.201.32 CA, Canada 30/Oct/2025:19:28:57 /travel/turkey/olimpos/?s=mb
77.48.28.218 CZ, Czech Republic 30/Oct/2025:19:28:57 /travel/turkey/olimpos/?s=mb
65.21.62.142 US, United States 30/Oct/2025:19:28:57 /travel/turkey/olimpos/?s=mb
57.128.95.182 BE, Belgium 30/Oct/2025:19:28:57 /travel/turkey/olimpos/?s=mb
57.128.95.182 BE, Belgium 30/Oct/2025:19:28:57 /travel/turkey/olimpos/?s=mb
88.99.233.240 DE, Germany 30/Oct/2025:19:28:57 /travel/turkey/olimpos/?s=mb
195.201.25.153 DE, Germany 30/Oct/2025:19:28:57 /travel/turkey/olimpos/?s=mb
157.180.127.199 SE, Sweden 30/Oct/2025:19:28:57 /travel/turkey/olimpos/?s=mb
65.109.49.87 US, United States 30/Oct/2025:19:28:56 /travel/turkey/olimpos/?s=mb
18.221.193.45 US, United States 30/Oct/2025:19:28:56 /travel/turkey/olimpos/?s=mb
84.246.91.254 SE, Sweden 30/Oct/2025:19:28:56 /travel/turkey/olimpos/?s=mb
87.255.63.166 NL, Netherlands 30/Oct/2025:19:28:56 /travel/turkey/olimpos/?s=mb
178.62.64.18 GB, United Kingdom 30/Oct/2025:19:28:56 /travel/turkey/olimpos/?s=mb
141.95.205.35 GB, United Kingdom 30/Oct/2025:19:28:55 /travel/turkey/olimpos/?s=mb
173.255.213.247 US, United States 30/Oct/2025:19:28:55 /travel/turkey/olimpos/?s=mb
72.85.247.67 US, United States 30/Oct/2025:19:28:55 /travel/turkey/olimpos/?s=mb
162.19.87.99 US, United States 30/Oct/2025:19:28:55 /travel/turkey/olimpos/?s=mb
108.173.160.229 CA, Canada 30/Oct/2025:19:28:55 /travel/turkey/olimpos/?s=mb
188.245.189.55 IR, Iran, Islamic Republic of 30/Oct/2025:19:28:54 /travel/turkey/olimpos/?s=mb
193.7.176.160 DE, Germany 30/Oct/2025:19:28:54 /travel/turkey/olimpos/?s=mb
91.194.60.179 FR, France 30/Oct/2025:19:28:54 /travel/turkey/olimpos/?s=mb
188.34.142.224 IR, Iran, Islamic Republic of 30/Oct/2025:19:28:54 /travel/turkey/olimpos/?s=mb
51.79.96.87 FR, France 30/Oct/2025:19:28:54 /travel/turkey/olimpos/?s=mb
99.154.128.1 US, United States 30/Oct/2025:19:28:53 /travel/turkey/olimpos/?s=mb
57.128.95.173 BE, Belgium 30/Oct/2025:19:28:53 /travel/turkey/olimpos/?s=mb
134.122.76.6 US, United States 30/Oct/2025:19:28:53 /travel/turkey/olimpos/?s=mb
51.75.127.208 FR, France 30/Oct/2025:19:28:53 /travel/turkey/olimpos/?s=mb
51.75.129.125 FR, France 30/Oct/2025:19:28:53 /travel/turkey/olimpos/?s=mb
88.99.91.59 DE, Germany 30/Oct/2025:19:28:53 /travel/turkey/olimpos/?s=mb
153.126.168.112 JP, Japan 30/Oct/2025:19:28:53 /travel/turkey/olimpos/?s=mb
79.219.54.69 DE, Germany 30/Oct/2025:19:28:53 /travel/turkey/olimpos/?s=mb
87.106.57.214 DE, Germany 30/Oct/2025:19:28:52 /travel/turkey/olimpos/?s=mb
195.201.202.11 DE, Germany 30/Oct/2025:19:28:52 /travel/turkey/olimpos/?s=mb
152.172.163.22 CL, Chile 30/Oct/2025:19:28:52 /travel/turkey/olimpos/?s=mb
208.113.130.160 US, United States 30/Oct/2025:19:28:52 /travel/china/hong-kong.html?s=mb
217.180.219.96 GB, United Kingdom 30/Oct/2025:19:28:52 /travel/turkey/olimpos/?s=mb
5.161.176.207 IR, Iran, Islamic Republic of 30/Oct/2025:19:28:52 /travel/turkey/olimpos/?s=mb
13.40.42.235 US, United States 30/Oct/2025:19:28:51 /travel/turkey/olimpos/?s=mb
99.9.136.110 US, United States 30/Oct/2025:19:28:51 /travel/turkey/olimpos/?s=mb
212.52.0.110 NL, Netherlands 30/Oct/2025:19:28:51 /travel/turkey/olimpos/?s=mb
82.98.169.88 ES, Spain 30/Oct/2025:19:28:51 /travel/turkey/olimpos/?s=mb
13.40.141.41 US, United States 30/Oct/2025:19:28:51 /travel/turkey/olimpos/?s=mb
209.145.54.83 US, United States 30/Oct/2025:19:28:51 /travel/turkey/olimpos/?s=mb
5.75.135.52 IR, Iran, Islamic Republic of 30/Oct/2025:19:28:51 /travel/turkey/olimpos/?s=mb
217.76.60.188 SE, Sweden 30/Oct/2025:19:28:51 /travel/turkey/olimpos/?s=mb
38.242.215.131 US, United States 30/Oct/2025:19:28:51 /travel/turkey/olimpos/?s=mb
46.4.252.37 DE, Germany 30/Oct/2025:19:28:50 /travel/turkey/olimpos/?s=mb
167.71.23.178 US, United States 30/Oct/2025:19:28:50 /travel/turkey/olimpos/?s=mb
65.109.134.148 US, United States 30/Oct/2025:19:28:50 /travel/turkey/olimpos/?s=mb
52.36.226.208 US, United States 30/Oct/2025:19:28:50 /travel/turkey/olimpos/?s=mb
167.235.146.56 US, United States 30/Oct/2025:19:28:50 /travel/turkey/olimpos/?s=mb
159.65.182.92 US, United States 30/Oct/2025:19:28:49 /travel/turkey/olimpos/?s=mb
144.48.38.94 AU, Australia 30/Oct/2025:19:28:49 /travel/turkey/olimpos/?s=mb
51.81.67.122 FR, France 30/Oct/2025:19:28:49 /travel/turkey/olimpos/?s=mb
5.78.43.69 IR, Iran, Islamic Republic of 30/Oct/2025:19:28:49 /travel/turkey/olimpos/?s=mb
159.69.72.33 DE, Germany 30/Oct/2025:19:28:49 /travel/turkey/olimpos/?s=mb
138.199.235.109 EU, Europe 30/Oct/2025:19:28:49 /travel/turkey/olimpos/?s=mb
37.27.248.47 IR, Iran, Islamic Republic of 30/Oct/2025:19:28:48 /travel/turkey/olimpos/?s=mb
37.27.222.192 IR, Iran, Islamic Republic of 30/Oct/2025:19:28:48 /travel/turkey/olimpos/?s=mb
93.92.199.194 RU, Russian Federation 30/Oct/2025:19:28:48 /travel/turkey/olimpos/?s=mb
142.132.227.141 CA, Canada 30/Oct/2025:19:28:48 /travel/turkey/olimpos/?s=mb
144.76.183.217 DE, Germany 30/Oct/2025:19:28:48 /travel/turkey/olimpos/?s=mb
91.98.43.143 IR, Iran, Islamic Republic of 30/Oct/2025:19:28:48 /travel/turkey/olimpos/?s=mb
199.167.29.8 US, United States 30/Oct/2025:19:28:47 /travel/turkey/olimpos/?s=mb
65.109.22.225 US, United States 30/Oct/2025:19:28:47 /travel/turkey/olimpos/?s=mb
192.99.237.212 CA, Canada 30/Oct/2025:19:28:47 /travel/turkey/olimpos/?s=mb
46.4.252.37 DE, Germany 30/Oct/2025:19:28:47 /travel/turkey/olimpos/?s=mb
49.12.170.238 IN, India 30/Oct/2025:19:28:47 /travel/turkey/olimpos/?s=mb
130.61.142.152 DE, Germany 30/Oct/2025:19:28:47 /travel/turkey/olimpos/?s=mb
116.203.63.138 IN, India 30/Oct/2025:19:28:47 /travel/turkey/olimpos/?s=mb
79.117.22.65 RO, Romania 30/Oct/2025:19:28:47 /travel/turkey/olimpos/?s=mb
143.244.177.66 US, United States 30/Oct/2025:19:28:46 /travel/turkey/olimpos/?s=mb
88.99.213.22 DE, Germany 30/Oct/2025:19:28:46 /travel/turkey/olimpos/?s=mb

Here's what's going on.

Each line above is a request from a client, extracted from Nginx's /var/www/logs/httpd-access.log file. The client IP address, timestamp, and requested path were selected with awk and the client IP address converted to a country if possible with geoiplookup.

Geolocate IP

You can use a service such as Abstract's IP geolocation to check if the conversion was successful, or if the client IP address is the exit portal of a VPN.

The first 3 octets or first 24 bits of the IP address are used to specify the hue, with chroma at 75% and intensity at 100%. The resulting red, green, and blue values are scaled to the range of 0-255 and printed as two-character hexadecimal in an HTML style string.

Low-numbered /8 networks appear as red, 20.0.0.0/8 through 40.0.0.0/8 are orange shifting to yellow, 50.0.0.0/8 through 110.0.0.0/8 are shades of green, the /16 networks 130.0.0.0/16 through about 180.0.0.0/16 are shades of blue, then it's shades of purple into magenta for the /24 networks 192.0.0.0/24 and up through 223.255.255.0/24.

The HTML file on the server has a line where PHP uses passthru() to call the following shell script:

#!/bin/sh

# Initial pipeline:
# tail		Just the last 200 (or slightly less after the grep)
# grep		... just the requests out of that
# cat | sort	... put into reverse order
# sed		... remove the quotes and square brackets
# awk		... print the IP address twice, timestamp, and requested path
# sed		... remove the first 3 dots to split first version of IP
#			address into octets, and remove any characters that
#			could cause trouble when inserted into this page
# I need to use the client IP address, field #5 at that point, to call
# geoiplookup.  So, send the initial pipeline into a while loop that
# assigns variables, sets a new variable, and then echoes the resulting
# collection into awk.
tail -200 /var/www/logs/access_log |
	grep 'GET.*200' |
	cat -n | sort -nr |
	sed -e 's/"/ /g' -e 's/\[//g' -e 's/\]//g' |
	awk '{print $2, $2, $5, $8}' |
	sed -e 's/\./ /' -e 's/\./ /' -e 's/\./ /' -e 's/[<>]//g' |
	while read IP1 IP2 IP3 IP4 CLIENTIP TIMESTAMP URL
	do
		COUNTRY=$( geoiplookup $CLIENTIP |
				sed 's/.*Edition: //' |
				sed 's/IP Address not found/Unknown/' )
		echo $IP1 $IP2 $IP3 $IP4 $CLIENTIP $COUNTRY $TIMESTAMP $URL |
		awk '{
			ip1 = $1;
			ip2 = $2;
			ip3 = $3;
			chroma = 0.75;
			hue = 6*(ip1*255*255 + ip2*255 + ip3)/(255*255*255);
			if (hue%2 > 1) {
				x = chroma*(1.0 - (hue%2 - 1));
			} else {
				x = chroma*(1.0 - (1 - hue%2));
			}
			if (hue < 1.0) {
				r = chroma;
				g = x;
				b = 0;
			} else if (hue < 2.0) {
				r = x;
				g = chroma;
				b = 0;
			} else if (hue < 3.0) {
				r = 0;
				g = chroma;
				b = x;
			} else if (hue < 4.0) {
				r = 0;
				g = x;
				b = chroma;
			} else if (hue < 5.0) {
				r = x;
				g = 0;
				b = chroma;
			} else {
				r = chroma;
				g = 0;
				b = x;
			}
			r = (r + 0.25)*255;
			g = (g + 0.25)*255;
			b = (b + 0.25)*255;

			printf("<div class=\"col-12 textleft\" ");
			printf("style=\"color:#000; background:#%02x%02x%02x;\"> ", r, g, b);
			for (i = 5; i <= NF; i++) {
				printf("%s ", $i);
			}
			printf("</div>\n");
		}'
	done 

Other Pages