UNIX / Linux keyboard.

How to enable compression with mod_gzip

Using the modified Apache web server in OpenBSD

Hosted with Apache. Hosted on OpenBSD.

Apache's httpd is a good web server. OpenBSD is a good operating system. How can we make the combination work even better?

We can add compression of the HTTP data transfer if the client's browser supports it (and unless they are using something truly ancient or bizarre, it should).

This should improve page load time and it may significantly reduce bandwidth utilization at both the server and client ends. How much of an improvement? Since images are already compressed, the improvement you get will depend on how much of a typical page is HTML (the content and the markup, including style details) as opposed to image comment.

But won't this increase the server CPU load? Surprisingly, not nearly as much as you would expect! That is, unless you are dynamically generating your pages, and maybe not even then. Linux Journal had an article about this.

If you are concerned, here is the result of using PHP to run the command:
top -d 1
on this server as it was generating this page. Really.

load averages:  1.17,  0.70,  0.49    21:00:06
33 processes:  1 running, 31 idle, 1 on processor
CPU states:  0.8% user,  0.0% nice,  3.0% system,  0.1% interrupt, 96.1% idle
Memory: Real: 22M/484M act/tot  Free: 1525M  Swap: 0K/262M used/tot
31189 www        2    0 3264K 5424K sleep    netio    76:51  0.05% httpd
 9343 www        2    0 3296K 5448K sleep    netcon   77:31  0.00% httpd
 7174 www        2    0 3268K 5400K sleep    netcon   77:06  0.00% httpd
28688 www        2    0 3260K 5408K sleep    netcon   76:35  0.00% httpd
21990 www        2    0 3296K 5428K sleep    netcon   76:30  0.00% httpd
 9145 www        2    0 3276K 5416K sleep    netcon   76:17  0.00% httpd
23750 www        2    0 3244K 5392K sleep    netcon   75:39  0.00% httpd
16275 www        2    0 3280K 5416K sleep    netcon   75:19  0.00% httpd
 2503 www        2    0 3180K 5328K sleep    netcon   74:58  0.00% httpd
21631 www       -6    0 3296K 5416K sleep    piperd   74:31  0.00% httpd
 2480 root       2    0 2936K 6428K sleep    select    3:03  0.00% httpd
23990 root       2    0  568K 1248K sleep    select   23:28  0.00% sshd
13511 root       2    0 1072K 1744K sleep    select    0:02  0.00% sendmail
 2794 _syslogd   2    0  592K  716K sleep    poll      8:49  0.00% syslogd
18640 root       2    0  484K  852K sleep    select    0:00  0.00% cron
20808 _ntp       2    0  468K  752K idle     poll      0:13  0.00% ntpd
    1 root      10    0  416K  296K sleep    wait      0:53  0.00% init
15980 root       2    0  472K  716K idle     poll      0:00  0.00% ntpd

You can click the Reload button and you will see that things change slightly. The time (end of first line) is local time at the server, UTC-4. I would expect the CPU to be almost 100% idle, unless you happen to have loaded this page while the server is busy running a scheduled job or some other infrequent task.

What are we working with?

OpenBSD comes with something that isn't exactly Apache's httpd. The web server is the result of an OpenBSD code audit, bug patching, and hardening of something out of the Apache 1.3 web server product line. As the OpenBSD documentation says, "The OpenBSD team has added default chrooting, privilege revocation, and other security-related improvements."

Step 1 — Install the mod_gzip package

The mod_gzip Apache module is not included in the stock OpenBSD distribution. It is part of the packages and ports system. If you have downloaded the full set of packages, you have the file you need. If not, it can be as simple as this if you use csh/tcsh:

# setenv PKG_PATH ftp://ftp.openbsd.org/pub/OpenBSD/`uname -r`/packages/`uname -m`
# pkg_add mod_gzip 

Or, for bash/ksh:

# PKG_PATH=ftp://ftp.openbsd.org/pub/OpenBSD/`uname -r`/packages/`uname -m`
# pkg_add mod_gzip 

Notice that those are backquotes for command substitution! The two uname commands run first, producing output similar to 4.7 and i386. Those results are then placed into the assignment of the shell variable.

Step 2 — Modify your Apache configuration file /var/www/conf/httpd.conf

Michael Schröpl has a nice description of how to get mod_gzip to work in many situations, click here to see that. I have based what I did on his work, with some additions required for OpenBSD or any other similarly configured chroot environment.

Become root, start editing /var/www/conf/httpd.conf, and search for the string mod_gzip. When you installed that package, a post-install script within the package should have added a line loading that module. If it did not, find the area of the file where modules are loaded and add a similar line (and, to help yourself in the future, a comment!) like the following:

# Add compression
LoadModule gzip_module    /usr/lib/apache/modules/mod_gzip.so 

Now add a stanza directly below that. It should be similar to the following, but you may want to adjust the minimum and maximum file sizes:

<IfModule mod_gzip.c>
	# Turn it on:
	mod_gzip_on                     Yes
	# Enable "partial content negotiation" and the serving
	# of compressed versions if those files are available,
	# along with the updating of the compressed versions
	# when the originals are updated:
	mod_gzip_can_negotiate          Yes
	mod_gzip_static_suffix          .gz
	AddEncoding                     gzip .gz
	mod_gzip_update_static          No
	# Allow the chunks to be joined into one compresable packet
	# with a HTTP head deleted:
	mod_gzip_dechunk                Yes
	# Specify a range of file sizes in bytes for compression.
	# There's no point in compressing tiny files, and compression
	# of enormous files will delay the start of transmission.
	mod_gzip_minimum_file_size      500
	mod_gzip_maximum_file_size      6000000
	# Allow any client speaking HTTP/1.0 and later and behaving
	# in a reasonable fashion:
	mod_gzip_min_http               1000
	mod_gzip_handle_methods         GET POST
	# Compress anything named *.html and of MIME type text/*:
	mod_gzip_item_include           file  \.html$
	mod_gzip_item_include           mime  ^text/.*
	# Go ahead and compress CSS and JavaScript.  This might confuse
	# really old Netscape, but no one should be running that now.
	mod_gzip_item_include           file  \.css$
	mod_gzip_item_include           file  \.js$
	mod_gzip_item_include           mime  ^appliation/x-javascript.*
	# Do NOT compress images, there should be no point!
	mod_gzip_item_exclude           mime  ^image/.$
	# Where is the working area for temporary files and the
	# compression cache?
	mod_gzip_temp_dir               /tmp
	# See Michael Schröpl's page for optional logging in case
	# you need to do some debugging.

Step 3 — Set up the chroot environment

Here is the trick required to get this to work on OpenBSD or any other chroot environment, something that isn't frequently mentioned in the documentation.

Remember that you told your compression module to use /tmp as a working area. However, the OpenBSD web server process is using chroot to run in an unusual environment (unless you have changed /etc/rc.conf to make it behave otherwise).

The httpd process is running in a "sandbox" of sorts. Its notion of the root of the file system isn't the real root. This means that there is no ".." directory, no way to go up one level and get out of the chroot jail (or gaol as it's spelled in Britain).

What the httpd process thinks is / is really the /var/www directory. So, we have to create something that will appear to be a proper /tmp for that process. Do this:

# cd /var/www
# mkdir tmp
# chmod 1777 tmp 

Step 4 — Let's see if that works!

Stop the running web server process:

# pkill httpd 

Make sure that really worked:

# ps axuww | egrep 'PID|httpd'
# lsof -i tcp:80 

You may need to give an active request a moment or two to complete. And lsof is in /usr/local/sbin if root doesn't have that in the path.

Once the coast is clear, start a fresh web server process:

# /usr/sbin/httpd 

Yes, I probably could have done this with a HUP signal, but killing it off and starting a new one makes it a little easier to avoid sysadmin confusion. If you are concerned about not refusing a single request, look into doing it this way:

# pkill -HUP httpd 

Who cares about mod_gzip, how did you do that PHP and top trick?

This page exists as an HTML file on the server containing a block like the below. The web server process executes the PHP code on the server and inserts the output in the page.

<pre style="background: #000000; color: #00ff00;
		font-size: 85%;
		padding: 5px 3px 5px 3px;
		border: medium solid orange;">
<?php @ passthru ('top -d 1 | grep -v "^$"'); ?>

And, because of the chroot issue discussed above I had to do the following on the server to get those two commands to work:

# cd /var/www
# mkdir usr usr/bin
# cp /bin/sh bin
# cp /usr/bin/grep /usr/bin/top usr/bin
# ldd bin/sh usr/bin/*
  [ ... hmm, need to include some shared libraries ... ]
# mkdir usr/lib usr/libexec
# cp /usr/lib/libc.so.* /usr/lib/libcurses.so.* /usr/lib/libz.so.* /usr/lib
# cp /usr/libexec/ld.so usr/libexec