The Project So FarGetting Started: Log Analysis and TLS Versions Configuring Nginx for Security and Performance
This page explains how to build both the cryptographic
toolkit OpenSSL and the web server Nginx from source code.
You shouldn't need to do this now, you should be able to proceed directly to configuring Nginx for an A+ Qualys score. That is, unless you're stuck using an old Linux distribution. If you are in that situation, then whatever requirement keeps you on an old Linux release might also prevent using software downloaded and built from source.
But, to help those people who do need to build their own OpenSSL or Nginx software, let's see how to do it!
What Do We Need?Try Google Cloud Platform and receive $50
When I first did this project in September, 2018, the Apache web server did not support TLS 1.3. It didn't matter if you had the needed shared libraries. Apache still thought that a directive to include TLS 1.3 was incorrect configuration syntax. From the Apache release notes:
This release is compatible with OpenSSL versions from 0.9.8a to 1.1.0 only, and does not support TLSv1.3. Future releases of httpd 2.4 are expected to add compatibility with OpenSSL 1.1.1 and enable support for TLSv1.3.
There were some patches that could make it happen, but that would make ongoing maintenance too much of a hassle.
Nginx was the clear choice of web server. It uses the OpenSSL cryptographic libraries to do TLS.FreeBSD on Google Compute Platform
I was doing this project on FreeBSD, where OpenSSL is included in the base operating system and Nginx is a standard package. However, FreeBSD 11.2-RELEASE included earlier versions: OpenSSL 1.0.2o and Nginx 1.14.0. (That version of Nginx supported TLS 1.3, but it was compiled to use the standard system OpenSSL libraries, which I didn't want to replace.) At the time, Mageia Linux had OpenSSL 1.0.2p, and Mint Linux had OpenSSL 1.1.0g.
The FreeBSD Ports system provides a way of getting more or less the same things. I needed to build Nginx and OpenSSL from source.
Getting The Software
First, download the current version of Nginx.
OpenSSL 1.1.1 had been released on September 11th, 2018, just 3 to 4 weeks after RFC 8446 was published, formally defining TLS 1.3. There had been a series of 28 drafts of that document. Some earlier versions of OpenSSL support Draft #28, which ended up being the final definition. But let's start with the official versions!
Three years later, in September 2021, OpenSSL 3.0.0 was released. Download the current version of OpenSSL.Verifying
Download the compressed archive files, along with
associated digital signatures and hashes.
Verify digital signatures and hashes, and make sure you're getting the real software from the original organization.
Here's what I get for the SHA-2-256 hashes of recent versions of the uncompressed archives:
$ openssl sha256 *.tar SHA256(nginx-1.21.6.tar)= 1d72f4ec78e6df031fda3657c8bb73908ac3c5f545da2e75c7506b72cd1ce8e3 SHA256(openssl-3.0.3.tar)= 7f70f1f48314633b6a3efb1e80dc8b39620480e6cf13a82867059c1c5f331271
About FIPS ComplianceFIPS PUB 140-2
Security Requirements for Cryptographic Modules
Simple mention of "FIPS" is a casual reference to a formal U.S. Government standard, FIPS-140-2 (or the Federal Information Processing Standard 140-2), published on 25 May 2001. That document enumerates some cryptographic algorithms for encryption and decryption, hashing, and hashed message authentication codes (or HMAC), and it also involves the certification, approval, or acceptance of implementations of those.
Anyone can speak of "FIPS" or "FIPS compliant" in various ways, often without carefully thinking about what they mean or what the terminology may imply.
Let's be careful. Saying that a software package or shared library "implements the FIPS cipher suite" or that it is "FIPS compliant" might mean any of the following, in increasing order of significance and formality:
- It implements all the algorithms defined in the current FIPS-140-* document.
- Like #1 but also that the source code has been audited by U.S. NIST (National Institute of Standards and Technology, considered the authority by other US government agencies) and they found it to be a complete and correct implementation of the cipher suite.
- Like #2 but also that the vendor (of the distribution, for Linux, or the organization for, e.g., FreeBSD) has been approved as a trusted group with an approved patch notification system, including digitally signed update packages. This includes both technical analysis of the code and organizational approval of the vendor.
Finally, you can, and for U.S. Government compliance you must,
disable the use of non-FIPS ciphers and hash functions.
First, do this for the kernel to restrict its use of algorithms
in IPsec and encrypted storage.
fips=1 to the kernel command line by
adding it to the
rebuilding the GRUB configuration files,
# grep GRUB_CMDLINE_LINUX /etc/default/grub GRUB_CMDLINE_LINUX_DEFAULT="quiet fips=1" GRUB_CMDLINE_LINUX="quiet fips=1" # update-grub # reboot
Beware that systemd meddling on
RHEL 8 and CentOS Stream 8
very likely breaks the above simple solution,
forcing you to deal with the file:
/boot/loader/entries/$(cat /etc/machine-id)-$(uname -r).conf
See these pages for hints:
Configuration Apache FIPS
Configuration OpenSSH FIPS
Then you will need to reconfigure applications to only use
the approved algorithms.
in this series for how to do that for Nginx,
httpd web server.
For the SSH service, edit
and specify FIPS algorithms for the
sshd_config manual page
Level #1 in my numbered list above is basically "Hey, we tried" and no more, while #3 is required in a U.S. Department of Defense setting. What do you need? If you are working for U.S. DoD or some other U.S. Government agencies, or industry working for them, you need #3. Otherwise, you must decide this for yourself.
Another way to think of this is that my #1 is about capability, but without any formal analysis or trust, while #3 is about analysis and certification of code and processes by a trusted entity. Unless you must operate within U.S. Government constraints, there is no simple "correct answer" here.
The OpenSSL 3.0 FIPS module was submitted for validation in September 2021, turned in to US NIST's Cryptographic Module Validation Program.
Specific implementations and packaging by Oracle, Canonical/Ubuntu, Red Hat, and others, plus the OpenSSL project itself, were under review by February 2022.
Building The Software
Where should the software be installed? I have to decide that before I start building it. The standard locations are:
For booting the system,
rescues, and maintenance:
/bin— user programs
/sbin— system programs
/lib— shared libraries
Full running environment:
/usr/bin— user programs
/usr/sbin— system programs
/usr/lib— shared libraries
/usr/share— manual pages, data
/usr/local/bin— user programs
/usr/local/sbin— system programs
/usr/local/lib— shared libraries
/usr/local/share— manual pages, data
I didn't want to use any of those!
I would put the new custom-built OpenSSL under
and the new custom-built Nginx under
They would create their own hierarchy of subdirectories,
share having the same meanings.
This way I could have multiple custom-built versions
installed at the same time.
I could test the very latest but still have the previous
one(s) available as a fall-back.
Then I could easily remove an older version I no longer need.
You also need to decide whether or not you want to
install the latest version of OpenSSL.
When you build Nginx, it compiles the OpenSSL code
to create static libraries
libcrypto.a and then embeds them
So, you don't have to independently build OpenSSL
to build the latest Nginx using the latest OpenSSL.
Extract The OpenSSL Archive
Extract the OpenSSL source code archive. In the following examples, I did this in my home directory.
$ tar xf openssl-3.0.3.tar.xz
The compile times reported in the following are what I saw
on my server, a FreeBSD system running in the
Google Cloud Platform.
dmesg immediately after booting
reports this about the CPU and memory:
[... lines omitted ...] CPU: Intel(R) Xeon(R) CPU @ 2.20GHz (1595.81-MHz K8-class CPU) Origin="GenuineIntel" Id=0x406f0 Family=0x6 Model=0x4f Stepping=0 Features=0x1f83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS,HTT> Features2=0xfefa3203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV> AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM> AMD Features2=0x121<LAHF,ABM,Prefetch> Structured Extended Features=0x1c2ffb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,FDPEXC,SMEP,BMI2,ERMS,INVPCID,RTM,NFPUSG,RDSEED,ADX,SMAP> Structured Extended Features3=0xac000400<MD_CLEAR,IBPB,STIBP,ARCH_CAP,SSBD> XSAVE Features=0x1<XSAVEOPT> IA32_ARCH_CAPS=0x4c<RSBA,SKIP_L1DFL_VME> TSC: P-state invariant Hypervisor: Origin = "KVMKVMKVM" real memory = 1073741824 (1024 MB) avail memory = 1004154880 (957 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: <Google GOOGAPIC> FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs FreeBSD/SMP: 1 package(s) x 1 core(s) x 2 hardware threads [... many more lines omitted ...]
This is easy.
Beginning with OpenSSL 3.0.2, Perl must be installed.
enable-fips parameter included,
it will build the FIPS module but it will not enforce its use.
$ cd openssl-3.0.3 $ ./config --prefix=/usr/local/openssl-3.0.3 \ --openssldir=/usr/local/openssl-3.0.3 \ enable-fips Operating system: amd64-whatever-freebsd Configuring OpenSSL version 3.0.3 for BSD-x86_64 Using os-specific seed configuration Creating configdata.pm Running configdata.pm Creating Makefile.in Creating Makefile ********************************************************************** *** *** *** OpenSSL has been successfully configured *** *** *** *** If you encounter a problem while building, please open an *** *** issue on GitHub <https://github.com/openssl/openssl/issues> *** *** and include the output from the following command: *** *** *** *** perl configdata.pm --dump *** *** *** *** (If you are new to OpenSSL, you might want to consult the *** *** 'Troubleshooting' section in the INSTALL file first) *** *** *** **********************************************************************
Build OpenSSL, If You Want ToOnce configured you can skip ahead
Continue with this section if you want to build and install the latest OpenSSL command and shared libraries. Or, if you just want to build Nginx with static libraries for the latest OpenSSL, you can skip ahead once you have configured OpenSSL.
The 1.1.1x series of OpenSSL required 12–14 minutes to compile on a single CPU.
The 3.0.x version required just over one hour! It required 1h 01m 29s the first time I compiled 3.0.0.
$ make [... much output, about one hour of compiling on a single CPU ...] $ su root -c 'make install' [... asked for root password, output of installing thousands of files, including generating manual pages with pod2man and mkpod2html.pl ...]
Once it's installed, you have to set the LD_LIBRARY_PATH environment variable before you can use the new binary.
$ /usr/local/openssl-3.0.0/bin/openssl version ld-elf.so.1: Shared object "libssl.so.3" not found, required by "openssl" $ bash $ export LD_LIBRARY_PATH=/usr/local/openssl-3.0.0/lib $ /usr/local/openssl-3.0.0/bin/openssl version OpenSSL 3.0.0 7 sep 2021 (Library: OpenSSL 3.0.0 7 sep 2021) $ exit
You can similarly ask for the manual pages from the new custom version.
$ man ciphers [... manual page for software included with the OS ...] $ man -M /usr/local/openssl-3.0.3/share/man ciphers [... manual page for custom-built version 3.0.3 ...]
You might have noticed that version 1.1.1* supported fewer ciphers than 1.0.2*. This is because the newer versions are more secure, by refusing to use weaker ciphers like DES, 3DES, and RC4. Compare the results of these two commands:
$ openssl ciphers | egrep --color 'DES|RC4' [... output including DES, 3DES, and RC4 variants ...] $ /usr/local/openssl-3.0.3/bin/openssl ciphers | egrep --color 'DES|RC4' [... no output ...]
Now Build Nginx
Now we can build Nginx,
telling it where to install itself,
what to build,
and where to find the OpenSSL code.
Notice that you tell it where the OpenSSL
source code is located, and not where the
openssl program and libraries and manual
pages were installed.
/home/cromwell in the below to your
home directory, or wherever you have the OpenSSL source code.
Notice the added
That's needed, as otherwise it will not use
enable-fips parameter included earlier.
$ cd $ tar xf nginx-1.21.6.tar.gz $ cd nginx-1.21.6 $ ./configure --prefix=/usr/local/nginx-1.21.6 \ --with-http_ssl_module \ --with-http_v2_module \ --with-http_geoip_module \ --with-http_gunzip_module \ --with-http_sub_module \ --with-openssl=/home/cromwell/openssl-3.0.3 \ --with-openssl-opt=enable-fips [... much output ...] Configuration summary + using system PCRE2 library + using OpenSSL library: /home/cromwell/openssl-3.0.3 + using system zlib library nginx path prefix: "/usr/local/nginx-1.21.6" nginx binary file: "/usr/local/nginx-1.21.6/sbin/nginx" nginx modules path: "/usr/local/nginx-1.21.6/modules" nginx configuration prefix: "/usr/local/nginx-1.21.6/conf" nginx configuration file: "/usr/local/nginx-1.21.6/conf/nginx.conf" nginx pid file: "/usr/local/nginx-1.21.6/logs/nginx.pid" nginx error log file: "/usr/local/nginx-1.21.6/logs/error.log" nginx http access log file: "/usr/local/nginx-1.21.6/logs/access.log" nginx http client request body temporary files: "client_body_temp" nginx http proxy temporary files: "proxy_temp" nginx http fastcgi temporary files: "fastcgi_temp" nginx http uwsgi temporary files: "uwsgi_temp" nginx http scgi temporary files: "scgi_temp" $ make [... much output, about 40–44 minutes of compiling ...] $ su root -c 'make install' [... asked for root password, much output ...]
Compiling Nginx also takes much longer with OpenSSL 3:
about 14–16 minutes with OpenSSL 1.1.1x,
but 40–44 minutes with 3.0.x.
This is because whether you already compiled and
installed OpenSSL or not, this recompiles the static libraries
and embeds them in the
$ file nginx nginx: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 13.0 (1300139), FreeBSD-style, with debug_info, not stripped $ ldd nginx nginx: libcrypt.so.5 => /lib/libcrypt.so.5 (0x80071f000) libpcre.so.1 => /usr/local/lib/libpcre.so.1 (0x800740000) libz.so.6 => /lib/libz.so.6 (0x8007e4000) libGeoIP.so.1 => /usr/local/lib/libGeoIP.so.1 (0x800800000) libc.so.7 => /lib/libc.so.7 (0x80084b000) libthr.so.3 => /lib/libthr.so.3 (0x800c5c000)
Now I will make symbolic links so generic names point to the current versions.
$ su # cd /usr/local # rm -f openssl nginx # ln -s openssl-3.0.3 openssl # ln -s nginx-1.21.6 nginx
And, to get the new version pointed to my configuration and content, change this as needed:
# cd /usr/local/nginx-1.21.6 # mv html html-original # ln -s /home/cromwell/www html # ln -s /home/cromwell/www/nginx/ssl_dhparam ssl_dhparam # rm conf/nginx.conf # ln -s /home/cromwell/www/nginx/nginx.conf conf/nginx.conf
Make sure that the new software, your configuration file, and the symbolic links are all ready to go:
# /usr/local/nginx/sbin/nginx -t nginx: the configuration file /usr/local/nginx-1.21.6/conf/nginx.conf syntax is ok nginx: configuration file /usr/local/nginx-1.21.6/conf/nginx.conf test is successful
I need to stop the running Apache process, and comment
out the line in
that starts it at boot time.
$ grep apache /etc/rc.conf ## apache24_enable=YES
I will create or modify
at boot time.
$ cat /etc/rc.local #!/bin/sh # Created just to start Nginx /usr/local/nginx/sbin/nginx
Initial ConfigurationDual ECC/RSA
The web server should be ready to go once we define
a simple configuration.
Here are the contents of a basic
This uses dual ECC and RSA keys and certificates.
for details on how to generate and install these,
and automatically renew them.
Start it, and make sure it's running on TCP ports
80 and 443.
You will see a master process running as
There should be a worker process running as
and listening on TCP/80 and TCP/443, highlighted here.
lsof will also show the worker
caught in the act of serving content.
# lsof -i tcp:80 -o -i tcp:443 COMMAND PID USER FD TYPE DEVICE OFFSET NODE NAME nginx 11117 root 9u IPv4 0xfffff80024dde820 0t0 TCP *:http (LISTEN) nginx 11117 root 10u IPv4 0xfffff80024e4a410 0t0 TCP *:https (LISTEN) nginx 11118 nobody 3u IPv4 0xfffff80024d9e820 0t0 TCP www.c.cromwell-intl.internal:https->74-216-249-130.dedicated.allstream.net:54781 (ESTABLISHED) nginx 11118 nobody 9u IPv4 0xfffff80024dde820 0t0 TCP *:http (LISTEN) nginx 11118 nobody 10u IPv4 0xfffff80024e4a410 0t0 TCP *:https (LISTEN) nginx 11118 nobody 13u IPv4 0xfffff80024900820 0t0 TCP www.c.cromwell-intl.internal:https->ec2-54-172-123-82.compute-1.amazonaws.com:37726 (ESTABLISHED) nginx 11118 nobody 14u IPv4 0xfffff80024d98820 0t0 TCP www.c.cromwell-intl.internal:https->220.127.116.11:25208 (ESTABLISHED) nginx 11118 nobody 15u IPv4 0xfffff80024b22820 0t0 TCP www.c.cromwell-intl.internal:https->d24-36-29-150.home1.cgocable.net:55150 (ESTABLISHED) nginx 11118 nobody 16u IPv4 0xfffff80024b20410 0t0 TCP www.c.cromwell-intl.internal:https->71-80-191-91.static.lsan.ca.charter.com:62074 (ESTABLISHED) nginx 11118 nobody 17u IPv4 0xfffff80024e18820 0t0 TCP www.c.cromwell-intl.internal:https->ip72-207-111-102.sd.sd.cox.net:56285 (ESTABLISHED) nginx 11118 nobody 18u IPv4 0xfffff80024d9e000 0t0 TCP www.c.cromwell-intl.internal:http->18.104.22.168.static.copel.net:44473 (ESTABLISHED) nginx 11118 nobody 19u IPv4 0xfffff80024384410 0t0 TCP www.c.cromwell-intl.internal:https->22.214.171.124:54251 (ESTABLISHED) nginx 11118 nobody 20u IPv4 0xfffff80024f1a000 0t0 TCP www.c.cromwell-intl.internal:https->mail.iph-bet.fr:49700 (ESTABLISHED) nginx 11118 nobody 22u IPv4 0xfffff80024d9d410 0t0 TCP www.c.cromwell-intl.internal:https->126.96.36.199.static.copel.net:54777 (ESTABLISHED) nginx 11118 nobody 23u IPv4 0xfffff800247e4820 0t0 TCP www.c.cromwell-intl.internal:https->CPE84948c4cc1b3-CM84948c4cc1b0.cpe.net.cable.rogers.com:54286 (ESTABLISHED) nginx 11118 nobody 24u IPv4 0xfffff80024d98000 0t0 TCP www.c.cromwell-intl.internal:https->CPE84948c4cc1b3-CM84948c4cc1b0.cpe.net.cable.rogers.com:54287 (ESTABLISHED) nginx 11118 nobody 25u IPv4 0xfffff800247f5820 0t0 TCP www.c.cromwell-intl.internal:https->CPE84948c4cc1b3-CM84948c4cc1b0.cpe.net.cable.rogers.com:54289 (ESTABLISHED) nginx 11118 nobody 26u IPv4 0xfffff8002490e410 0t0 TCP www.c.cromwell-intl.internal:https->CPE84948c4cc1b3-CM84948c4cc1b0.cpe.net.cable.rogers.com:54288 (ESTABLISHED) nginx 11118 nobody 27u IPv4 0xfffff80024dde410 0t0 TCP www.c.cromwell-intl.internal:https->188.8.131.52:61366 (ESTABLISHED) nginx 11118 nobody 28u IPv4 0xfffff80024384820 0t0 TCP www.c.cromwell-intl.internal:https->184.108.40.206:45182 (ESTABLISHED) nginx 11118 nobody 29u IPv4 0xfffff80024d99000 0t0 TCP www.c.cromwell-intl.internal:https->dynamicip-94-181-133-82.pppoe.penza.ertelecom.ru:55386 (ESTABLISHED) nginx 11118 nobody 56u IPv4 0xfffff80024e4a000 0t0 TCP www.c.cromwell-intl.internal:https->220.127.116.11:30154 (ESTABLISHED) python2.7 12908 root 7u IPv4 0xfffff80024d9f000 0t0 TCP www.c.cromwell-intl.internal:55720->metadata.google.internal:http (ESTABLISHED)
It Should Be Running, But Old Tools Might Not Realize That
The problem is that our measurement tools may fail to show the TLS functionality. Your web browser, the (default) OpenSSL package, the Wireshark protocol analyzer, and online scanners should all be unable to test or recognize TLS 1.3. If not, you need to update your tools!
Make sure you can test your server from a distance:
$ openssl s_client -tls1_3 cromwell-intl.com:443 CONNECTED(00000005) --- Certificate chain 0 s:CN = cromwell-intl.com i:C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3 1 s:C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3 i:O = Digital Signature Trust Co., CN = DST Root CA X3 --- Server certificate -----BEGIN CERTIFICATE----- MIIFdzCCBF+gAwIBAgISAzBCb9tcMGcrVV2eqVLf7J39MA0GCSqGSIb3DQEBCwUA MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD [... lines deleted ...] sboGITUCwIdEGAPawfevVlv0xKG2g0oZK3dqhTXNIOuq3euzHf+NA+iUXVrO6ll7 Vy/rjUpz225dyEBtPULkwrVvN5bpNKutKgH8HE++mII47+sv3ef+jRs6b3/95m5f Oz7WyhK2c3R0mGo= -----END CERTIFICATE----- subject=CN = cromwell-intl.com issuer=C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3 --- No client certificate CA names sent Peer signing digest: SHA384 Peer signature type: ECDSA Server Temp Key: X25519, 253 bits --- SSL handshake has read 2986 bytes and written 321 bytes Verification error: unable to get local issuer certificate --- New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384 Server public key is 384 bit Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated Early data was not sent Verify return code: 20 (unable to get local issuer certificate) --- QUIT
Yes! It works!
I captured that traffic with Wireshark. On a recent distribution, Wireshark can figure it out.
Also try testing it with the Qualys web server evaluation scanner. Their development scanner at dev.ssllabs.com may be able to do additional tests.
We Can Do BetterTuning HTTPS
In an earlier series of pages I describe how to configure an Apache web server to score an A+ grade from Qualys and from online HTTP header analyzers. Here's how to do the same thing with Nginx.
I pass two variables to the PHP back end processor, which allows me to put the TLS protocol version and cipher in the standard footer on each page. The code in the footer looks like this:
<p style="text-align:right; font-size:0.80rem;"> <?php /* The trick here is to pass these values to the PHP processor * in the Nginx configuration file: * fastcgi_param TLS_PROTOCOL $ssl_protocol; * fastcgi_param TLS_CIPHER $ssl_cipher; */ echo('Protocol: ' . $_SERVER["SERVER_PROTOCOL"] . '<br />'); echo('Crypto: ' . $_SERVER["TLS_PROTOCOL"] . ' / ' . $_SERVER["TLS_CIPHER"] ); ?> </p>
The result, specific to your connection to view this page, is:
Crypto: TLSv1.3 / TLS_AES_256_GCM_SHA384
You need at least one key pair, with the public key wrapped in a digital certification issued by a trusted CA (or Certificate Authority). Another page of mine shows how to generate and maintain dual ECC and RSA certificates from the Let's Encrypt project, automatically renewing them.
Configuring the Server
Now you're ready for the last step:
Configuring Nginx for an A+ Qualys Score