Rack of Ethernet switches.

Installing Dual TLS Certificates and Running TLS

Steps Toward The Goal

On earlier pages I showed how to set up a Google Compute Engine virtual machine, run FreeBSD, and set up a basic Apache/PHP web server. Now let's set up HTTPS!

On this page I will:
1: Set up both RSA and ECC key pairs.
2: Obtain Let's Encrypt certificates for both.
3: Set up automatic renewal.

On the final page I will tune the Apache HTTP headers.

Why TLS?

TLS, the successor to SSL, authenticates a web server and provides confidentiality and integrity of the data.

In October 2017, Google's Chrome browser began marking HTTP pages as Not Secure if they contain text entry boxes. Google had warned the previous year that Chrome would eventually mark all HTTP pages as Not Secure, a later warning placed that change in July 2018. Once Chrome does something, the other browsers quickly follow.

Google also announced back in 2014 that HTTPS already had a slight effect on a page's rank in search results, and the weight would probably increase. In other words, if you don't run HTTPS, you will be ranked lower in search results. Similarly, once Google's search algorithm starts doing something, the other search engines follow.

So, even if a web site is purely informational, you still want to set up TLS.

Public-Key Security

Asymmetric cryptography, also called public-key cryptography, bases its security on a trapdoor function. That's a function that is easy to compute in one direction but enormously difficult to compute in the opposite direction. RSA, which was developed in the late 1970s, relies on the difficulty of factoring the product of two very large prime numbers.

It is easy to multiply integers, even ones with hundreds of digits, but it is impractically difficult to start with such a product and figure out which two large prime numbers went into it.

Then people worried: what if someone develops a general-purpose quantum computer? Shor's Algorithm could quickly factor very large numbers if you have such a platform on which to run it.

Around this time, people were starting to use mobile devices for Internet access, but smart phones with fast multi-core CPUs had not yet appeared.

So, Elliptic Curve Cryptography or ECC suddenly became popular. Its trapdoor function is based on a discrete logarithm, entirely different from RSA's factoring. Analysis by NSA and NIST showed that ECC provides equal security with much smaller keys than RSA, requiring much less computation.

So, there were two advantages for ECC: much higher performance, and expected resistance to sudden obsolescence when quantum computers appear. Certificate authorities began issuing dual certificates for sites: one based on ECC which newer clients would prefer for performance, and RSA as a fall-back.

Since then, cryptographers have discovered that ECC will be just as susceptible as RSA to attack by quantum computers. But ECC still has a performance advantage.

Let's Encrypt

Let's Encrypt is a certificate authority founded by the Electronic Frontier Foundation, the Mozilla Foundation, the University of Michigan, Akamai Technologies, and Cisco Corporation.

The Let's Encrypt CA gives you a certificate that's good for 90 days. The short certificate lifetime makes automated renewal important.

ACME, the Automated Certificate Management Environment, is a protocol for interacting with the Let's Encrypt CA. You use the certbot program to carry out the various steps. Install the py27-certbot package to get it.

Creating the RSA Certificate

This is very easy! For my site I simply did this: 

# certbot certonly --webroot \
	-w /usr/local/www/htdocs/ \
	-d cromwell-intl.com -d www.cromwell-intl.com

I told it where the web document root was located, and then listed the domain names. Yes, clients will be redirected from www.cromwell-intl.com to cromwell-intl.com as needed, but they must first make a secure connection to the server and ask for the longer name with "www.".

There is some narrative output, and you are asked for an email address in case they need to send you an urgent renewal or security notice. You agree to the terms of service, then answer whether it's OK to share your email address with the EFF, and then it's done!

I didn't tell it anything about the cryptography, so it generated and installed a 2048-bit RSA key pair.

What Did You Get?

The key pair, certificate, and associated files have been created and saved under /usr/local/etc/letsencrypt. Let's see what files are there. 

# cd /usr/local/etc/letsencrypt
# tree -F
.
|-- accounts/
|   |-- acme-staging.api.letsencrypt.org/
|   |   `-- directory/
|   |       `-- d72ae2a5cf968487add7cbdece6e3aab/
|   |           |-- meta.json
|   |           |-- private_key.json
|   |           `-- regr.json
|   `-- acme-v01.api.letsencrypt.org/
|       `-- directory/
|           `-- 5f78856fecb3b21a157f41d986716e2c/
|               |-- meta.json
|               |-- private_key.json
|               `-- regr.json
|-- archive/
|   `-- cromwell-intl.com/
|       |-- cert1.pem
|       |-- chain1.pem
|       |-- fullchain1.pem
|       `-- privkey1.pem
|-- csr/
|   `-- 0000_csr-certbot.pem
|-- keys/
|   `-- 0000_key-certbot.pem
|-- live/
|   `-- cromwell-intl.com/
|       |-- README
|       |-- cert.pem -> ../../archive/cromwell-intl.com/cert1.pem
|       |-- chain.pem -> ../../archive/cromwell-intl.com/chain1.pem
|       |-- fullchain.pem -> ../../archive/cromwell-intl.com/fullchain1.pem
|       `-- privkey.pem -> ../../archive/cromwell-intl.com/privkey1.pem
`-- renewal/
    `-- cromwell-intl.com.conf

14 directories, 18 files

Creating the ECC Certificate

Now I will create an ECC private key and certificate. You need a reasonably recent version of openssl. See what yours is capable of: 

$ openssl ecparam -list_curves | less

I will use elliptic curve P-384, designated secp384r1, as that is the strongest elliptic curve included in NSA Suite B cryptography. See the U.S. NIST SP 800-57 "Recommendation for Key Management" for its definition, and this comparison of relative strength against brute-force attack:

Key Length in Bits for Approximately Equal Resistance to Brute-Force Attacks, per NIST/NSA
Security
Strength		 Symmetric
(3DES, AES)		 Asymmetric
(RSA, DSA)		 Elliptic
Curve
80 80 1024 160
112 112 2048 224
128 128 3072 256
192 192 7680 384
256 256 15,360 512

The first time I used certbot I let it generate an RSA key pair. Now I need to generate an ECC certificate-signing request. Start by generating an ECC private key: 

$ openssl ecparam -genkey -name secp384r1 | openssl ec -out ecc-privkey.pem

Before generating the CSR or Certificate Signing Request, I must slightly change the OpenSSL configuration to enable multiple names, both cromwell-intl.com and www.cromwell-intl.com.

  1. Edit /etc/ssl/openssl.cnf.
  2. Find and uncomment the entry:
    req_extensions = v3_req
  3. Add a line below that:
    subjectAltName = @alt_names
  4. Add a new stanza at the end of the file:
    ## Added
    [alt_names]
    DNS.1 = www.cromwell-intl.com
    DNS.2 = cromwell-intl.com

Now I can generate the CSR: 

$ openssl req -new -sha256 -key ecc-privkey.pem -nodes -outform pem -out ecc-csr.pem
[... output deleted ...]
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:Indiana
Locality Name (eg, city) []:West Lafayette
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Cromwell International
Organizational Unit Name (eg, section) []:Cromwell International
Common Name (e.g. server FQDN or YOUR name) []:cromwell-intl.com
Email Address []:bob.cromwe11@comcast.net
[... output deleted ...]

Now ask Let's Encrypt to generate a certificate. This time we pass it our new CSR. 

$ certbot certonly -w /usr/local/www/htdocs \
	-d cromwell-intl.com -d www.cromwell-intl.com \
	--email bob.cromwe11@comcast.net \
	--csr ecc-csr.pem --agree-tos

This gives us three new files in the local directory:

0000_cert.pem = The certificate itself
0000_chain.pem = The signing chain
0001_chain.pem = The full chain including our certificate

Storing Both Certificates

It took some research after initial frustration to learn that certbot is very fussy about file names when it comes to renewal.

You have archive/example.com with the actual files, and live/example.com with symbolic links pointing to them. You can rename the archive and live directories, but the files must have specific names.

The "archive" directory, or whatever you end up naming it, must have files named precisely cert1.pem, chain1.pem, fullchain1.pem, and privkey1.pem.

The "live" directory, again possibly renamed, must have symbolic links with those names minus the "1", precisely cert.pem, chain.pem, fullchain.pem, and privkey.pem.

First, I rearranged the existing hierarchy under /usr/local/etc/letsencrypt:

  1. Rename the existing "archive" and "live" directories rsa-archive and rsa-live.
  2. Recreate the symbolic links in rsa-live/cromwell-intl.com to point to the relocated "archive" files.
  3. Edit renewal/cromwell-intl.com.conf and make corresponding changes to the paths.
  4. Rename that file rsa-cromwell-intl.com.conf.
  5. Verify that renewal still works:
    certbot renew --dry-run

Next, create new directories ecc-archive/cromwell-intl.com and ecc-live/cromwell-intl.com and:

  1. Move the ECC files into the ecc-archive area, changing the names as required.
  2. Create the symbolic links under ecc-live.
  3. Rename the RSA files in csr and keys, and move the corresponding ECC files into those areas.
  4. Copy the file in renewal to ecc-cromwell-intl.com.conf and edit that new file so it refers to the ECC files.

The result of all this is the following, where:
yellow indicates renamed files and changed file content,
green indicates (re)created symbolic links,
blue indicates new files and directories, and
grey indicates unchanged files 

# cd /usr/local/etc/letsencrypt
# tree -F
.
|-- accounts/
|   |-- acme-staging.api.letsencrypt.org/
|   |   `-- directory/
|   |       `-- d72ae2a5cf968487add7cbdece6e3aab/
|   |           |-- meta.json
|   |           |-- private_key.json
|   |           `-- regr.json
|   `-- acme-v01.api.letsencrypt.org/
|       `-- directory/
|           `-- 5f78856fecb3b21a157f41d986716e2c/
|               |-- meta.json
|               |-- private_key.json
|               `-- regr.json
|-- csr/
|   |-- ecc-csr.pem
|   `-- rsa-csr.pem
|-- ecc-archive/
|   `-- cromwell-intl.com/
|       |-- cert1.pem
|       |-- chain1.pem
|       |-- fullchain1.pem
|       `-- privkey1.pem
|-- ecc-live/
|   `-- cromwell-intl.com/
|       |-- cert.pem -> ../../ecc-archive/cromwell-intl.com/cert1.pem
|       |-- chain.pem -> ../../ecc-archive/cromwell-intl.com/chain1.pem
|       |-- fullchain.pem -> ../../ecc-archive/cromwell-intl.com/fullchain1.pem
|       `-- privkey.pem -> ../../ecc-archive/cromwell-intl.com/privkey1.pem
|-- keys/
|   |-- ecc-privkey.pem
|   `-- rsa-privkey.pem
|-- renewal/
|   `-- rsa-cromwell-intl.com.conf
|-- rsa-archive/
|   `-- cromwell-intl.com/
|       |-- cert1.pem
|       |-- chain1.pem
|       |-- fullchain1.pem
|       `-- privkey1.pem
`-- rsa-live/
    `-- cromwell-intl.com/
        |-- README
	|-- cert.pem -> ../../rsa-archive/cromwell-intl.com/cert1.pem
	|-- chain.pem -> ../../rsa-archive/cromwell-intl.com/chain1.pem
	|-- fullchain.pem -> ../../rsa-archive/cromwell-intl.com/fullchain1.pem
	`-- privkey.pem -> ../../rsa-archive/cromwell-intl.com/privkey1.pem

18 directories, 28 files

# cat renewal/rsa-cromwell-intl.com.conf
# renew_before_expiry = 30 days
version = 0.18.2
archive_dir = /usr/local/etc/letsencrypt/rsa-archive/cromwell-intl.com
cert = /usr/local/etc/letsencrypt/rsa-live/cromwell-intl.com/cert.pem
privkey = /usr/local/etc/letsencrypt/rsa-live/cromwell-intl.com/privkey.pem
chain = /usr/local/etc/letsencrypt/rsa-live/cromwell-intl.com/chain.pem
fullchain = /usr/local/etc/letsencrypt/rsa-live/cromwell-intl.com/fullchain.pem

# Options used in the renewal process
[renewalparams]
authenticator = webroot
installer = None
account = 5f78856fecb3b21a157f41d986716e2c
webroot_path = /usr/local/www/htdocs,
[[webroot_map]]
www.cromwell-intl.com = /usr/local/www/htdocs
cromwell-intl.com = /usr/local/www/htdoc

Now test this with certbot renew --dry-run.

Examining the Certificates

We can use the openssl tool to parse and display the certificates. 

# openssl x509 -in ecc-live/cromwell-intl.com/cert.pem -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            03:29:85:65:fd:e2:55:c4:46:47:9f:14:94:8f:5a:66:70:52
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=Let's Encrypt, CN=Let's Encrypt Authority X3
        Validity
            Not Before: Oct 10 18:22:54 2017 GMT
            Not After : Jan  8 18:22:54 2018 GMT
        Subject: CN=cromwell-intl.com
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (384 bit)
                pub: 
                    04:70:6f:1c:57:94:8c:d4:2c:30:7c:e3:a3:ec:3b:
                    c9:fd:15:bb:fe:3f:b5:f6:85:4d:f1:6f:6f:4a:26:
                    a8:46:7c:f2:ac:c6:50:f7:8c:f2:8f:62:17:3f:00:
                    7e:27:d0:bb:92:20:db:2a:97:91:f2:51:c5:69:65:
                    76:32:09:fe:9b:72:3d:8e:91:5d:4b:3e:5b:72:0e:
                    b6:69:54:a8:17:06:60:7a:d1:39:03:de:c6:3c:a7:
                    e2:66:65:9c:2f:da:ef
                ASN1 OID: secp384r1
                NIST CURVE: P-384
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier: 
                A8:EC:CF:36:F7:98:BB:E0:87:00:20:16:64:1B:74:6E:8F:67:29:B3
            X509v3 Authority Key Identifier: 
                keyid:A8:4A:6A:63:04:7D:DD:BA:E6:D1:39:B7:A6:45:65:EF:F3:A8:EC:A1

            Authority Information Access: 
                OCSP - URI:http://ocsp.int-x3.letsencrypt.org
                CA Issuers - URI:http://cert.int-x3.letsencrypt.org/

            X509v3 Subject Alternative Name: 
                DNS:cromwell-intl.com, DNS:www.cromwell-intl.com
            X509v3 Certificate Policies: 
                Policy: 2.23.140.1.2.1
                Policy: 1.3.6.1.4.1.44947.1.1.1
                  CPS: http://cps.letsencrypt.org
                  User Notice:
                    Explicit Text: This Certificate may only be relied upon by Relying Parties and only in accordance with the Certificate Policy found at https://letsencrypt.org/repository/

    Signature Algorithm: sha256WithRSAEncryption
         76:74:63:e3:eb:44:f8:47:a0:d9:c2:7e:35:cc:1c:6c:cf:b9:
         46:3b:37:4c:a0:f8:5f:fa:fa:b3:03:a7:be:23:70:6e:54:b8:
         50:b1:ef:d6:7c:da:5b:18:f0:6d:6d:af:64:6b:8d:8d:2c:34:
         d8:ab:ba:01:86:3a:30:57:fc:78:72:dd:c8:52:91:1d:1d:16:
         e8:f3:45:6b:81:31:30:eb:c0:b9:24:aa:1f:bd:f4:16:be:45:
         4c:9f:cd:9b:2d:a2:b1:a0:34:f5:f6:e8:f5:a5:cb:e6:70:b7:
         d3:7a:6f:34:8e:5b:4c:29:59:33:76:d0:c8:74:aa:d8:4a:39:
         0c:6e:6e:a9:0b:a7:76:eb:da:31:dc:d1:82:7a:a9:f5:b6:4b:
         99:f0:ee:ad:8c:2a:86:22:f8:4c:60:ad:62:f3:cf:a4:fd:1a:
         f3:33:20:c1:15:c0:95:6e:db:12:d0:89:11:ff:f2:c6:5c:2e:
         3e:93:95:81:56:a3:15:ef:a6:cc:2d:16:0b:6e:0b:5e:44:b4:
         b1:bc:3c:1e:39:55:a1:15:22:e3:f2:9a:99:06:19:23:ce:db:
         e2:8f:1e:b7:3d:0d:83:eb:49:ea:ad:74:ad:e1:c3:48:01:cf:
         f5:70:df:70:d4:5a:70:9c:0f:30:b8:f3:6d:2f:79:18:63:63:
         5a:94:14:22

# openssl x509 -in rsa-live/cromwell-intl.com/cert.pem -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            03:55:0c:4e:00:54:90:3a:40:1b:e1:d5:4e:27:c2:a0:d7:ec
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=Let's Encrypt, CN=Let's Encrypt Authority X3
        Validity
            Not Before: Sep 27 19:01:00 2017 GMT
            Not After : Dec 26 19:01:00 2017 GMT
        Subject: CN=cromwell-intl.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:b5:a8:8c:f6:69:5f:1b:42:e3:8f:a4:f5:c6:1f:
                    2d:7b:db:b6:97:d7:99:e2:92:b9:1f:5e:a0:6a:78:
                    cb:bd:fb:99:b9:e8:5a:c1:90:a3:5f:45:68:9c:3e:
                    56:11:36:16:ea:ed:4e:e8:7e:54:66:af:bb:93:c9:
                    c3:71:1a:8a:23:ce:02:11:a0:33:6b:19:2b:dc:08:
                    88:ab:dc:aa:e4:9f:52:8a:dd:69:ac:f0:11:31:ed:
                    ee:2e:1a:75:8a:20:39:73:77:5c:ab:a9:67:d7:e2:
                    5b:c8:58:4f:1c:6a:96:10:0c:f5:a8:52:de:ff:6a:
                    b0:73:74:f8:0f:ee:7f:f6:f6:57:99:12:ee:54:e9:
                    f6:dd:cc:e8:b5:80:44:9e:44:d7:00:97:5f:1d:1b:
                    dc:de:c7:bf:05:7b:c6:e9:81:6b:f4:b6:eb:6a:a0:
                    af:d3:60:98:a4:33:92:ea:97:b7:6a:f7:71:22:fb:
                    99:71:ed:1f:cf:2c:ea:e4:73:21:ac:aa:49:ea:de:
                    e7:14:1e:00:97:8e:e1:33:25:86:1b:8e:c7:39:fb:
                    a3:7e:0f:2b:ac:da:e5:41:cb:43:5a:70:f7:35:8a:
                    b1:7e:db:0b:04:59:22:50:19:63:72:2a:3d:e0:9f:
                    5d:61:33:6d:99:d8:04:22:9e:d7:20:ff:e3:a6:b8:
                    97:3b
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier: 
                D7:5B:09:39:10:A6:C6:0F:78:57:0B:60:08:86:4A:5B:DA:42:27:33
            X509v3 Authority Key Identifier: 
                keyid:A8:4A:6A:63:04:7D:DD:BA:E6:D1:39:B7:A6:45:65:EF:F3:A8:EC:A1

            Authority Information Access: 
                OCSP - URI:http://ocsp.int-x3.letsencrypt.org
                CA Issuers - URI:http://cert.int-x3.letsencrypt.org/

            X509v3 Subject Alternative Name: 
                DNS:cromwell-intl.com, DNS:www.cromwell-intl.com
            X509v3 Certificate Policies: 
                Policy: 2.23.140.1.2.1
                Policy: 1.3.6.1.4.1.44947.1.1.1
                  CPS: http://cps.letsencrypt.org
                  User Notice:
                    Explicit Text: This Certificate may only be relied upon by Relying Parties and only in accordance with the Certificate Policy found at https://letsencrypt.org/repository/

    Signature Algorithm: sha256WithRSAEncryption
         23:7a:be:18:c5:56:ac:e3:6e:35:e2:c3:11:fb:b7:60:ab:0e:
         26:f4:f4:a6:af:90:c5:8d:16:73:43:fc:19:85:8a:a0:30:47:
         48:3b:f2:c5:99:77:2e:97:20:d6:20:ea:48:22:b6:64:99:65:
         64:63:cb:e9:f0:28:02:09:9c:c3:63:b9:10:03:1d:fc:d1:81:
         1a:cd:1a:ab:6a:2c:08:02:ee:1c:28:8d:86:43:90:da:8b:a1:
         bb:35:ac:b7:bf:7e:c0:1b:53:54:e7:5d:00:d0:f4:ae:09:47:
         fd:8c:47:b3:21:4f:b0:0d:5f:e0:b9:73:de:39:43:c0:8c:37:
         0f:cc:c3:ec:b7:15:32:e6:f2:80:68:47:41:88:39:c9:6c:a8:
         9e:ed:fd:c3:eb:73:c0:5f:38:21:5e:0e:77:59:9e:a2:bb:dc:
         78:bb:72:4f:cc:e0:f4:0c:7c:d6:d7:3a:e9:95:af:7a:0d:45:
         53:b8:ca:63:80:66:87:f6:b0:9a:dd:66:12:12:61:b5:79:9a:
         e9:0f:50:57:5c:0d:ca:ff:1f:79:b7:c2:a1:9b:15:07:a2:36:
         cd:60:da:41:c0:eb:3b:79:d6:c1:f2:95:3a:81:96:43:f3:3c:
         0a:9b:2c:d4:f8:e3:64:94:69:dc:1b:a0:1f:d3:45:74:66:1a:
         10:76:e3:7e

Automated Renewal

The problem is that the certbot program cannot renew a certificate for an ECC public key. It really doesn't renew anything, its "renewal" process generates a new RSA key pair and obtains a certificate with the new public key.

The --csr option to specify a certificate signing request works only with the certonly subcommand, as described on the certbot manual page.

If you create a configuration file in the renewal directory for your ECC key pair, it will simply create an additional RSA key pair and create a certificate for the public key. It places the results in files that you expect to hold ECC keys and certificate.

Be careful developing your solution

You can only request 5 duplicate certificates per week. So, develop any scripts with the certificate renewals commented out, until you're pretty sure that it's going to work. See the rate limit documentation for details.

The solution for automated renewal

Create the following script in /root/renew-all-certs: 

#!/bin/sh

LOGFILE=/root/certbot-output
ARCHIVE=/usr/local/etc/letsencrypt/ecc-archive/cromwell-intl.com

## "Renew" the RSA certificate -- this actually generates a fresh
## 2048-bit RSA key pair and creates a certificate from the public key.
echo "RSA renewal ====================================" > $LOGFILE
certbot --force-renewal renew >> $LOGFILE 2>&1
## Use the Certificate-Signing Request for the existing ECC
## public key, request a new certificate.  You can read a CSR with:
## $ openssl req -noout -text -in /path/to/csr.pem
echo "ECC renewal ====================================" >> $LOGFILE
certbot certonly --non-interactive --webroot \
	-w /usr/local/www/htdocs \
	-d cromwell-intl.com -d www.cromwell-intl.com \
	--email bob.cromwell@comcast.net \
	--csr /usr/local/etc/letsencrypt/csr/ecc-csr.pem \
	--agree-tos >> $LOGFILE 2>&1
## The above creates three files in the local directory.
## Move them into place.
echo "Installing files ===============================" >> $LOGFILE
mv -fv 0000_cert.pem  $ARCHIVE/cert5.pem >> $LOGFILE
mv -fv 0000_chain.pem $ARCHIVE/chain5.pem >> $LOGFILE
mv -fv 0001_chain.pem $ARCHIVE/fullchain5.pem >> $LOGFILE
## Restart the web server so it's now using the new files.
echo "Apache restart =================================" >> $LOGFILE
/usr/local/etc/rc.d/apache24 stop >> $LOGFILE 2>&1
/usr/local/etc/rc.d/php-fpm stop >> $LOGFILE 2>&1
/usr/local/etc/rc.d/php-fpm start >> $LOGFILE 2>&1
/usr/local/etc/rc.d/apache24 start >> $LOGFILE 2>&1

Now set up a crontab job to run your script once a week. 

# crontab -l
#       day of       day of
# min hr month month  week  command
  40  21   *     *    fri   /root/renew-all-certs

Here is the resulting output: 

RSA renewal ====================================
Saving debug log to /var/log/letsencrypt/letsencrypt.log

-------------------------------------------------------------------------------
Processing /usr/local/etc/letsencrypt/renewal/rsa-cromwell-intl.com.conf
-------------------------------------------------------------------------------
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for cromwell-intl.com
http-01 challenge for www.cromwell-intl.com
Waiting for verification...
Cleaning up challenges

-------------------------------------------------------------------------------
new certificate deployed without reload, fullchain is
/usr/local/etc/letsencrypt/rsa-live/cromwell-intl.com/fullchain.pem
-------------------------------------------------------------------------------

-------------------------------------------------------------------------------

Congratulations, all renewals succeeded. The following certs have been renewed:
  /usr/local/etc/letsencrypt/rsa-live/cromwell-intl.com/fullchain.pem (success)
-------------------------------------------------------------------------------
ECC renewal ====================================
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Performing the following challenges:
http-01 challenge for cromwell-intl.com
http-01 challenge for www.cromwell-intl.com
Using the webroot path /usr/local/www/htdocs for all unmatched domains.
Waiting for verification...
Cleaning up challenges
Server issued certificate; certificate written to /root/0000_cert.pem
Cert chain written to <fdopen>
Cert chain written to <fdopen>
IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /root/0001_chain.pem
   Your cert will expire on 2018-05-17. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

Installing files ===============================
0000_cert.pem -> /usr/local/etc/letsencrypt/ecc-live/cromwell-intl.com/cert.pem
0000_chain.pem -> /usr/local/etc/letsencrypt/ecc-live/cromwell-intl.com/chain.pem
0001_chain.pem -> /usr/local/etc/letsencrypt/ecc-live/cromwell-intl.com/fullchain.pem
Apache restart =================================
Stopping apache24.
Waiting for PIDS: 22998.
Stopping php_fpm.
Waiting for PIDS: 22989.
Performing sanity check on php-fpm configuration:
[17-Aug-2018 20:25:59] NOTICE: configuration file /usr/local/etc/php-fpm.conf test is successful

Starting php_fpm.
Performing sanity check on apache24 configuration:
Syntax OK
Starting apache24.

Enabling HTTPS

Edit the httpd.conf configuration file and add the following to the end of the file, changing the hostname and file system paths as needed. Make sure to use the file fullchain.pem, which contains the full certificate chain, and not cert.pem which has just your site's certificate. 

# Put these directives at the global level:
LoadModule ssl_module libexec/apache24/mod_ssl.so
Listen 443

# Put these within individual VirtualHost stanzas
# if you are hosting several sites on one server.
<VirtualHost *:443>
    ServerName cromwell-intl.com
    SSLEngine on
    # ECC secp384r1
    SSLCertificateFile "/usr/local/etc/letsencrypt/ecc-live/cromwell-intl.com/fullchain.pem"
    SSLCertificateKeyFile "/usr/local/etc/letsencrypt/ecc-live/cromwell-intl.com/privkey.pem"
    # RSA
    SSLCertificateFile "/usr/local/etc/letsencrypt/rsa-live/cromwell-intl.com/fullchain.pem"
    SSLCertificateKeyFile "/usr/local/etc/letsencrypt/rsa-live/cromwell-intl.com/privkey.pem"
</VirtualHost>

Initial Test

Restart Apache and verify that you can connect with both HTTP and HTTPS.

You could run a Qualys server test, but at this point it would probably get a "B" grade at best. We will tune the configuration, but first let's set up redirection.

Redirect to HTTPS and Remove "www."

The goal is to accept all connections, redirecting all of these:
http://cromwell-intl.com/some/path/
http://www.cromwell-intl.com/some/path/
https://www.cromwell-intl.com/some/path/
to this:
https://cromwell-intl.com/some/path/

Add the following to your .htaccess file in the root of the web site. Don't duplicate the RewriteEngine line. 

# Do NOT duplicate the following line if it
# already exists earlier in the file.
RewriteEngine on

# Remove "www." and redirect HTTP to HTTPS
# Use a standard variable and a tagged regular expression to
# replace the URL with "https://", the host name, and the
# path minus any leading "www.":
RewriteCond %{HTTP_HOST} ^www\.(.*)$ [NC]
RewriteRule ^(.*)$ https://%1/$1 [R=301,L]
# If they asked for the non-www name but with HTTP,
# build a new HTTPS URL with the host name and path:
RewriteCond %{HTTPS} off
RewriteRule ^(.*)$ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]

# We added the following in an earlier step.
# If the client asks for a non-existent *.html file, rewrite it so
# it isn't passed to the PHP engine causing a "Primary script unknown"
# log entry and a plain "File not found" page.
RewriteCond %{REQUEST_FILENAME} \.html$
RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_URI} !-f
RewriteRule (.*) - [H=text/html]

Yes, I did the redirection with the site-specific .htaccess file. I could have instead done it with slightly different syntax in the server-wide httpd.conf configuration file. For the single site, given its size and traffic, I didn't see a big advantage of one method over the other.

Now test all combinations of the redirections: http vs https, www. vs not, and include requests for nonexistent files and directories.

Improving the TLS Configuration

Mozilla Config Generator Apache SSL Howto

Apache has a good SSL/TLS how-to document. Even more useful, Mozilla has a configuration generator. You select your server, its version, and your OpenSSL versions, and then you select the security profile.

Which security profile should you use? It depends...

Let's say you're setting up a server for use within your organization, and you have full control of the desktop systems and any portable laptops that could be used to connect in from outside. I recommend the strictest "Modern" profile for that case. All your client machines will need to be fairly current, but that should already be the case.

However, let's say that you want to be open to most all clients from the public. That's my situation. I have ads on my site. While it would be nice if everyone used up-to-date operating systems and browsers, I don't want to block or even inconvenience people with outdated platforms.

I used the "Intermediate" profile as a starting point. Here is what I added toward the end of the httpd.conf configuration file, before and outside the VirtualHost stanza, so it will apply to all virtually hosted web sites I eventually set up on this server. 

# TLS only, no SSL
SSLProtocol all -SSLv2 -SSLv3
# Specify ciphers in a preferred order.  I reordered what the configuration
# generator gave me, putting 3DES ("DES-CBC3") at the end.
SSLCipherSuite ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:DES-CBC3-SHA:!DSS
SSLHonorCipherOrder     on
# Disable compression and session tickets
SSLCompression          off
SSLSessionTickets       off
# Enable OCSP Stapling
LoadModule socache_shmcb_module libexec/apache24/mod_socache_shmcb.so
SSLUseStapling On
SSLStaplingCache "shmcb:logs/ssl_stapling(32768)"
# Enable session resumption (caching)
SSLSessionCache "shmcb:logs/ssl_scache"
# Insist on HSTS or HTTP Strict Transport Security
Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"

And with that, an A+ evaluation from the Qualys analyzer:

Screenshot of Qualys SSL Labs A+ analysis of a HTTPS web server

The server supports 3DES as a fallback position if none of the stronger ciphers are supported. That allows connections from IE 8 on XP. While 3DES is flagged in the report as weak, at least at the time that I did this it did not lower the score. Unlike RC4, which capped the grade at B at the time.

DNS CAA or Certification Authority Authorization provides a way to indicate in a DNS record just who is allowed to be a CA for the site. See this Qualys blog for details on what CAA is, and how the CA/Browser Forum has mandated its use.

Google Domains did not yet support CAA records when I started this project in July 2017, as you can see below. Google Cloud DNS, a different product, did.

Screenshot of Google Domains dashboard showing available record types.

By February 2018, they had added CAA records:

Screenshot of Google Domains dashboard with CAA records available.

Let's test it: 

$ dig @localhost cromwell-intl.com CAA

; <<>> DiG 9.10.6-P1 <<>> @localhost cromwell-intl.com CAA
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 26186
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;cromwell-intl.com.		IN	CAA

;; ANSWER SECTION:
cromwell-intl.com.	3570	IN	CAA	128 issue "letsencrypt.org"

;; AUTHORITY SECTION:
cromwell-intl.com.	78163	IN	NS	ns-cloud-c1.googledomains.com.
cromwell-intl.com.	78163	IN	NS	ns-cloud-c4.googledomains.com.
cromwell-intl.com.	78163	IN	NS	ns-cloud-c3.googledomains.com.
cromwell-intl.com.	78163	IN	NS	ns-cloud-c2.googledomains.com.

;; Query time: 0 msec
;; SERVER: ::1#53(::1)
;; WHEN: Wed Feb 14 14:44:24 EST 2018
;; MSG SIZE  rcvd: 198

Final Step

I was very happy with this result! There are just a few more things to do.

Proceed to the final step to see how to tune the HTTP(S) headers.

Final step: HTTPS Headers