Rack of Ethernet switches.

FreeBSD on Google Compute Engine

Steps Toward The Goal

On the previous page I showed how to set up a Google Compute Engine virtual machine within the Google Cloud Platform. I used FreeBSD for good performance, stability, and minimal complexity.

In this stage I will set up the FreeBSD system:
1: Add an unprivileged user.
2: Set up SSH authentication with keys only.
3: Check the file systems and networking.
4: Install some packages: Apache, PHP, and others.
5: Fix a clock problem.

On following pages I will install free Let's Encrypt TLS certificates for both RSA and ECC, and then adjust the Apache configuration for a good score from the authoritative Qualys server analysis.

Adding a User

You can connect in over SSH as a user using cryptographic key-based authentication. That user can become root in an interactive session with:
sudo bash

Now you can create a user with a name of your choice, using useradd.

  Do not assign passwords to any users!  

Continue using SSH keys for all authentication into your system. Automated password-guessing attacks constantly arrive from all across the Internet. There is no need for the risk posed by supporting weak password authentication. You can find the list of supported authentication keys by running this command on your server:

$ ssh -Q key
ssh-ed25519
ssh-ed25519-cert-v01@openssh.com
ssh-rsa
ssh-dss
ecdsa-sha2-nistp256
ecdsa-sha2-nistp384
ecdsa-sha2-nistp521
ssh-rsa-cert-v01@openssh.com
ssh-dss-cert-v01@openssh.com
ecdsa-sha2-nistp256-cert-v01@openssh.com
ecdsa-sha2-nistp384-cert-v01@openssh.com
ecdsa-sha2-nistp521-cert-v01@openssh.com 

If you add your new user to group wheel, they will be able to become root by simply running the command su because of the contents of /etc/pam.d/su.

File Systems, Device Detection, and Networking

Let's look at the file systems, and then see what's in the kernel ring buffer. The following is after I had installed the web site, which takes up 1.7 GB.

The dmesg output is edited down to show the CPU, memory, timer devices, and the disk.

$ df -hT
Filesystem       Type     Size    Used   Avail Capacity  Mounted on
/dev/gpt/rootfs  ufs       28G    5.5G     20G    21%    /
devfs            devfs    1.0K    1.0K      0B   100%    /dev
$ swapctl -l
Device:       1024-blocks     Used:
/dev/gpt/swapfs   1048576     15788

$ dmesg | less
Copyright (c) 1992-2017 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.1-RELEASE-p1 #0: Wed Aug  9 11:55:48 UTC 2017
    root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
[... output deleted ...]
CPU: Intel(R) Xeon(R) CPU @ 2.20GHz (1837.61-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x406f0  Family=0x6  Model=0x4f  Stepping=0
  Features=0x1f83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS,HTT>
  Features2=0xfef83203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended Features=0xc2b7a<TSCADJ,BMI1,HLE,AVX2,FDPEXC,BMI2,ERMS,RTM,NFPUSG,RDSEED,ADX>
  XSAVE Features=0x1<XSAVEOPT>
  TSC: P-state invariant
Hypervisor: Origin = "KVMKVMKVM"
real memory  = 643825664 (614 MB)
avail memory = 580870144 (553 MB)
[... output deleted ...]
random: unblocking device.
ioapic0 <Version 1.1> irqs 0-23 on motherboard
Timecounter "TSC" frequency 1837606598 Hz quality 1000
random: entropy device external interface
[... output deleted ...]
atrtc0: <AT realtime clock> port 0x70-0x71,0x72-0x77 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0xb008-0xb00b on acpi0
[... output deleted ...]
attimer0: <AT timer> at port 0x40 on isa0
Timecounter "i8254" frequency 1193182 Hz quality 0
attimer0: Can't map interrupt.
ppc0: cannot reserve I/O port range
Timecounters tick every 1.000 msec
usb_needs_explore_all: no devclass
nvme cam probe device init
da0 at vtscsi0 bus 0 scbus0 target 1 lun 0
da0: <Google PersistentDisk 1> Fixed Direct Access SPC-4 SCSI device
da0: 4294966.784MB/s transfers
da0: Command Queueing enabled
da0: 30720MB (62914560 512 byte sectors)
Trying to mount root from ufs:/dev/gpt/rootfs [rw]...
vtnet0: link state changed to UP

The CPU and memory are as we expected. The disk device clearly shows that we're running on a Google-specific virtualized cloud platform. I don't think it really transfers data at 4.3 terabyte/second...

For a little more on the virtual hardware platform:

# lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371AB/EB/MB PIIX4 ISA (rev 03)
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:03.0 Non-VGA unclassified device: Red Hat, Inc. Virtio SCSI
00:04.0 Ethernet controller: Red Hat, Inc. Virtio network device 

As for the three Timecounter devices, we will need to look deeper into those. But first, let's look at the network environment. The Ethernet interface is detected as vtnet0. The /32 netmask seems wrong, but it works.

$ ifconfig
vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1460
        options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 42:01:0a:8a:00:02
        hwaddr 42:01:0a:8a:00:02
        inet6 fe80::4001:aff:fe8a:2%vtnet0 prefixlen 64 scopeid 0x1
        inet 10.138.0.2 netmask 0xffffffff broadcast 10.138.0.2
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet 10Gbase-T <full-duplex>
        status: active
lo: lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 1460
[... output deleted ...]

$ netstat -nr
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.138.0.1         UGS      vtnet0
10.138.0.1/32      42:01:0a:8a:00:02  US       vtnet0
web                link#1             UHS         lo0
10.138.0.2/32      link#1             U        vtnet0
127.0.0.1          link#2             UH          lo0

Internet6:
Destination        Gateway            Flags     Netif Expire
::/96              ::1                UGRS        lo0
::1                link#2             UH          lo0
::ffff:0.0.0.0/96  ::1                UGRS        lo0
fe80::/10          ::1                UGRS        lo0
fe80::%vtnet0/64   link#1             U        vtnet0
fe80::4001:aff:fe8 link#1             UHS         lo0
fe80::%lo0/64      link#2             U           lo0
fe80::1%lo0        link#2             UHS         lo0
ff02::/16          ::1                UGRS        lo0

$  cat /etc/resolv.conf
# Generated by resolvconf
search c.cromwell-intl.internal google.internal
nameserver 169.254.169.254

You're in a private network, a VPC or Virtual Private Cloud. It's something like the 10.138.0.0/24 network with just your server and a (virtual) router.

In an earlier step you reserved an external IP address. You assigned it to your VM and created the appropriate DNS records as shown on the previous page. The router does NAT for your VPC to that public address. Google will have PTR records in place, resolving to the googleusercontent.com domain.

Packages

The FreeBSD image comes with several packages added to the basic install. Make sure that you don't remove either sudo or pkg, as you would need to back up and start over!

I know that because I initially wanted to use the Nginx web server. Installing that required many additional packages. When I gave up on Nginx and decided to go with Apache, I "cleaned up" by removing all added packages. I did not realize that pkg itself was an added package!

OK, lesson learned. Delete the image, redeploy, and set up SSH again.

I found these 22 packages on the freshly deployed image:

bash, ca_root_nss, curl, firstboot-freebsd-update, firstboot-growfs, flock, gettext-runtime, google-cloud-sdk, google-daemon, google-startup-scripts, indexinfo, libffi, libnghttp2, panicmail, pkesh, pkg, python, python2, python27, readline, sudo.

Package Management for BSD and Linux

I first installed all available updates for the existing packages.

I added bind-tools and lsof for troubleshooting. Those brought along just 4 required libraries. I also added vim for personal preferences, copying my ~/.vimrc file into place for both my user account and root. That required another 99 packages to satisfy dependencies!

Then I added the packages needed for Apache/PHP web service: apache24, mod_php71, and php72, and their dependencies.

Correcting Clock Problems

I soon noticed that there was a huge clock drift! Within one minute the system clock would be off by several seconds. A Google groups discussion gave me the needed hint.

# sysctl kern.timecounter.hardware
kern.timecounter.hardware: TSC
# sysctl kern.timecounter.choice
kern.timecounter.choice: i8254(0) ACPI-fast(900) TSC(1000) dummy(-1000000)
# sysctl kern.timecounter.hardware=ACPI-fast
kern.timecounter.hardware: TSC -> ACPI-fast

That fixed my problem, so I added a line to /etc/sysctl.conf.

I verified that /etc/rc.conf already contained a line reading:
ntpd_enable=YES
and I added a new line:
ntpdate_enable=YES

The second of those will resynchronize the system clock at each boot using ntpdate, and the first will keep the clock in sync while running with the ntpd daemon.