FreeBSD on Google Compute Engine

Steps Toward The Goal

On the previous page I showed how to set up a Google Compute Engine virtual machine within the Google Cloud Platform. I used FreeBSD for good performance, stability, and minimal complexity.

In this stage I will set up the FreeBSD system:
1: Add an unprivileged user.
2: Set up SSH authentication with keys only.
3: Check the file systems and networking.
4: Install some packages: Apache, PHP, and others.
5: Fix a clock problem.

Try Google Cloud Platform and receive $50 On following pages I will install free Let's Encrypt TLS certificates for both RSA and ECC, and then adjust the Apache configuration for a good score from the authoritative Qualys server analysis.

Adding a User

You can connect in over SSH as a user using cryptographic key-based authentication. That user can become root in an interactive session with:
sudo bash

Now you can create a user with a name of your choice, using useradd.

  Do not assign passwords to any users!  

Continue using SSH keys for all authentication into your system. Automated password-guessing attacks constantly arrive from all across the Internet. There is no need for the risk posed by supporting weak password authentication. You can find the list of supported authentication keys by running this command on your server:

$ ssh -Q key
ssh-ed25519
ssh-ed25519-cert-v01@openssh.com
ssh-rsa
ssh-dss
ecdsa-sha2-nistp256
ecdsa-sha2-nistp384
ecdsa-sha2-nistp521
ssh-rsa-cert-v01@openssh.com
ssh-dss-cert-v01@openssh.com
ecdsa-sha2-nistp256-cert-v01@openssh.com
ecdsa-sha2-nistp384-cert-v01@openssh.com
ecdsa-sha2-nistp521-cert-v01@openssh.com 

If you add your new user to group wheel, they will be able to become root by simply running the command su because of the contents of /etc/pam.d/su.

File Systems, Device Detection, and Networking

Let's look at the file systems, and then see what's in the kernel ring buffer. The following is after I had installed the web site, which takes up 1.7 GB.

The dmesg output is edited down to show the CPU, memory, timer devices, and the disk.

$ df -hT
Filesystem       Type     Size    Used   Avail Capacity  Mounted on
/dev/gpt/rootfs  ufs       28G    5.5G     20G    21%    /
devfs            devfs    1.0K    1.0K      0B   100%    /dev
$ swapctl -l
Device:       1024-blocks     Used:
/dev/gpt/swapfs   1048576     15788

$ dmesg | less
---<<BOOT>>---
Copyright (c) 1992-2021 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 13.0-RELEASE #0 releng/13.0-n244733-ea31abc261f: Fri Apr  9 04:24:09 UTC 2021
    root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64
FreeBSD clang version 11.0.1 (git@github.com:llvm/llvm-project.git llvmorg-11.0.1-0-g43ff75f2c3fe)
VT(vga): text 80x25
CPU: Intel(R) Xeon(R) CPU @ 2.20GHz (2200.06-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x406f0  Family=0x6  Model=0x4f  Stepping=0
  Features=0x1f83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS,HTT>
  Features2=0xfefa3203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended Features=0x1c2ffb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,FDPEXC,SMEP,BMI2,ERMS,INVPCID,RTM,NFPUSG,RDSEED,ADX,SMAP>
  XSAVE Features=0x1<XSAVEOPT>
  TSC: P-state invariant
Hypervisor: Origin = "KVMKVMKVM"
real memory  = 643825664 (614 MB)
avail memory = 586088448 (558 MB)
Event timer "LAPIC" quality 100
ACPI APIC Table: <Google GOOGAPIC>
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
random: unblocking device.
ioapic0 <Version 1.1> irqs 0-23
Timecounter "TSC-low" frequency 1100027555 Hz quality 1000
KTLS: Initialized 1 threads
random: entropy device external interface
[... output deleted ...]
vtvga0: <VT VGA driver>
cryptosoft0: <software crypto>
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS>
acpi0: <Google GOOGRSDT>
acpi0: Power Button (fixed)
acpi0: Sleep Button (fixed)
cpu0: <ACPI CPU> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71,0x72-0x77 irq 8 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0xb008-0xb00b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
isab0: <PCI-ISA bridge> at device 1.0 on pci0
isa0: <ISA bus> on isab0
pci0: <bridge> at device 1.3 (no driver attached)
virtio_pci0: <VirtIO PCI (legacy) SCSI adapter> port 0xc000-0xc03f mem 0xfebfe000-0xfebfe07f irq 11 at device 3.0 on pci0
vtscsi0: <VirtIO SCSI Adapter> on virtio_pci0
virtio_pci1: <VirtIO PCI (legacy) Network adapter> port 0xc040-0xc07f mem 0xfebff000-0xfebff03f irq 11 at device 4.0 on pci0
vtnet0: <VirtIO Networking Adapter> on virtio_pci1
vtnet0: Ethernet address: 42:01:0a:8a:00:03
vtnet0: netmap queues/slots: TX 1/2048, RX 1/2048
000.000113 [ 450] vtnet_netmap_attach       vtnet attached txq=1, txd=2048 rxq=1, rxd=2048
[... output deleted ...]
attimer0: <AT timer> at port 0x40 on isa0
Timecounter "i8254" frequency 1193182 Hz quality 0
attimer0: Can't map interrupt.
attimer0: non-PNP ISA device will be removed from GENERIC in FreeBSD 14.
Timecounters tick every 10.000 msec
[... output deleted ...]
usb_needs_explore_all: no devclass
Trying to mount root from ufs:/dev/gpt/rootfs [rw]...
Root mount waiting for: CAM
da0 at vtscsi0 bus 0 scbus0 target 1 lun 0
da0: <Google PersistentDisk 1> Fixed Direct Access SPC-4 SCSI device
da0: 300.000MB/s transfers
da0: Command Queueing enabled
da0: 30720MB (62914560 512 byte sectors)
mountroot: waiting for device /dev/gpt/rootfs...
[... output deleted ...]
vtnet0: link state changed to UP

The CPU and memory are as we expected. The disk and other devices clearly show that we're running on a Google-specific virtualized cloud platform.

For a little more on the virtual hardware platform:

# lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371AB/EB/MB PIIX4 ISA (rev 03)
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:03.0 Non-VGA unclassified device: Red Hat, Inc. Virtio SCSI
00:04.0 Ethernet controller: Red Hat, Inc. Virtio network device 

As for the three Timecounter devices, we will need to look deeper into those. But first, let's look at the network environment. The Ethernet interface is detected as vtnet0. The /32 netmask seems wrong, but it works.

$ ifconfig
vtnet0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1460
        options=4c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,TXCSUM_IPV6>
        ether 42:01:0a:8a:00:03
        inet6 fe80::4001:aff:fe8a:3%vtnet0 prefixlen 64 scopeid 0x1
        inet 10.138.0.3 netmask 0xffffffff broadcast 10.138.0.3
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo: lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 1460
[... output deleted ...]

$ netstat -nr
Routing tables

Internet:
Destination    Gateway      Flags     Netif Expire
default        10.138.0.1   UGS      vtnet0
10.138.0.1     link#1       UHS      vtnet0
10.138.0.3     link#1       UH       vtnet0
127.0.0.1      link#2       UH          lo0

Internet6:
Destination                     Gateway     Flags     Netif Expire
::/96                           ::1         UGRS        lo0
::1                             link#2      UHS         lo0
::ffff:0.0.0.0/96               ::1         UGRS        lo0
fe80::/10                       ::1         UGRS        lo0
fe80::%vtnet0/64                link#1      U        vtnet0
fe80::4001:aff:fe8a:3%vtnet0    link#1      UHS         lo0
fe80::%lo0/64                   link#2      U           lo0
fe80::1%lo0                     link#2      UHS         lo0
ff02::/16                       ::1         UGRS        lo0

$  cat /etc/resolv.conf
# Generated by resolvconf
search c.cromwell-intl.internal google.internal
nameserver 169.254.169.254

You're in a private network, a VPC or Virtual Private Cloud. It's something like the 10.138.0.0/24 network with just your server and a (virtual) router.

In an earlier step you reserved an external IP address. You assigned it to your VM and created the appropriate DNS records as shown on the previous page. The router does NAT for your VPC to that public address. Google will have PTR records in place, resolving to the googleusercontent.com domain.

Packages

The FreeBSD image comes with several packages added to the basic install. Make sure that you don't remove either sudo or pkg, as you would need to back up and start over!

I know that because I initially wanted to use the Nginx web server. Installing that required many additional packages. When I gave up on Nginx and decided to go with Apache, I "cleaned up" by removing all added packages. I did not realize that pkg itself was an added package!

OK, lesson learned. Delete the image, redeploy, and set up SSH again.

I found these 22 packages on the freshly deployed image:

bash, ca_root_nss, curl, firstboot-freebsd-update, firstboot-growfs, flock, gettext-runtime, google-cloud-sdk, google-daemon, google-startup-scripts, indexinfo, libffi, libnghttp2, panicmail, pkesh, pkg, python, python2, python27, readline, sudo.

I first installed all available updates for the existing packages.

I added bind-tools and lsof for troubleshooting. Those brought along just 4 required libraries. I also added vim for personal preferences, copying my ~/.vimrc file into place for both my user account and root. That required another 99 packages to satisfy dependencies!

Then I added the packages needed for Apache/PHP web service: apache24, mod_php73, and php73, and their dependencies.

Correcting Clock Problems

I soon noticed that there was a huge clock drift! Within one minute the system clock would be off by several seconds. A Google groups discussion gave me the needed hint.

# sysctl kern.timecounter.hardware
kern.timecounter.hardware: TSC
# sysctl kern.timecounter.choice
kern.timecounter.choice: i8254(0) ACPI-fast(900) TSC(1000) dummy(-1000000)
# sysctl kern.timecounter.hardware=ACPI-fast
kern.timecounter.hardware: TSC -> ACPI-fast

That fixed my problem, so I added a line to /etc/sysctl.conf.

I verified that /etc/rc.conf already contained a line reading:
ntpd_enable=YES
and I added a new line:
ntpdate_enable=YES

The second of those will resynchronize the system clock at each boot using ntpdate, and the first will keep the clock in sync while running with the ntpd daemon.