Linux / FreeBSD keyboard.

Mirroring Data and Making Backups with rsync

Mirroring Data and Synchronizing File Systems with rsync

The rsync command can easily and quickly meet several goals on Linux, BSD, and other platforms. You can use it locally, within one host, but it's especially useful for mirroring data from one host to another. You can run it on either the source or destination host.

Backups are absolutely vital. You can use rsync to make a complete mirror image of a file system on another system.

You can recursively copy a directory structure while preserving file permissions and modification and access timestamps. Symbolic links in terms of both absolute and relative paths will be recreated. (So would device-special files, named pipes, and sockets, but those are much less likely to be an issue)

Web site maintenance can become much easier. You can make changes and additions to a local copy, and then use rsync to synchronize what's on the server with the local copy.

The rsync command only transfers data as needed, making it very efficient. And, it transfers via SSH for security.

Advantages: rsync > ssh+tar

Mirroring Data with ssh and tar

You can also use a combination of ssh and tar to make a mirror image of a directory tree, either from local to remote or remote to local. However, a great advantage of rsync is that it only transfers what it needs to.

By default, rsync makes only a quick check based on size or modification time to decide what it needs to transfer. It deals with timezones as needed — my web server runs on UTC and my home system and laptop on local time, but this still works regardless of how the systems are spread across the Internet. The file system actually stores the time and offset from UTC, use the stat command to verify that for yourself. However, you can use the --checksum option to instead analyze content to decide what needs to be transferred.

How to Use rsync

Below is an example of using rsync. I have just edited several files on my web site, editing the files on my local machine. Now I want to upload all the changes to my server. If the changes include creating new directories, files, or symbolic links, all of those will be created on the remote system. As for the options:

Archive mode: Recursively follow the directory tree structure preserving owners, groups, permissions, modification timestamps, symbolic links, and device-special files. You have to do this as root to create device nodes or change ownership. Add the -U or --atimes option to also preserve access timestamps.

Verbose mode, narrate the list of transferred files.

Show the progress for individual files. This is very useful when large files are involved, to assure you that it's working. It also shows the network performance.

--exclude xyz
Exclude paths containing the string xyz. You can use this multiple times. The following example will not copy vi/vim swap files named *.swp or directories named old-css or old-js.

Here we go with the command, and the output. All the additions and updates in the local /var/www/html directory tree will be copied to the same location on the host Thousands of files that don't need to be uploaded, won't be.

$ rsync -av --progress	\
	--exclude '*.swp'	\
	--exclude '/old-css/'	\
	--exclude '/old-js/'	\
sending incremental file list
         63,071 100%   59.48MB/s    0:00:00 (xfr#1, ir-chk=1041/1055)
         30,250 100%  113.18kB/s    0:00:00 (xfr#2, ir-chk=1133/3867)
          6,623 100%   24.78kB/s    0:00:00 (xfr#3, ir-chk=1079/3867)
          7,009 100%   26.23kB/s    0:00:00 (xfr#4, ir-chk=1067/3867)

sent 455,142 bytes  received 1,992 bytes  130,609.71 bytes/sec
total size is 2,249,384,384  speedup is 4,920.62

To That Location? Or Into That Location?

In the above example, the destination already exists. I must include the trailing slash on the source. That is, use /var/www/html/ instead of simply /var/www/html.

If I leave that off, then the destination gets new a subdirectory /var/www/html/html containing an extra copy of everything else from one level up.

The trailing / on the source means "Copy the contents of this directory and not the directory itself."

These two commands are equivalent:

$ rsync -av --progress	\
	--exclude '*.swp'	\
	--exclude '/old-css/'	\
	--exclude '/old-js/'	\

$ rsync -av --progress	\
	--exclude '*.swp'	\
	--exclude '/old-css/'	\
	--exclude '/old-js/'	\

Connect via SSH?

Yes, you want to use SSH. And by default you will, on modern systems.

rsync is quite old, dating from the era of rsh and rlogin. No one uses rsh and rlogin, or at least they shouldn't! Use ssh instead!

How to Set Up and Use SSH

Similarly, there has been an rsync daemon. Usually inetd listened on TCP port 873, and started rsyncd as needed. In the past, rsync would connect to its own service by default. However, modern versions of rsync use SSH by default. I can't think of a good reason to use anything else.


Setting up Kodi/OSMC on a Raspberry Pi

Later, I wanted to make a backup copy of a large collection of media files. Photographs, video files, and DVD UDF image files.

When I play one of my DVDs, I first extract it into an image file so I can watch it with Kodi or OSMC. I very much prefer the interface, and generally find it easier and more convenient. Here's a segment of output when mirroring media files to another system.

[... lines omitted ...]
iso images/
iso images/2001: A Space Odyssey.iso
  6,923,206,656 100%   97.16MB/s    0:01:07 (xfr#178, to-chk=169/399)
iso images/A Scanner Darkly.iso
  6,870,347,776 100%   98.72MB/s    0:01:06 (xfr#179, to-chk=168/399)
iso images/Abraham Lincoln Vampire Hunter.iso
  6,748,639,232 100%   97.34MB/s    0:01:06 (xfr#180, to-chk=167/399)
iso images/Ad Astra.iso
  7,037,278,208 100%   94.20MB/s    0:01:11 (xfr#181, to-chk=166/399)
iso images/American Hustle.iso
  6,199,312,384 100%   94.76MB/s    0:01:02 (xfr#182, to-chk=165/399)
iso images/Andromeda Strain.iso
  4,519,952,384 100%   97.60MB/s    0:00:44 (xfr#183, to-chk=164/399)
iso images/Apocalypse Now Redux.iso
  8,319,780,864 100%   95.70MB/s    0:01:22 (xfr#184, to-chk=163/399)
iso images/Arrival.iso
  7,966,900,224 100%   94.97MB/s    0:01:20 (xfr#185, to-chk=162/399)
iso images/Atomic Blonde.iso
  6,010,949,632 100%   96.20MB/s    0:00:59 (xfr#186, to-chk=161/399)
[... lines omitted ...]

It isn't clear whether "MB" is intended to report traditional units of MB or 106 bytes, or MiB, units of 220 bytes. MB = 1,000,000 bytes while MiB = 1,048,576 bytes. Doing the math, it's in between the two. However, it's significantly closer to units of 220, which I use to calculate network speed.

Or maybe it's MB including the TCP, IP, and Ethernet headers? I don't care, I'm getting roughly 80% of 1 Gbps.

The reported speed isn't useful for small files that transfer in a few seconds or less. These large files, however, provide a measure of performance.

This was on a gigabit switch. 94.2 to 98.7 MiB per second means 790 to 828 Mbps, or 79.0% to 82.8% of the full bandwidth. That's not bad, especially when you consider the protocol overhead and the fact that this is not using Ethernet jumbo frames. NetApp has a nice analysis.

The top command shows that while ssh and rsync are keeping the CPU cores busy, this transfer isn't limited by processing speed. During this snapshot, ssh was using 84.4% of the available time of one CPU core, rsync was using 32.1% of another. Results on the server, where it was the sshd server instead of the ssh client, were very similar.

$ top
top - 11:07:23 up 1 day, 21:11,  6 users,  load average: 2.22, 2.08, 2.03
Tasks: 285 total,   2 running, 283 sleeping,   0 stopped,   0 zombie
%Cpu(s): 17.8 us,  4.7 sy,  0.0 ni, 63.8 id,  2.2 wa,  0.0 hi, 11.5 si,  0.0 st
MiB Mem :  11892.1 total,    145.6 free,   1661.9 used,  10084.6 buff/cache
MiB Swap:   6144.0 total,   6029.2 free,    114.8 used.   9592.0 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  43374 cromwell  20   0   10280   7348   5464 S  84.4   0.1  62:03.08 ssh
  43373 cromwell  20   0   73720  17600   2292 S  32.1   0.1  31:31.94 rsync
   1465 cromwell  20   0 4068652 139752  76744 R   4.3   1.1  47:57.37 cinnamon
    989 root      20   0  858440  62740  48688 S   2.6   0.5  35:13.12 Xorg
    101 root      20   0       0      0      0 S   1.3   0.0   1:15.36 kswapd0
   1648 cromwell  20   0  937136  72084  42544 S   1.3   0.6  19:02.48 audacio+
  45387 cromwell  20   0 4724388 186388 117928 S   1.3   1.5   0:36.92 chrome
[... lines omitted ...]

Digging Deeper

The iostat command measures storage I/O in transfers per second (tps); kilobytes read, written, and discarded per second; and the total read and written during that measurement cycle.

Here I'm running it to output a report every 10 seconds, indefinitely until interrupted with Control-C. The same as with vmstat, you should ignore the first block of output. It is intended to be cumulative since boot time, but that isn't usually of interest. Worse yet, at least for vmstat it isn't at all accurate.

Here are the first two useful blocks of information on the client, where the data is being read for transmission to the other end. As you can see, it's being read from the /dev/sdb device.

$ iostat 10
Linux 5.4.0-65-generic (clienthost)   02/21/21        _x86_64_        (4 CPU)
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.78    0.00    0.92    0.58    0.00   96.72

Device     tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
sda       0.68        11.85        69.94         0.00    1928364   11385973          0
sdb      22.60      3240.46        55.14         0.00  527539060    8976908          0

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          19.00    0.00   15.53    2.84    0.00   62.63

Device     tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
sda       3.00        20.00        14.00         0.00        200        140          0
sdb     580.20     98649.60        10.80         0.00     986496        108          0

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          19.09    0.00   16.91    2.39    0.00   61.61

Device     tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
sda       2.90         0.00        15.60         0.00          0        156          0
sdb     602.70    102860.80         0.00         0.00    1028608          0          0

Linux interface names

The ethstat command may be of interest. Again, ignore the first block of output. We get Mbps and packets per second in and out. The interface enp4s0 is moving the heavy traffic.

$ ethstat
total:        0.48 Mb/s In    85.81 Mb/s Out -    631.3 p/s In    7312.4 p/s Out
    eno1:     0.01 Mb/s In     0.00 Mb/s Out -      1.1 p/s In       0.7 p/s Out
  enp4s0:     0.46 Mb/s In    85.81 Mb/s Out -    630.2 p/s In    7311.7 p/s Out
  enp5s0:     0.00 Mb/s In     0.00 Mb/s Out -      0.0 p/s In       0.0 p/s Out
total:        4.68 Mb/s In   879.72 Mb/s Out -   6193.4 p/s In   74653.5 p/s Out
    eno1:     0.13 Mb/s In     0.00 Mb/s Out -     11.5 p/s In       7.1 p/s Out
  enp4s0:     4.55 Mb/s In   879.71 Mb/s Out -   6181.9 p/s In   74646.4 p/s Out
  enp5s0:     0.00 Mb/s In     0.00 Mb/s Out -      0.0 p/s In       0.0 p/s Out
total:        4.69 Mb/s In   891.44 Mb/s Out -   6199.3 p/s In   75361.6 p/s Out
    eno1:     0.13 Mb/s In     0.00 Mb/s Out -     11.1 p/s In       7.4 p/s Out
  enp4s0:     4.56 Mb/s In   891.44 Mb/s Out -   6188.2 p/s In   75354.2 p/s Out
  enp5s0:     0.00 Mb/s In     0.00 Mb/s Out -      0.0 p/s In       0.0 p/s Out

If you frequently do large transfers, you might look into further performance measurements and possible tuning.

If You're Synchronizing a GoDaddy Web Site

The good news is that GoDaddy doesn't bill for bandwidth.

The bad news is that GoDaddy SSH connectivity is unreliable and very outdated.

I still had hosted there, but I wouldn't do that again. Three consulting clients also have sites there. So, for the time being I had to deal with GoDaddy being outdated and unreliable.

I talked to their technical support on the line. They informed me that the Linux-based shared hosting service I was using could not be updated. If I wanted security software that wasn't 9.5 years old, I would have to move to their "dedicated hosting" service which was significantly more expensive.

Deploying FreeBSD on Google Cloud Platform Nginx, OpenSSL, and Quantum Safe

OK, then I just need a short-term workaround until I can migrate a couple of sites to my Google Cloud Platform server running Nginx on FreeBSD.

Dealing with GoDaddy's Outdated SSH

I updated my laptop from Linux Mint 20.3 to Linux Mint 21 in August 2022. That meant updating from OpenSSH OpenSSH_8.2p1 to OpenSSH_8.9p1. GoDaddy was still using OpenSSH_5.3p1 from February 2013, 9.5 years in the past. Now connections from my updated laptop to a GoDaddy host failed. Asking for details showed this:

$ ssh -v uptime
[... much output ...]
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: diffie-hellman-group-exchange-sha256
debug1: kex: host key algorithm: (no match)
Unable to negotiate with port 22: no matching host key type found.
 Their offer: ssh-rsa,,ssh-dss
How Elliptic-Curve Cryptography Works

The short answer is that GoDaddy's outdated OpenSSH doesn't support ephemeral keys with elliptic-curve cryptography and thus forward secrecy, while OpenSSH_8.9p1 no longer supports the less secure ssh-rsa or ssh-dss by default.

At least at the time of writing this and using OpenSSH_8.9p1, you can enable ssh-rsa or ssh-dss explicitly and then the connection will work. Also include the -v option to see the details:

$ ssh -oHostKeyAlgorithms=+ssh-rsa ssh -V
OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013

And so, my workaround was to add host-specific exceptions to my ~/.ssh/config file. I could have done this with a Host stanza for each GoDaddy host, or with two Host stanzas with two hosts each. I listed all four in one, it's up to you.

$ cat ~/.ssh/config
Host *
  IdentityFile ~/.ssh/id_ed25519
  IdentityFile ~/.ssh/id_ecdsa

# These are needed because in August 2022 with Mint using OpenSSH_8.9p1,
# GoDaddy was still using OpenSSH_5.3p1 from Feb 2013.
  HostKeyAlgorithms +ssh-rsa

I also needed to edit ~/.ssh/known_hosts and remove the lines for those outdated GoDaddy hosts.

And now, using that workaround to connect from my updated laptop to an outdated GoDaddy server:

$ ssh -V
OpenSSH_8.9p1 Ubuntu-3, OpenSSL 3.0.2 15 Mar 2022
$ ssh -v uptime
[... much output ...]
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: diffie-hellman-group-exchange-sha256
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: aes128-ctr MAC: compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: compression: none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(2048<3072<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_GROUP received
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: SSH2_MSG_KEX_DH_GEX_REPLY received
debug1: Server host key: ssh-rsa SHA256:bS8oN8AH5+dgNQGuyB+EcpSQAVdJqP2tw+nrNuMJVrY
[... more output ...]

Here's what we would hope to see, communicating with an up-to-date server:

$ ssh -v ssh -V
[... much output ...]
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: MAC: <implicit> compression: none
debug1: kex: client->server cipher: MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:BbuCDDVQOHhVK8WdVfZBKboAp29GEwmqfaRSTWIlRLw
[... more output ...]
OpenSSH_8.8p1, OpenSSL 1.1.1o-freebsd  3 May 2022
debug1: channel 0: free: client-session, nchannels 1
Transferred: sent 2944, received 3100 bytes, in 0.2 seconds
Bytes per second: sent 12486.3, received 13147.9
debug1: Exit status 0

Further suggestions for dealing with legacy versions of OpenSSH are here

Dealing with GoDaddy's Unreliable SSH

The unreliability seems to be due to overly sensitive load balancers through which you access their web servers via SSH and thus scp or rsync. It easily takes over an hour on the phone to reach someone who has any idea what you're talking about. You want to avoid that problem.

Add a --timeout=10 option. My sychronize-websites script contains the following: Linux-based hosting at GoDaddy puts your web root in your ~/public_web directory:

[...lines deleted...]
rsync -av --progress --exclude '*.swp' --timeout=10 \
echo '-------------------------------------------------------------------------'
rsync -av --progress --exclude '*.swp' --timeout=10 \
echo '-------------------------------------------------------------------------'
rsync -av --progress --exclude '*.swp' --timeout=10 \
echo '-------------------------------------------------------------------------'
rsync -av --progress --exclude '*.swp' --timeout=10 \
echo '-------------------------------------------------------------------------'
rsync -av --progress --exclude '*.swp' \
[...lines deleted...] 

I frequently see output like this, where one or more GoDaddy servers timed out:

$ synchronize-websites
sending incremental file list

sent 77,483 bytes  received 136 bytes  51,746.00 bytes/sec
total size is 303,892,020  speedup is 3,915.18
sending incremental file list
rsync error: timeout in data send/receive (code 30) at io.c(137) [receiver=3.0.9]
rsync: connection unexpectedly closed (86 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(235) [sender=3.1.3]
sending incremental file list

sent 46,495 bytes  received 466 bytes  18,784.40 bytes/sec
total size is 159,567,423  speedup is 3,397.87
sending incremental file list

sent 43,049 bytes  received 496 bytes  29,030.00 bytes/sec
total size is 912,999,178  speedup is 20,966.80
sending incremental file list
         18,859 100%   17.32MB/s    0:00:00 (xfr#1, ir-chk=1079/3867)

sent 463,173 bytes  received 1,167 bytes  132,668.57 bytes/sec
total size is 2,249,420,775  speedup is 4,844.34

Or, instead, this:

[...lines deleted...]
kex_exchange_identification: Connection closed by remote host
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(235) [sender=3.1.3]
[...lines deleted...] 

Give it a moment, then try it again. Be careful! Rapidly repeated failures get your client IP address blacklisted, and it takes at least an hour on the phone to get the problem escalated to higher-level support staff who can see and fix these things. Even when you start by explaining exactly what's going on.