Upgrading Red Hat Enterprise Linux / CentOS / Oracle

RHEL/CentOS/Oracle 5–6–7–8–9 Changes

You encounter enormous changes when you change to the next major version of Red Hat's Enterprise Linux product suite. Here are my notes on what changes when you upgrade from one major release of Red Hat Enterprise Linux, or its clone CentOS, or Oracle Linux. In other words, how to upgrade from RHEL or CentOS or Oracle Linux 5 to 6 to 7 to 8 to 9.

The changes from 5 to 6 were large, and those from 6 to 7 were even larger. Then 7–8 and 8–9 were more of an evolution.

Remember that it's not as if Red Hat is changing all of these things. Their installers and their graphical configuration tools, sure, those are the distribution's changes. But much of this is the result of the many underlying projects changing. So this collection of pages also applies to Oracle Linux, where the major release numbers 5, 6, 7, 8, 9 largely align with RHEL and CentOS, and really to any distribution as its components have updated over the past several years.

1 — Installation and Exploring the Newly Installed System

The order of events and the logic of the installation completely change with every major release.

Linux has always supported much larger systems than I have had to worry about. For example, RHEL 7 had 48/46 bits of virtual/physical memory address, and the kernel did 4 levels of page tables. That limited virtual memory to 256 TiB and physical memory to 64 TiB, although 64 terabytes of RAM was way beyond any hardware I dealt with.

Well, RHEL 8 has moved the goalposts much further out. Its kernel supports 57/52 bits of virtual/physical memory address, and does 5 levels of page tables. So, it could support up to 128 PiB of virtual address space and 4 PiB of physical memory.

RHEL 8 has better support for deployment in containers and in cloud settings. The new Composer lets you created customized images in several formats, including those prepared for deployment onto public cloud providers.

The RHEL 8 web console, Cockpit, lets you create and manage virtual machines. The QEMU full system virtualization control introduces a sandboxing feature with RHEL 8, limiting the system calls QEMU can make.

Here are some Cockpit screenshots:

The change from RHEL 6 to 7 was the most sweeping. Not because of what Red Hat did, but because that spans the change from init to systemd and the many /usr/*bin/*ctl programs, and the Ethernet device name shift that came with IProute2

The commands and configuration files used to configure RHEL 7 were often completely different from RHEL 6. That didn't happen in going from RHEL 7 to RHEL 8. RHEL 8 seems to me to be about updating existing components.

Many updates are quite significant:
Kernel 3.10.0 to 4.18.0,
GCC 4.8.5 to 8.2.1,
Python 2.7.5 to 3.6.6,
PHP 5.4.16 to 7.2.11 (and 7.1.20),
Ruby 2.0.0 to 2.5,
OpenSSL 1.0.2k to 1.1.1,
iptables v1.4.21 to v1.8.0
(however, nftables has replaced iptables),
qemu-kvm 1.5.3 to 2.12.0,
and others.

Red Hat did a lot of work between the beta release in November 2018 and the final release in May 2019. The kernel stayed at the same "upstream" or original version, 4.18.0, but it went from 4.18.0-32 to 4.18.0-80. So, there were 48 rounds of configuration changes and kernel tuning and patching and modifying. Python changed from 3.6.6 to 3.6.8, Ruby from 2.5 to 2.5.3, and iptables from 1.8.0 to 1.8.2.

Other updates are relatively minor:
OpenSSH 7.4p1 to 7.8p1,
Samba 4.8.3 to 4.9.1,
Bash to 4.2.46 to 4.4.19,
and others.

RHEL 8 splits the packaging into two sets: BaseOS (1,660 packages, 905 MB) and AppStream (4,987 packages, 5.3 GB). Versus 9,007 packages and 6.5 GB on the CentOS 7 Everything-1511 ISO.

As for web servers, RHEL 7 gave you Apache 2.4.6. RHEL 8 has Apache 2.4.37 and Nginx 1.14.0. Both of those support TLS 1.3. The 8 beta release had Apache 2.4.35, which does not. Apache 2.4.37, the first version of Apache to support TLS 1.3, became generally available on October 23. This was just about 3 weeks before RHEL 8-beta released.

X improves from one version to the next. With RHEL 5 you have just whatever was auto-detected during the installation, probably 800×600. RHEL 6 should be close to full screen size, RHEL 7 and later should be full screen size.

RHEL 8 instead uses the Wayland display server by default. Package xorg-x11-server-Xwayland provides /usr/bin/Xwayland, while package xorg-x11-server-Xorg provides the classic /usr/bin/Xorg.

Once you get it installed, the configuration of Gnome within RHEL hides the terminal emulator in ever more inconvenient corners of the menus. Gnome 3 supports using the Windows Start Key^* to bring up a "run this command" text box that allows searching. It can be confusing, you can bring up a command by a search for something that isn't really the command's name. This left me confused as to why I could start the firewall administration tool by typing firewall at text box, which is really searching on some metadata, but I couldn't run that graphical tool using a command starting firewall.

The default user PATH environment variable changes:

RHEL 5:

/usr/kerberos/bin:/usr/local/bin:/bin:/usr/X11R6/bin:~/bin

No sbin components, so remember to always add "-" or "-l" or "--login" to your use su command so your following system administration commands work!

RHEL 6:

/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:~/bin

What an odd beginning! There are 12 programs there. Stranger yet, /usr/lib64/qt4/bin exists with 35 programs, 7 of which have the same names as those under qt-3.3. The newer version is not in the user's path. I don't know what's going on here...

RHEL 7:

/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:~/.local/bin:~/bin

It's strange that /bin and /sbin are included, as they are links to /usr/bin and /usr/sbin, respectively. And I don't know who would use ~/.local/bin to house their personal programs.

RHEL 8:

~/.local/bin:~/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin

That's the same set as RHEL 7, minus the redundant /bin and /sbin, and reordered.

The Bash shell environment on RHEL 8 behaved strangely. It was case insensitive at times, and these two commands worked differently:

$ ls [NO]*
... all files starting with "N" or "O" ...
$ ls [N-O]*
... all files starting with "n", "N", "o", or "O" ...

The environment variables LANG, LC_ALL, and LOCALE are correctly (for me) set to en_US.utf8, C, and en_US.utf8, respectively, on RHEL/CentOS/Oracle Linux 7. However, on RHEL 8 only LANG was set. Fix: add a line to ~/.bash_profile:

$ cat ~/.bash_profile
... lines deleted ...
# User specific environment and startup programs
export LC_ALL=C

The manual page indices always need to be rebuilt so man -k will work after the installation. There will be a cron job to do that, or you can do it manually. This used the makewhatis utility for ages, but with RHEL 7 mandb replaces makewhatis. The file /etc/cron.daily/man-db.cron makes this happen every night. With RHEL 8 it appears that this is enabled in /etc/sysconfig/man-db but there is no corresponding cron job. All the same, you get the indices. It seems that there is an hourly cron job, but I don't see signs of it.

When you upgrade from RHEL 5 to 6, you will notice that the network configuration has become rather complicated. See the IP networking section later for what's behind this. Among other things, be careful to de-select Require addressing for this connection to complete for IPv4 and also IPv6 if you use it. You get more choices about storage starting with RHEL 6, some of it a bit exotic for typical users. If you customize the package selection, watch out! It is very tempting to click the Next button prematurely. That immediately starts an installation process, and the only way to abort it is to reset or power-cycle the machine and start over.

When you upgrade from 6 to 7, it becomes more complicated yet. The graphical configuration tool for networking now includes tabs for 802.1x security and higher performance data center type LAN technology.

The distribution had grown to the point that RHEL 6 no longer fit onto one single-layer DVD. DVD1 is the main installation media, it's 4.0 GB for the RHEL 6 workstation and the single CentOS image. DVD2 is the "optional" media, it's just 1.2 GB. DVD2 for the most part contains the -doc and -devel packages, but it also contains some of the intranet/LAN server packages. For example, samba-swat and lpd-cups.

RHEL 5 gave you a choice about using SELinux and the iptables and ip6tables firewall rules. RHEL 6 and later force these on you, preconfigured and enabled.

The firstboot program that runs on the first post-installation graphical boot starting with RHEL 6 insists that you create a user. However, the user creation tool is poorly designed and it does not let you fully control all the attributes of this user. Workaround: Create a dummy user to make it happy, create your real users, and then delete the dummy user.

If you're using VMware, Red Hat has included drivers that give you the benefits of VMware's VMtools, such as the ability to simply move the mouse out of the VM window without first pressing <Ctrl><Alt>. Do not install VMtools, as that can make things worse instead of better. My experience was that VMtools took away my control of the display settings and recreated the mouse and keyboard focus problem. Just install RHEL or CentOS or Oracle Linux and enjoy.

The desktop graphical user interface goes through major changes, especially going from RHEL 6 to 7. Both the default Gnome and the optional KDE graphical interface go through major version changes. RHEL 8 dropped the KDE window manager. Linux Mint also dropped KDE after its version 18.3.

Gnome 3.8 is surprisingly resource-hungry. You will probably want to put autospawn = no in your ~/.pulse/client.conf file and also in /etc/skel. Otherwise pulseaudio will always run and always restart, and sometimes it will impose a surprising load on the CPU.

By RHEL 7 many more packages had moved to having one main configuration file plus a collection, like what xinetd has done for some time. For example, sudo puts its system-wide settings in /etc/sudoers and then reads all the user-specific files /etc/sudoers.d/*. Rsyslog reads /etc/rsyslog.conf and then all the files in /etc/rsyslog.d/*.

This isn't anything specific to Red Hat, it is a Linux-wide trend that became common between the releases of RHEL 6 and 7. This is also similar to the shells that for some time now first use /etc/profile and then /etc/profile.d/*, followed by ~/.profile.

This is good, take advantage of it. The intent is that you don't touch the distribution-provided file /etc/*.conf. When the package updates, rpm discovers that the main configuration file is still in pristine form and you have no confusing *.rpmnew, *.rpmold, and so on to track down and manually merge changes. Let the distribution's provided configuration file do whatever they intended, and their your locally created files can "correct" any system-wide settings you want to change.

With RHEL 7 there is a significant difference between running vi and vim. Many distributions, including earlier RHEL, really ran vim, the improved version with more capabilities, when you typed just vi. (this was through some trickery with /etc/alternatives).

vim is your friend, it really helps by coloring the syntax in configuration files, programming languages, HTML, and much more. I had the benefit of being in the habit of always typing vim so I got the better one on non-Linux OSes like OpenBSD and Solaris. Adopt my habit!