Upgrading Red Hat Enterprise Linux / CentOS / Oracle

RHEL/CentOS/Oracle 6–7–8–9 Changes

Here are my notes on what changes when you upgrade from one major release of Red Hat Enterprise Linux, or was its clone CentOS, or its derivative Oracle Linux. In other words, how to upgrade from RHEL or CentOS or Oracle Linux 6 to 7 to 8 to 9.

The changes 6 to 7 were enormous. Then 7–8 and 8–9 were more of an evolution.

Remember that it's not as if Red Hat is changing all of these things. Their installers and their graphical configuration tools, sure, those are the distribution's changes. But much of this is the result of the many underlying projects changing. So this collection of pages also applies to Oracle Linux, where the major release numbers 6, 7, 8, 9 largely align with RHEL and CentOS, and really to any distribution as its components have updated over the past several years.

CentOS ceased to be a RHEL clone with CentOS 7. There was a CentOS Linux 8 but it ended at the end of 2021. Now there is CentOS Stream Linux, and its versions are quite similar to the corresponding RHEL versions. But basically CentOS Stream has taken the role of Fedora, and Fedora still exists for, well, ... I'm not sure exactly what its role is. Even sketchier beta testing and development, I suppose. It makes Oracle Linux look even more attractive to me.

1 — Installation and Exploring the Newly Installed System

The order of events and the logic of the installation completely change with every major release.

Linux has always supported much larger systems than I have had to worry about. For example, RHEL 7 had 48/46 bits of virtual/physical memory address, and the kernel did 4 levels of page tables. That limited virtual memory to 256 TiB and physical memory to 64 TiB, although 64 terabytes of RAM was way beyond any hardware I dealt with.

Well, RHEL 8 moved the goalposts much further out. Its kernel supports 57/52 bits of virtual/physical memory address, and does 5 levels of page tables. So, it could support up to 128 PiB of virtual address space and 4 PiB of physical memory.

With RHEL 8, Red Hat started emphasizing deployment in containers, in cloud settings, and as a virtualization host. The new Composer lets you created customized images in several formats, including those prepared for deployment onto public cloud providers.

RHEL 8 added the web console, Cockpit. It lets you create and manage virtual machines. The QEMU full system virtualization control introduces a sandboxing feature with RHEL 8, limiting the system calls QEMU can make.

Here are some Cockpit screenshots, note that it listens on TCP/9090:

The change from RHEL 6 to 7 was the most sweeping. Not because of what Red Hat did, but because that spans the change from init to systemd and the many /usr/*bin/*ctl programs, and the Ethernet device name shift that came with IProute2

The commands and configuration files used to configure RHEL 7 were often completely different from RHEL 6. That didn't happen in going from RHEL 7 to 8 to 9. RHEL 8 and 9 seem to me to be about updating existing components.

Many 7–8–9 updates are quite significant:
Kernel 3.10.0 to 4.18.0 to 5.14.0,
GCC 4.8.5 to 8.2.1 to 11.2.1,
Python 2.7.5 to 3.6.6 to 3.9.10,
PHP 5.4.16 to 7.2.11 to 8.0.13,
Ruby 2.0.0 to 2.5 to 3.0.2,
OpenSSL 1.0.2k to 1.1.1 to 3.0.1,
iptables v1.4.21 to v1.8.0 and then nftables 0.9.0 to 0.9.8,
qemu-kvm 1.5.3 to 2.12.0 to 6.2.0,
and many other significant changes.

At RHEL 8 the packaging splits into two sets: BaseOS (1,660 packages, 905 MB) and AppStream (4,987 packages, 5.3 GB). Versus 9,007 packages and 6.5 GB on the CentOS 7 Everything-1511 ISO.

As for web servers, RHEL 7 gave you Apache 2.4.6. RHEL 8 has Apache 2.4.37 and Nginx 1.14.0. Both of those support TLS 1.3. The 8 beta release had Apache 2.4.35, which does not. Apache 2.4.37, the first version of Apache to support TLS 1.3, became generally available on October 23. This was just about 3 weeks before RHEL 8-beta released. RHEL 9 has Apache 2.4.51 and Nginx 1.20.1.

X improves from one version to the next. With RHEL 5 you have just whatever was auto-detected during the installation, probably 800×600. RHEL 6 should be close to full screen size, RHEL 7 and later should be full screen size.

RHEL 8 and later instead use the Wayland display server by default. Package xorg-x11-server-Xwayland provides /usr/bin/Xwayland, while package xorg-x11-server-Xorg provides the classic /usr/bin/Xorg.

Once you get it installed, the configuration of Gnome within RHEL hides the terminal emulator in ever more inconvenient corners of the menus. Gnome 3 supports using the Windows Start Key^* to bring up a "run this command" text box that allows searching. It can be confusing, you can bring up a command by a search for something that isn't really the command's name. This left me confused as to why I could start the firewall administration tool by typing firewall at text box, which is really searching on some metadata, but I couldn't run that graphical tool using a command starting firewall.

The default user PATH environment variable has changed. To include an even earlier version to see how much it has changed:

RHEL 5:

/usr/kerberos/bin:/usr/local/bin:/bin:/usr/X11R6/bin:~/bin

No sbin components, so you had to remember to always add "-" or "-l" or "--login" to your su command so your following system administration commands work!

RHEL 6:

/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:~/bin

What an odd beginning! There are 12 programs there. Stranger yet, /usr/lib64/qt4/bin exists with 35 programs, 7 of which have the same names as those under qt-3.3. The newer version is not in the user's path. I don't know what's going on here...

RHEL 7:

/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:~/.local/bin:~/bin

It's strange that /bin and /sbin are included, as they are links to /usr/bin and /usr/sbin, respectively. And I don't know who would use ~/.local/bin to house their personal programs.

RHEL 8, 9:

~/.local/bin:~/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin

That's the same set as RHEL 7, minus the redundant /bin and /sbin, and reordered.

The Bash shell environment on RHEL 8 behaves strangely. It is case insensitive at times, and these two commands work differently:

$ ls [NO]*
... all files starting with "N" or "O" ...
$ ls [N-O]*
... all files starting with "n", "N", "o", or "O" ...

The environment variables LANG, LC_ALL, and LOCALE are correctly (for me) set to en_US.utf8, C, and en_US.utf8, respectively, on RHEL/CentOS/Oracle Linux 7. However, on RHEL 8 and 9 only LANG are set. Fix: add a line to ~/.bash_profile:

$ cat ~/.bash_profile
... lines deleted ...
# User specific environment and startup programs
export LC_ALL=C

The manual page indices always need to be rebuilt so man -k will work after the installation. There will be a cron job to do that, or you can do it manually. This had used the makewhatis utility, but with RHEL 7 mandb replaces makewhatis. The file /etc/cron.daily/man-db.cron makes this happen every night. With RHEL 8 it appears that this is enabled in /etc/sysconfig/man-db but there is no corresponding cron job. All the same, you get the indices. It seems that there is an hourly cron job, but I don't see signs of it.

When you upgrade from RHEL 6 to 7, network configuration becomes more complicated. The graphical configuration tool for networking now includes tabs for 802.1x security and higher performance data center type LAN technology.

The distribution had grown to the point that RHEL 6 no longer fit onto one single-layer DVD. DVD1 is the main installation media, it's 4.0 GB for the RHEL 6 workstation and the single CentOS image. DVD2 is the "optional" media, it's just 1.2 GB. DVD2 for the most part contains the -doc and -devel packages, but it also contains some of the intranet/LAN server packages. For example, samba-swat and lpd-cups.

RHEL 5 gave you a choice about using SELinux and the iptables and ip6tables firewall rules. RHEL 6 and later force these on you, preconfigured and enabled.

The firstboot program that runs on the first post-installation graphical boot starting with RHEL 6 insists that you create a user. However, the user creation tool is poorly designed and it does not let you fully control all the attributes of this user. Workaround: Create a dummy user to make it happy, create your real users, and then delete the dummy user.

If you're using VMware, Red Hat has included drivers that give you the benefits of VMware's VMtools, such as the ability to simply move the mouse out of the VM window without first pressing <Ctrl><Alt>. Do not install VMtools, as that can make things worse instead of better. My experience was that VMtools took away my control of the display settings and recreated the mouse and keyboard focus problem. Just install RHEL or CentOS or Oracle Linux and enjoy.

The desktop graphical user interface goes through major changes, especially going from RHEL 6 to 7. Both the default Gnome and the optional KDE graphical interface go through major version changes. RHEL 8 dropped the KDE window manager. Linux Mint also dropped KDE after its version 18.3.

Gnome is surprisingly resource-hungry. You will probably want to put autospawn = no in your ~/.pulse/client.conf file and also in /etc/skel. Otherwise pulseaudio will always run and always restart, and sometimes it will impose a surprising load on the CPU.

By RHEL 7 many more packages had moved to having one main configuration file plus a collection, like what xinetd had done for some time. For example, sudo puts its system-wide settings in /etc/sudoers and then reads all the user-specific files /etc/sudoers.d/*. Rsyslog reads /etc/rsyslog.conf and then all the files in /etc/rsyslog.d/*.

This isn't anything specific to Red Hat, it is a Linux-wide trend that became common between the releases of RHEL 6 and 7. This is also similar to the shells that for some time now first use /etc/profile and then /etc/profile.d/*, followed by ~/.profile.

This is good, take advantage of it. The intent is that you don't touch the distribution-provided file /etc/*.conf. When the package updates, rpm discovers that the main configuration file is still in pristine form and you have no confusing *.rpmnew, *.rpmold, and so on to track down and manually merge changes. Let the distribution's provided configuration file do whatever they intended, and their your locally created files can override any system-wide settings you want to change.

With RHEL 7 there is a significant difference between running vi and vim. Many distributions, including earlier RHEL, really ran vim, the improved version with more capabilities, when you typed just vi. (this was through some trickery with /etc/alternatives).

vim is your friend, it really helps by coloring the syntax in configuration files, programming languages, HTML, and much more. I had the benefit of being in the habit of always typing vim so I got the better one on non-Linux OSes like OpenBSD and Solaris. Adopt my habit!