Upgrading Red Hat Enterprise Linux / CentOS / Oracle Linux

RHEL/CentOS/Oracle Linux Migration: Networking

There have been enormous changes in Linux networking as we went through recent versions of popular distributions. Some of the changes, like the move to the ip command in the IProute2 package, have been underway for a while. Others, like the changes to the network interface names, caught many people by surprise.

The previous page described the relatively small changes in package management and building software from source.

8 — Networking: IP Configuration

Kernel support for hardware

RHEL 5

Used /etc/modprobe.conf and /etc/modprobe.d/* to control which kernel module is used for which device, and used its parameters to control which of two Ethernet interfaces with the same chipset became eth0 versus eth1.

RHEL 6

Used /etc/udev/rules.d/70-persistent-net-rules to remember Ethernet devices by MAC address. Replace a card or (move onto new hardware) and a single Ethernet card is no longer eth0, it is now eth1.

RHEL 7–9

Udev activity has been absorbed into systemd.

Network interface names change going from 6 to 7, see details below.

RHEL 8 dropped support for some older network interface chipsets.

Network Interface Names

Starting with RHEL 7 (really with v197 systemd), the traditional interface names eth0, wlan0, and so on have been replaced. See the details here, and even more details here, but the interface names are now based on en for Ethernet or wl for wireless LAN, followed by one of:

1. Firmware-provided index numbers for on-board devices: eno1 for the first on-board Ethernet.
   However, note that this Fedora page says "Ethernet ports embedded on server motherboards will be named em<port_number>, ...," so em1 or similar.
2. Firmware-provided PCI Express hotplug index numbers: ens1 for the first hotplug Ethernet.
3. Physical locations: enp2s0 for an Ethernet device at PCI bus address 02:00.0.
   However, note that this Fedora page says that "... ports on PCI cards will be named p<slot_number>p<port_number>, corresponding to the chassis labels," so p3p1 or similar.
4. Chained physical locations: wlp0s2f1u4 for a USB WLAN device, with the USB controller at PCI bus address 00:02.0 and using USB function 1 and port 4 (meaning that on USB bus 1 it's device 4).
5. MAC address: enx0011951E8EB6 for a device with MAC address 00:11:95:1E:8E:B6.

Version #1 applies if that information is available, falling back to #2 and then #3 or #4, and finally defaulting to an unpredictable detection order and the old style named ethN and wlanN. The MAC address version is only used if the administrator configures it.

On VMware, a bug leads to the nonsensical numbered name eno16777736. Wow, where are the other 16,777,735 Ethernet ports? One workaround is to edit the *.vmx line and change the number found here:
ethernet0.pciSlotNumber="33"
from 33 to a smaller unused number such as 20. That virtual Ethernet device will appear at PCI address 00:14.0 (which is base 16, 0x14 = 20), and it will be enp0s20. Experiment: Not all numbers work, many will result in 16777736 or other absurdly large numbers.

For example, I boot my laptop from a CentOS live CD and see the following. Both the wireless LAN and Ethernet controllers are on the PCI bus, but the WLAN device gets a motherboard device name and the Ethernet device gets a PCI device name.

# lspci | grep -i net
02:00.0 Network controller: Realtek Semiconductor Co., Ltd. RTL8191SEvA Wireless LAN Controller (rev 10)
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8101E/RTL8102E PCI Express Fast Ethernet controller (rev 02)
# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode default qlen 1000
    link/ether 2c:27:d7:c5:d3:7b brd ff:ff:ff:ff:ff:ff
3: wlo1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DORMANT qlen 1000
    link/ether 68:a34:c4:70:f1:73 brd ff:ff:ff:ff:ff:ff

The biosdevname command is intended to take a kernel device name as an argument and return the BIOS-given name it "should" be. Or at least that's what its manual page says. I honestly don't know what's going on here, I get a completely different name for the Ethernet port and no output at all for the wireless device in either the new or traditional name:

# biosdevname -i enp3s0
p1p1
# biosdevname -i wlo1
# biosdevname -i wlan0

My desktop has a wireless LAN interface plugged into a USB port:

$ lspci | grep USB
00:02.0 USB controller: NVIDIA Corporation MCP61 USB 1.1 Controller (rev a3)
00:02.1 USB controller: NVIDIA Corporation MCP61 USB 2.0 Controller (rev a3)
01:09.0 USB controller: OPTi Inc. 82C861 (rev 10)
$  lsusb
Bus 003 Device 004: ID 148f:5370 Ralink Technology, Corp. RT5370 Wireless Adapter
$  lsusb -t
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/10p, 480M
    |__ Port 1: Dev 2, If 0, Class=Mass Storage, Driver=usb-storage, 480M
    |__ Port 4: Dev 4, If 0, Class=Vendor Specific Class, Driver=rt2800usb, 480M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ohci-pci/2p, 12M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ohci-pci/10p, 12M
    |__ Port 7: Dev 6, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
$ ip addr
[...]
4: wlp0s2f1u4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether c8:3a:35:cf:3b:b9 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.102/24 brd 192.168.1.255 scope global wlp0s2f1u4
       valid_lft forever preferred_lft forever
    inet6 fe80::ca3a:35ff:fecf:3bb9/64 scope link 
       valid_lft forever preferred_lft forever

You can inventory the network devices by searching the /sys/devices hierarchy:

# ls -F /sys/class/net
wlo1@  enp3s0@  lo@

# ls -FL /sys/class/net/[we]*
/sys/class/net/wlo1:
addr_assign_type  device@  ifalias    netdev_group  queues/       type
address           dev_id   ifindex    operstate     speed         uevent
addr_len          dormant  iflink     phy80211@     statistics/   wireless/
broadcast         duplex   link_mode  phys_port_id  subsystem@
carrier           flags    mtu        power/        tx_queue_len

/sys/class/net/enp3s0:
addr_assign_type  device@  ifalias    netdev_group  speed         uevent
address           dev_id   ifindex    operstate     statistics/
addr_len          dormant  iflink     phys_port_id  subsystem@
broadcast         duplex   link_mode  power/        tx_queue_len
carrier           flags    mtu        queues        type

See this page for much more on Sysfs and the /sys file system. The files in /sys/class/net are symbolic links pointing into the PCI device tree.

TCP/IP Commands

The traditional commands ifconfig, arp, route, and netstat by the Iproute2 package and its ip and ss. This has been happening gradually on all distributions, but it becomes much more obvious and important with RHEL 7.

The old Net-tools package including ifconfig uses the older-style IOCTL calls to retrieve information from the kernel and cannot be relied upon. The new IProute2 package uses the kernel's Netlink interface. It does what you ask, and you can believe what it tells you. I have more details elsewhere, but here is a short summary of what changes when you compare RHEL 7 to earlier versions:

NetworkManager and firewalld

Two local services or daemons have appeared. RHEL 6 has NetworkManager, RHEL 7 has that plus firewalld. Both were developed in the Fedora community and they seem like they might be nice for mobile devices. But...

NetworkManager does a very good job of handling situations where wired and wireless networks appear, disappear, and change. But.... Is an operating system costing hundreds to a few thousands of dollars per CPU socket pair per year going to be used in a notebook carried from coffeeshop to coffeeshop? Meanwhile, NetworkManager slows down the process to the point that restarting the networking can take 30 seconds or more.

My recommendation for RHEL 6 and 7 has been to disable the NetworkManager service. However, in RHEL 8 the core ifup and ifdown scripts have been replaced by NetworkManager components, so we're stuck with it. The good news is that its much better in RHEL 8 and later.

RHEL 7 adds the firewalld daemon, which may appear comfortably familiar to Windows users. But...

Again, I am not going to advise anyone to pay hundreds to thousands of dollars per year for an OS for a notebook, and your servers worth the expenditure are not going to be connected to fluctuating networks. Most servers don't need terribly complex firewall configurations. Design and carefully analyze and test a script you call from rc.local or for which you create a systemd unit.

To make my recommendations happen:

RHEL 5

Has neither NetworkManager nor firewalld, no fix needed.

RHEL 6

# /etc/init.d/NetworkManager stop
# chkconfig NetworkManager off 

RHEL 7

# systemctl stop NetworkManager firewalld
# systemctl disable NetworkManager firewalld 

RHEL 8, 9

Useful Networkmanager and firewalld, no fix needed. Learn the firewall-cmd command.

IP Configuration and Scripts

The ifup and ifdown scripts and the systemd service network disappear at RHEL 8. You're left with nothing but the NetworkManager systemd service and the nmcli command. Now, in place of ifdown and ifup with interfacename being something like enp0s3 you use:

# nmcli connection down interfacename
# nmcli connection up interfacename

At RHEL 9, the configuration scripts
/etc/sysconfig/network-scripts/ifcfg-*
are replaced with:
/etc/NetworkManager/system-connections/interfacename.nmconnection
Those files are in INI format with stanzas, an example appears below.

Up through RHEL 8, you configure interfaces in the simple files /etc/sysconfig/network-scripts/ifcfg-interfacename. Other networking configuration goes into other files under /etc/. Below is a table showing how to set up this configuration:

• Fully-qualified domain name = server.example.com.
• IPv4 address = 1.2.3.4/24.
• IPv4 default router = 1.2.3.254.
• IPv6 should be autoconfigured with link-local fe80:: address plus any globally routable assignment it can pick up from the IPv6 router.
• DNS servers = 1.2.3.253 and 1.2.3.252. Attempts to resolve client will first try client.example.com and if that fails, client.subdom.example.com.

Notice that /etc/resolv.conf is the only one that doesn't change when progressing from RHEL 5 to 8! However, if NetworkManager is running, and you simply edit that file, it will not have the desired effect. That information goes into the interface configuration file(s).

The interface files ifcfg-* are shown in two forms, what is created during the installation, and all that is really needed. Simplify your ifcfg-* scripts for ease of maintenance and reduction of errors. The IPv4 address and netmask define the network and broadcast. Bizarre and mysterious things happen if your changes specify a network and/or broadcast that does not agree with the logic of the IP address and netmask. Things like hosts on your LAN are not routable, but you do have connectivity to other networks.

The HWADDR entry will cause problems when you replace hardware. It was intended to solve the problem that really was solved with predictable address names.

The UUID entry will cause further problems through the meddling of NetworkManager.

I have also reordered the lines in the file for a more logical sequence, and removed the unneeded double-quotes for RHEL 7 and later.

search example.com subdom.example.com
nameserver 1.2.3.253 1.2.3.252 

127.0.0.1   localhost.localdomain  localhost
::1         localhost6  localhost6.localdomain6
1.2.3.4     server.example.com  server 

NETWORKING=yes
NETWORKING_IPV6=yes
HOSTNAME=server.example.com
GATEWAY=1.2.3.254 

DEVICE=eth0
BOOTPROTO=static
BROADCAST=1.2.3.255
HWADDR=00:11:95:1e:8e:b6
IPADDR=1.2.3.4
IPV6INIT=yes
IPV6_AUTOCONF=yes
NETMASK=255.255.255.0
NETWORK=1.2.3.0
ONBOOT=yes 

DEVICE=eth0
ONBOOT=yes
BOOTPROTO=static
IPADDR=1.2.3.4
NETMASK=255.255.255.0
IPV6INIT=yes
IPV6_AUTOCONF=yes 

search example.com subdom.example.com
nameserver 1.2.3.253 1.2.3.252 

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6 

NETWORKING=yes
HOSTNAME=server.example.com 

DEVICE=eth0
HWADDR=01:02:03:04:05:06
TYPE=Ethernet
UUID=2ec2be26-d402-4712-80d9-440d6b224413
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=none
IPADDR=1.2.3.4
NETMASK=255.255.255.0
GATEWAY=1.2.3.254
DNS1=1.2.3.253
DNS2=1.2.3.252
IPV6INIT=yes
USERCTL=no 

DEVICE=eth0
ONBOOT=yes
BOOTPROTO=none
IPADDR=1.2.3.4
NETMASK=255.255.255.0
GATEWAY=1.2.3.254
IPV6INIT=yes 

search example.com subdom.example.com
nameserver 1.2.3.253 1.2.3.252 

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6 

server.example.com 

TYPE="Ethernet"
BOOTPROTO="none"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="no"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_FAILURE_FATAL="no"
NAME="enp0s3"
UUID="2ec2be26-d402-4712-80d9-440d6b224413"
DEVICE="enp0s3"
ONBOOT="yes"
IPADDR="1.2.3.4"
PREFIX="24"
GATEWAY="1.2.3.254"
DNS1="1.2.3.253"
DNS2="1.2.3.252"
DOMAIN="example.com"
IPV6_PEERDNS="yes"
IPV6_PEERROUTES="yes"
IPV6_PRIVACY="no" 

DEVICE=enp0s3
ONBOOT=yes
BOOTPROTO=none
IPADDR=1.2.3.4
PREFIX=24
DEFROUTE=yes
GATEWAY=1.2.3.254
DNS1=1.2.3.253
DNS2=1.2.3.252
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_ADDR_GEN_MODE=stable-privacy 

If you want a static IPv6 configuration for your Internet-facing IPv6 server, change the IPv6 details to something like the following. It will pick up IPv6 routing information from the router:

IPV6INIT=yes
IPV6_AUTOCONF=no
IPV6ADDR=2001:1800:1234:90::c8/64 

Keep in mind that an IPv6 address is laid out like the following, using my example address above for illustration. Remember that leading zeros can be deleted in each 16-bit block, and one string of consecutive zeros can be collapsed into "::", so the final "::c8" really represents 64 bits:

With RHEL 9 the configuration changes to INI style files with stanzas. The GNOME project has a page describing the various properties you can set.

$ cat etc/NetworkManager/system-connections/enp0s3.nmconnection
[connection]
id=enp0s3
uuid=3cb09699-4c57-3499-8134-efbecab038b4
type=ethernet
autoconnect-priority=-999
interface-name=enp0s3
timestamp=1661181433

[ethernet]

[ipv4]
method=auto

[ipv6]
addr-gen-mode=eui64
method=auto

[proxy]

Virtual Network Interfaces vs Multiple IP Addresses

Let's say you want to assign multiple IP addresses to one physical interface. This changes when you go from RHEL 6 to 7. The configuration is done in files in the directory /etc/sysconfig/network-scripts, the files are named ifcfg-interface. There are several changes to deal with in going to RHEL/CentOS/Oracle Linux 7. First, commands and interface names:

• The interface names change.
• You now check and set interface configuration with ip instead of ifconfig.
• You now control services with systemctl instead of init scripts.

Second, in RHEL 5 and 6, you set up virtual interfaces. The physical interface was eth0 and the virtual interfaces were eth0:0, eth0:1, and so on. Create a separate file for each virtual interface. You saw separate stanzas for the physical interface and all the virtual interfaces in the ifconfig output, although only the physical one has the physical details. If a virtual interface is on a different logical IP network, there will be another entry in the routing table via the physical interface. See example output below.

In RHEL 7 and later, you assign additional addresses and netmasks to the interface. There is just enp0s3 (or whatever), it has multiple IP addresses. Add address/netmask entries to the one configuration file. Again, if one address/netmask is on another local IP network, there will be an added line in the routing table, see the example output below.

# more ifcfg-eth*
:::::::::::::
ifcfg-eth0
:::::::::::::
DEVICE=eth0
ONBOOT=yes
IPADDR=10.0.0.100
NETMASK=255.255.255.0

:::::::::::::
ifcfg-eth0:0
:::::::::::::
DEVICE=eth0:0
ONBOOT=yes
IPADDR=10.0.0.101
NETMASK=255.255.255.0

:::::::::::::
ifcfg-eth0:1
:::::::::::::
DEVICE=eth0:0
ONBOOT=yes
IPADDR=10.0.1.100
NETMASK=255.255.255.0

# /etc/init.d/network restart
# ifconfig
eth0:  Link encap:Ethernet  HWaddr 00:11:95:1E:8E:B6
	inet addr: 10.0.0.100 Bcast:10.0.0.255 Mask: 255.255.255.0
	inet6 addr: fe80::6e62:6dff:feb2:f841/64 Scope:Link
	UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
	RX packets:5695666 errors:0 dropped:0 overruns:0 frame:0
	TX packets:5269107 errors:0 dropped:0 overruns:0 carrier:0
	collisions:0 txqueuelen:1000 
	RX bytes:2828275570 (2.6 GiB)  TX bytes:2753415143 (2.5 GiB)

eth0:0  Link encap:Ethernet  HWaddr 00:11:95:1E:8E:B6
	inet addr: 10.0.0.101 Bcast:10.0.0.255 Mask: 255.255.255.0
	UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

eth0:1  Link encap:Ethernet  HWaddr 00:11:95:1E:8E:B6
	inet addr: 10.0.1.100 Bcast:10.0.0.255 Mask: 255.255.255.0
	UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

lo	Link ....

# route
Destination  Gateway     Genmask        Flags Metric Ref  Use Iface
default      10.0.0.254  0.0.0.0        UG    5      0      0 eth0
10.0.0.0     *           255.255.255.0  U     0      0      0 eth0
10.1.1.0     *           255.255.255.0  U     0      0      0 eth0

# more ifcfg-e*
:::::::::::::
ifcfg-enp0s3
:::::::::::::
DEVICE=enp0s3
ONBOOT=yes
IPADDR=10.0.0.100
PREFIX=24
IPADDR0=10.0.0.101
PREFIX0=24
IPADDR1=10.0.0.102
PREFIX1=24
IPADDR2=10.0.1.100
PREFIX2=24

# systemctl restart network
# ip addr
1: lo: ...
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> ...
    link/ether 00:0c:29:cf:62:79 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.100/24 brd.1.1.255 scope global enp0s3
       valid_lft forever preferred_lft forever
    inet 10.0.1.100/24 brd.1.1.255 scope global enp0s3
       valid_lft forever preferred_lft forever
    inet 10.0.0.101/24 brd.1.1.255 scope global secondary enp0s3
       valid_lft forever preferred_lft forever
    inet 10.0.0.102/24 brd.1.1.255 scope global secondary enp0s3
       valid_lft forever preferred_lft forever
    inet6 ...
# ip route
default via 10.1.1.254 dev enp0s3  proto static  metric 1024
10.0.0.0/24 dev enp0s3  proto kernel  scope link  src 10.0.0.100
10.0.1.0/24 dev enp0s3  proto kernel  scope link  src 10.0.1.100

Filtering with iptables / ip6tables or nft

Recall that you got default packet filtering rules with iptables and ip6tables starting with RHEL 6. The RHEL 6 iptables default rules are:

Chain INPUT (policy ACCEPT)
target    prot opt source        destination         
ACCEPT    all  --  anywhere      anywhere      state RELATED,ESTABLISHED 
ACCEPT    icmp --  anywhere      anywhere            
ACCEPT    all  --  anywhere      anywhere            
ACCEPT    tcp  --  anywhere      anywhere      state NEW tcp dpt:ssh 
REJECT    all  --  anywhere      anywhere      reject-with icmp-host-prohibited 

Chain FORWARD (policy ACCEPT)
target    prot opt source        destination         
REJECT    all  --  anywhere      anywhere      reject-with icmp-host-prohibited 

Chain OUTPUT (policy ACCEPT)
target    prot opt source        destination          

The RHEL 6 ip6tables default rules are:

Chain INPUT (policy ACCEPT)
target    prot opt source        destination         
ACCEPT    all       anywhere     anywhere      state RELATED,ESTABLISHED 
ACCEPT    ipv6-icmp anywhere     anywhere            
ACCEPT    all       anywhere     anywhere            
ACCEPT    tcp       anywhere     anywhere      state NEW tcp dpt:ssh 
REJECT    all       anywhere     anywhere      reject-with icmp6-adm-prohibited 

Chain FORWARD (policy ACCEPT)
target    prot opt source        destination         
REJECT    all       anywhere     anywhere      reject-with icmp6-adm-prohibited 

Chain OUTPUT (policy ACCEPT)
target    prot opt source        destination          

With RHEL 7 the outputs of iptables -L and ip6tables -L are typically over 200 lines long. The rule sets are maintained by a daemon:

# systemctl status firewalld
firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled)
   Active: active (running) since Tue 20xx-xx-xx 18:54:04 UTC; 22min ago
 Main PID: 785 (firewalld)
   CGroup: /system.slice/firewalld.service
           +-/usr/bin/python /usr/sbin/firewalld --nofork --nopid

The daemon is a Python program that listens on a Unix domain socket. Verify that for yourself with this:

# lsof -p $( pgrep firewalld )
... 

When you start a service through systemd, it can contact the firewall daemon through that socket. "Hello, this is the SSH daemon sshd starting. Please let me accept connections on TCP/22."

With RHEL 8, nftables replaces iptables (and ip6tables, arptables, and ebtables) and the firewalld daemon now uses nftables as its backend.

Those changes aren't obvious. The iptables-translate and related tools translate existing rule sets into nftables syntax.

Up through RHEL 7:

# iptables -L -v ; ip6tables -L -v
[... output ...]
# iptables -t nat -L -v ; ip6tables -t nat -L -v
[... output ...]

RHEL 8:

# nft list table filter
[... output ...]
# nft list table nat
[... output ...]

# iptables-save > /tmp/iptables.dump
# iptables-restore-translate -f /tmp/iptables.dump
# ip6tables-save > /tmp/ip6tables.dump
# ip6tables-restore-translate -f /tmp/ip6tables.dump

Virtual Bridges

Red Hat was really pushing virtualization and especially containers by the time RHEL 7 came out, so a default installation would probably include the libvirt package and its dependencies even if you didn't explicitly ask for it. That leads to a virtual bridge interface virbr0 at 192.168.122.1/24.

The presence of virbr0 and the 192.168.122.0/24 network will lead to errors in the /var/log/firewalld log file.

You can remove all the related packages:

RHEL 7

# rpm --erase '*libvirt*'	\
	gnome-boxes virt-top virt-viewer virt-manager	\
	virt-manager-common virt-install libguestfs
# yum autoremove 

RHEL 8

The extra virt-* virtualization packages weren't installed by default on RHEL 8.

# rpm --erase '*libvirt*' gnome-boxes
# yum autoremove 

Do that, reboot, and there is no longer a virbr0 interface, a route to the 192.168.122.0/24 network, or error messages appended to /var/log/firewalld.