UNIX / Linux command-line interface.

Upgrading Red Hat Enterprise Linux / CentOS

RHEL/CentOS 5–6–7 Migration: Networking

There have been enormous changes in Linux networking as we went through recent versions of popular distributions. Some of the changes, like the move to the ip command in the IProute2 package, have been underway for a while. Others, like the changes to the network interface names, caught many people by surprise.

Previous: Software
Package Management

The previous page described the relatively small changes in package management and building software from source.

8 — Networking: IP Configuration

Kernel support for hardware

RHEL 5

Used /etc/modprobe.conf and /etc/modprobe.d/* to control which kernel module is used for which device, and used its parameters to control which of two Ethernet interfaces with the same chipset became eth0 versus eth1.

RHEL 6

Used /etc/udev/rules.d/70-persistent-net-rules to remember Ethernet devices by MAC address. Replace a card or (move onto new hardware) and a single Ethernet card is no longer eth0, it is now eth1.

RHEL 7

Udev activity has been absorbed into systemd.

Network interface names change, see details below.

Network Interface Names

Starting with RHEL 7 (really with v197 systemd), the traditional interface names eth0, wlan0, and so on have been replaced. See the details here, and even more details here, but the interface names are now based on en for Ethernet or wl for wireless LAN, followed by one of:

  1. Firmware-provided index numbers for on-board devices: eno1 for the first on-board Ethernet.
    However, note that this Fedora page says "Ethernet ports embedded on server motherboards will be named em<port_number>, ...," so em1 or similar.
  2. Firmware-provided PCI Express hotplug index numbers: ens1 for the first hotplug Ethernet.
  3. Physical locations: enp2s0 for an Ethernet device at PCI bus address 02:00.0.
    However, note that this Fedora page says that "... ports on PCI cards will be named p<slot_number>p<port_number>, corresponding to the chassis labels," so p3p1 or similar.
  4. Chained physical locations: wlp0s2f1u4 for a USB WLAN device, with the USB controller at PCI bus address 00:02.0 and using USB function 1 and port 4 (meaning that on USB bus 1 it's device 4).
  5. MAC address: enx0011951E8EB6 for a device with MAC address 00:11:95:1E:8E:B6.

Version #1 applies if that information is available, falling back to #2 and then #3 or #4, and finally defaulting to an unpredictable detection order and the old style named ethN and wlanN. The MAC address version is only used if the administrator configures it.

On VMware, a bug leads to the nonsensical numbered name eno16777736. Wow, where are the other 16,777,735 Ethernet ports? One workaround is to edit the *.vmx line and change the number found here:
ethernet0.pciSlotNumber="33"
from 33 to a smaller unused number such as 20. That virtual Ethernet device will appear at PCI address 00:14.0 (which is base 16, 0x14 = 20), and it will be enp0s20. Experiment: Not all numbers work, many will result in 16777736 or other absurdly large numbers.

For example, I boot my laptop from a CentOS 7 live cd and see the following. Both the wireless LAN and Ethernet controllers are on the PCI bus, but the WLAN device gets a motherboard device name and the Ethernet device gets a PCI device name.

# lspci | grep -i net
02:00.0 Network controller: Realtek Semiconductor Co., Ltd. RTL8191SEvA Wireless LAN Controller (rev 10)
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8101E/RTL8102E PCI Express Fast Ethernet controller (rev 02)
# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode default qlen 1000
    link/ether 2c:27:d7:c5:d3:7b brd ff:ff:ff:ff:ff:ff
3: wlo1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DORMANT qlen 1000
    link/ether 68:a34:c4:70:f1:73 brd ff:ff:ff:ff:ff:ff

The biosdevname command is intented to take a kernel device name as an argument and return the BIOS-given name it "should" be. Or at least that's what its manual page says. I honestly don't know what's going on here, I get a completely different name for the Ethernet port and no output at all for the wireless device in either the new or traditional name:

# biosdevname -i enp3s0
p1p1
# biosdevname -i wlo1
# biosdevname -i wlan0

My desktop has a wireless LAN interface plugged into a USB port:

$ lspci | grep USB
00:02.0 USB controller: NVIDIA Corporation MCP61 USB 1.1 Controller (rev a3)
00:02.1 USB controller: NVIDIA Corporation MCP61 USB 2.0 Controller (rev a3)
01:09.0 USB controller: OPTi Inc. 82C861 (rev 10)
$  lsusb
Bus 003 Device 004: ID 148f:5370 Ralink Technology, Corp. RT5370 Wireless Adapter
$  lsusb -t
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/10p, 480M
    |__ Port 1: Dev 2, If 0, Class=Mass Storage, Driver=usb-storage, 480M
    |__ Port 4: Dev 4, If 0, Class=Vendor Specific Class, Driver=rt2800usb, 480M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ohci-pci/2p, 12M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ohci-pci/10p, 12M
    |__ Port 7: Dev 6, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
$ ip addr
[...]
4: wlp0s2f1u4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether c8:3a:35:cf:3b:b9 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.102/24 brd 192.168.1.255 scope global wlp0s2f1u4
       valid_lft forever preferred_lft forever
    inet6 fe80::ca3a:35ff:fecf:3bb9/64 scope link 
       valid_lft forever preferred_lft forever
wl = Wireless LAN
p0s2 = Through PCI bus address 00:02.0 to...
f1u4 = USB function 1, port 4

You can inventory the network devices by searching the /sys/devices hierarchy:

# ls -F $( find /sys/devices -name net )
/sys/devices/pci0000:00/0000:00:1c.0/0000:02:00.0/net:
wlo1/

/sys/devices/pci0000:00/0000:00:1c.1/0000:03:00.0/net:
enp3s0/

/sys/devices/virtual/net:
lo/

# ls -F /sys/devices/pci*/*/*/net/*
/sys/devices/pci0000:00/0000:00:1c.0/0000:02:00.0/net/wlo1:
addr_assign_type  device@  ifalias    netdev_group  queues/       type
address           dev_id   ifindex    operstate     speed         uevent
addr_len          dormant  iflink     phy80211@     statistics/   wireless/
broadcast         duplex   link_mode  phys_port_id  subsystem@
carrier           flags    mtu        power/        tx_queue_len

/sys/devices/pci0000:00/0000:00:1c.0/0000:03:00.0/net/enp3s0:
addr_assign_type  device@  ifalias    netdev_group  speed         uevent
address           dev_id   ifindex    operstate     statistics/
addr_len          dormant  iflink     phys_port_id  subsystem@
broadcast         duplex   link_mode  power/        tx_queue_len
carrier           flags    mtu        queues        type

See this page for much more on Sysfs and the /sys file system.

TCP/IP Commands

TCP/IP Commands on
Linux, Unix, OS X, Android,
Windows, and Cisco IOS

The traditional commands ifconfig, arp, route, and netstat by the Iproute2 package and its ip and ss. This has been happening gradually on all distributions, but it becomes much more obvious and important with RHEL 7.

The old Net-tools package including ifconfig uses the older-style IOCTL calls to retrieve information from the kernel and cannot be relied upon. The new IProute2 package uses the kernel's Netlink interface. It does what you ask, and you can believe what it tells you. I have more details elsewhere, but here is a short summary of what changes when you compare RHEL 7 to others:

Task Traditional Unix, OS X
RHEL 5/6
Android Linux
RHEL 7
LAN statistics netstat -i ip -s link
ifstat
MAC–IP mapping arp -a ip neigh
IP address, netmask ifconfig [-a] ip addr
ip -6 addr
IP routing table route
netstat -r
ip route
ip -6 route
UDP/TCP activity netstat -a ss -a

NetworkManager and firewalld

Two local services or daemons have appeared. RHEL 6 has NetworkManager, RHEL 7 has that plus firewalld. Both were developed in the Fedora community and they seem like they might be nice for mobile devices. But...

NetworkManager does a very good job of handling situations where wired and wireless networks appear, disappear, and change. But.... Is an operating system costing hundreds to a few thousands of dollars per CPU socket pair per year going to be used in a notebook carried from coffeeshop to coffeeshop? Meanwhile, NetworkManager slows down the process to the point that restarting the networking can take 30 seconds or more.

My recommendation for RHEL 6 and later is to disable the NetworkManager service.

RHEL 7 adds the firewalld daemon, which may appear comfortably familiar to Windows users. But...

Again, I am not going to advise anyone to pay hundreds to thousands of dollars per year for an OS for a notebook, and your servers worth the expenditure are not going to be connected to fluctuating networks. Most servers don't need terribly complex firewall configurations. Design and carefully analyze and test a script you call from rc.local or for which you create a systemd unit.

RHEL 5

Has neither NetworkManager nor firewalld, no fix needed.

RHEL 6

# /etc/init.d/NetworkManager stop
# chkconfig NetworkManager off 

RHEL 7

# systemctl stop NetworkManager firewalld
# systemctl disable NetworkManager firewalld 

IP Configuration and Scripts

Configure interfaces in the simple files /etc/sysconfig/network-scripts/ifcfg-interfacename. Other networking configuration goes into other files under /etc/. Here is how to configure:

Notice that /etc/resolv.conf is the only one that doesn't change!

The interface files ifcfg-* are shown in two forms, what is created during the installation, and all that is really needed. Simplify your ifcfg-* scripts for ease of maintenance and reduction of errors. The IPv4 address and netmask define the network and broadcast. Bizarre and mysterious things happen if your changes specify a network and/or broadcast that does not agree with the logic of the IP address and netmask. Things like hosts on your LAN are not routable, but you do have connectivity to other networks.

The HWADDR entry will cause problems when you replace hardware. It was intended to solve the problem that really was solved with predictable address names.

The UUID entry will cause further problems through the meddling of NetworkManager.

I have also reordered the lines in the file for a more logical sequence, and removed the unneeded double-quotes for RHEL 7.

RHEL 5

/etc/resolv.conf

search example.com subdom.example.com
nameserver 1.2.3.253 1.2.3.252 

/etc/hosts

127.0.0.1   localhost.localdomain  localhost
::1         localhost6  localhost6.localdomain6
1.2.3.4     server.example.com  server 

/etc/sysconfig/network

NETWORKING=yes
NETWORKING_IPV6=yes
HOSTNAME=server.example.com
GATEWAY=1.2.3.254 

/etc/sysconfig/network-scripts/ifcfg-e* as provided:

DEVICE=eth0
BOOTPROTO=static
BROADCAST=1.2.3.255
HWADDR=00:11:95:1e:8e:b6
IPADDR=1.2.3.4
IPV6INIT=yes
IPV6_AUTOCONF=yes
NETMASK=255.255.255.0
NETWORK=1.2.3.0
ONBOOT=yes 

/etc/sysconfig/network-scripts/ifcfg-e* as simplified and recommended:

DEVICE=eth0
ONBOOT=yes
BOOTPROTO=static
IPADDR=1.2.3.4
NETMASK=255.255.255.0
IPV6INIT=yes
IPV6_AUTOCONF=yes 

RHEL 6

/etc/resolv.conf

search example.com subdom.example.com
nameserver 1.2.3.253 1.2.3.252 

/etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6 

/etc/sysconfig/network

NETWORKING=yes
HOSTNAME=server.example.com 

/etc/sysconfig/network-scripts/ifcfg-e* as provided:

DEVICE=eth0
HWADDR=01:02:03:04:05:06
TYPE=Ethernet
UUID=2ec2be26-d402-4712-80d9-440d6b224413
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=none
IPADDR=1.2.3.4
NETMASK=255.255.255.0
GATEWAY=1.2.3.254
DNS1=1.2.3.253
DNS2=1.2.3.252
IPV6INIT=yes
USERCTL=no 

/etc/sysconfig/network-scripts/ifcfg-e* as simplified and recommended:

DEVICE=eth0
ONBOOT=yes
BOOTPROTO=none
IPADDR=1.2.3.4
NETMASK=255.255.255.0
GATEWAY=1.2.3.254
IPV6INIT=yes 

You will also find versions of /etc/hosts, /etc/resolv.conf, /etc/sysconfig/network, and /etc/sysconfig/network-scripts-ifcfg-* in the directories /etc/sysconfig/networking/devices and /etc/sysconfig/networking/profiles/default. Stop and disable NetworkManager or you are likely to have mysterious intermittent problems when it "corrects" or "refreshes" your configuration.

RHEL 7

/etc/resolv.conf

search example.com subdom.example.com
nameserver 1.2.3.253 1.2.3.252 

/etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6 

/etc/hostname

server.example.com 

/etc/sysconfig/network-scripts/ifcfg-e* as provided:

TYPE="Ethernet"
BOOTPROTO="none"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="no"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_FAILURE_FATAL="no"
NAME="enp0s3"
UUID="2ec2be26-d402-4712-80d9-440d6b224413"
DEVICE="enp0s3"
ONBOOT="yes"
IPADDR="1.2.3.4"
PREFIX="24"
GATEWAY="1.2.3.254"
DNS1="1.2.3.253"
DNS2="1.2.3.252"
DOMAIN="example.com"
IPV6_PEERDNS="yes"
IPV6_PEERROUTES="yes"
IPV6_PRIVACY="no" 

/etc/sysconfig/network-scripts/ifcfg-e* as simplified and recommended, assuming you have also disabled NetworkManager:

DEVICE=enp0s3
ONBOOT=yes
BOOTPROTO=none
IPADDR=1.2.3.4
PREFIX=24
DEFROUTE=yes
GATEWAY=1.2.3.254
IPV6INIT=yes
IPV6_AUTOCONF=yes 

If you want a static IPv6 configuration for your Internet-facing IPv6 server, change the IPv6 details to something like the following. It will pick up IPv6 routing information from the router:

IPV6INIT=yes
IPV6_AUTOCONF=no
IPV6ADDR=2001:1800:1234:90::c8/64 

Keep in mind that an IPv6 address is laid out like the following, using my example address above for illustration. Remember that leading zeros can be deleted in each 16-bit block, and one string of consecutive zeros can be collapsed into "::", so the final "::c8" really represents 64 bits:

3 bits 45 bits 16 bits 64 bits
001 global routing prefix subnet ID        interface ID       
2001:1800:1234 0090 0000:0000:0000:00c8

Virtual Network Interfaces vs Multiple IP Addresses

Let's say you want to assign multiple IP addresses to one physical interface. This changes when you go from RHEL 6 to 7. The configuration is done in files in the directory /etc/sysconfig/network-scripts, the files are named ifcfg-interface. There are several changes to deal with in going to RHEL/CentOS 7. First, commands and interface names:

Second, in RHEL 5 and 6, you set up virtual interfaces. The physical interface was eth0 and the virtual interfaces were eth0:0, eth0:1, and so on. Create a separate file for each virtual interface. You saw separate stanzas for the physical interface and all the virtual interfaces in the ifconfig output, although only the physical one has the physical details. If a virtual interface is on a different logical IP network, there will be another entry in the routing table via the physical interface. See example output below.

Note: The RHEL 7 method was also possible in RHEL 6, but all the documentation I saw told us to create these other files and define virtual interfaces.

In RHEL 7, you assign additional addresses and netmasks to the interface. There is just enp0s3 (or whatever), it has multiple IP addresses. Add address/netmask entries to the one configuration file. Again, if one address/netmask is on another local IP network, there will be an added line in the routing table, see the example output below.

RHEL 5 / 6

Add extra files for the other interfaces, change the IP and the device name in each:

# more ifcfg-eth*
:::::::::::::
ifcfg-eth0
:::::::::::::
DEVICE=eth0
ONBOOT=yes
IPADDR=10.0.0.100
NETMASK=255.255.255.0

:::::::::::::
ifcfg-eth0:0
:::::::::::::
DEVICE=eth0:0
ONBOOT=yes
IPADDR=10.0.0.101
NETMASK=255.255.255.0

:::::::::::::
ifcfg-eth0:1
:::::::::::::
DEVICE=eth0:0
ONBOOT=yes
IPADDR=10.0.1.100
NETMASK=255.255.255.0

# /etc/init.d/network restart
# ifconfig
eth0:  Link encap:Ethernet  HWaddr 00:11:95:1E:8E:B6
	inet addr: 10.0.0.100 Bcast:10.0.0.255 Mask: 255.255.255.0
	inet6 addr: fe80::6e62:6dff:feb2:f841/64 Scope:Link
	UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
	RX packets:5695666 errors:0 dropped:0 overruns:0 frame:0
	TX packets:5269107 errors:0 dropped:0 overruns:0 carrier:0
	collisions:0 txqueuelen:1000 
	RX bytes:2828275570 (2.6 GiB)  TX bytes:2753415143 (2.5 GiB)

eth0:0  Link encap:Ethernet  HWaddr 00:11:95:1E:8E:B6
	inet addr: 10.0.0.101 Bcast:10.0.0.255 Mask: 255.255.255.0
	UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

eth0:1  Link encap:Ethernet  HWaddr 00:11:95:1E:8E:B6
	inet addr: 10.0.1.100 Bcast:10.0.0.255 Mask: 255.255.255.0
	UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

lo	Link ....

# route
Destination  Gateway     Genmask        Flags Metric Ref  Use Iface
default      10.0.0.254  0.0.0.0        UG    5      0      0 eth0
10.0.0.0     *           255.255.255.0  U     0      0      0 eth0
10.1.1.0     *           255.255.255.0  U     0      0      0 eth0

RHEL 7

One file per physical interface:

# more ifcfg-e*
:::::::::::::
ifcfg-enp0s3
:::::::::::::
DEVICE=enp0s3
ONBOOT=yes
IPADDR=10.0.0.100
PREFIX=24
IPADDR0=10.0.0.101
PREFIX0=24
IPADDR1=10.0.0.102
PREFIX1=24
IPADDR2=10.0.1.100
PREFIX2=24

# systemctl restart network
# ip addr
1: lo: ...
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> ...
    link/ether 00:0c:29:cf:62:79 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.100/24 brd.1.1.255 scope global enp0s3
       valid_lft forever preferred_lft forever
    inet 10.0.1.100/24 brd.1.1.255 scope global enp0s3
       valid_lft forever preferred_lft forever
    inet 10.0.0.101/24 brd.1.1.255 scope global secondary enp0s3
       valid_lft forever preferred_lft forever
    inet 10.0.0.102/24 brd.1.1.255 scope global secondary enp0s3
       valid_lft forever preferred_lft forever
    inet6 ...
# ip route
default via 10.1.1.254 dev enp0s3  proto static  metric 1024
10.0.0.0/24 dev enp0s3  proto kernel  scope link  src 10.0.0.100
10.0.1.0/24 dev enp0s3  proto kernel  scope link  src 10.0.1.100

If you run ifconfig you will only see the first address, but the system uses all of the configured IP/netmask addresses by responding to ICMP Echo Request and accepting TCP connections. Again, this is because the Net-tools package including ifconfig uses the older-style IOCTL calls, while the new Iproute2 package uses the kernel's Netlink interface.

Notice that the first one in each logical IP network is listed simply as of global scope and is the source in the routing table, while the following ones are listed as secondary.

What I configured as IPADDR and IPADDR2 were the first addresses on the 10.0.0.0/24 and 10.0.1.0/24 networks, respectively, while IPADDR1 and IPADDR2 were additional or secondary addresses also on 10.0.0.0/24.

Filtering with iptables / ip6tables

Recall that you got default packet filtering rules with iptables and ip6tables starting with RHEL 6. The RHEL 6 iptables default rules are:

Chain INPUT (policy ACCEPT)
target    prot opt source        destination         
ACCEPT    all  --  anywhere      anywhere      state RELATED,ESTABLISHED 
ACCEPT    icmp --  anywhere      anywhere            
ACCEPT    all  --  anywhere      anywhere            
ACCEPT    tcp  --  anywhere      anywhere      state NEW tcp dpt:ssh 
REJECT    all  --  anywhere      anywhere      reject-with icmp-host-prohibited 

Chain FORWARD (policy ACCEPT)
target    prot opt source        destination         
REJECT    all  --  anywhere      anywhere      reject-with icmp-host-prohibited 

Chain OUTPUT (policy ACCEPT)
target    prot opt source        destination          

The RHEL 6 ip6tables default rules are:

Chain INPUT (policy ACCEPT)
target    prot opt source        destination         
ACCEPT    all       anywhere     anywhere      state RELATED,ESTABLISHED 
ACCEPT    ipv6-icmp anywhere     anywhere            
ACCEPT    all       anywhere     anywhere            
ACCEPT    tcp       anywhere     anywhere      state NEW tcp dpt:ssh 
REJECT    all       anywhere     anywhere      reject-with icmp6-adm-prohibited 

Chain FORWARD (policy ACCEPT)
target    prot opt source        destination         
REJECT    all       anywhere     anywhere      reject-with icmp6-adm-prohibited 

Chain OUTPUT (policy ACCEPT)
target    prot opt source        destination          

With RHEL 7 the outputs of iptables -L and ip6tables -L are typically over 200 lines long. The rule sets are maintained by a daemon:

# systemctl status firewalld
firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled)
   Active: active (running) since Tue 20xx-xx-xx 18:54:04 UTC; 22min ago
 Main PID: 785 (firewalld)
   CGroup: /system.slice/firewalld.service
           +-/usr/bin/python /usr/sbin/firewalld --nofork --nopid

The daemon is a Python program that listens on a Unix domain socket. Verify that for yourself with this:

# lsof -p $( pgrep firewalld )
... 

When you start a service through systemd, it can contact the firewall daemon through that socket. "Hello, this is the SSH daemon sshd starting. Please let me accept connections on TCP/22."

Virtual Bridges

Red Hat was really pushing virtualization and especially containers by the time RHEL 7 came out, so a default installation would probably include the libvirt package and its dependencies even if you didn't explicitly ask for it. That leads to a virtual bridge interface virbr0 at 192.168.122.1/24.

The presence of virbr0 and the 192.168.122.0/24 network will lead to errors in the /var/log/firewalld log file.

You can remove all the related packages:

# rpm --erase $( rpm -qa | sort | grep libvirt )	\
	gnome-boxes virt-top virt-viewer virt-manager	\
	virt-manager-common virt-install libguestfs 

Do that, reboot, and there is no longer a virbr0 interface, route to the 192.168.122.0/24 network, or error messages appended to /var/log/firewalld.

Next: Network Services

Now that we have the IP interfaces configured, we can move on to the relatively simple task of controlling network services.

Next: Network Services