Performance Tuning on Linux — Storage Hardware

Select Appropriate Hardware

Select the appropriate RAM and disk technology to have a chance of getting the system performance you want. Solid-state "disks" are much faster than rotating magnetic disks, but for most of us their cost is still much too high for the amount of long-term storage we want. We must have some magnetic disk storage, so we must choose the controller or interface carefully. Controllers have much higher potential throughput than rotating disks can fill on average, but there are still significant performance advantages to using faster interfaces.

Storage I/O Rates

RAM is much faster than rotating magnetic disks. DDR SDRAM memory is commonly used in servers, desktops, and notebook computers. Let's compare the numbers.

RAM I/O Speeds

DDR SDRAM = Double Data Rate Synchronous Dynamic Random-Access Memory

DDR memory uses "double pumping" to transfer data on both the leading and falling edges of the clock signal. The advantage is that the lower clock frequency reduces the requirements for signal purity in the circuitry connecting the memory to the controller. The disadvantage is the requirement for stricter control of the timing of clock and data signals.

The data transfer is 64 bits or 8 bytes wide, so DDR SDRAM yields a maximum transfer rate of:

DDR transfer rate = (memory bus clock) × (2 transfers/cycle) × (8 bytes/transfer)

The original DDR SDRAM, now called DDR1, has been followed by DDR2, DDR3, and DDR4. The later generations allow higher bus speeds and can make more transfers per internal clock cycle, so "DDR" is a misnomer — it's double for DDR and then doubled again for each successive generation.

DDR2 transfer rate = (memory bus clock) × (2 clock multiplier) × (2 transfers/cycle) × (8 bytes/transfer)

DDR3 transfer rate = (memory bus clock) × (4 clock multiplier) × (2 transfers/cycle) × (8 bytes/transfer)

DDR4 transfer rate = (memory bus clock) × (8 clock multiplier) × (2 transfers/cycle) × (8 bytes/transfer)

The different generations are physically incompatible, a motherboard with DDR3 memory slots can use DDR3 DIMMs only. The position of the notch varies with the DDR generation, see the two sides of the DDR3 modules below. The signal timing and voltages also differ between the types.

Disk I/O Rates

The IDE (or, eventually, PATA) design dates from 1986. The first PATA disks only had 16 MB/sec peak transfer rates. That was doubled and later was further increased to as fast as 167 MB/sec. PATA use began to decline with the release of SATA in 2003. In December 2013 Western Digital was the last manufacturer to end production of general-purpose PATA disks.

Disk controllers and interfaces

The following table lists maximum theoretical speeds of various controllers and interfaces. The storage devices themselves will very likely not be able to utilize the full bandwidth, especially not under typical use. Mechanical disks have a maximum performance of about 180 MB/second. Solid-state drives can do about 225 MB/second in sequential reads and writes. However, controller and bus buffering provide several significant performance advantages to using connection technology with speeds higher than the maximum raw disk I/O.

Realize that a faster controller must be on an adequately fast bus. Interfaces like SATA 2.0 or USB 3.0 must be on a PCI Express (or PCIe) bus.

As for SD and Micro SD cards, the class is the minimum guaranteed sustained write speed in megabytes per second. Class 10 SD cards can write at a maximum of 10 MB/second.

Disk performance measures

Rotating disks make us wait for two physical movements. First the heads must move in or out to the correct track. Then the disk must rotate the needed sector under the heads. Unless you are doing something very unusual like making an image for forensic analysis, disk I/O will be random because user processes make random use of the file system. That randomness means that the heads usually have to move.

Seek time is the time required for the disk to prepare the read/write apparatus. For solid-state drives, this is simply the time required to prepare the circuitry to access a specific address.

Rotating mechanical disks force us to wait for the actuator arm to move the head assembly to the circular track where the data is to be read or written. We must also wait for the heads to settle into place so they do not read or write "off track", but this settle time is typically less than 0.1 millisecond and is included in any measurement of seek time.

Rotational latency is the average time we must wait for the disk to rotate the needed sector under the read/write head. The faster the spindle turns, the shorter this is. The average rotational latency is a little over 7 milliseconds for a 4,200 rpm risk, dropping to 3 milliseconds at 10,000 rpm and 2 milliseconds at 15,000 rpm. Solid-state drives obviously have zero rotational latency.

Combining seek time and rotational latency, we're looking at averages of about 12 milliseconds on desktops and 5–6 milliseconds on servers, versus about 0.1 millisecond with solid-state drives.

Disk performance trends

Disk transfer speed has not kept up with disk storage capacity. Disks have rapidly grown larger. Meanwhile the I/O speed has increased but it has not kept up with the growth in capacity. The result is storage devices that take longer and longer to fully read, and writing is even slower.

Looking closer at speed, seek time is not growing as fast as transfer rate.

Seek time is related to latency — how long it takes to start the transfer data.

Transfer speed is related to throughput — how much data per second once it starts moving.

Performance Versus Power Saving

Your disks may support Advanced Power Management, which could spin down a disk to save power and then require a wait for it to spin back up.

Possible settings are 1 through 254, lower being more power saving and higher being better performance (with 255 meaning "disable power management", but not understood by all hardware).

Similarly, -M queries and sets the Automatic Acoustic Management setting, with 254 the loudest and fastest and 128 and lower quiet and slow.

Check with the hdparm command and set it with similar syntax.

# for DISK in /dev/sd?
> do
>   hdparm -BM $DISK
> done

# for DISK in /dev/sd?
> do
>   hdparm -B 254 -M 254 $DISK
> done 

LVM is not RAID

You may accidentally get a performance boost from striping across physical volumes in your Logical Volume Management or LVM, but that is not the point. If you need RAID for performance or integrity, use real RAID.

The point of LVM is to make expansion easy, which is not needed when you can add more disks to a RAID array. And even without hardware RAID, the Btrfs file system will make LVM unnecessary.

And next...

Next we'll tune the kernel for I/O on our selected hardware.