Linux servers.

Performance Tuning on Linux — Storage Hardware

Select Appropriate Hardware

Select the appropriate RAM and disk technology to have a chance of getting the system performance you want. Solid-state "disks" are much faster than rotating magnetic disks, but for most of us their cost is still much too high for the amount of long-term storage we want. We must have some magnetic disk storage, so we must choose the controller or interface carefully. Controllers have much higher potential throughput than rotating disks can fill on average, but there are still significant performance advantages to using faster interfaces.

Storage I/O Rates

RAM is much faster than rotating magnetic disks. DDR SDRAM memory is commonly used in servers, desktops, and notebook computers. Let's compare the numbers.

RAM I/O Speeds

DDR SDRAM = Double Data Rate Synchronous Dynamic Random-Access Memory

DDR memory uses "double pumping" to transfer data on both the leading and falling edges of the clock signal. The advantage is that the lower clock frequency reduces the requirements for signal purity in the circuitry connecting the memory to the controller. The disadvantage is the requirement for stricter control of the timing of clock and data signals.

The data transfer is 64 bits or 8 bytes wide, so DDR SDRAM yields a maximum transfer rate of:

DDR transfer rate = (memory bus clock) × (2 transfers/cycle) × (8 bytes/transfer)

The original DDR SDRAM, now called DDR1, has been followed by DDR2, DDR3, and DDR4. The later generations allow higher bus speeds and can make more transfers per internal clock cycle, so "DDR" is a misnomer — it's double for DDR and then doubled again for each successive generation.

DDR2 transfer rate = (memory bus clock) × (2 clock multiplier) × (2 transfers/cycle) × (8 bytes/transfer)

DDR3 transfer rate = (memory bus clock) × (4 clock multiplier) × (2 transfers/cycle) × (8 bytes/transfer)

DDR4 transfer rate = (memory bus clock) × (8 clock multiplier) × (2 transfers/cycle) × (8 bytes/transfer)

Name Memory clock
I/O bus clock
Peak transfer rate
DDR-200 100.0 100.0 1,600.0
DDR-266 133.3 133.3 2,133.3
DDR-333 166.7 166.7 2,666.7
DDR-400 200.0 200.0 3,200.0
DDR2-400 100.0 200.0 3,200.0
DDR2-533 133.3 266.7 4,266.7
DDR2-667 166.7 333.3 5,333.3
DDR2-800 200.0 400.0 6,400.0
DDR2-1066 266.7 533.3 8,533.3
DDR3-800 100.0 400.0 6400.0
DDR3-1066 133.3 533.3 8533.3
DDR3-1333 166.7 666.7 10,666.7
DDR3-1600 200.0 800.0 12,800.0
DDR3-1866 233.3 933.3 14,933.3
DDR3-2133 266.7 1066.7 17,066.7
DDR4-1600 200.0 800.0 12,800.0
DDR4-1866 233.3 933.3 14,933.3
DDR4-2133 266.7 1,066.7 17,066.7
DDR4-2400 300.0 1,200.0 19,200.0
DDR4-2666 333.3 1,333.3 21,333.3
DDR4-2933 366.7 1,466.7 23466.7
DDR4-3200 400.0 1,600.0 25,600.0

The different generations are physically incompatible, a motherboard with DDR3 memory slots can use DDR3 DIMMs only. The position of the notch varies with the DDR generation, see the two sides of the DDR3 modules below. The signal timing and voltages also differ between the types.

DDR3 SDRAM, 1333 MHz, 4 GB DIMMs

Disk I/O Rates

Note that this page is not about RAID! Hardware RAID arrays are their own complex topic which I'm not getting into.

The IDE (or, eventually, PATA) design dates from 1986. The first PATA disks only had 16 MB/sec peak transfer rates. That was doubled and later was further increased to as fast as 167 MB/sec. PATA use began to decline with the release of SATA in 2003. In December 2013 Western Digital was the last manufacturer to end production of general-purpose PATA disks.

Disk controllers and interfaces

The following table lists maximum theoretical speeds of various controllers and interfaces. The storage devices themselves will very likely not be able to utilize the full bandwidth, especially not under typical use. Mechanical disks have a maximum performance of about 180 MB/second. Solid-state drives can do about 225 MB/second in sequential reads and writes. However, controller and bus buffering provide several significant performance advantages to using connection technology with speeds higher than the maximum raw disk I/O.

Realize that a faster controller must be on an adequately fast bus. Interfaces like SATA 2.0 or USB 3.0 must be on a PCI Express (or PCIe) bus.

Interface Peak transfer rate
1.1 1.5
Fast 10
ATA 16.7
Ultra 20
UDMA 33 33
Ultra Wide 40
Ultra-2 40
2.0 60
UDMA 66 66.7
Ultra-2 Wide 80
UDMA 100 100
UDMA 133 133
1.0 150
Ultra-3 160
UDMA 167 167
SAS 300
2.0 300
Ultra-320 320
3.0 600
3.1 600
SAS 2 600
3.0 625
Ultra-640 640
SAS 3 1,200
3.1 1,250
3.2 1,969

As for SD and Micro SD cards, the class is the minimum guaranteed sustained write speed in megabytes per second. Class 10 SD cards can write at a maximum of 10 MB/second.

Disk performance measures

Rotating disks make us wait for two physical movements. First the heads must move in or out to the correct track. Then the disk must rotate the needed sector under the heads. Unless you are doing something very unusual like making an image for forensic analysis, disk I/O will be random because user processes make random use of the file system. That randomness means that the heads usually have to move.

Seek time is the time required for the disk to prepare the read/write apparatus. For solid-state drives, this is simply the time required to prepare the circuitry to access a specific address.

Rotating mechanical disks force us to wait for the actuator arm to move the head assembly to the circular track where the data is to be read or written. We must also wait for the heads to settle into place so they do not read or write "off track", but this settle time is typically less than 0.1 millisecond and is included in any measurement of seek time.

Technology Average seek time, milliseconds
Solid-state drive 0.08 — 0.16
High-performance disk for a server < 4
Typical desktop ~ 9
Laptop ~ 12

Rotational latency is the average time we must wait for the disk to rotate the needed sector under the read/write head. The faster the spindle turns, the shorter this is. The average rotational latency is a little over 7 milliseconds for a 4,200 rpm risk, dropping to 3 milliseconds at 10,000 rpm and 2 milliseconds at 15,000 rpm. Solid-state drives obviously have zero rotational latency.

Combining seek time and rotational latency, we're looking at averages of about 12 milliseconds on desktops and 5–6 milliseconds on servers, versus about 0.1 millisecond with solid-state drives.

People write graduate theses and other technical papers on the topic of modeling typical user file system access patterns. Jonathan Grier's paper Detecting Data Theft Using Stochastic Forensics describes how to detect data theft through unusual patterns of file access. Someone like Bradley Manning or Ed Snowden copies everything, but on typical systems about 90% of the files aren't used over the course of a year. In an interview Grier said "When you look at routine usage, you see a nice graph — a long-tailed pattern — where you use a few things, but most things gather dust. When you copy, you break that pattern. Because when you copy, you don't cherry-pick, you just get in and get out. And that has a uniform pattern, which is going to look unusual."

Disk performance trends

Disk transfer speed has not kept up with disk storage capacity. Disks have rapidly grown larger. Meanwhile the I/O speed has increased but it has not kept up with the growth in capacity. The result is storage devices that take longer and longer to fully read, and writing is even slower.

Looking closer at speed, seek time is not growing as fast as transfer rate.

Seek time is related to latency — how long it takes to start the transfer data.

Transfer speed is related to throughput — how much data per second once it starts moving.

Performance Versus Power Saving

Your disks may support Advanced Power Management, which could spin down a disk to save power and then require a wait for it to spin back up.

Possible settings are 1 through 254, lower being more power saving and higher being better performance (with 255 meaning "disable power management", but not understood by all hardware).

Similarly, -M queries and sets the Automatic Acoustic Management setting, with 254 the loudest and fastest and 128 and lower quiet and slow.

Check with the hdparm command and set it with similar syntax.

# for DISK in /dev/sd?
> do
>   hdparm -BM $DISK
> done

# for DISK in /dev/sd?
> do
>   hdparm -B 254 -M 254 $DISK
> done 

LVM is not RAID

You may accidentally get a performance boost from striping across physical volumes in your Logical Volume Management or LVM, but that is not the point. If you need RAID for performance or integrity, use real RAID.

The point of LVM is to make expansion easy, which is not needed when you can add more disks to a RAID array. And even without hardware RAID, the Btrfs file system will make LVM unnecessary.

And next...

Next we'll tune the kernel for I/O on our selected hardware.

To the Linux / Unix Page