Linux / UNIX Command Fundamentals and the File System

Linux / UNIX Commands

So you want to start using the powerful Linux / UNIX command-line interface, but you don't know where to start. Here's a suggested set of things to know how to do. But first, just a quick note on nomenclature to avoid unneeded confusion — UNIX is a general family of operating systems of which Linux is just one example. Learn the commands once, and you know them everywhere. Everything on this page should be true for any vaguely UNIX-like operating system. This includes:

• Commercial examples like Solaris, HP-UX, IRIX, AIX, Tru64, and so on
• Any distribution of Linux
• All other free UNIX-family operating systems like FreeBSD, OpenBSD, and NetBSD.
• Apple OS X

Your Goal

Here is a quick inventory of what you need to learn in order to be a reasonably capable Linux command-line user. For the most part, these are lists of commands. You need to understand how to run them with appropriate parameters and options, and how to interpret their output. All these and more are explained further down this page.

Navigate the file system:   cd,   pwd

Understand fundamental directories:   .,   ..,   ~,   ~joe

Manipulate directories:   mkdir,   rmdir

Understand information about files and directories:   ls,   ls -a,   ls -l
In the ls -l output, be able to interpret the permission mask. Also notice the owner, group, size in bytes, and modification timestamp, although those are very obvious.

Use file name wildcards:   *,   ?,   [abc],   [a-f]

Copy, move (or rename), and remove files:   cp,   mv,   rm

Read file contents:   cat,   more,   less

Compare file contents:   diff,   cmp

Redirect input and output streams:   >,   <,   >>,   |

Extract lines containing specified strings from files or output streams:   grep

Change file permissions:   chmod

Find files based on name or other attributes:   find

Edit text files: You can start with the very simple nano or pico, but eventually you need to learn vi, or really its replacement, the much improved vim.

Read the manual pages:   man

There's much more to learn, but the above list is a good start. Someone can't honestly claim to be a Linux/Unix command-prompt user without knowing all of the above,

If I Had To Interview People

Let's say there's a job posting for a position where you will be writing software in a Linux environment. You will be collaborating with a group of people on the project. Here are some things I would expect a new hire to be able to do or answer with no problem:

1: What is the list of all directories on the system with "log" anywhere in their name? Since you don't know the root password, and the needed search will create a large number of unavoidable error messages, throw away the error output.

2: Same thing, except either "log" or "Log".

3: In the directory /var/log, what are the last 20 lines in the most recently modified file? Half credit if this takes two commands and human intervention.

4: In the file /var/log/messages, how many times does the string apache appear?

5: In the file /var/log/messages, display the last line containing the string apache.

6: Your file ~/myfile is owned by you and group coders. Leaving all other existing permissions alone, change permissions so that others (not you, not group coders) have none of read, write, or execute permission.

7: Your file ~/otherfile is owned by you and group coders. Set its permissions so that:

• You can read, write, and execute it.
• Group coders members can read and execute it, but they cannot write it.
• Everyone else can read it, but not write or execute it.

8: You are suddenly worried about insider threats. Change permissions to remove all access for all users other than you for your home directory and all of its contents. You have thousands of files.

9: Without modifying its content, change the modification timestamp of the file current-file in your home directory to be right now.

10: The file /tmp/list contains lines that look like this:
.M......C    /boot
SM5....T.    /boot/grub.conf
.M.....T.    /etc/passwd
S.5....T.    /var/lib/unbound/root.key
.....UG..    /var/run/avahi-daemon
.......T.    /boot/kernel-5.4.42-v7.1.el7.img
You want to extract just those 9-character patterns at the beginning of each line. That is, the output should look like this, with just the first 9 characters from each line:
.M......C
SM5....T.
.M.....T.
S.5....T.
.....UG..
.......T.

11: Run commandA and save its output in /tmp/output, and then run commandB and append its output to that same file. Don't destroy the output of commandA!

12: The file /etc/passwd contains one line per user, with fields specifying various attributes. One field is their preferred command shell. Extract the list of users who use the /bin/bash shell, sorted into alphabetical order.

13: What is the list of all files owned by the user julie named *.pptx?

14: Run the command complex and save all of its output, both standard output and standard error, in the file /tmp/all-output.

15: How do you save your changes and exit your vi (or, better yet, vim) session?

16: How do you abort your vi (or vim) session, exiting without saving changes?

Answers

1:
$ find / -name '*log*' 2> /dev/null

2:
$ find / -name '*[Ll]og*' 2> /dev/null

3:
$ tail -20 $(ls -t /var/log/* | head -1)
or
$ tail -20 $(ls -tr /var/log/* | tail -1)

Or run:
$ ls -t /var/log | head
or
$ ls -tr /var/log | tail
and examine the output to find the newest file. Then use tail on that file.

4:
$ grep -c /var/log/messages
or, less elegantly,
$ grep /var/log/messages | wc -l

5:
$ grep apache /var/log/messages | tail -1

6:
$ chmod o-rwx ~/myfile

7:
$ chmod 754 ~/otherfile

8:
$ chmod -R go-rwx ~

9:
$ touch ~/current-file

10:
$ cut -c1-9 /tmp/list
That does what the question asks, the first nine characters from each line. If the entire file looks like the example, you could get the same output from either of the following. Personally, casual use of awk or regular expressions would impress me more than fussing about the distinction between "first 9 characters" and "first white-space-delimited token".
$ awk '{print $1}' /tmp/list
or
$ sed 's/ .*//' /tmp/list

11:
$ commandA > /tmp/output
$ commandB >> /tmp/output

12:
$ grep /bin/bash /etc/passwd | sort
Extra credit for just the user names themselves:
$ grep /bin/bash /etc/passwd | sed 's/:.*//' | sort
or
$ grep /bin/bash /etc/passwd | awk -F: '{print $1}' | sort
Hire them immediately if they suggest:
$ awk -F: '/\/bin\/bash/ {print $1}' /etc/passwd | sort

13:
$ find / -user julie -name '*.pptx' 2> /dev/null

14:
$ complex > /tmp/all-output 2>&1

15:
Press <Escape> and then
ZZ
or
:wq
or
:w
:q

16:
Press <Escape> and then
:q!

Getting Started and Learning More

This page is just a starting point, it mentions some of the most fundamental commands and provides examples of some simple and commonly needed tasks. The examples are just a glimpse of what they can accomplish, some of the commands have so many options that the overwhelming variety of possible command-line parameters could be considered a bug, as the command may be too complex to fully memorize.

This isn't really a problem! UNIX is designed around tools which each do one very specific thing well. Learn the main purposes of some core commands, and remember that you can always refresh your memory or learn more about options with the on-line manual pages.

The O'Reilly book Unix in a Nutshell is a good introductory text. Yes, there is also a Linux in a Nutshell title, but it uses a lot of its content showing you how to use fairly intuitive graphical interfaces. Unix in a Nutshell is much better for learning the command-line environment and powerful tools.

Another good introduction is The Linux Command Line: A Complete Introduction, which is available as a free download here.

Let's say that you want to remove a file but you you do not remember the precise command. You can ask for a list of commands related to the topic. Ask for those one-line summaries of commands containing the string remove, where -k specifies by keyword:

$ man -k remove 

You will see a lot of output, including commands to remove directories, to remove users, and arcane things for system administrators such as removing logical volume groups and named semaphores. But as you look down the alphabetical list of output you will see the line I have highlighted here:

[...  lots of output  ...]
removef              (1)  - remove a file from software database
remque [insque]      (3)  - insert/remove an item from a queue
rm                   (1)  - remove files or directories
rmdir                (1)  - remove empty directories
rmvq                 (8)  - remove a message from a queue
[... lots more output ...] 

So you see that you probably need the command rm but you want to read a little about it. You need to verify that it's really the correct command and to see how to use it and possibly how to specify some optional behavior. Ask to see its detailed manual page:

$ man rm 

Let's get started! Skilled Windows users will see that many features of the Windows command line interface were modeled directly on the UNIX command line interface — some things work exactly the same, and others are analogies.

File Name Wildcards

This is probably easiest to understand as a list of examples. You would use these literally on the command line, typing something like:
$ ls blah.*
or:
$ cp blah.[gjp]* backup-directory
You will see examples of these types of uses in later sections of this page.

* matches any string, including nothing at all.

foo* matches foo, foot, and football.

UNIX is stricter than Windows about matching. A Windows user would expect this command to remove all the files:
$ rm *.*
However, it does what you ask. It only removes files with a "." in their name.
$ ls
Index.html image.jpg logo.gif notes
$ rm *.*
$ ls
notes
The file notes has no "." in its name, so it was not removed.

? matches any single character.

foo? matches food and foot, but not football.

[abc...] matches one of any character in the list.

foo[ldt] matches fool, food, and foot.

You can specify ranges. foo[a-c] matches fooa, foob, and fooc.

foo[a-cX-Z] matches fooa, foob, fooc, fooX, fooY, and fooZ.

foo[a-cmx-z] matches fooa, foob, fooc, foom, foox, fooy, and fooz.

Be careful! foo[d,t]ball matches foodball and football, but it also matches foo,ball because that comma is treated as just another character.

[^...] matches any single character not in the list.

foo[^ldt] matches everything foo? would match except for fool, food, and foot.

Brace expansion is a powerful shortcut that can compress a series of similar commands into just one. Instead of typing this:
$ mkdir /home/cromwell/project/subdir1
$ mkdir /home/cromwell/project/subdir2
$ mkdir /home/cromwell/project/subdir3
you could simply type this:
$ mkdir /home/cromwell/project/{subdir1,subdir2,subdir3}
or, even shorter:
$ mkdir /home/cromwell/project/subdir{1,2,3}

Very Fundamental Directories: . and .. and ~

The directory named ".", that is, just one dot, is your current working directory. What is it really? It's wherever you are at the moment. As Buckaroo Banzai said, "No matter where you go, there you are."

Related to that, the directory named ".." is the parent of your current working directory. If your current working directory is /home/cromwell then:
.  is /home/cromwell and
.. is /home.

There is just one exception — when you are at the root of the file system there is no parent. That is, when your current working directory is / then you can't go up to its parent.

Actually, there is a directory named ".." in the root of the file system, but it points to the same location as it has the same i-node. If you don't believe me, verify this for yourself:
$ ls -ldi /.*

The special directory name ~ refers to your home directory, while ~julie would be the home directory of the user julie. Home directories are specified in the file /etc/passwd, but you can simply use ~ and the shell automatically looks it up.

Running Commands and Your PATH

You do not need to be in any particular directory to run a program, so long as either of the following is true.

1:   The program is in a directory listed in your PATH environment variable, or

2:   You specify the full path to the program. This could be an absolute path:
$ /usr/bin/vim
or a relative path. In this example, mydir is a subdirectory of your current working directory and it contains a program named myprogram:
$ mydir/myprogram
That relative path could start with ".", to mean "Run the program that's right here, not a program by that name installed in a standard system binary area":
$ ./myprogram

You can see your current PATH environment variable setting with the following command. You will see that the components are separated by colons:

$ echo $PATH 

When you just type a command, the shell scans through the elements of your PATH looking for the first match. The first match wins — when the shell finds an executable program by that name in a PATH component, it runs that program. If no match is found, an error is reported. The precise error depends on your command shell, but it will be something like:
foo: Command not found

A reasonable PATH value for users includes the following directories. Note that X might be installed under X11R6 as shown here, or just X11, or even just X. You will have to investigate and think just a little:
/bin
/usr/bin
/usr/X11R6/bin
/usr/local/bin

A reasonable PATH value for root includes the above directories and adds the following:
/sbin
/usr/sbin
/usr/local/sbin

Notice that "." should never be in the path for root as that would allow a malicious user to lay a trap. That means that root needs to specify the path to a command in the local directory, for example, ./programname.

You can run anything from anywhere if you provide the details. For example, let's say that your Apache web server saves its logs in a bunch of files in the directory /var/www/logs and you want to process them with some program named /var/www/tools/log-analyzer, a program that reads the files listed as its command-line parameters. You want to run that analysis and save the output in a file named result in your home directory. All you have to do is this:

$ /var/www/tools/log-analyzer /var/www/logs/* > ~/result

Redirecting I/O — Writing To and Reading From Arbitrary Files

Again, this is probably easiest to understand by example, once you realize that any process has at least three standard I/O streams: input, output, and error. They are sometimes called stdin, stdout, and stderr, especially by C/C++ programmers.

The standard output stream is general output created by the process — its contents may be the main intended result of the program. The standard error stream is available for a second parallel stream of output intended for use in reporting errors. As you will see below, they can be sent to the same place, which could be the terminal screen where you started the program, or to a file, or into another program as its standard input. Or, the two output streams could be sent to different places. It's up to you, Unix gives you fine control!

Editing Text Files With vi

Now you're ready to run some programs! But those programs will need data, and the programs may be simple shell scripts that you create. This means that you need to edit simple text files.

This means that you really need to learn some standard Unix text editor, either vi or emacs. Sorry, that's just the way that it is.

Note that vi may really be vim, which stands for "Vi Improved", on your system, or else both vi and vim may be available. If you use a variety of Unix systems, you might want to get into the habit of at least attempting to run vim and falling back to the slightly less friendly vi if necessary.

You can find a good on-line tutorial of vi at YoLinux.com, it goes deeper than this page does. Wikibooks has a book on-line, "Learning the vi editor". Other useful pages include vim.org and thomer.com.

You may also have the vimtutor program installed on your system, or you may be able to add it.

Crucial things to know about vi (and vim) include:

It has two modes: command and text insertion. When you're in text insertion mode, pressing keys just inserts those characters into the file at the current location of the cursor. When you're in command mode, pressing keys causes commands to happen.

Be careful, as always it is a bad idea to execute random commands. Typing away when you think you're inserting text but those characters are being interpreted as commands can have strange results at best, disastrous at worse.

To go from text insertion mode to command mode, just press the <Escape> key.

Now, when you're in command mode, there are many commands, but these are the vital ones plus a few more especially useful ones. The vital ones are those with green backgrounds. The good news is that you only need to know a handful of commands to be fairly useful. You will want to learn more so you can use the editor much more efficiently:

Listing Files and Directories

The Swiss Army Chainsaw is the ls command. It has many options, to the point that the BUGS section of the GNU manual page has at times referred to the large number of options and the inability of users to accurately remember and use most of them as a bug. It would properly be a design flaw, but there's a point in there somewhere.

Useful applications of ls include:

Now, to read that long listing:

$ ls -lFd [Dp]* /tmp/file1OjOkm /dev/ttyS0 /dev/sda1 /usr/bin/passwd /usr/bin/man
brw-rw----.  1 root     disk      8,  1 Apr 04 20:43 /dev/sda1
crw-rw----.  1 cromwell uucp      4, 64 Apr 04 20:43 /dev/ttyS0
srwxrwxr-x.  1 cromwell wheel         0 Apr 06 11:41 /tmp/file1OjOkm=
-rwxr-sr-x.  1 root     mail      47360 Dec 14 17:55 /usr/bin/mail*
-r-s--x--x.  1 root     shadow    22104 Dec 14 13:56 /usr/bin/passwd*
drwxr-x---. 12 cromwell wheel    126976 Apr 07 14:14 Documents/
lrwxrwxrwx.  1 cromwell cromwell     18 Jan 28 14:40 pete@ -> purdue.jpeg
-rw-r--r--.  1 cromwell wheel    314943 Jan 22 12:16 purdue.jpeg

^\________/  ^    ^          ^      ^   \__________/     ^
|    |       |    |          |      |        |           |
|    |       |    |          |      |        |       File name
|    |       |  User       Group    |    Time stamp
|    |       |                      |
|    |       |                   Size in bytes for regular files.
|    |       |                   Major,minor device numbers for
|    |       |                   device special files.
|    |       |
|    |   Number of links pointing to this in the file system.  "Documents" is
|    |   a directory with 11 subdirectories.  There is one link in the current
|    |   directory pointing to it by the name "Documents" plus one more link
|    |   in each of those 11 subdirectories pointing to it by the name "..".
|    |
| Permission mask:  1st three for the user r = read
|                   2nd three for group    w = write
|                   3nd three for others   x = execute for files
|                                              search for directories
|                                          s = setuid/setgid, set user ID and
|                                              set group ID
|                                          - = permission not granted
| If there is a final "." on the permission mask, as seen here, it means that
| additional NSA Security-Enhanced Linux ACL (Access Control List) restrictions
| apply.  See the SELinux ACLs with:  ls --context
| Similarly, on BSD and Solaris a "@" or "+" character indicates extended ACLs.
| See them with:  getfacl
|
|  So, "Documents" has permissions rwxr-x---.  cromwell  wheel
|    Subject to unexpected restrictions by the extended SELinux ACLs:
|    Owner cromwell can read, write and execute (search)
|    Group wheel members can read and execute (search)
|    Others have no permissions
|
|  "/usr/bin/passwd" has permissions r-s--x--x.  root  shadow:
|    Numeric mode 4511.
|    Subject to unexpected restrictions by the extended SELinux ACLs:
|    Anyone can execute it, but that one process will have the effective user
|    ID of "root" so it can read and write /etc/shadow.
|
|  "/usr/bin/mail" has permissions rwxr-sr-x.  root  mail
|    Numeric mode 2555.
|    Subject to unexpected restrictions by the extended SELinux ACLs:
|    Anyone can execute it, but that one process will have the effective group
|    ID of "mail" so it can manipulate the directory /var/spool/mail.
|
File type:  -  =  regular file
            d  =  directory
            l  =  symbolic link
            s  =  socket
            b  =  block (buffered) special device
            c  =  character (raw) special device

Manipulating Files and Directories

Examining File Contents

Changing File Permissions

Permissions can be specified in octal. Yes, octal, base 8. It will eventually become second nature, but to the uninitiated this is like Cypher in The Matrix when he waves his hand at the screens of cascading code and says he no longer sees the character patterns but instead what they represent. "All I see now is blonde, brunette, redhead." Only six patterns are really useful:

  rwx  ->  111  =  7
  rw-  ->  110  =  6
  r-x  ->  101  =  5
  r--  ->  100  =  4
  --x  ->  001  =  1
  ---  ->  000  =  0

Once you learn this, you will see rwxr-x--- and read it as 750. Really. And you will immediately understand it as "Full access for the owner, all but writing for the group, nothing else."

Permissions can be specified explicitly in octal, as above, or symbolically:

                [ugo][+-=][rwxs]
                  ^    ^    ^
                  |    |    |
  +---------------+    |    +-------+
  |                    |            |
  |                    |            |
u = user (owner)     + = add      r = read
g = group            - = remove   w = write
o = other (world)    = = equal    x = execute (or search, for directories)
                                  X = search (only if it's a directory)
                                  s = set-UID or set-GID

So, with all that background we're ready for examples:

Who Am I and Where Am I?

Command History

Identifying and Controlling Processes

Network Commands

For far more details and explanations of how to read some sample output, see my page with network commands for various operating systems. Here is a short version. Some of these will require you to have /sbin and maybe /usr/sbin in your PATH. Why? Because with a lot of other commands in those directories, they are administrator and power user tasks but not the sort of the thing the average user does. See the above section on PATH for details on this.

Controlling System State

For Linux with systemd, meaning most any Linux distribution after 2013–2014:

For UNIX family in general and Linux before systemd:

Managing Users and Groups

All these commands must be run as root:

Here are the files defining users:

Changing File Ownership and Group

With one exception, chgrp, all these commands must be run as root:

Finding Files and Directories

Most people seem to find the find command the most confusing, with a manual page that may not be terribly helpful. Here is an attempt to present some useful examples.

Do any users own a file containing the string "mystery", ignoring case?

find /home -type f -exec grep -i mystery {} /dev/null \;

Notice the bizarre syntax of find -exec — the command you want to execute for every file matching the search criteria (in this case, under /home and a regular file) must appear between -exec and the delimiter formed by an escaped semicolon. The command will be run once for every file found, with {} replaced by the name of that file.

The example here always passes a second file name argument, /dev/null. The grep command can read that empty file in no time at all, but the presence of a second file name means that any output will be prefaced with the file name where the string was found. Compare the output of the two grep commands in this sequence:

echo foo > /tmp/bar
$ grep foo /tmp/bar
foo
$ grep foo /tmp/bar /dev/null
/tmp/bar:foo 

There is usually a way to tell your grep command to include that file name even with a single file name on the command line, but the precise way to do that varies from one implementation to the other, and it isn't always possible at all.

Where The Pieces Go

This is hard to predict in detail, but the following table provides generally useful guidance.

See my page on file system design for performance and security for more detail, and suggestions of how to design a partitioning scheme.