Linux server motherboard.

How to Build an RPM Package from a Hierarchy of Files and Directories

Turning /usr/local into an RPM Package

A consulting project had me needing to easily convert a /usr/local hierarchy into a package that could be copied to multiple systems and installed.

Better yet, it could be incorporated into an ISO image, a so-called "respin" based on an existing installation image. That could be coupled with a kickstart configuration, making media that installs itself including our newly customized /usr/local hierarchy. Programs in bin, system programs in sbin, associated data, manual pages, and other documentation in share, and so on.

Most of the individual files are shell scripts, many of them under active development. A typical day sees updates to multiple scripts and data files, and a fresh package needs to be created as soon as the new versions have been tested. So, I need an automated process to create the rpm.

Here's how I did it. I couldn't easily find a useful how-to, and while I enjoy figuring things out, I'm annoyed when I have to figure out the same thing a second time. So, like many of my open-source and cybersecurity pages, these are notes to myself for the next time I have to solve this problem. Hopefully you will also find it useful.

Project Goals

I was developing shell scripts used to automate some system administration — hardening the system and various reconfiguration tasks.

There would be a few large binary files included in the collection. One for an anti-malware product, another for a security scanner. They were zip archive files, containing binary RPM files. The software was there, but the installation was non-trivial. So, I had written some shell scripts to go through the multiple levels of extracting archives, running small utilities, renaming files, and so on.

How to create and use patch files for RPM packages

Nothing would be compiled during the process of assembling this package. This makes this project unlike the typical RPM package creation. See my other page for how to patch and compile software for building a binary package. My plan was to maintain a copy of what should be /usr/local on the target system. I would be modifying scripts and adding and modifying data. I would actually keep this copy under my home directory.

/home/cromwell/usr = root of the model /usr/local directory tree.

/home/cromwell/rpmbuild = small directory tree where I would build the RPM package.

I needed to write a shell script to automatically rebuild the RPM spec file and then use that to create a new package file. That script is below.

Getting Started

It's easy to set up an area for building RPM packages. 

$ cd
$ rpmdev-setuptree
$ tree -F rpmbuild/
rpmbuild/
|-- BUILD/
|-- RPMS/
|-- SOURCES/
|-- SPECS/
`-- SRPMS/

Then, I set up my /usr/local template under my home directory: 

$ cd
$ for d in bin sbin share src
> do
>   mkdir -pv ~/usr/local/$d
> done
mkdir: created directory '/home/cromwell/usr'
mkdir: created directory '/home/cromwell/usr/local'
mkdir: created directory '/home/cromwell/usr/local/bin'
mkdir: created directory '/home/cromwell/usr/local/sbin'
mkdir: created directory '/home/cromwell/usr/local/share'
mkdir: created directory '/home/cromwell/usr/local/src'
$ tree -F usr
/home/cromwell/usr
`-- local/
    |-- bin/
    |-- sbin/
    |-- share/
    `-- src/

Red Hat Enterprise Linux also includes etc, games, include, lib, lib64, and libexec in its /usr/local area. You can create as much or as little as you want. Nested subdirectories will work just fine.

Populate the File System

Create files and subdirectories. Because we will automate the package creation, you can add, delete, move, and rename things at will.

My script will preserve the permissions, so carefully set those up. Again, you can change them and rebuild.

My plan was to make everything owned by root, ownership and group are set file-by-file in the script. If you want the resulting files to be owned by a variety of owners, then you will need to use chown and chgrp on those files as root. If you them to have ownership and permissions such that you can't read them, things get more complicated. You will need to do the package creation as root, at least in the automated approach I used.

The Script

Here it is. The resulting package will be named usrlocal, that and other details should be fairly easy to find and adjust in this script. Greyed content needs to be carefully changed. 

#!/bin/bash

cd
echo "Creating /usr/local RPM"

############################################################################
# Everything in this tar file will go into the package.
# So, if you only want some pieces, replace the final
# "usr" with something more specific.  E.g.:
#   tar cf rpmbuild/SOURCES/usrlocal.tar usr/bin usr/sbin
############################################################################
echo "Creating rpmbuild/SOURCES/usrlocal.tar"
tar cf rpmbuild/SOURCES/usrlocal.tar usr

############################################################################
# Change this one definition if needed.
############################################################################
specFile=/home/cromwell/rpmbuild/SPECS/usrlocal.spec

############################################################################
# Begin the spec file with the preamble, %description, and %prep sections
############################################################################
echo "Generating $specFile"
cat > $specFile << EOF
Name:		usrlocal
# Because I anticipate frequent incremental updates, for now
# I will set the version to 1 and use "YYYYMMDD_hhmmss" as
# the release string.
Version:	1
Release:	$(date "+%Y%m%d_%H%M%S")
Summary:	Special project /usr/local hierarchy

Group:		System
License:	GPL
URL:		https://whatever.example.com/path/to/project
Source0:	/home/cromwell/factory/rpmbuild/SOURCES/usrlocal.tar

BuildRequires:	tar
Requires:	bash
Requires:	openssl
Requires:	pam_pkcs11

BuildRoot:	\${BUILDROOT}/usrlocal

%description

A collection of scripts and data to reconfigure Linux systems.

Commands for root: /usr/local/sbin/*

Commands for users: /usr/local/bin/*

Documentation:  /usr/local/share/documentation/*

Manual pages:  /usr/local/share/man/man*/*

%prep
%setup -n usr/local

# The collection of data is quite large, and it's already compressed.
# So, use these two directives to turn off the usual attempt to compress
# it further.  This uses gzip for compression, passing it -0 which means
# to simply store content as it is.  With a few hundred megabytes of data,
# this reduces package build time from several minutes to about 15-20 seconds.
%define _source_payload w0.gzdio
%define _binary_payload w0.gzdio

# Empty %build so rpmbuild won't try to run ./configure,
# set compiler flags, and compile something.
%build

# Now explain how to install the pieces.
%install
EOF

############################################################################
# Create all the directories, preserving original permissions.
############################################################################
for d in $( find usr -type d | sort )
do
	mode=$(stat -c "%a" $d)
	n="$(echo $d | sed 's@.*/factory/@@')"
	echo "mkdir -p \$RPM_BUILD_ROOT/$d" >> $specFile
	echo "chmod $mode \$RPM_BUILD_ROOT/$d" >> $specFile
	# If you want to also set the owner and group as they are
	# in your template, uncomment the following three lines.
	# userName=$(stat -c "%U" $d)
	# groupName=$(stat -c "%G" $d)
	# echo "chown $userName.$groupName \$RPM_BUILD_ROOT/$d" >> $specFile
done
echo "" >> $specFile

############################################################################
# Loop through the files adding "install -m ..." directives.
############################################################################
for f in $(find usr -type f | sort)
do
	mode=$(stat -c "%a" $f)
	echo "install -m $mode $f \$RPM_BUILD_ROOT/$f" >> $specFile
done
echo "" >> $specFile
echo "%files" >> $specFile

############################################################################
# Loop through the files again, setting permissions, and optionally,
# user and group.  Yes, we could have done the above loop like this
# and done this work there:
#    for f in $(find usr -type f | sort)
#    do
#	mode=$(stat -c "%a" $f)
#	u=$(stat -c "%U" $f)
#	g=$(stat -c "%G" $f)
#	echo "install -m $mode -o $u -g $g $f \$RPM_BUILD_ROOT/$f" >> $specFile
#    done
# However, that means that "install" will be trying to set the owner
# and group of the file, and only root can change the owner.  This
# second loop adds directives that only happen during the installation,
# which is done by root.
############################################################################
for f in $(find usr -type f | sort)
do
	mode=$(stat -c "%a" $f)
	# The following line will install every file to be
	# owned by user root, group root:
	echo "%attr($mode,root,root) /$f" >> $specFile
	# Comment out the above line, and uncomment the following three lines
	# to preserve owner and group as they were in the model hierarchy.
	# userName=$(stat -c "%U" $f)
	# groupName=$(stat -c "%G" $f)
	# echo "%attr($mode,$userName,$groupName) /$f" >> $specFile
done
cat >> $specFile << EOF

# We don't need the rest of these, included here for completeness.
# %doc

# %clean
# rm -rf \$RPM_BUILD_ROOT/usr

# %changelog

EOF

############################################################################
# Now use that spec file to create the package.  We just do "-bb"
# for a "binary build", meaning the executable (in our case, scripts
# and data), and not "-ba" for "build all" which would also create
# a SRPM file, containing the tar file and the spec file.
#  -- Remove "--quiet" for a lot of narration.
#  -- Remove "--clean" to leave working data in rpmbuild/BUILD.
############################################################################
echo "Creating an RPM package file."
cd rpmbuild/SPECS
rpmbuild --quiet --clean -bb usrlocal.spec
ls -l ../RPMS/x86_64

Putting the RPM into a Respin Image

Let's say you have downloaded an ISO image, mounted it, and made a copy that you can modify. Because of the way ISO and UDF file systems are built, you can't modify them. You can make a complete copy in a modifiable file system, like Btrfs or XFS, modify that, and create a new ISO or UDF file system. 

$ mount /path/to/distro.iso /media
$ cp -a /media dvd-image

Now you can copy your new RPM file into the appropriate place. 

$ cp rpmbuild/RPMS/x86_64/usrlocal-1-20240419_202928.rpm dvd-image/Packages/rhel-7-server-rpms/Packages/u/

And now you can recreate the metadata, so the resulting image can be used as a repository. This may take 30 to 60 seconds. Add the -v option for loads of narration. 

$ cd dvd-image
$ createrepo --simple-md-filenames -g repodata/*comps*xml .
Spawning worker 0 with 3888 pkgs
Workers Finished
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete

Now, to sign the repository.

Let's say that you created a PGP keyring as root, so the keyring is stored in /root/.gnupg.

Next you copied that to your usr/local archive, to somewhere like:
~/usr/local/share/package-signing.

If you copied it as root and made no change, then only root can read that directories and the files it contains. You will have to do the signing as root. And, if that is to be copied into the RPM file, you will have to build the RPM as root. That makes things more complicated yet, as rpmbuild will insist on looking for /root/rpmbuild until you make configuration changes to convince it otherwise.

It's easier if you copy it and then use chown to make you the owner, so you can read it. Then you will be able to create the signature within the new DVD image. But you will notice a warning message about insecure ownership and permissions on the signing key.

The choice is yours. When you're ready, the needed command sequence will be something like this: 

$ rm -f repodata/repomd.xml.asc
$ gpg --homedir ~/usr/local/share/package-signing \
	--detach-sign --armor repodata/repomd.xml
$ file repodata/repomd.xml*
repodata/repomd.xml:     XML 1.0 document, ASCII text
repodata/repomd.xml.asc: PGP signature

Other Linux and open-source topics