Lab machines are normally setup only to install "signed" packages. For RPMs, this is done by signing each package with a single key. For .debs, a file containing the checksum of the packages is signed, again with a single key. Currently the same key is used for both, e.g.

radyr:~: gpg < /usr/groups/linux/redhat/cl/.gnupg/28BB5403.asc 
pub  1024D/28BB5403 2011-06-13 Signer of RPM and .deb packages in the cl.cam.ac.uk repositories <unix-admin@cl.cam.ac.uk>
sub  2048g/1133F975 2011-06-13 [expires: 2012-06-12]

RPM systems use a package name of gpg-pubkey-$DSAkeyID-$date:

toton:~: rpm -qa  --nosignature --qf '%{VERSION}-%{RELEASE} %{SUMMARY}\n' gpg-pubkey\* | grep cl.cam
28bb5403-4df5a37a gpg(Signer of RPM and .deb packages in the cl.cam.ac.uk repositories <unix-admin@cl.cam.ac.uk>)
3294fb03-4def5605 gpg(Signer of RPM and .deb packages in the cl.cam.ac.uk repositories <unix-admin@cl.cam.ac.uk>)

debian systems use apt-key for management:

radyr:~: sudo apt-key list | grep -1 cl.cam
pub   2048R/3294FB03 2011-06-08 [expires: 2012-06-07]
uid                  Signer of RPM and .deb packages in the cl.cam.ac.uk repositories <unix-admin@cl.cam.ac.uk>
sub   2048R/ED9D5ADB 2011-06-08 [expires: 2012-06-07]
pub   1024D/28BB5403 2011-06-13 [expires: 2012-06-12]
uid                  Signer of RPM and .deb packages in the cl.cam.ac.uk repositories <unix-admin@cl.cam.ac.uk>
sub   2048g/1133F975 2011-06-13 [expires: 2012-06-12]

Below are commands needed when the key expired and a new one had to be generated.


/usr/groups/linux/redhat/cl/RPM-noarch/make-rpm uses ../.gnupg-rpm/ to sign RPMs, and /usr/groups/linux/redhat/cl/RPM-noarch/make-deb uses /usr/groups/linux/debian/repos/Makefile to call gpg with GPGHOME=../.gnupg-deb/ in /usr/groups/linux/redhat/cl/.gnupg/ to generate a new Release.gpg

The two pairs of files should normally be the same, the "current" one, although there may be a "not yet in use" key which is in the process of being distributed to machines, signed with the "current" one.

Smooth migration

Although both systems only allow a single signing key, both allow machines to accept multiple keys. Things are much smoother if the new one is installed before the old one expires. This would mean that "active" machines would make a smooth transition, but it would still be necessary to manually install the new key for any machines which did not pick up the package with the new key before the old one expired. The new package or digest should be signed with the old key. When there are two keys in the keyring, care is needed to ensure the correct one is used.

For RPMs, it is only necessary to sign the RPM containing the new key with the old key. Other RPMs can be signed with the new key as they will install the new key RPM and then be able to access later RPMs. As such, so long as the new CL-release RPM is created while the old signature is still the default, no action is required. A slight problem is that if an 'update' is done and some RPMs are not accepted as they have the new key, the new key will not be installed as the whole update fails. In such cases, just the new key should be updated, e.g.

cl-asuser yum update -y CL-release

For .debs, the key signs a digest of all the signatures, so the old key should be used until it expires or (nearly) all machines have taken the new key.

RPM signing uses the 'default' (oldest?) key in .gnupg/secring.gpp. debian uses .gnupg-deb/secring.gpg as specfied in /usr/groups/linux/debian/repos/Makefile.

The plan is to generate two keys per key lifetime to maximise the overlap and mimimize the need to HACK things, such as manually adding keys or resigning RPMs.


Key generation and migration is (as of Feb 2017) handled by the script /usr/groups/linux/redhat/cl/gpg/cycle-keys.sh which runs daily from cron on radyr.

This maintains a symbolic link to the current key used for signing RPM packages and the APT repository in /usr/groups/linux/redhat/cl/gpg/live.

If the live key will expire within the next 180 days, a new key will be generated and linked from /usr/groups/linux/redhat/cl/gpg/prepublish and included in CL-release and cl-debian packages. The new prepublished key will become trusted by hosts next time they update these packages.

If the live key will expire within the next 7 days, the prepublished key will become the live key.

In theory this process does not need our input, but it would be wise to check that the email generated whenever key generation or rollover happens does not indicate errors.

Manual Key Generation

/!\ The rest of this page has been largely superceded by the cycle-keys.sh script. The former manual process is described below for historical interest and in case the script malfunctions.

To generate a new key pair1 in pubring.gpg and secring.gpg run a command such as

cd /usr/groups/linux/redhat/cl&& rm .gnupg/{sec,pub}ring.gpg && sudo HOME=. gpg --gen-key --batch keygen.tmpl

which should 'just do it'.

Alternatively, omit the "--batch keygen.tmpl" and do it manually: select the default type ("DSA and Elgamal" rather than "RSA and RSA" which is OK for .deb, but may fail for RPMs) and size ("What keysize do you want? (2048)"). Set a period ("Key is valid for? (0) 1y") (may need a "y" to confirm the expiry date), and give the name ("Real name: Signer of RPM and .deb packages in the cl.cam.ac.uk repositories") and email address ("Email address: unix-admin@cl.cam.ac.uk"), but leave the comment blank ("Comment:"). It then needs a confirmation ("Change (N)ame, (C)omment, (E)-mail or (O)kay/(Q)uit? o") and a password. If it fails "gpg: no writable public keyring found: eof", "Key generation failed: eof", check ownership of the directory, run under sudo or some such.

gpg: key 73A7A982 marked as ultimately trusted
public and secret key created and signed.

gpg: checking the trustdb
gpg: public key of ultimately trusted key 69D17960 not found
gpg: public key of ultimately trusted key 28BB5403 not found
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   4  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 4u
gpg: next trustdb check due at 2014-09-16
pub   1024D/73A7A982 2014-02-26 [expires: 2015-02-26]
      Key fingerprint = C337 F93E 7FD3 A7AA ADC0  64DB EFB0 C355 73A7 A982
uid                  Signer of RPM and .deb packages in the cl.cam.ac.uk repositories <unix-admin@cl.cam.ac.uk>
sub   2048g/25568B55 2014-02-26 [expires: 2015-02-26]

Ensure the files are CO readable

sudo chmod g+r      .gnupg/{sec,pub}ring.gpg
sudo chgrp sysadmin .gnupg/{sec,pub}ring.gpg

Generate an ASCII armoured version for possible later use by clients

HOME=. gpg -a --export --comment "cl.cam.ac.uk package signing key"  > .gnupg/RPM-GPG-KEY-CL.asc

To keep things tidy, consider creating a directory EXP-$year-$month/ and put the dated copies of the key rings, ASCII armoured public key, etc in it (see existing EXP-*/ directories).

RPM package

To generate a new CL-release RPM

K=$(HOME=/usr/groups/linux/redhat/cl gpg --list-keys | sed -n 's/ 20..-..-.*//;s/^pub.*\///p'); echo K is $K 
cp -p /usr/groups/linux/redhat/cl/.gnupg/RPM-GPG-KEY-CL.asc /tmp/ && sudo cp -p /tmp/RPM-GPG-KEY-CL.asc $G-$K && sudo ln -f $G-$K $G 
cl-asuser restorecon -v $G-$K $G 
vi /usr/groups/linux/redhat/cl/RPM-noarch/.rel.CL-release # to increase the release
KEYID=-$K /usr/groups/linux/redhat/cl/RPM-noarch/make-rpm release

.deb package

To generate a new cl-debian package (note that make-rpm and make-deb are normally run on a RPM and .deb based machine respectively)

K=$(HOME=/usr/groups/linux/redhat/cl gpg --list-keys | sed -n 's/ 20..-..-.*//;s/^pub.*\///p'); echo K is $K
cp -p /usr/groups/linux/redhat/cl/.gnupg/RPM-GPG-KEY-CL.asc /tmp && sudo cp -p /tmp/RPM-GPG-KEY-CL.asc $G-$K && sudo ln -f $G-$K $G
vi /usr/groups/linux/redhat/cl/RPM-noarch/.rel.CL-debian # to increase the release
vi /usr/groups/linux/redhat/cl/RPM-noarch/CL-debian.spec # add to %{oldkeys} any keys to be tidied
KEYID=-$K docreate=n /usr/groups/linux/redhat/cl/RPM-noarch/make-rpm debian # may fail on a Ubuntu machine. Works on CentOS
/usr/groups/linux/redhat/cl/RPM-noarch/make-deb debian

Key roll over

When the 'new' key (probably around six months old) is to become the 'current' key, delete the old 'current' key (used by RPM), install the 'new' key for .deb signing, and make it easy to install manually the 'new' key for systems which missed a 'routine' update.

Delete old key

Once the old key 99DC7507 is no longer needed, (having kept a safety copy) delete it from the key rings to avoid using it by mistake:

tar czf $D/.gnupg-$(date +%Y-%m-%d).tgz $D/.gnupg/; chmod 0 $D/.gnupg-$(date +%Y-%m-%d).tgz
HOME=$D gpg --list-keys; HOME=$D gpg --list-secret-keys
HOME=$D gpg --delete-secret-keys $OLDKEY; HOME=$D gpg --delete-keys $OLDKEY

Installing (RPM)

To allow machines easily to install the key, publish it on the web server

cp -p /usr/groups/linux/redhat/cl/.gnupg/$K.asc /anfs/www/html/RPM/RPM-GPG-KEY-CL
ln -svf RPM-GPG-KEY-CL /anfs/www/html/RPM/KEY

allowing a machine which has not managed a smooth transition to use the command:

sudo rpm --import http://www.cl.cam.ac.uk/RPM/RPM-GPG-KEY-CL

An alternative is to run "yum update" as usual, fetching the new RPM. This can then be installed using a command such as

sudo rpm -Uvh --nosignature /var/cache/yum/CL/packages/CL-release*.noarch.rpm

Also check which are in the RPM system and delete any unneeded ones

rpm -qa --nodigest --nosignature --qf '%{VERSION}-%{RELEASE} %{SUMMARY}\n' gpg-pubkey\*
cl-asuser rpm -e --allmatches gpg-pubkey-$K

Installing (.deb)

Debian packages use /usr/groups/linux/redhat/cl/.gnupg-deb/.

Update the .deb signing key and arrange that new installs get the new key

V=14 OLD=$$; cd /usr/groups/linux/redhat/cl/.gnupg-deb/ 
for t in *.gpg; mv $t $t-$OLD; done; cp -pv ../gnupg/*.gpg .
cp -pv ../.gnupg/RPM-GPG-KEY-CL.asc /anfs/www/html/deb/cl.cam.ac.uk-archive-keyring.gpg
cp -pv /usr/groups/linux/debian/repos/dists/CL/all/binary-i386/cl-debian_1-${V}_all.deb /anfs/www/html/deb/cl.deb

allowing a machine which has not managed a smooth transition to use commands such as:

sudo apt-key add /anfs/www/html/deb/KEY
wget -q -O - http://www.cl.cam.ac.uk/deb/cl.cam.ac.uk-archive-keyring.gpg | sudo apt-key add -
wget -qO- www/deb/KEY|sudo apt-key add -
d=~/tmp/0BF79378;mkdir -p $d;for host in $(ruptime|grep -w up|while read a b;do find /var/spool/rwho/whod.$a* -mmin -30|sed 's/.*whod.//';done);do test ! -e $d/$host&&echo -n Try $d/$host ...\ &&sudo ssh $host "! type apt-key>/dev/null||apt-key list|grep 0BF79378||wget -qO- www/deb/KEY|apt-key add -"&& touch $d/$host;done

Alternatively copy the key file (e.g.) to /tmp/newkey and run

sudo apt-key add /tmp/RPM-GPG-KEY-CL

It should also be possible to tell the system to ignore signatures, but this might also add other untrusted packages

sudo apt-get upgrade --force-yes -y

Check it worked using "sudo apt-key list".

Resigning RPMs

Whereas debian signs the checksums, RPMs include the signature. As such, if any machines need to install packages signed with the old key, the package needs to be resigned using the new key

cd /usr/groups/linux/redhat/relics && ./resign shared/$RPM


  • 1 Key generation may need xprop and pinentry-gui to be installed, and ./.Xauthority to point to $HOME

SysInfo/PackageKey (last edited 2017-02-03 16:04:14 by MalcolmScott)