A caching scheme to reduce downloads for Linux package updates

As I administer sites with a lot of (nearly) identical Linux hosts, I thought a lot about the issue of reducing the download for package updates. Even if you have unlimited data quota for your site, downloading the updates multiple times for multiple machines takes time.

I looked at various schemes suggested on the Internet. One is to set up a satellite repository. This is reliable but involves a lot of configuration. Also you may download a lot of package updates that are not used at your site.

Another class of scheme involves interposing a proxy. It could be a specialised proxy for the distro, or a general proxy like squid. This can work well but is sometimes defeated by load balancers or smart mirrors where you are not guaranteed to hit the same repository every download, as the actual repository used could vary.

Eventually I came up with a scheme that is applicable to three distros that I have tried it on: CentOS, Debian (and derivatives) and openSUSE. The key observation is that fact that the updaters in these distros have a package cache. If packages are added to the cache on the side, the updater will skip downloading them. I describe this as a caching scheme, not a proxying scheme as no change is made to the updater configuration files and normal, independent update works.

The scheme:

There are three parts to the scheme.

1. A download script runs on a designated master host. This runs during off-peak hours. We use the updater program in download but not install mode so we will only download packages we need. It's also necessary to set the repository to retain packages.

2. A synchronisation script runs on the other hosts to pull the new updates to their cache. This is run as soon as possible after the download.

3. Various utility scripts are used to fire off updates on all hosts. I make use of parallel ssh and ssh-agent to make life easy.

Let's use a concrete example to illustrate this.

CentOS:

1. Download:

The download script is this:

yum --disableexcludes=updates --downloadonly -y update

You need to install the auxiliary package yum-downloadonly. The --disableexcludes=updates is to disable the regular expressions that prevent the installation of the kernel on this host. On this host I disabled kernel updates due to the need to reboot and recompile VirtualBox modules whenever the kernel is updated, and as this is a server that engineers use constantly, I delay kernel updates until downtime can be scheduled to do them manually. Other software that may need recompilation on kernel update include proprietary video drivers.

To set the repository to retain packages, add

keepcache=1

to /etc/yum.conf

2. Synchronisation:

Next I set up a rsync server. Initially I used rsync over ssh, but this requires setting up a login account, restricted commands, etc. for security reasons. Package updates are not sensitive data so I just share the cache directory with rsyncd. Please consult your distro documentation on how to enable the rsyncd daemon. Usually it's run out of xinetd. Here is /etc/rsyncd.conf:

secrets file = /etc/rsyncd.secrets
motd file = /etc/rsyncd.motd
read only = yes
list = yes
uid = nobody
gid = nobody

[yum]
comment = YUM cache
path = /var/cache/yum
filter = + *.rpm + */ - *

Note the filter expression to share RPM files and directories but not files of other types.

The synchronisation script, call it syncfrommaster, on the other hosts is:

rsync -avi masterhost::yum /var/cache/yum

In the cron job that runs this command I insert a random delay so that the hosts don't all try to synchronise at the same time and overload the master.

sleep $(($RANDOM \% 900)); /root/bin/syncfrommaster

The % is escaped with \ in cron scripts by the way.

I arrange for this job to be run an hour after the download script.

3. Update.

You have probably noticed that it is possible that an update will be published in the time between the master pulling it down and you applying the updates. In this case each host merely downloads the package on its own, and nothing breaks. However you may wish to run a script to cause the cache to be updated and the hosts to catch up.

#!/bin/sh
yum --disableexcludes=updates --downloadonly -y update
pssh -h /root/linux-servers -P -p 1 -t -1 /root/bin/syncfrommaster

This first command is just the download script again. The second command makes use of the parallel ssh program to fire off syncfrommaster on each host. Before you run this you need to run ssh-agent bash, followed by ssh-add to make it possible to run pssh without supplying a login password for each host. This assumes have already installed ssh keys on each host.

4. Finally you can update all the hosts in one go:

pssh -h /root/linux-servers -P -p 1 -t -1 yum update -y

You need the -y as pssh cannot run interactive commands. You might want to run

yum update

on the master host first to check that the updates will go as expected.

It is possible that the master doesn't have all the packages in the union of all packages installed on all hosts. No harm done, the host that has distinct packages will download it on its own. Or you may wish to install that package on the master so the next update will be covered.

openSUSE:

1. Download:

zypper -n up -l -d

To set the repository to retain packages, run

zypper mr -k <repo>

for the repositories must be retained, usually repo-update and any extras such as packman.

2. Synchronisation:

/etc/rsyncd.conf (I only show the stanza added to the existing config file)

[zypp]

comment = Zypp archive

path = /var/cache/zypp/packages

read only = yes

list = yes

Notice that read only and list are in the zypp section. It can be either in the global or module section.

Synchronisation script:

rsync -avi masterhost::zypp /var/cache/zypp/packages

3. Update:

zypper up

With this info you can work out the catchup script and parallel ssh scripts. As in CentOS the -y flag also works to make zypper non-interactive.

Debian:

1. Download:

apt-get update; apt-get upgrade -d -y

2. Synchronisation:

/etc/rsyncd.conf:


[apt]
comment = APT archive
path = /var/cache/apt/archives
read only = yes
list = yes
filter = - lock

Synchronisation script:

rsync -avi masterhost::apt /var/cache/apt/archives

3. Update:

apt-get upgrade

and you can work out the catchup script and parallel ssh scripts. Again -y works to make apt-get upgrade non-interactive.

Notes:

To keep the explanation simpler I have not added the shell directives to redirect output to /dev/null. If cron jobs generate output, root or you will get mailed the output. This could be useful for debuging and to keep an eye on things, but gets tiresome after a while, given that this all works well and falls back to normal update if the caching is not effective. You will need 2>&1 to redirect stderr also.

You will notice that packages will accumulate in the cache after a while. While Debian commendably has apt-get autoclean, the other two RPM based distros do not have an analogous feature. You might just have a cron job remove packages older than a certain number of days.

Install tmm, matplotlib, scipy and numpy on CentOS/RHEL 5

I hope this post saves you some searching if you have to go through the same steps as I did.

A user at this site wanted to run tmm for simulations on CentOS 5 servers.

CentOS/RHEL 5 unfortunately comes with python 2.4, too old for the required modules. Fortunately there is an easy way to install python 2.6 side by side and that is to get the python26 package from EPEL. If you do:

yum search python26

you will get a list of python26 packages. Aside from the base python26 package, I also needed python26-numpy. This will pull in any dependencies. When you have installed python26 you must remember to invoke python scripts with python26 or to edit the #! first line.

The scipy module was installed from source, that was the easiest way. Then I installed the tmm module. BTW in all the python extension installs I used python26 setup.py install rather than any package manager.

After that the test.py program in the tmm distribution ran. But my user pointed out that examples.py didn't run. This requires matplotlib. Ok, installed that from source. Now the function sample1() in examples.py ran, but generated no output. More searching showed that you have to specify the backend in ~/.matplotlib/matplotlibrc like this:

backend: TkAgg
interactive: true

(I'm leaving out some useless experiments involving the Postscript backend.) This generated an error about Tkinter not loading. I went back and looked at the setup step for matplotlib and noticed that when I ran:

python26 setup.py

it said I didn't have the Tkinter header files installed so it wasn't building the interface for the Tkinter backend. So I did yum install tk-devel, built and installed matplotlib again, and finally I got the example program to display a plot.

Whew!

Xsane generates large PDFs, use gscan2pdf instead

A friend and I both noticed that Xsane, which is the usual application used with flatbed scanners, generates large PDFs. An investigation showed this is an old problem. One of the comments, by the author in fact, suggested gscan2pdf.

Being a Perl program, this has a large set of dependencies, and there were reports that not all were available in openSUSE repos, so I decided to install from source instead of using a contributed RPM. My strategy was to run perl Makefile.PL, see what missing dependencies were listed and then install then firstly from the standard openSUSE repos, otherwise from CPAN. Some dependencies only show up at runtime, so start it from a terminal and watch it for diagnostic messages.

These dependencies were satisfied using zypper:

tiff
unpaper
ImageMagick-devel
sane-backends-devel
perl-PerlMagick
perl-Try-Tiny
perl-PDF-API2
perl-Font-TTF
perl-Goo-Canvas
perl-Gtk2-ImageView
perl-Config-General
perl-Log-Log4perl
perl-Readonly

These dependencies were satisfied using CPAN:

Gtk2-Ex-PodViewer Gtk2-Ex-Simple-List IO-stringy Set-IntSpan Proc-ProcessTable

These lists are not exhaustive because 1. I may have already had some installed and 2. I did not install dependencies for DjVu or OCR features. So regard this as a starting point.

In retrospect installing from the openSUSE contributed RPM may have been smooth. Perhaps somebody can try that and report. If you are running Debian or derivatives you will probably have no problem getting all the dependencies.

Preliminary results show that gscan2pdf generates much smaller files than Xsane so this is what I'll be using from now on and I encourage you to try gscan2pdf instead of Xsane.

warn of SSL certificate expiry

So you have been tasked to write a cron job to warn when your site's SSL certificate(s) will soon expire and send email to the responsible person. An expired SSL certificate is at best embarrassing and at worst can cause significant business disruption. There are various ways to do this task.

Nagios has a plugin to check and warn about imminently expiring certificates.

If you have the openssl tools installed you can do it from the command line as shown in this blog post. You should be able to parse the returned date string with the GNU date command and convert it to a number of seconds since the Unix epoch for comparison with the current date to check if expiry is imminent.

I preferred to use Perl. There is a very useful Perl module Net::SSL::ExpireDate by Masaaki Hirose which provided exactly what I wanted. The example is self-explanatory.

On a Debian system other CPAN modules are required, fortunately all of them already in Debian so you only have install those with apt-get, then build, test and install Net::SSL::ExpireDate. Here are the modules I had to install, however there may be other dependencies that were already installed on my system, so this list may not be exhaustive.

libclass-accessor-perl
libcrypt-openssl-x509-perl
libdatetime-perl
libtimedate-perl
libtime-duration-parse-perl
libuniversal-require-perl

The Perl script I wrote was 13 lines long.

enable ext4 for RHEL/Centos 5.6+ installs

This tip doesn't get enough publicity so I'm repeating it here:

Installing on ext4

Note the limitation on boot filesystems. I used it for the /home filesystem for better performance.

And if you forget during install time, you can still convert ext3 to ext4 afterwards, a search will find you many pages on how to do this.

vncserver won't start on CentOS 5

CentOS/RHEL has a vncserver script which is configured to start instances for users specified in /etc/sysconfig/vncservers. The format is documented in the comments in that file.

I copied this from a host that I was making a copy of, ran /etc/init.d/vncserver start and it failed. Running the script again with sh -vx /etc/init.d/vncserver start showed me that it was failing on the first user at this line:

runuser -l ${USER} -c "cd ~${USER} && [ -f .vnc/passwd ] && vncserver :${DISP} ${VNCUSERARGS}"

Of course! I had forgotten to copy the .vnc/passwd file for each user across so the servers were failing to start. If this is a new machine, then you need to create passwords for each user first. It's a pity that the script doesn't inform you what really happened.

As an aside, I modified this line:

[ "$RETVAL" -ne 0 ] && break

to:

[ "$RETVAL" -ne 0 ] && echo "Warning: ${USER}:${DISP} not started"

so that the script would not die at the first user that caused an error.

But the script is not very sophisticated and doesn't allow for example starting VNC for a single user that you've added to the configuration.

You don't need iTunes

Well, with a provocative title like that I should explain that I have tricked you a bit. You don't need iTunes to reset a first generation iPod Shuffle (gumstick) if it has gone out of its mind. Mine was flashing the green and orange LEDs alternately.

The usual advice is to install iTunes and to repair the iPod filesystem. However if you don't mind resetting everything, and there is little to lose on a 512MB or 1GB Shuffle, you can go straight to this reset utility from Apple and not have to download and install the big iTunes software package. At the same time this will update the software to the final version released. As the article states, you may need to try a few times. Mine worked on the second attempt.

I don't think this tip has a large audience as most first generation Shuffles are dead now as those models were released in 2005. Mine's still going strong though. I wonder when the flash memory and rechargeable cells will wear out.

Incidentally I can highly recommend the open source shuffle DB software. With this Python script installed in the Shuffle, and a Python interpreter on the host, you don't need complicated tune management software. Just copy the files onto the Shuffle and then run the program to build a playlist that the Shuffle will understand.