Upgrading from Puppet 4 to Puppet 6

One day I discovered that the puppetlabs-release-pc1 repository had stopped working. After some searching I learnt that these repositories had been deprecated. Things sure move fast in the configuration management world. Only seems like yesterday that I migrated from 3 to 4.

A bit of reading convinced me that an upgrade to 6 should be straightforward, as the pain of the language update was over, and at this site I did not use advanced features like puppetdb. The one area of large difference was the change from puppet cert to puppetserver ca commands.

The machines comprise a whole bunch of CentOS 6 and 7 servers, and one Debian Stretch server. First on the master I removed the old release RPM with:

yum remove -y puppetlabs-release-pc1

Then I installed the new puppet release RPM:

rpm -i puppet-release-el-6.rpm

followed by a

yum update -y

This updated the puppetserver and puppet-agent packages to the latest version. The server started fine and agents could connect to it. So on all the CentOS machines, I did the same thing and updated the puppet-agent package. The package should be puppet-release-el-7.rpm for the CentOS 7 machines of course.

On Debian Stretch it was a bit trickier. I had installed the Debian Puppet packages which use non-standard directory paths. So I had to first remove the old Stretch packages before adding the APT repository with:

dpkg -i puppet-release-stretch.deb

Then I installed the Puppetlabs agent package:

apt update
apt install -y puppet-agent

On the first run of puppet agent --test --noop it couldn't contact the master. This is due to the puppet.conf being under /etc/puppetlabs rather than /etc/puppet (I don't use a CNAME for puppet here). I added the lines:

[agent]
server = master.example.com

and it connected to the master. But this time it generated a new PEM certificate then said there was a mismatch with the master's copy. I located the old PEM and copied it over the new PEM, and the mismatch went away. This is, as mentioned, due to the Debian package using non-standard directory paths. By the way the first run also loaded all the configuration files from the master into the local cache which is in a standard directory now. At some point I should clean up the old non-standard puppet directories.

On the master the puppetserver ca commands didn't work, giving a 403 not allowed error. A bit of searching turned up this blog entry. However the instructions that say to modify auth.conf can't be used verbatim. If you add the indicated rule, the server will not start. This I suspected was due to it being a duplicate of an existing rule. Instead what you should do is add the line:

allow: master.example.com   # <- add your puppet master certname

to the existing rule for the match:

path: "/puppet-ca/v1/certificate_status"

Restarting the server worked and I could run commands like puppetserver ca list --all Business as usual again.

NVIDIA driver, libglx.so and hardware acceleration crashing X server

My users on CentOS complained that Firefox would crash the X server after I updated the package xorg-x11-server-Xorg.

A search returned suggestions to disable hardware acceleration in Firefox. However a user mentioned that when he reinstalled the NVIDIA driver, it gave the warning libglx.so is not a symbolic link before setting things right.

An investigation showed that the NVIDIA installer replaces the file /usr/lib64/xorg/modules/extensions/libglx.so with a symlink to /usr/lib64/xorg/modules/extensions/libglx.so.X.Y where X.Y is the NVIDIA package version. So every time the xorg-x11-server-Xorg package is reinstalled, it replaces the symlink and hardware acceleration fails, crashing the X server.

The same problem for Debian is documented in this forum thread.

I could blacklist xorg-x11-server-Xorg in yum but I rather not do that. Since I may forget when an update of that package happens, I wrote a shell script to restore the symlink if removed and a cron job to call it periodically. But I'm also looking to make a Puppet stanza to do this.

Apache won't start on CentOS 5 because self-signed certificate used by mod_nss expired

You are not likely to hit this error with the usual configuration since mod_ssl is the one normally used. However mod_nss is used by the Fedora Directory Server (aka 389 Directory Server now) in CentOS for the console for LDAP authentication (centos-idm-console). Possibly also used by RHEL.

One day I found that Apache would not start. The error log indicated that the certificate had expired. I searched and searched for how to generate a new certificate and tried various things. Too many steps and too hard.

Then I had an idea. The certificate must have been generated when the package mod_nss was installed. Let's try reinstalling it. First we move the old database directory out of the way:

mv /etc/httpd/alias /etc/httpd/alias.old

Then reinstall mod_nss:

yum reinstall mod_nss

and voila, after a while, a new database with a fresh certificate was generated and I could start Apache.

CentOS/RHEL: dig axfr works but zone transfer fails?

I set up named as a secondary with a stanza like this:

zone "example.com" IN {
        type slave;
        masters { 10.0.0.1; };
        file "example.com.zone";
        notify yes;
};

When I restarted named the log messages in /var/log/messages told me that the transfer failed. However I could do:

dig @10.0.0.1 axfr example.com

just fine. What was the problem?

Looking in /var/named/chroot/var/named I saw that there was a directory named slaves. Should I be using that? Looking at the permissions, I saw:

total 88
drwxr-x--- 4 root  named 4096 Jan  7  2013 .
drwxr-x--- 6 root  named 4096 Jan  7  2013 ..
drwxrwx--- 2 named named 4096 Mar  4 23:23 data
-rw-r----- 1 root  named  208 Apr 28  2005 localdomain.zone
-rw-r----- 1 root  named  195 Apr 28  2005 localhost.zone
-rw-r----- 1 root  named  427 Apr 28  2005 named.broadcast
-rw-r----- 1 root  named  424 Apr 28  2005 named.ip6.local
-rw-r----- 1 root  named  426 Apr 28  2005 named.local
-rw-r----- 1 root  named 1892 Feb 26  2008 named.root
-rw-r----- 1 root  named  427 Apr 28  2005 named.zero
drwxrwx--- 2 named named 4096 Mar  5 00:59 slaves

Yes, I bet I'm supposed to use the directory slaves because that's writable by named but the top zone directory isn't, so I changed the line in the stanza to:

        file "slaves/example.com.zone";

and it worked.

You may see advice on the Web to change permissions of the top zone directory. Don't do that. Work with the structure and permissions that the distribution makers have developed.

A caching scheme to reduce downloads for Linux package updates

As I administer sites with a lot of (nearly) identical Linux hosts, I thought a lot about the issue of reducing the download for package updates. Even if you have unlimited data quota for your site, downloading the updates multiple times for multiple machines takes time.

I looked at various schemes suggested on the Internet. One is to set up a satellite repository. This is reliable but involves a lot of configuration. Also you may download a lot of package updates that are not used at your site.

Another class of scheme involves interposing a proxy. It could be a specialised proxy for the distro, or a general proxy like squid. This can work well but is sometimes defeated by load balancers or smart mirrors where you are not guaranteed to hit the same repository every download, as the actual repository used could vary.

Eventually I came up with a scheme that is applicable to three distros that I have tried it on: CentOS, Debian (and derivatives) and openSUSE. The key observation is that fact that the updaters in these distros have a package cache. If packages are added to the cache on the side, the updater will skip downloading them. I describe this as a caching scheme, not a proxying scheme as no change is made to the updater configuration files and normal, independent update works.

The scheme:

There are three parts to the scheme.

1. A download script runs on a designated master host. This runs during off-peak hours. We use the updater program in download but not install mode so we will only download packages we need. It's also necessary to set the repository to retain packages.

2. A synchronisation script runs on the other hosts to pull the new updates to their cache. This is run as soon as possible after the download.

3. Various utility scripts are used to fire off updates on all hosts. I make use of parallel ssh and ssh-agent to make life easy.

Let's use a concrete example to illustrate this.

CentOS:

1. Download:

The download script is this:

yum --disableexcludes=updates --downloadonly -y update

You need to install the auxiliary package yum-downloadonly. The --disableexcludes=updates is to disable the regular expressions that prevent the installation of the kernel on this host. On this host I disabled kernel updates due to the need to reboot and recompile VirtualBox modules whenever the kernel is updated, and as this is a server that engineers use constantly, I delay kernel updates until downtime can be scheduled to do them manually. Other software that may need recompilation on kernel update include proprietary video drivers.

To set the repository to retain packages, add

keepcache=1

to /etc/yum.conf

2. Synchronisation:

Next I set up a rsync server. Initially I used rsync over ssh, but this requires setting up a login account, restricted commands, etc. for security reasons. Package updates are not sensitive data so I just share the cache directory with rsyncd. Please consult your distro documentation on how to enable the rsyncd daemon. Usually it's run out of xinetd. Here is /etc/rsyncd.conf:

secrets file = /etc/rsyncd.secrets
motd file = /etc/rsyncd.motd
read only = yes
list = yes
uid = nobody
gid = nobody

[yum]
comment = YUM cache
path = /var/cache/yum
filter = + *.rpm + */ - *

Note the filter expression to share RPM files and directories but not files of other types.

The synchronisation script, call it syncfrommaster, on the other hosts is:

rsync -avi masterhost::yum /var/cache/yum

In the cron job that runs this command I insert a random delay so that the hosts don't all try to synchronise at the same time and overload the master.

sleep $(($RANDOM \% 900)); /root/bin/syncfrommaster

The % is escaped with \ in cron scripts by the way.

I arrange for this job to be run an hour after the download script.

3. Update.

You have probably noticed that it is possible that an update will be published in the time between the master pulling it down and you applying the updates. In this case each host merely downloads the package on its own, and nothing breaks. However you may wish to run a script to cause the cache to be updated and the hosts to catch up.

#!/bin/sh
yum --disableexcludes=updates --downloadonly -y update
pssh -h /root/linux-servers -P -p 1 -t -1 /root/bin/syncfrommaster

This first command is just the download script again. The second command makes use of the parallel ssh program to fire off syncfrommaster on each host. Before you run this you need to run ssh-agent bash, followed by ssh-add to make it possible to run pssh without supplying a login password for each host. This assumes have already installed ssh keys on each host.

4. Finally you can update all the hosts in one go:

pssh -h /root/linux-servers -P -p 1 -t -1 yum update -y

You need the -y as pssh cannot run interactive commands. You might want to run

yum update

on the master host first to check that the updates will go as expected.

It is possible that the master doesn't have all the packages in the union of all packages installed on all hosts. No harm done, the host that has distinct packages will download it on its own. Or you may wish to install that package on the master so the next update will be covered.

openSUSE:

1. Download:

zypper -n up -l -d

To set the repository to retain packages, run

zypper mr -k <repo>

for the repositories must be retained, usually repo-update and any extras such as packman.

2. Synchronisation:

/etc/rsyncd.conf (I only show the stanza added to the existing config file)

[zypp]

comment = Zypp archive

path = /var/cache/zypp/packages

read only = yes

list = yes

Notice that read only and list are in the zypp section. It can be either in the global or module section.

Synchronisation script:

rsync -avi masterhost::zypp /var/cache/zypp/packages

3. Update:

zypper up

With this info you can work out the catchup script and parallel ssh scripts. As in CentOS the -y flag also works to make zypper non-interactive.

Debian:

1. Download:

apt-get update; apt-get upgrade -d -y

2. Synchronisation:

/etc/rsyncd.conf:


[apt]
comment = APT archive
path = /var/cache/apt/archives
read only = yes
list = yes
filter = - lock

Synchronisation script:

rsync -avi masterhost::apt /var/cache/apt/archives

3. Update:

apt-get upgrade

and you can work out the catchup script and parallel ssh scripts. Again -y works to make apt-get upgrade non-interactive.

Notes:

To keep the explanation simpler I have not added the shell directives to redirect output to /dev/null. If cron jobs generate output, root or you will get mailed the output. This could be useful for debuging and to keep an eye on things, but gets tiresome after a while, given that this all works well and falls back to normal update if the caching is not effective. You will need 2>&1 to redirect stderr also.

You will notice that packages will accumulate in the cache after a while. While Debian commendably has apt-get autoclean, the other two RPM based distros do not have an analogous feature. You might just have a cron job remove packages older than a certain number of days.

enable ext4 for RHEL/Centos 5.6+ installs

This tip doesn't get enough publicity so I'm repeating it here:

Installing on ext4

Note the limitation on boot filesystems. I used it for the /home filesystem for better performance.

And if you forget during install time, you can still convert ext3 to ext4 afterwards, a search will find you many pages on how to do this.

Samba, SELinux and the homes share

As has been documented here and many other places, on Redhat, CentOS, Fedora and other distros that come with SELinux enabled, you have to enable the SELinux boolean samba_enable_home_dirs if you export the homes share. I did this but I still could not connect to the share. The usual error message is NT_STATUS_BAD_NETWORK_NAME.


It turns out that you need samba_export_all_ro or more likely, samba_export_all_rw enabled to mount the share. This was on CentOS 5.5. It could be that on this distro and version both come disabled by default.


Make sure though that SELinux is the cause. There are many other reasons for NT_STATUS_BAD_NETWORK_NAME. A quick way to check is to set SELinux to permissive temporarily to attempt the connection.

Force yum to determine the fastest mirrors again

We are going to ship this CentOS machine overseas where the fastest mirrors will be different. We want yum to work that out on deployment there. A quick search of the files in the package yum-fastestmirror found the cache file: /var/cache/yum/timedhosts.txt


Just delete or rename this file and yum will do a speed test the next time it is run.


The plugin yum-fastestmirror appears to be an EPEL addon, so may not be installed by default on RHEL or Fedora.

Modifying synbak to use lzop or xz for compression

For backing up a server I installed the synbak program on CentOS, partly because it was a standard package from rpmforge, and partly because it is uncomplicated to use, basically just shell scripts. I noticed that when I used gzip as the compression method, it tied up a whole CPU doing the compression. So I decided to add lzop support. It's very easy, just add these two lines to /usr/share/synbak/method/tar/tar.sh:

--- /usr/share/synbak/method/tar/tar.sh.orig    2010-06-10 05:18:14.000000000 +1000
+++ /usr/share/synbak/method/tar/tar.sh 2012-04-12 09:46:36.000000000 +1000
@@ -30,6 +30,8 @@
        tar)    backup_method_opts_default="--totals -cpf"  ; backup_name_extension="tar" ;;
        gz)     backup_method_opts_default="--totals -zcpf" ; backup_name_extension="tar.gz" ;;
        bz2)    backup_method_opts_default="--totals -jcpf" ; backup_name_extension="tar.bz2" ;;
+       lzo)    backup_method_opts_default="--use-compress-program=lzop --totals -cpf" ; backup_name_extension="tar.lzo" ;;
+       xz)     backup_method_opts_default="--use-compress-program=xz --totals -cpf" ; backup_name_extension="tar.xz" ;;
        *)      report_text extra_option_wrong && exit 1 ;;
   esac


You run it like this: synbak -s serverconfig -m tar -M lzo

I added xz support for good measure, but this is untested. Make sure that backup_method_opts in your config file doesn't have -z or the tar command will throw an error.


Naturally you must have the lzop and/or xz packages installed.


Using lzop, tar and lzop take about equal amounts of CPU time, most of the time tar was I/O bound, and backups finished faster. I also see that the memory footprint of lzop is small. The downside that is the compressed output is larger, about 31% more in my case. Decompression is supposed to be very fast, which is useful when restoring. I haven't used decompression and I hope I won't ever have to.