Prefer IPv4 for zypper updates

I had been battling server timeouts from a local mirror of OpenSUSE repositories when I noticed that if I telneted to the troublesome server it attempted to contact it via its IPv6 address. My Internet retailer doesn't support IPv6 connections, although my Linux distro and modem are capable. At the moment there's no need as there is no shortage of IPv4 addresses in this country.

A little search found that it's due to the curl library trying IPv6 addresses first. There is a little known environment variable in libzypp to control this. I put

ZYPP_MEDIA_CURL_IPRESOLVE=4

in my environment settings and I no longer had server timeouts.

If you are using sudo to run zypper, look into the env_keep option  in the sudoers file, and check that the env_reset option is enabled (the default), for example:

Defaults env_keep += "ZYPP_MEDIA_CURL_IPRESOLVE"

 

Travails installing openSUSE on Samsung 2012 Chromebook (Snow)

I have a 2012 Chromebook made by Samsung, codenamed Snow. I like this machine as it is light, has a decent sized screen, and is suited for travel. Google stopped issuing updates to ChromeOS for this Chromebook in July 2018. So I was pleased to see that openSUSE Linux developers had issued builds for this hardware. For one thing I might be able to do things on the Chromebook not possible from ChromeOS.

So I went to the wiki page that had instructions for preparing boot media for it. First thing I had to do was enable developer mode. I lost my personalisation of the Chromebook but I didn't have much anyway, and most of the config was backed up to the cloud.

Ever optimistic, I chose the Leap KDE build image. I got it to boot and then the troubles began. I needed to get a network connection to update the packages, and this needed to be set up from the desktop. But the Plasma desktop kept crashing. Most of the time it would restart by itself after a hiatus at the desktop. But sometimes the machine froze and the only to get out was by turning off the machine by holding down the power button. On power up, sometimes it wouldn't boot far enough to show kernel messages and then it was power off again. Perhaps it was trying to repair the filesystem and I should have waited longer, but I did wait a long time. Finally I managed to download all the updates, some 600 packages in all. But by this time the machine wouldn't boot up anymore. Perhaps the CPU had overheated.

After a rest for the machine and for me, I decided I was too ambitious so I tried the JeOS (Just enough OS) image. Unfortunately this turned out to be NQeOS (Not Quite enough OS) as it didn't boot to a GUI. I might have been able to configure the WLAN interface from the command line but IHeOI (I Had enough Of It).

So I decided to try booting the LXQT image, I reasoned that perhaps without Plasma the desktop would be more stable. After quite a few tries and retries with the WLAN interface, I managed to fetch all the update packages, some 570 IIRC, and I installed them with zypper up -y from the CLI.

Then there was some to and fro with NetworkManager and wickedd. I decided to go for wickedd as this works at boot rather than when the user logs in on the desktop. But both were active after install causing some clashes.

So I have a desktop of sorts which can stay up for a while. I may keep LXQT, even though I'm used to KDE on my desktop, because this is a low RAM machine (1GB) after all.

But here's the main problem: after all that work I could not locate some key packages like a web browser, e.g. Firefox. Perhaps it's in a repo I don't know about, certainly it's not in the main repos. Looks like these builds are for people who are willing to experiment and not ready for normal usage. In retrospect perhaps I should have tried a Tumbleweed image as this might have all the updates to build date rolled in. I also see there's are experimental 15.1 builds which I might try and report back.

Other approaches I could have tried are Crouton, which keeps the kernel used by ChromeOS but runs applications over it. I don't know if this can be installed on a SD card. To the openSUSE developers' credit, getting a boot image which supports all of the peripherals I needed is a fair achievement. There is Internet documentation on how to install Debian on it but there is a lot of cross compilation and manual configuration.

The main motivation for installing a Linux distro on the Chromebook is to be able to get access to things like VPN software with a decent sized desktop and multiple terminal windows. But this motivation has faded a bit as I don't need as much to connect to my home or work machines during travels. If I do, my smartphone, with its 4GB RAM, and lots of VPN apps under Android, coupled with a Bluetooth keyboard, is much more capable, with the caveat that the display is smaller, even though the resolution is higher. So that only leaves local development when on the road as a rationale for carrying a Linux enabled Chromebook.

A caching scheme to reduce downloads for Linux package updates

As I administer sites with a lot of (nearly) identical Linux hosts, I thought a lot about the issue of reducing the download for package updates. Even if you have unlimited data quota for your site, downloading the updates multiple times for multiple machines takes time.

I looked at various schemes suggested on the Internet. One is to set up a satellite repository. This is reliable but involves a lot of configuration. Also you may download a lot of package updates that are not used at your site.

Another class of scheme involves interposing a proxy. It could be a specialised proxy for the distro, or a general proxy like squid. This can work well but is sometimes defeated by load balancers or smart mirrors where you are not guaranteed to hit the same repository every download, as the actual repository used could vary.

Eventually I came up with a scheme that is applicable to three distros that I have tried it on: CentOS, Debian (and derivatives) and openSUSE. The key observation is that fact that the updaters in these distros have a package cache. If packages are added to the cache on the side, the updater will skip downloading them. I describe this as a caching scheme, not a proxying scheme as no change is made to the updater configuration files and normal, independent update works.

The scheme:

There are three parts to the scheme.

1. A download script runs on a designated master host. This runs during off-peak hours. We use the updater program in download but not install mode so we will only download packages we need. It's also necessary to set the repository to retain packages.

2. A synchronisation script runs on the other hosts to pull the new updates to their cache. This is run as soon as possible after the download.

3. Various utility scripts are used to fire off updates on all hosts. I make use of parallel ssh and ssh-agent to make life easy.

Let's use a concrete example to illustrate this.

CentOS:

1. Download:

The download script is this:

yum --disableexcludes=updates --downloadonly -y update

You need to install the auxiliary package yum-downloadonly. The --disableexcludes=updates is to disable the regular expressions that prevent the installation of the kernel on this host. On this host I disabled kernel updates due to the need to reboot and recompile VirtualBox modules whenever the kernel is updated, and as this is a server that engineers use constantly, I delay kernel updates until downtime can be scheduled to do them manually. Other software that may need recompilation on kernel update include proprietary video drivers.

To set the repository to retain packages, add

keepcache=1

to /etc/yum.conf

2. Synchronisation:

Next I set up a rsync server. Initially I used rsync over ssh, but this requires setting up a login account, restricted commands, etc. for security reasons. Package updates are not sensitive data so I just share the cache directory with rsyncd. Please consult your distro documentation on how to enable the rsyncd daemon. Usually it's run out of xinetd. Here is /etc/rsyncd.conf:

secrets file = /etc/rsyncd.secrets
motd file = /etc/rsyncd.motd
read only = yes
list = yes
uid = nobody
gid = nobody

[yum]
comment = YUM cache
path = /var/cache/yum
filter = + *.rpm + */ - *

Note the filter expression to share RPM files and directories but not files of other types.

The synchronisation script, call it syncfrommaster, on the other hosts is:

rsync -avi masterhost::yum /var/cache/yum

In the cron job that runs this command I insert a random delay so that the hosts don't all try to synchronise at the same time and overload the master.

sleep $(($RANDOM \% 900)); /root/bin/syncfrommaster

The % is escaped with \ in cron scripts by the way.

I arrange for this job to be run an hour after the download script.

3. Update.

You have probably noticed that it is possible that an update will be published in the time between the master pulling it down and you applying the updates. In this case each host merely downloads the package on its own, and nothing breaks. However you may wish to run a script to cause the cache to be updated and the hosts to catch up.

#!/bin/sh
yum --disableexcludes=updates --downloadonly -y update
pssh -h /root/linux-servers -P -p 1 -t -1 /root/bin/syncfrommaster

This first command is just the download script again. The second command makes use of the parallel ssh program to fire off syncfrommaster on each host. Before you run this you need to run ssh-agent bash, followed by ssh-add to make it possible to run pssh without supplying a login password for each host. This assumes have already installed ssh keys on each host.

4. Finally you can update all the hosts in one go:

pssh -h /root/linux-servers -P -p 1 -t -1 yum update -y

You need the -y as pssh cannot run interactive commands. You might want to run

yum update

on the master host first to check that the updates will go as expected.

It is possible that the master doesn't have all the packages in the union of all packages installed on all hosts. No harm done, the host that has distinct packages will download it on its own. Or you may wish to install that package on the master so the next update will be covered.

openSUSE:

1. Download:

zypper -n up -l -d

To set the repository to retain packages, run

zypper mr -k <repo>

for the repositories must be retained, usually repo-update and any extras such as packman.

2. Synchronisation:

/etc/rsyncd.conf (I only show the stanza added to the existing config file)

[zypp]

comment = Zypp archive

path = /var/cache/zypp/packages

read only = yes

list = yes

Notice that read only and list are in the zypp section. It can be either in the global or module section.

Synchronisation script:

rsync -avi masterhost::zypp /var/cache/zypp/packages

3. Update:

zypper up

With this info you can work out the catchup script and parallel ssh scripts. As in CentOS the -y flag also works to make zypper non-interactive.

Debian:

1. Download:

apt-get update; apt-get upgrade -d -y

2. Synchronisation:

/etc/rsyncd.conf:


[apt]
comment = APT archive
path = /var/cache/apt/archives
read only = yes
list = yes
filter = - lock

Synchronisation script:

rsync -avi masterhost::apt /var/cache/apt/archives

3. Update:

apt-get upgrade

and you can work out the catchup script and parallel ssh scripts. Again -y works to make apt-get upgrade non-interactive.

Notes:

To keep the explanation simpler I have not added the shell directives to redirect output to /dev/null. If cron jobs generate output, root or you will get mailed the output. This could be useful for debuging and to keep an eye on things, but gets tiresome after a while, given that this all works well and falls back to normal update if the caching is not effective. You will need 2>&1 to redirect stderr also.

You will notice that packages will accumulate in the cache after a while. While Debian commendably has apt-get autoclean, the other two RPM based distros do not have an analogous feature. You might just have a cron job remove packages older than a certain number of days.