Alias for hexdump for canonical output

The man page for hexdump from util-linux states:

-C, --canonical

Canonical hex+ASCII display. Display the input offset in hexadecimal, followed by sixteen space-separated, two-column, hexadecimal bytes, followed by the same sixteen bytes in %_p format enclosed in '|' characters. Invoking the program as hd implies this option.

This also happens to be my preferred format for working with microprocessor code. However my Linux distro's package chose not to provide a link or alias from hd to hexdump.

No matter, I just made a symlink called hd in my private bin directory to /usr/bin/hexdump and it works as described.

k3b, mkisofs, and hard linked files

As it's the end of the year, I decided to back up photos from my trips onto DVD. A du -sh of the directories showed that they would fit onto a standard DVD. But when I went to create the image in k3b, it wanted a double layer DVD.

I knew what the problem was, in those albums I had made heavy use of hard links to provide multiple access paths to photos without using extra disk space.

Checking the man page of mkisofs showed that there were a pair of options, --cache-inodes and the negative form. I added this option to the arguments passed by k3b to mkisofs and bingo, the space of the image halved, even though the preview claimed that it required almost 8GiB of DVD.

Looking at the verbose output of k3b on the terminal showed that --no-cache-inodes is the default, presumably to avoid problems with dumping filesystems that don't have working inodes.

Travails installing openSUSE on Samsung 2012 Chromebook (Snow)

I have a 2012 Chromebook made by Samsung, codenamed Snow. I like this machine as it is light, has a decent sized screen, and is suited for travel. Google stopped issuing updates to ChromeOS for this Chromebook in July 2018. So I was pleased to see that openSUSE Linux developers had issued builds for this hardware. For one thing I might be able to do things on the Chromebook not possible from ChromeOS.

So I went to the wiki page that had instructions for preparing boot media for it. First thing I had to do was enable developer mode. I lost my personalisation of the Chromebook but I didn't have much anyway, and most of the config was backed up to the cloud.

Ever optimistic, I chose the Leap KDE build image. I got it to boot and then the troubles began. I needed to get a network connection to update the packages, and this needed to be set up from the desktop. But the Plasma desktop kept crashing. Most of the time it would restart by itself after a hiatus at the desktop. But sometimes the machine froze and the only to get out was by turning off the machine by holding down the power button. On power up, sometimes it wouldn't boot far enough to show kernel messages and then it was power off again. Perhaps it was trying to repair the filesystem and I should have waited longer, but I did wait a long time. Finally I managed to download all the updates, some 600 packages in all. But by this time the machine wouldn't boot up anymore. Perhaps the CPU had overheated.

After a rest for the machine and for me, I decided I was too ambitious so I tried the JeOS (Just enough OS) image. Unfortunately this turned out to be NQeOS (Not Quite enough OS) as it didn't boot to a GUI. I might have been able to configure the WLAN interface from the command line but IHeOI (I Had enough Of It).

So I decided to try booting the LXQT image, I reasoned that perhaps without Plasma the desktop would be more stable. After quite a few tries and retries with the WLAN interface, I managed to fetch all the update packages, some 570 IIRC, and I installed them with zypper up -y from the CLI.

Then there was some to and fro with NetworkManager and wickedd. I decided to go for wickedd as this works at boot rather than when the user logs in on the desktop. But both were active after install causing some clashes.

So I have a desktop of sorts which can stay up for a while. I may keep LXQT, even though I'm used to KDE on my desktop, because this is a low RAM machine (1GB) after all.

But here's the main problem: after all that work I could not locate some key packages like a web browser, e.g. Firefox. Perhaps it's in a repo I don't know about, certainly it's not in the main repos. Looks like these builds are for people who are willing to experiment and not ready for normal usage. In retrospect perhaps I should have tried a Tumbleweed image as this might have all the updates to build date rolled in. I also see there's are experimental 15.1 builds which I might try and report back.

Other approaches I could have tried are Crouton, which keeps the kernel used by ChromeOS but runs applications over it. I don't know if this can be installed on a SD card. To the openSUSE developers' credit, getting a boot image which supports all of the peripherals I needed is a fair achievement. There is Internet documentation on how to install Debian on it but there is a lot of cross compilation and manual configuration.

The main motivation for installing a Linux distro on the Chromebook is to be able to get access to things like VPN software with a decent sized desktop and multiple terminal windows. But this motivation has faded a bit as I don't need as much to connect to my home or work machines during travels. If I do, my smartphone, with its 4GB RAM, and lots of VPN apps under Android, coupled with a Bluetooth keyboard, is much more capable, with the caveat that the display is smaller, even though the resolution is higher. So that only leaves local development when on the road as a rationale for carrying a Linux enabled Chromebook.

EFI System Partition in soft RAID1

One reason you might want to put the EFI System Partition (ESP) in a RAID1 array on a computer with Linux soft RAID is to have redundancy when booting. If one disk fails, you want the boot to continue from the other disk.

At first I thought this wasn't possible since a RAID1 partition wouldn't have the specific FAT filesystem and GUID required by the specification. However the fact that the CentOS 7 install media offered the choice of putting the ESP on a RAID1 array and that it actually works, made me doubt my hypothesis.

The key to this that the CentOS 7 installer uses RAID metadata format 1.0, which is located at the end of the partition. Thus it doesn't clash with the beginning of the partition, which is where the BIOS will check to see if the partition is an ESP. However most Linux partition tools will detect it first as a RAID member so it's not immediately obvious that it's an ESP.

There are some caveats to this scheme. All writing of the ESP must be done while it's mounted as a RAID array so that there is no discrepancy between the two members. If the only OS on the disks is Linux, this won't be a problem. But don't use this scheme if the ESP also boots other operating systems that don't know about Linux RAID.

For CentOS when you look at the choice of boot devices in the BIOS, you should see two disk boot candidates, both labelled CentOS.

On the machines I used, HP z230 workstations, I found that I had to disable Legacy Boot or errors reading the boot sectors would be triggered.

The bottom line is I now have workstations with soft RAID1 whose disks are fully redundant. If one disk fails, the other will continue to boot and run with degraded arrays for each of the partitions.

grub2 error: failure reading sector 0x0 from 'hd0'. Press any key to continue

After I had installed CentOS 7 as the only OS on a HP z230 workstation in UEFI boot mode, I got this message before booting. It was actually the last of three errors:

error: failure reading sector 0xfc from 'hd0'. 
error: failure reading sector 0xe0 from 'hd0'. 
error: failure reading sector 0x0 from 'hd0'

Boot would resume from the hard disk after a timeout, but the pause was unacceptable and would worry users.

A search showed many articles like this but none solved my problem. I tried various things: refreshing grub.cfg, disabling the CD/DVD (thinking it might be trying to read the optical drive), checking if having the ESP in a EFI system partition in a RAID1 array was disallowed. (I figured out how ESP can work with RAID1, and its limitations, but that's for another blog entry.) None of my experiments worked.

However the linked to web page alluded to turning off Secure Boot so I went into that part of the BIOS setup. I found that it was already turned off but there was a setting there for Legacy Boot which was enabled. So I turned it off to see what would happen. Lo and behold, the error messages ceased, and UEFI boot worked as expected. Also the Boot Order menu stopped showing a Legacy section.

Since debugging the innards of the GRUB2 loader is beyond me, I can only surmise that the presence of Legacy Boot entries in the BIOS makes GRUB2 try reading the sectors in question but since the disk is formatted with GPT partitions and UEFI is in force, the sector reads fail, for some definition of fail. Maybe somebody can figure out the significance of the sectors 0xfc, 0xe0, and 0x0.

Rebuild kernel modules for VirtualBox on Linux automatically

I prefer to install packages distributed by VirtualBox rather than packages built by the distro maintainers as I don't have to wait for a new package when a new kernel is released. Unfortunately this involves rebuilding the vbox kernel modules after the new kernel is booted. So I decided to devise a way to automate this.

First I wrote a script to check if the vbox modules are loaded and if not to run the setup script. It's just this:

#!/bin/sh

sleep 120
if ! lsmod | grep -s vbox > /dev/null
then
       /usr/lib/virtualbox/vboxdrv.sh setup
fi

The sleep is needed as in some init systems cron comes up even before all the modules are loaded. Next, I installed a crontab entry that runs the script above once at bootup, using the @reboot notation:

@reboot /root/bin/vboxdrv

This goes into root's crontab, using crontab -e.

And voilà, that's all that's needed.

Installing Linuxmint 18 on RAID 1

TL;DR: Can't be done easily.

I had a PC with dual disks that I was running under openSUSE Leap 42.1 in RAID 1. I wanted to install Linuxmint 18 on it before giving it away.

First problem I encountered was that the partitioner in the installer knows nothing about RAID.

After some reading I found that the solution is to use gparted to create the RAID partitions first. But gparted didn't know how to do this. I figured out that the mdadm tool was missing, so I did:

apt-get install mdadm

Note that this installs to RAM as it's a live filesystem so needs to be redone if the install is restarted. Also there are some post-install script errors but mdadm gets installed.

Now with gparted I could create the RAID partitions and assemble them. Alternatively it can be done from the command line:

mdadm --assemble --scan

Now md0 and md1 appeared on the list of "disks" in the partitioner and I could assign them to / and /home.

The system installation went fine until it was time to install GRUB. It failed when it couldn't do

grub-install /dev/sdb /dev/sda

Changing the target to just /dev/sda and clicking Continue did nothing. This seems to be an installer bug.

So I did this from the command line:

mount /dev/md0 /mnt
grub-install --root-directory=/mnt /dev/sda
umount /mnt

When I rebooted from disk it stopped at the GRUB prompt. So I tried to boot manually.

grub> linux /boot/vmlin.... --root=/dev/md0
grub> initrd /boot/initrd...
grub> boot

The ... are where you should use TAB completion to select the kernel and initramfs images respectively. Unfortunately this failed when systemd tried to mount the root partition. The reason is that the raid modules are not present in the stock initramfs file. This was built by the Linuxmint 18 distributors.

At this point I gave up. Obviously Linuxmint 18 isn't designed to support this kind of RAID installation. If you want to go further you should rebuild the initramfs with the RAID modules included. Do let us know if you overcome this next hurdle.

And remember if you get it to work, you should install the mdadm package permanently, as it's not part of the default package set.

ssh hangs at SSH2_MSG_KEX_DH_GEX_GROUP trying to connect to servers behind Cisco firewall

Today I was unable to ssh to some CentOS servers behind a Cisco firewall. I was connected using AnyConnect. When I ran ssh with -v, it showed me that it stopped at expecting SSH2_MSG_KEX_DH_GEX_GROUP.

A search on the Internet turned up this article: Natty Narwhal: Problems connecting to servers behind (Cisco) firewalls using ssh. Shortening the Cipher and MAC list as suggested solved the problem. Apparently due to overflowing some packet size limit somewhere. I'll leave it to the experts to work out what it is about Cisco and ssh.

A caching scheme to reduce downloads for Linux package updates

As I administer sites with a lot of (nearly) identical Linux hosts, I thought a lot about the issue of reducing the download for package updates. Even if you have unlimited data quota for your site, downloading the updates multiple times for multiple machines takes time.

I looked at various schemes suggested on the Internet. One is to set up a satellite repository. This is reliable but involves a lot of configuration. Also you may download a lot of package updates that are not used at your site.

Another class of scheme involves interposing a proxy. It could be a specialised proxy for the distro, or a general proxy like squid. This can work well but is sometimes defeated by load balancers or smart mirrors where you are not guaranteed to hit the same repository every download, as the actual repository used could vary.

Eventually I came up with a scheme that is applicable to three distros that I have tried it on: CentOS, Debian (and derivatives) and openSUSE. The key observation is that fact that the updaters in these distros have a package cache. If packages are added to the cache on the side, the updater will skip downloading them. I describe this as a caching scheme, not a proxying scheme as no change is made to the updater configuration files and normal, independent update works.

The scheme:

There are three parts to the scheme.

1. A download script runs on a designated master host. This runs during off-peak hours. We use the updater program in download but not install mode so we will only download packages we need. It's also necessary to set the repository to retain packages.

2. A synchronisation script runs on the other hosts to pull the new updates to their cache. This is run as soon as possible after the download.

3. Various utility scripts are used to fire off updates on all hosts. I make use of parallel ssh and ssh-agent to make life easy.

Let's use a concrete example to illustrate this.

CentOS:

1. Download:

The download script is this:

yum --disableexcludes=updates --downloadonly -y update

You need to install the auxiliary package yum-downloadonly. The --disableexcludes=updates is to disable the regular expressions that prevent the installation of the kernel on this host. On this host I disabled kernel updates due to the need to reboot and recompile VirtualBox modules whenever the kernel is updated, and as this is a server that engineers use constantly, I delay kernel updates until downtime can be scheduled to do them manually. Other software that may need recompilation on kernel update include proprietary video drivers.

To set the repository to retain packages, add

keepcache=1

to /etc/yum.conf

2. Synchronisation:

Next I set up a rsync server. Initially I used rsync over ssh, but this requires setting up a login account, restricted commands, etc. for security reasons. Package updates are not sensitive data so I just share the cache directory with rsyncd. Please consult your distro documentation on how to enable the rsyncd daemon. Usually it's run out of xinetd. Here is /etc/rsyncd.conf:

secrets file = /etc/rsyncd.secrets
motd file = /etc/rsyncd.motd
read only = yes
list = yes
uid = nobody
gid = nobody

[yum]
comment = YUM cache
path = /var/cache/yum
filter = + *.rpm + */ - *

Note the filter expression to share RPM files and directories but not files of other types.

The synchronisation script, call it syncfrommaster, on the other hosts is:

rsync -avi masterhost::yum /var/cache/yum

In the cron job that runs this command I insert a random delay so that the hosts don't all try to synchronise at the same time and overload the master.

sleep $(($RANDOM \% 900)); /root/bin/syncfrommaster

The % is escaped with \ in cron scripts by the way.

I arrange for this job to be run an hour after the download script.

3. Update.

You have probably noticed that it is possible that an update will be published in the time between the master pulling it down and you applying the updates. In this case each host merely downloads the package on its own, and nothing breaks. However you may wish to run a script to cause the cache to be updated and the hosts to catch up.

#!/bin/sh
yum --disableexcludes=updates --downloadonly -y update
pssh -h /root/linux-servers -P -p 1 -t -1 /root/bin/syncfrommaster

This first command is just the download script again. The second command makes use of the parallel ssh program to fire off syncfrommaster on each host. Before you run this you need to run ssh-agent bash, followed by ssh-add to make it possible to run pssh without supplying a login password for each host. This assumes have already installed ssh keys on each host.

4. Finally you can update all the hosts in one go:

pssh -h /root/linux-servers -P -p 1 -t -1 yum update -y

You need the -y as pssh cannot run interactive commands. You might want to run

yum update

on the master host first to check that the updates will go as expected.

It is possible that the master doesn't have all the packages in the union of all packages installed on all hosts. No harm done, the host that has distinct packages will download it on its own. Or you may wish to install that package on the master so the next update will be covered.

openSUSE:

1. Download:

zypper -n up -l -d

To set the repository to retain packages, run

zypper mr -k <repo>

for the repositories must be retained, usually repo-update and any extras such as packman.

2. Synchronisation:

/etc/rsyncd.conf (I only show the stanza added to the existing config file)

[zypp]

comment = Zypp archive

path = /var/cache/zypp/packages

read only = yes

list = yes

Notice that read only and list are in the zypp section. It can be either in the global or module section.

Synchronisation script:

rsync -avi masterhost::zypp /var/cache/zypp/packages

3. Update:

zypper up

With this info you can work out the catchup script and parallel ssh scripts. As in CentOS the -y flag also works to make zypper non-interactive.

Debian:

1. Download:

apt-get update; apt-get upgrade -d -y

2. Synchronisation:

/etc/rsyncd.conf:


[apt]
comment = APT archive
path = /var/cache/apt/archives
read only = yes
list = yes
filter = - lock

Synchronisation script:

rsync -avi masterhost::apt /var/cache/apt/archives

3. Update:

apt-get upgrade

and you can work out the catchup script and parallel ssh scripts. Again -y works to make apt-get upgrade non-interactive.

Notes:

To keep the explanation simpler I have not added the shell directives to redirect output to /dev/null. If cron jobs generate output, root or you will get mailed the output. This could be useful for debuging and to keep an eye on things, but gets tiresome after a while, given that this all works well and falls back to normal update if the caching is not effective. You will need 2>&1 to redirect stderr also.

You will notice that packages will accumulate in the cache after a while. While Debian commendably has apt-get autoclean, the other two RPM based distros do not have an analogous feature. You might just have a cron job remove packages older than a certain number of days.

What good is the --target option of cp?

If you look at the man page for cp(1) on operating systems where the GNU tools are used, such as Linux, you will see there is an third form that uses the -t option or alternatively the equivalent long form --target.

cp [OPTION]... -t DIRECTORY SOURCE...

So what good is this when you already can do:

cp [OPTION] SOURCE... DIRECTORY

Here's a reason. Suppose the SOURCE list is large and comes from a file or another command. So you have to use the xargs command to invoke cp as many times as necessary to consume the list, without running into command line argument limits. Assuming the source list is one per line, you could do something like this:

xargs -d '\n' cp -pr -t destdir < listofsources

The -t allows the destination directory to be put before the source arguments in the cp command. Without it,  you would have to resort to the interpolation feature of xargs, i.e. -I {}

Modifying synbak to use lzop or xz for compression

For backing up a server I installed the synbak program on CentOS, partly because it was a standard package from rpmforge, and partly because it is uncomplicated to use, basically just shell scripts. I noticed that when I used gzip as the compression method, it tied up a whole CPU doing the compression. So I decided to add lzop support. It's very easy, just add these two lines to /usr/share/synbak/method/tar/tar.sh:

--- /usr/share/synbak/method/tar/tar.sh.orig    2010-06-10 05:18:14.000000000 +1000
+++ /usr/share/synbak/method/tar/tar.sh 2012-04-12 09:46:36.000000000 +1000
@@ -30,6 +30,8 @@
        tar)    backup_method_opts_default="--totals -cpf"  ; backup_name_extension="tar" ;;
        gz)     backup_method_opts_default="--totals -zcpf" ; backup_name_extension="tar.gz" ;;
        bz2)    backup_method_opts_default="--totals -jcpf" ; backup_name_extension="tar.bz2" ;;
+       lzo)    backup_method_opts_default="--use-compress-program=lzop --totals -cpf" ; backup_name_extension="tar.lzo" ;;
+       xz)     backup_method_opts_default="--use-compress-program=xz --totals -cpf" ; backup_name_extension="tar.xz" ;;
        *)      report_text extra_option_wrong && exit 1 ;;
   esac


You run it like this: synbak -s serverconfig -m tar -M lzo

I added xz support for good measure, but this is untested. Make sure that backup_method_opts in your config file doesn't have -z or the tar command will throw an error.


Naturally you must have the lzop and/or xz packages installed.


Using lzop, tar and lzop take about equal amounts of CPU time, most of the time tar was I/O bound, and backups finished faster. I also see that the memory footprint of lzop is small. The downside that is the compressed output is larger, about 31% more in my case. Decompression is supposed to be very fast, which is useful when restoring. I haven't used decompression and I hope I won't ever have to.

Tethering a Linux machine inside VirtualBox

Yes you can.


Today I needed to test an openconnect VPN connection while inside a LAN. At this site my desktop is Windows but I needed to check connectivity from outside for Linux users, using the openconnect and NetworkManager-openconnect packages.


Let's see, I could plug my smartphone into the USB port of the desktop, forward the USB connection to Fedora 16 inside VirtualBox and I should be able to connect to usb0 and I would have a WAN connection from outside. Right?


To cut to the chase, it just works.


In the VirtualBox, make sure USB forwarding is enabled in the VM settings. Plug in the smartphone and turn on USB tethering. Ignore Windows suggestions to install hardware drivers for your smartphone. When the VM is running, there will be a USB icon on the bottom bar. Choose the USB device that is your smartphone. Windows will suggest installing a VirtualBox USB driver. Do that. On Linux a usb0 device should appear in the network manager and after disabling the eth0 device which forwards to Windows, you can connect to it. You should then get a DHCP lease from your smartphone and be connected to the outside world.


It seems you have to install the VirtualBox USB driver every time the VM is started, not sure why.


This should work on other distros. For example I know openconnect works on Debian and Ubuntu. It should also work for other VPN technologies, such as openvpn. The USB network driver is called cdc-ether, by the way.

Cloning a Linux distro inside a VirtualBox VM to a real machine

Recently I tried out LinuxMint Helena LXDE inside a VirtualBox virtual machine. Very nice it was too. I decided that I wanted it installed on a real machine. Being loath to do another install on the real machine I decided to do a network clone, a technique that I have refined over time.


The key idea is to boot both machines with a rescue CD and mount their respective disks. I use SystemRescueCD because it's very up to date with all kinds of filesystems and modules. Then we create a network pipe with netcat and tar the source machine filesystem onto the target machine. I won't specify every step in detail; you have to fill in some details yourself.


First boot the source machine and make a note of the partition filesystems and sizes. In the case of LinuxMint LXDE, it's one of the standard Ubuntu setups, / in /dev/sda1 and an extended partition /dev/sda2 containing a swap in /dev/sda5. I noted the size of the swap, and that / was ext4.


Then boot the target machine. First setup the networking and note the address assigned to the network interface. Then using fdisk, create the same layout. However you can make the disk larger or smaller, so keep the swap partition the same size (I assume your VM was created with the same amount of memory as your real machine; otherwise use the 2xRAM rule) but adjust /dev/sda1 to suit. This is one advantage of this cloning method, you can increase the size of the disk. Then create the target filesystems:

mkfs.ext4 /dev/sda1
mkswap /dev/sda5

Now on both systems, mount the root filesystem and any other filesystems on it. We have only / so this will do:

mkdir /disk; mount /dev/sda1 /disk

Go to the top of the target filesystem and start a netcat listener piping into tar:

cd /disk; netcat -l 2222 | tar zxvf -

On the source machine, mount the source filesystem.

mkdir /disk; mount /dev/sda1 /disk

Go to the top of the source filesystem and start a tar feeding into netcat:

cd /disk; tar zcvf - * | netcat 192.168.1.123 2222

Subsitute the real IP address for 192.168.1.123 above. You will see the files being packed up and sent over. On the target machine you will see the files extracting.


Netcat doesn't terminate at end of input so you have to judge when there are no more files to be tar'red and interrupt it. Usually it's the last file alphabetically, something like /var/xxx or /vmlinuz, depending.


Now you have to reinstall GRUB on the target machine. LinuxMint Helena uses GRUB2, so

grub2-install --root-directory=/disk /dev/sda

You also have to adjust the UUIDs of the partitions or it will not boot. Look at the UUIDs in /disk/etc/fstab, then use tune2fs and mkswap to fix:

tune2fs -U xxxx-xxxx /dev/sda1
mkswap -U yyyy-yyyy /dev/sda5

where the xxxx-xxxx and yyyy-yyyy are the long UUIDs. I recommend cut and paste from /disk/etc/fstab to avoid keying errors.


Another thing you might want to fix up is the label of the ethernet interface. Edit /etc/udev/rules.d/70-persistent-net.rules and comment out the entry for eth0 which will be for a pcnet32, the default "hardware" for VirtualBox VMs and promote the eth1 entry to eth0. This is cosmetic since the system will work fine with eth1 as the network interface.


Other distros may require more fixing up of sound or video configuration. Fortunately LinuxMint (and Ubuntu distros in general) work out these things and adapt more or less automatically.


Now you should be able to boot the target machine. Have fun.


It is also possible to do the reverse, but as the VM is inside a private network, you have to use bridging, or ssh forwarding to pipe the tar stream into the VM. This is one way to virtualise a real install.