k3b, mkisofs, and hard linked files

As it's the end of the year, I decided to back up photos from my trips onto DVD. A du -sh of the directories showed that they would fit onto a standard DVD. But when I went to create the image in k3b, it wanted a double layer DVD.

I knew what the problem was, in those albums I had made heavy use of hard links to provide multiple access paths to photos without using extra disk space.

Checking the man page of mkisofs showed that there were a pair of options, --cache-inodes and the negative form. I added this option to the arguments passed by k3b to mkisofs and bingo, the space of the image halved, even though the preview claimed that it required almost 8GiB of DVD.

Looking at the verbose output of k3b on the terminal showed that --no-cache-inodes is the default, presumably to avoid problems with dumping filesystems that don't have working inodes.

Travails installing openSUSE on Samsung 2012 Chromebook (Snow)

I have a 2012 Chromebook made by Samsung, codenamed Snow. I like this machine as it is light, has a decent sized screen, and is suited for travel. Google stopped issuing updates to ChromeOS for this Chromebook in July 2018. So I was pleased to see that openSUSE Linux developers had issued builds for this hardware. For one thing I might be able to do things on the Chromebook not possible from ChromeOS.

So I went to the wiki page that had instructions for preparing boot media for it. First thing I had to do was enable developer mode. I lost my personalisation of the Chromebook but I didn't have much anyway, and most of the config was backed up to the cloud.

Ever optimistic, I chose the Leap KDE build image. I got it to boot and then the troubles began. I needed to get a network connection to update the packages, and this needed to be set up from the desktop. But the Plasma desktop kept crashing. Most of the time it would restart by itself after a hiatus at the desktop. But sometimes the machine froze and the only to get out was by turning off the machine by holding down the power button. On power up, sometimes it wouldn't boot far enough to show kernel messages and then it was power off again. Perhaps it was trying to repair the filesystem and I should have waited longer, but I did wait a long time. Finally I managed to download all the updates, some 600 packages in all. But by this time the machine wouldn't boot up anymore. Perhaps the CPU had overheated.

After a rest for the machine and for me, I decided I was too ambitious so I tried the JeOS (Just enough OS) image. Unfortunately this turned out to be NQeOS (Not Quite enough OS) as it didn't boot to a GUI. I might have been able to configure the WLAN interface from the command line but IHeOI (I Had enough Of It).

So I decided to try booting the LXQT image, I reasoned that perhaps without Plasma the desktop would be more stable. After quite a few tries and retries with the WLAN interface, I managed to fetch all the update packages, some 570 IIRC, and I installed them with zypper up -y from the CLI.

Then there was some to and fro with NetworkManager and wickedd. I decided to go for wickedd as this works at boot rather than when the user logs in on the desktop. But both were active after install causing some clashes.

So I have a desktop of sorts which can stay up for a while. I may keep LXQT, even though I'm used to KDE on my desktop, because this is a low RAM machine (1GB) after all.

But here's the main problem: after all that work I could not locate some key packages like a web browser, e.g. Firefox. Perhaps it's in a repo I don't know about, certainly it's not in the main repos. Looks like these builds are for people who are willing to experiment and not ready for normal usage. In retrospect perhaps I should have tried a Tumbleweed image as this might have all the updates to build date rolled in. I also see there's are experimental 15.1 builds which I might try and report back.

Other approaches I could have tried are Crouton, which keeps the kernel used by ChromeOS but runs applications over it. I don't know if this can be installed on a SD card. To the openSUSE developers' credit, getting a boot image which supports all of the peripherals I needed is a fair achievement. There is Internet documentation on how to install Debian on it but there is a lot of cross compilation and manual configuration.

The main motivation for installing a Linux distro on the Chromebook is to be able to get access to things like VPN software with a decent sized desktop and multiple terminal windows. But this motivation has faded a bit as I don't need as much to connect to my home or work machines during travels. If I do, my smartphone, with its 4GB RAM, and lots of VPN apps under Android, coupled with a Bluetooth keyboard, is much more capable, with the caveat that the display is smaller, even though the resolution is higher. So that only leaves local development when on the road as a rationale for carrying a Linux enabled Chromebook.

sizeof operator in C and C++ should be better known

Having seen some code examples on the Internet where sizeof should be used, I thought I'd offer a short reminder of a couple of useful cases:

1. A EPROM burner program to run on Arduino where the data was embedded as a byte array in the sketch which is then compiled and downloaded to run:

What was written:

const int progSize = 1024;
const byte hexProg[progSize] = { ... };
...
for (i = 0; i < progSize; i++) {
...
}

What should have been written:

const byte hexProg[] = { ... };
...
for (i = 0; i < sizeof(hexProg); i++) {
...
}

This has the advantage of automatically adapting to the number of bytes that actually need to be programmed instead of always 1024.

2. An Arduino program to test a LCD display:

What was written:

char row1[] = "I'm Arduino";
...
for (int positionCounter1 = 0; positionCounter1 < 26; positionCou
nter1++) {

This prints junk after the greeting because it strays past the end of the string, which is also a security hazard.

What should have been written:

for (int positionCounter1 = 0; positionCounter1 < sizeof(row1) - 1; positionCounter1++) {

The - 1 is to not print the NUL byte at the end of the string. A reminder, if the string is say UTF-8, sizeof gives the number of bytes in the string, not the number of characters. But usually the number of bytes is what you want.

Also do not make this mistake:

char *row1 = "I'm Arduino";

In this case sizeof(row1) will be sizeof(char *) on the machine, not the length of the phrase I'm Arduino.

A couple of other idioms.

Getting the number of elements in an array:

int divisors[] = { 2273, 2145, 2025, 1911, ... };
const int ndivisors = sizeof(divisors)/sizeof(int);

Or even better:

const int ndivisors = sizeof(divisors)/sizeof(divisors[0]);

which means you can change the type of the divisors array to be other than an int without changing the second line.

Allocating the right amount of memory from heap:

double *readings;

...

readings = malloc(NREADINGS * sizeof(double));

And a reminder, sizeof is a compile-time operator, so there is no runtime cost to using it. If the expression provided to sizeof cannot be evaluated at compile-time, the compiler will tell you.

A comparison of the TM1637 protocol with I2C protocol

The TM1637 is a Chinese made IC designed to drive up to 6 7-segment LEDs. It offers serial data input so that only 4 wires are need to connect to the display: POWER, GND, CLOCK and DATA. In fact it can also scan a keyboard matrix and return the key status to the controller. It can be found on integrated 4 digit displays available on eBay from Chinese sellers for a dollar or two. It follows on from the trend by MAXIM display chips serving similar functions. It's an attractive component if your microcontroller project needs a display and you don't want to use up lots of lines, drivers and resistors driving LED digits yourself, not to mention the coding of the multiplexing. A display board incorporating this controller costs only a little more than the displays on it.

The serial protocol has already been explained in this blog post. It's fairly easy to implement in the language of your choice, provided you can do low level bit-banging. This post is to explain the differences between and I2C protocol and the custom protocol designed by Titan Micro Electroncs, the manufacturer. TM explicitly says their protocol (call it TMP) is not compatible with I2C, but it's easy to see it has similarities. You can see that TM pared the protocol down to only what they needed.

These are the major differences:

• I2C has addresses so a master can drive multiple slaves. TMP doesn't have this. So I2C is a bus and TMP is point-to-point. This means each display needs two lines. No daisy-chaining either. This also means TM didn't have to pay to get a slave address assigned to their chip. BTW I2C assigns the lowest bit in the address byte to indicate the direction of the data flow. Alternatively you may regard the slave as having two consecutive addresses.
• I2C specifies that after the address comes the register address within the slave. TMP is flatter, you send a command followed by any data expected.
• I2C sends the most significant bit first, TMP sends the least significant bit first. This only shows up in the low-level bit-banging.
• I2C has the concept of clock-stretching. TMP has no way for the slave to tell the master to wait. It isn't needed as the chip will respond within the specified time. There is also no need to worry about a pull-up resistor as a result.
• I2C specifies various clock speed tiers. TMP only implements their speed, a max of about 250kHz, which is fast enough.
• As it's a custom protocol you won't be able to use any silicon or firmware assists for I2C that may be present.

Here's a project of mine that uses a display board with this controller.

Login to standalone Samba server from Windows 7 (and above)

At a workplace we have a server in the DMZ running Samba and other things and therefore not joined to the Windows domain. It has a small set of local users (no, no, we would never share a CIFS over the Internet) whose passwords are managed with smbpasswd. When a user tries to connect to a Samba share, the Windows desktop "helpfully" prefills the login name as domain\username. The login fails because the Samba server is not in the domain. This did not happen in XP.

Extremely simple solution, opt to login as a different user and use:

\username

to login. For some reason this is not widely known. NET USE will also work but then the user has to use a COMMAND window.

It works for me and it also worked for this person.

Avoid calling basename or dirname by using parameter expansion

These short cuts have been documented under parameter expansion for a long time but are not well known because people are used to writing basename and dirname.

Instead of using:

file=$(basename "$path")

use

file=${path##*/}

Instead of using:

dir=$(dirname "$path")

use

dir=${path%/*}

The advantage is not calling basename as an extra process. Although basename might be implemented as a builtin in some shells.

See the linked page for further short cuts.

Chrome/Chromium is storing the url and referrer of downloaded files in extended attributes

When I turned on extended attributes for a rsync command (-X option) I noticed that it proposed to copy attributes for a bunch of files it had not before. Most of those files appeared to be PDFs or zip files. So I wondered what those attributes were. Here is an example:

$ getfattr -d work/uC/asm7000.zip
# file: work/uC/asm7000.zip
user.xdg.origin.url="http://cd.textfiles.com/emspro1/ASMUTIL/ASM7000.ZIP"
user.xdg.referrer.url="http://cd.textfiles.com/emspro1/ASMUTIL/"

Oh! That's not good! The origin of this file is harmless, but what if it had been a file downloaded from a sensitive location or the URL contained sensitive parameters. I may want to give the file to somebody without revealing the URL I got it from, which may contain side channel information. You can imagine your own scenario.

I should add this is only an issue for you if you have enabled extended attributes on your filesystem and mounted with that option. Look for the option user_xattr in /etc/fstab lines. But maybe Android does enable extended attributes. That would be a big worry; your phone may be retaining information you didn't know about.

Doing a search found this recent Debian bug report with no resolution. Also these attributes are listed as in current use by freedesktop.org. And it looks like wget too stores the url. Update 2019-03-18: I have been informed that wget from 1.20.1 has options to control storage of the attributes.

This really should be behaviour that can be turned off in Chrome/Chromium. Until then, I'm not backing up extended attributes for my own files. You should also screen extended attributes in files you transfer to other systems.

It does have one use though. If you forgot where you got a file from, you can recover the URL.

Addendum: Here are some situations on Linux showing whether the extended attributes will be transferred and potentially revealed to others.

• Email as attachment: Extended attributes are not included. But see archive files further on.
• Zip archive: Extended attributes are not captured.
• Tar archive: Extended attributes could be copied, depending on command line options and the compiled default. It seems to be not on by default on my system (openSUSE).
• Transfer on DOS or Windows filesystem, e.g. flash storage: As far as I can tell these extended attributes are not stored on FAT or NTFS.
• Transfer on CD/DVD: ISO9660 can store extended attributes but mkisofs doesn't seem to store these despite mention of GETXATTR and SETXATTR error message suppression. I haven't found information on UDF regarding extended attributes yet.
• Transfer on external storage with Linux filesystem: This depends on whether the filesystem has user_xattr enabled and it is mounted with this enabled, and the copy command transfers it. cp -p doesn't but cp -a does. rsync without -X doesn't.
• Uploaded to the cloud: This would depend on the client doing the upload. Also see here.

More testing is required. Send additions and corrections by commenting on this blog post.

The true origin of "the NUXI Problem"

A bit of computer history today.

The Cathedral and Bazaar website defines the NUXI Problem and various other web pages have elaborated on how it is due to endianess of the CPU interacting with data transfer.

All well and good, but the name given to the problem dates from before data transfer. In those days, telecommunications was slow and data transfer was by magnetic tape. So it wasn't due to encountering problems with FTP or something like that, Internet Protocol had not come to Unix yet.

Rather it was named after the glitch observed on the console upon booting up a port of Unix V6 to, my unreliable memory tells me, the Interdata 7/32. The ports were done independently at two places, Bell Labs and University of Woolongong. Here is the story from the veteran of the UW port.

The PDP/11 is little-endian and the recipient of the port was big-endian so when Unix V6 booted up, the message on the console was supposed to be:

UNIX V6...

which came out as:

NUXI...

Obviously a fix had to be applied to the order of bytes within words in compiler generated code for C strings.

I don't know which team encountered this symptom first. The tale was told to me by Piers Lauder, the Unix guru at the time at Basser Dept. of CS at the U of Sydney.

Ancient EPROMS: 2516, 2716, 2532, 2732

I needed to burn a 16kb (2kB) EPROM as part of a restoration project on an old piece of equipment. When I couldn't burn the EPROMs on my TL-866 programmer, I started investigating. Here's what I found out, summarised for your benefit.

Programming voltage

Practically all EPROMs of this era require 25V programming voltage. That's why my TL-866 couldn't do it, it failed on the first byte, it can only go up to 21V. I can understand why it's not supported, only a tiny fraction of chips require 25V programming. Maybe your programmer supports 25V. However some parts suffixed with A are 21V programmable so try that voltage first. I didn't have any A parts.

Pinouts

This article goes into detail about the differences. To summarise:

For the 2716, the Texas Instruments equivalent is 2516. The TI 2716 is a different beast, fortunately quite rare.

For the 2732, this time TI came out with the 2532, with a different pinout from the more widespread 2732. So TI 2532 != other 2732. Again my programmer cannot program 2532s, not just insufficient voltage but different algorithm too. If you need to only read them, say you want to put 2732 EPROMs in a machine using 2532 EPROMs, it's possible to make an adaptor that reroutes three pins. The article is only for subscribers, but if you click on the PDF image of the PCB, it will be obvious what they have done to the three pins 18, 20 and 21.

From the 2764 onwards, TI fell in line.

EEPROM substitutes

So I had no programmable EPROMs in my spares box, but fortunately I had a Xicor 2816 which is an EEPROM with pinout compatible with the 2716. It speeds up my development too as I don't need to go through the UV erase step.

28pin to 24 pin adaptor

If you can't get hold of a suitable chip or programmer, another way out is to build an adaptor using two IC sockets and patch wire so that you can use the much more widespread 28 pin EPROMs (2764 and above). Fortunately I don't have to do this (yet).

pdftk cannot open some protected PDFs

I receive some documents as protected PDFs. I got tired of inputting the password every time I viewed them with okular so I decided to batch unprotect all the ones I have. This is on a computer which only I use.

I tried the recommended pdftk command:

pdftk protected.pdf input_pw secret output plain.pdf

but it kept telling me OWNER PASSWORD REQUIRED. Eventually I decided it was pdftk's fault, not mine, so I switched to qpdf, using the command:

qpdf --password=secret --decrypt protected.pdf plain.pdf

and that worked.

A web search showed various complaints about this. I haven't investigated when or why the problem occurs.

Perl complaining about not being able to set locale?

If you are getting this error when running Perl, usually from a ssh session:

perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
       LANGUAGE = (unset),
       LC_ALL = (unset),
       LC_CTYPE = "en_US.UTF-8",
       LANG = "en_AU.UTF-8"
   are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_AU.UTF-8").

You are missing the locale mentioned in LC_CTYPE. Run:

dpkg-reconfigure locales

select the missing locale, in this case en_US.UTF-8 and generate it. Voilà, error fixed.

crond runs /bin/sh not /bin/bash

Got caught by this today. Cron jobs normally run /bin/sh, not /bin/bash. It's probably required POSIX behaviour.

Normally most commands run out of cron are not affected because they are not dependent on bash, or the shell script starts with #!/bin/bash. However commands like vncserver which start up desktops (e.g. with the @reboot time spec) can be sensitive to the shell used because of environment scripts in /etc/profile.d.

Therefore if you want to ensure that the cron job works the same whether run interactively or from a crontab, specify the shell. E.g.

@reboot SHELL=/bin/bash /usr/bin/vncserver ...

VM in VirtualBox using unbound WiFi interface on host

I have a Linux host with 2 interfaces, a wired gigabit Ethernet and WiFi. Currently only the wired interface is connected. I wanted to experiment using WiFi interface by a VM guest in VirtualBox. There are various scenarios for this. I might want to test network routing by using the WiFi in conjunction with a smartphone offering tethering. Or access the Internet using from a VM on the desktop using the smartphone when the broadband connection is down.

The short answer is yes, this is eminently possible. The key points to take away are that, 1. the host has to configure the WiFi interface, i.e. load the appropriate driver and authenticate to the access point but not assign an IP address to it, and 2. the guest has to use bridging to access the host's WiFi interface. In fact it will appear to be a wired interface to the guest. All the usual IP configuration methods are available, static or DHCP, depending on what the access point allows. Also note that the MAC address seen by the gateway will be the MAC of the guest, not the MAC of the WiFi interface.

XFS partition created under CentOS 7 cannot be mounted on CentOS 6

Found this out the hard way. Error message is invalid FS type, i.e. not backward compatible. You have to backup the contents and restore to a partition created under CentOS 6.

Deploying SSSD for authentication against Windows AD in CentOS 6

In an earlier post I discussed in general terms deploying SSSD on CentOS 7 which is fairly straightforward once the right choices have been made.

I have now successfully deployed it on CentOS 6. This blog post by one of the authors of SSSD gave me the initial clues. The key point is that while sssd and associated packages are available on CentOS 6, realmd is not so you have to do those steps manually using adcli.

There was one big problem though, the join command didn't work for me! When it came to that step I got an error message saying I had insufficient permissions to modify the computer entry in AD. I tried various permutations but still got the same error.

Finally I decided to examine the progress messages of realm -v join on a CentOS 7 machine to see what it was doing under the hood. It turned out to be a command of this form:

net -s /tmp/sssd-smb.tmpfile ads join example.com

So it was using the Samba net command, not adcli for joining!

The temporary file providing the settings to net was of the form:

[global]
workgroup = EXAMPLE
netbios name = MYWORKSTATION-N
realm = EXAMPLE.COM
kerberos method = system keytab
security = ads

Note that the netbios name should be truncated to 15 characters. The reason net is used is explained in the last comment of this bug report.

Now I don't know if realm switched to calling net when it detected that adcli would not work, or uses net since that blog post was written, to satisfy servers that don't allow creation/modification of computer entries with LDAP. I think the latter because realmd comes from the CentOS repo while adcli comes from EPEL repo so can't be a dependency. I suppose I could look at the code.

The next command that realm runs is:

net -s /tmp/sssd-smb.tmpfile ads keytab create

Once that is done, you will be able to run klist -k and get output. Then follow the rest of the blog post to set up sssd. If you were running /etc/passwd authentication, remember to delete those entries in passwd, group, shadow, and gshadow so that they don't mask the user entries from AD. Restart nscd to flush any cached entries afterwards. In the long term you want to follow RedHat's recommendations referred to in my CentOS 7 blog post to disable nscd caching for user and group entries.

Software whose authors won't update to keep up with advances

Now and then I am frustrated by software that doesn't implement modern functionality. So I'm creating a dynamic list here of the ones I've come across, if only to get the annoyance off my chest. If subsequent changes prove me wrong, let me know in the comments and I'll be more than happy to remove the software from this list.

Fetchmail

Google keeps nagging me to disable less secure apps for my accounts. But I can't do this yet because fetchmail doesn't implement OAuth2. Apparently somebody has submitted changes to implement this but it isn't in the mainline code. I've also started looking at the Perl script fetchgmail. This however requires a lot of Perl modules to support so it was a chore to find the ready-made packages for my distro and use CPAN for the rest. But I may switch to this.

Addendum: A solution to this is to get an application specific password from Gmail settings just for fetchmail. In this case you are not exposing your login password in fetchmail config files.

Filezilla

Filezilla is a useful tool for sftp between hosts. At my workplace the Linux hosts are joined to the domain, and so support gssapi-with-mic from Windows workstations. Quite handy since the user is already authenticated to the domain. Openssh has supported gssapi-with-mic for quite a while, as has Putty. Again someone has submitted changes to Filezilla to implement this feature, but this has not been taken up.

Experimenting with ext4 encryption on openSUSE Leap 42.3

I decided to encrypt some of my directories both on internal hard disks and external hard disks. One of the reasons for encrypting on an external hard disk is so that I can keep an offsite copy of my files without worrying that somebody who gets hold of the disk can read the files.

The best guide I found was the ArchWiki one. It's succinct but all that is essential is there.

For openSUSE, the kernel is recent enough but the stock e2fsprogs 1.42 package is not. You'll need to download 1.43 sources and compile. These utilities must be the 1.43 version: tune2fs (for enabling the encrypt feature), e2fsck (to deal with the new feature), and of course e4crypt (for adding a key to the kernel keychain). The other utilities can be the stock ones.

A short word about keys. When you add a key with e4crypt, it will attempt to use the key with all filesystems for which the password matches. So if you have been using the same password on several filesystems, all the matching ones will be unlocked.

If you have unlocked a directory and then delete the key from the keychain with keyctl, the encrypted directories on the mounted filesystem remain unlocked. Access will be lost only when you umount.

When assigning policies to directories, you need to use the one for the filesystem it resides on. To avoid confusion if possible you should clear the kernel keychain, and then e4crypt add_key for the target filesystem, then use the corresponding descriptor. Otherwise if you use the descriptor for another filesystem by accident, you will not be able to read the directory when the other filesystem has not been unlocked.

These commands of keyctl will be of most use: show, clear, new_session. You might also want to specify the keyring. Most likely it's @us.

Reinstating the nouveau driver after having installed the NVidia driver on CentOS 7

Due to the NVidia proprietary driver not working with 3D acceleration on VNC, which meant that users had to use MATE (GNOME 2) on VNC sessions, and couldn't use vino-server for the :0 session if the console desktop was GNOME 3, we decided to try the nouveau driver instead.

The steps for removing the NVidia driver are in this post for Fedora. For CentOS 7 the corresponding steps are:

nvidia-uninstall
yum install -y xorg-x11-drv-nouveau (just in case it wasn't installed)
rm -f /etc/modprobe.d/nvidia-installer-disable-nouveau.conf
rm -f /usr/lib/modprobe.d/nvidia-installer-disable-nouveau.conf

dracut -f

systemctl reboot

Note the removal of the second occurrence of the modprobe blacklist file created by the NVidia installer. Without removing this, the blacklist is still in place in the initramfs.

The symptom was that the nouveau driver was loaded only when the X server was started, not at boot, which caused it to not detect anything but one monitor and only one mode setting, as the output of xrandr showed.

How to send email that will expire (Outlook only)

I get notification email from Linux cron jobs. I wanted some way to automatically mark mail as no longer relevant after a week. I recently discovered that there are a couple of mail headers defined for this purpose in RFCs: Expires: and the older, superseded Expiry-date: which is what Outlook uses. They are not honoured by most mail readers though so this tip is specific to Outlook.

I discovered sample code from Internet search. Here is my version in Perl which does the job.


use Net::SMTP;
...
my $expire_time = strftime "%d %b %y %T %Z", localtime(time() + 86400 * 7);
...

$smtp = Net::SMTP->new($mailserver);

...

$smtp->datasend("Expires: $expire_time\n");

$smtp->datasend("Expiry-date: $expire_time\n");

The Net::SMTP module allows one to send arbitrary headers. You can see the required format of the date time string from the strftime() call.

Expired emails are not deleted, but are displayed in desktop Outlook with strikethrough. No effect that I know of in other mail readers, not even the Internet Office 365.