Monday, 23 February 2026

Reliably recording Internet radio programs

In a previous blog, I've explained how to use mplayer to dump Internet radio programs to a file.

In practice the recording sometimes terminated before the time was up, which could be due to the server breaking the connection, Internet glitches and so forth. Interactive station tuners will attempt to reconnect. We want to do the same with our mplayer scripts.

Here is a solution I came up with, which I will explain below the code. Some code before the main code has been elided. Assume that the bash script is called with 3 arguments, the duration, the destination file, and the streaming URL.

#!/bin/bash

pipe="/run/user/$(id -u)/mplayerpipe$$"
READER_PID=''
MIN_DURATION=30
STRETCH_TIME=15         # seems to lose time when a new segment is started

cleanup() {
       rm -f "$pipe"
       # don't use builtin kill so that we can send to process group
       [[ -n "$READER_PID" ]] && echo "Killing group $READER_PID" && /usr/bin/kill -TERM -- "$READER_PID" $(pgrep -f "cat $pipe")
}

if [[ "$1" -lt "$MIN_DURATION" ]]
then
       echo Minimum duration "$MIN_DURATION" seconds
       exit 1
fi
trap cleanup EXIT SIGINT SIGTERM
mkfifo -m 600 "$pipe"
coproc READER { exec bash -c "while [[ -p '$pipe' ]]; do cat '$pipe'; echo 'Restart reader' 1>&2; done" > "$2"; }
now=$(date +%s)
later=$((now + $1))
left=$((later - now))
while [[ "$left" -ge "$MIN_DURATION" ]]
do
       # echo and sleep for testing
       echo mplayer -prefer-ipv4 -noconsolecontrols -slave -vo null -vc null -endpos "$left" -dumpaudio -dumpfile "$pipe" -really-quiet -loop 0 "$stream"
       # sleep 10
       mplayer -prefer-ipv4 -noconsolecontrols -slave -vo null -vc null -endpos "$left" -dumpaudio -dumpfile "$pipe" -really-quiet -loop 0 "$stream"
       now=$(date +%s)
       left=$((later - now + STRETCH_TIME))
done
sleep 2

The idea is we establish a named pipe in the standard location /run/user/<uid>/and mplayer writes to this pipe. At the same time we start a coprocess which is a bash inline script calling cat repeatedly to append to the destination file. Coprocesses are explained in the bash man page. Here we call it READER and its pid is put in READER_PID by convention. Note that we exec the bash coprocess so that the pid is that of the code fragment, not the bash parent.

When mplayer terminates prematurely, the amount of time left to go is computed and it's restarted with that duration. The cat process will receive EOF, and new one is started to continue to read from the pipe.

We need to clean up the named pipe at the end of the program so we install a signal handler cleanup() to do that. It removes the named pipe, and kills the group pid, which the kill command says can be done by prepending - to the pid (making it negative). Note that for this behaviour we use the kill command in /usr/bin/kill, not the bash builtin. However I couldn't get the group kill to work so I settled for using pgrep to find the cat process to get the pid.

You may ask why not try to trick mplayer into appending to a file by >> output redirection, then passing /dev/stdout as the dumpfile argument. It doesn't work. /dev/stdout is a symlink to the real file and mplayer will open it in overwrite not append mode.

After I wrote that script, I decided to try another tack. The script relies on Linux named pipes, so isn't cross-platform.

Here is the second attempt, which is a Python program. I'll show only the main routine, the needed auxiliary definitions and functions can be inferred.

def main():
   """ Main program """

   mplayer_cmd = sys.argv.copy()
   mplayer_cmd[0] = MPLAYER

   # find index of -dumpfile argument
   dumpfile_pos = find_index_of("-dumpfile", mplayer_cmd)
   if dumpfile_pos < 0:
       sys.exit("-dumpfile argument not found")
   dumpfile_name = mplayer_cmd[dumpfile_pos]
   insert_pos = dot_index(dumpfile_name)
   base = dumpfile_name[0:insert_pos]
   tail = dumpfile_name[insert_pos:]
   seq = 'aa'
   mplayer_cmd[dumpfile_pos] = first_name = base + seq + tail

   # find index of -endpos argument
   end_pos = find_index_of("-endpos", mplayer_cmd)
   if end_pos < 0:
       sys.exit("-endpos argument not found")
   if not mplayer_cmd[end_pos].isdigit():
       sys.exit("End posiiton not a number")

   left = int(mplayer_cmd[end_pos])  # get original endpos
   now = time.time()
   later = now + left                  # when we should stop
   # substitute endpos
   mplayer_cmd[end_pos] = str(left)
   while left >= MIN_DURATION:
       print(f'{" ".join(mplayer_cmd)}')
       before = time.time()
       # for debugging just sleep
       if TESTING:
           time.sleep(10)
       else:
           result = subprocess.run(mplayer_cmd, check=False)
           if DEBUG:
               print(result)

       if seq == 'zz':
           print("Can't handle more than 676 segments")
           break

       now = time.time()
       if bad_session(mplayer_cmd[dumpfile_pos], seq, before, now):
           continue

       # set up for next session
       seq = next_segment(seq)
       # substitute dumpfile
       mplayer_cmd[dumpfile_pos] = dump_name(base, seq, tail)
       # substitute endpos
       left = round(later - now) + STRETCH_TIME
       mplayer_cmd[end_pos] = str(left)

   # gather all the segments
   # rename first segment
   if DEBUG:
       print(f'{first_name} -> {dumpfile_name}')
   try:
       os.replace(first_name, dumpfile_name)
   except FileNotFoundError:
       print(f'Warning: {first_name} not found')
   # append remaining segments
   if seq > 'ab':
       append_segments(base, seq, tail, dumpfile_name)


if __name__ == '__main__':
   main()
We write a series of files called <basename>aa.aac to <basename>zz.aac and at the end of the program we rename the first one <basename>.aac and append all the others to it. Up to 676 segments are allowed. Hopefully you will not have so many connection breaks. All the string manipulations are to find the endpos and dumpfile arguments in the original invocation and replace those with new values for each run of mplayer. Thus this script which I called mplayer_retry is a plugin replacement for the original mplayer invocation.

This script has the advantage of not requiring more than standard file operations, nor named pipes, so is cross-platform.

Unfortunately it seems that in practice, some seconds of program are lost when the connection breaks. Only thing for it is to complain to the service provider to have more a robust streaming service. 

Monday, 3 November 2025

E Unibus Unix

Today I was reminded of what I think might have been a fortune cookie from Bell Labs Unix: E Unibus Unix. Unibus was the backplane bus of the DEC PDP-11 series of minicomputers which was the main platform for Unix development for many years.

It's of course a play on the motto E pluribus unum (Out of many, one) which is on the great seal of the USA. We won't go into if that motto is still valid these days. 

Thursday, 2 January 2025

Installing mplayer on dietpi on Raspberry Pi

In a previous article, I explained how to use mplayer to record Internet radio broadcast non-interactively.

I got it working on my workhorse PC, but it suffered from using 100% of one core as mentioned in that article. I have 12 cores so this wasn't a disaster. But I thought I could shove the job to a less important computer and also have a backup means of recording. I have a very old Raspberry Pi 2B which was idle.

I tried installing the latest Raspberry Pi OS, but I couldn't write the entire image on the micro SD card. I think the USB to micro SD adaptor got too hot and caused sector errors towards the end. Maybe I should have limited the writing rate. Anyway I decided to use a lighter RPi distro: dietpi.

This installed and booted up fine. I had an issue with the default NTP server until I specified a regional pool. Then I encountered a series of problems:

Apt repos need to be signed

It couldn't update from the default Debian Bookworm archive because the signing key wasn't present. Normally this is provided by the distro but since this is dietpi they only provided keys for the repos they used. Or they provided an old key.

Cut to the chase: Install all the relevant Bookworm repos, you can find a definitive list of them at at various sites. If possible use a local mirror for the repos. Don't forget bookworm-security, but this will come from security.debian.org, not a mirror.

When you do an apt update it will complain about various unsigned repos. Note down the key fingerprints for the next stage.

Apt-key is deprecated

Ignore any tutorials that talk about using apt-key to install the required keys. For security reasons, apt-key is deprecated. The new way of doing it is:

Download the GPG keys for all the repos missing keys. You'll need to find a suitable keyserver with the Debian keys.

Feed each through gpg to dearmor the keys and write the output to a suitably named file in /etc/apt/trusted.gpg.d/ Here's what I have:

root@DietPi:/etc/apt/trusted.gpg.d# ls
bookworm-security.gpg       debian-bookworm-archive.gpg  dietpi.asc
bullseye-security.gpg       debian-bookworm-stable.gpg   raspberrypi-archive-stable.gpg
deb-multimedia-keyring.asc  debian-bullseye-archive.gpg  raspbian-archive-keyring.gpg

I haven't given the details to avoid duplication and because I could have forgotten some bits. You can find them in up-to-date tutorials.

You need Deb multimedia

Mplayer uses some codecs that are not supplied in Debian, so you have to get them from Deb Multimedia. Use a mirror if you can. You need to install the signing key for this in the same manner shown above.

Finally install mplayer

Do an apt update and then apt install mplayer. If you have any issues at the update step, fix those. I came across issues like mirrors no longer existing, or didn't specify their Debian repo domain in their list of alternate domain names which caused failure on verification.

Also any other packages with problems could block the installation. For example I had issues with libgomp1 where it had a spurious dependency. I actually hacked /var/lib/dpkg/status with a text editor to bypass this.

What made this exercise worthwhile

When I use mplayer on the RPi to record an Internet radio station I was surprised to find that it didn't eat up 100% of a core like on my workstation. So I ran a strace on mplayer on my workstation and saw that it was looping on reading file descriptor 0 (stdin). Recalling that mplayer reads single keystrokes from the controlling terminal to control the playback, I reasoned that it must be doing that in non-interactive mode and looping on failure. So I found the -noconsolecontrols and -slave options to mplayer and adding these to the command made the CPU usage normal again.

Recording Internet radio with mplayer

It's not widely known, but mplayer can be used to listen to Internet radio stations.

If you just want to listen, my recommendation is to install pyradio which is a curses based command player. For a widget I can recommend radiotray-ng. For Plasma desktops there is plasma5-radiotray. They use mplayer and other programs like vlc to do the heavy lifting.

But the subject of this blog article is recording, and I usually do this from a cronjob or crontab entry for unattended recording of periodic programs. Cutting to the chase, this is the command line you need, with explanations below.

mplayer -prefer-ipv4 -noconsolecontrols -slave -vo null -vc null -endpos "$1" -dumpaudio -dumpfile "$2" -really-quiet -profile pyradio "$stream"

-prefer-ipv4 is because I have a DNS client that returns IPv6 entries but I have only IPv4 connectivity

-noconsolecontrols -slave prevent mplayer from reading for commands and polling for single keypresses for commands. I think only the first is needed, but the second can't hurt. In non-interactive mode, there is no terminal and mplayer goes into a busy loop, using up 100% of a CPU core

-vo null -vc null disable the video output and codecs

-endpos is followed by the number of seconds to record. It's the first argument to the shell script this command is in

-dumpaudio -dumpfile are followed by the file to write the raw audio data to, typically it's AAC format. It's the second argument to the shell script

-really-quiet suppresses pretty much all messages

-profile pyradio specifies a profile in ~/.mplayer/config. It consists of this stanza:

[pyradio]
softvol=1
softvol-max=300
volstep=1
volume=80

"$stream" is the URL the station broadcasts on. A site like https://streamurl.link/ could be useful for finding this for the station you are interested in.

Some stations use a playlist URL, in which case "$stream" should be replaced by -playlist "$playlist"

I've found that typically the audio data is about 8 kB/s for AAC.

Wednesday, 26 June 2024

An interesting anomaly in my car player re MP3 and AAC

I discovered by accident that my 10-year old car's entertainment system can accept .aac suffixed files on USB flash memory sticks to play. But when I tried to play an ISO9660 data CD containing AAC files instead of MP3 files, it said it could not find any MP3 files on the CD.

Since it's the same entertainment unit which also accepts input from Bluetooth, and analog AUX 3.5 mm stereo jack for a total of 4 input sources, it seems strange that it can handle AAC files, but only from the USB flash memory.

I tried naming the files suffixed as .m4a. No joy, still could not find any MP3 files on the CD.

Ok, I'll try to fool it. I renamed the AAC files to have .mp3 suffix. Now it doesn't complain that there are no MP3 files, but regards them as invalid, skipping through them without playing.

From this I infer that there are at least two decoder paths, the one for the CD drive that can only play MP3 files, and the one for the USB flash memory that can play both MP3 and AAC.

Incidentally I think this might be the last car player I own that will play CDs. For my next car I'll probably play from USB flash memory, or from my phone via Bluetooth. These days when you mention CDs to people below a certain age, they go: what?

Saturday, 6 January 2024

Clever function names

It came back to me today that in the language that influenced Python, ABC, developed at the CWI to teach programming, there were 2 string operations described as behead and curtail. I suppose the person who thought up the verbs was chuffed. They were probably too bloody for general consumption so these days in programming languages other verbs are used to describe the operations, or means like string slicing are used.

Sunday, 5 November 2023

Excluding devices from pipewire control

In my previous blog post I described how I used pipewire to add a bluetooth dongle to the audio outputs of my computer.

Unfortunately activating pipewire had the result of it taking control of all the sound interfaces on my computer. This caused a problem with a line-in port which I use for recording audio at scheduled times. It turns out that now and then pipewire will reset the port for maximum gain, wrecking the recording.

I searched for how to exclude this port from pipewire's influence and this was the best answer. I discovered the PCI bus id of the sound card (actually onboard sound device) in question and wrote an override script just like the one described with my card's id. Unfortunately the whole device has to be disabled, not just the line-in port, but that's ok for me. I still have an HDMI port which I can connect to my amplifier instead of using the analog line-out port. So thanks to the author of that ZenLinux blog for the insight. It's a pity such a simple task has to be so complicated, maybe future wireplumber developments will make this simpler.

Monday, 11 September 2023

Getting a bluetooth capable amplifier working with a bluetooth USB dongle on Linux

A while back I bought one of those $10 tiny class D amplifiers that can drive my bookshelf speakers with up to 30W (depending on power supply voltage, between 12V and 24V). An old laptop supply supplies this. It can be fed from a 3.5 audio socket or from bluetooth. Works like a charm. May not be audiophile but good enough for me.

I've been feeding it from an older mobile phone that mounts a SMB share from my workhorse computer. But as I would like to automate it so that I can use music to wake me up, I bought a bluetooth USB dongle from one of many AliExpress sellers. It contains a Realtek chipset that I checked is supported in Linux and supports BT 5.3 profile.

How to set the dongle up? Plugging it in elicited messages in the syslog, as dmesg showed. I used the KDE desktop widget (probably bluedevil) to connect to it and command it to pair and connect to amplifier. A prerequisite is that bluetoothd should be running. I discoved that the configuration can also be done from the CLI using bluez-tools. But once discovered and paired, nothing more needs to be done on this front as the setup is static.

How to send audio to it? I thought there might be a PCM device under /dev like for sound cards. But no. It seems that you need a service like pulseaudio to drive it. So I got stuck into pulseaudio documentation. Along the way I discovered that my distro prefers pipewire. So I installed the pipewire packages and this also obsoleted the corresponding pulseaudio packages. One of the packages pipewire-pulseaudio supplies compatible functionality.

Pipewire is a process that should keep running to handle all this. How to fire it off? It turns out that it's preferable to run it with user permissions, using the user instance of systemd, e.g.

systemctl --user start pipewire.service

How to make sure it runs when the user logs in? You activate the systemd units like this:

systemctl --user enable pipewire.service pipewire.socket

Then start both the service and the socket the first time.

How to send audio to pipewire? I struggled with this under pulseaudio, it wouldn't show up in players like vlc or amarok. But once I switched to pipewire, the bluetooth audio sink was just there and I could control the volume and make it the default audio playback.

How to specify this device to mplayer so that I can play from a cron job? I just had to specify it to the -ao option.

mplayer -ao pulse audio.mp3

Notice that mplayer sees pipewire the same as pulseaudio. I can put this in a profile in .mplayer/config named after the amplifier ID:

[xinyi]
ao=pulse

then I can specify:

mplayer -profile xinyi audio.mp3

The advantage is that I can specify other settings in the profile.

Wednesday, 6 September 2023

Pay attention to the size of ipsets for nftables

I was preparing an ipset for blocking access by geographical region using firewalld. Firewalld can work with either the traditional iptables or the newer nftables.

I created the ipset file using this Makefile stanza:

%.load: ipset_%.txt
        -firewall-cmd --permanent --delete-ipset=$(@:.load=)
        firewall-cmd --permanent --new-ipset=$(@:.load=) --type=hash:net
        firewall-cmd --permanent --ipset=$(@:.load=) --add-entries-from-file=$<
        systemctl reload firewalld

Here % is replaced by the two letter country code for the region I want to allow.

In the public zone I have a rich rule which inverts the test. This is the result of the firewall-cmd I issued:

  <rule family="ipv4">
    <source ipset="ok" invert="True"/>
    <service name="theservice"/>
    <log/>
    <drop/>
  </rule>

Later I discovered that that the countries not in ok weren't blocked. Troubleshooting found these symptoms:

  • firewall-cmd --state returned failed, despite systemctl stating that the service had started
  • There was a message about python-nftables failing in systemd logs
  • nft list showed nothing in the chains or rulesets

Searching on these symptoms got hits but none of the solutions worked. The only thing I learnt is that firewalld calls the python3-nftables routines to manipulate the nftables. I thought it might be the top line in the Python script which read #!/usr/bin/python which invokes python2 on my system, but adding 3 made no difference presumably because firewalld calls it with /usr/bin/python3 <name of script>. Someone suggested this python3 module was broken in a particular distro release because it didn't include a needed flag so he went back to iptables as the backend. I didn't want to do this. Also the report was for a previous distro release so over a year old.

Playing around with ipset which deals directly with the kernel to create the sets manually revealed that the default maxelem of 65336 was not sufficient for my set. After some experimentation I figured that I had to increase this parameter at creation. This boiled down to including this final option on the firewall-cmd line:

firewall-cmd --permanent --new-ipset=$(@:.load=) --type=hash:net --option=maxelem=262144

Now restarting firewalld took a bit longer due to having to digest lots of entries, and there was no more failure of python-nftables. Inspecting the ruleset using nft worked and showed that the nftables rule was in place.

It would have been easier if I had understood the relationship of all the components of this setup better, but things are obvious in hindsight.

Wednesday, 12 July 2023

Prefer IPv4 for zypper updates

I had been battling server timeouts from a local mirror of OpenSUSE repositories when I noticed that if I telneted to the troublesome server it attempted to contact it via its IPv6 address. My Internet retailer doesn't support IPv6 connections, although my Linux distro and modem are capable. At the moment there's no need as there is no shortage of IPv4 addresses in this country.

A little search found that it's due to the curl library trying IPv6 addresses first. There is a little known environment variable in libzypp to control this. I put

ZYPP_MEDIA_CURL_IPRESOLVE=4

in my environment settings and I no longer had server timeouts.

If you are using sudo to run zypper, look into the env_keep option  in the sudoers file, and check that the env_reset option is enabled (the default), for example:

Defaults env_keep += "ZYPP_MEDIA_CURL_IPRESOLVE"



 

Sunday, 2 April 2023

Displaying problematic characters for Windows like the colon in Samba shares

I have a Samba share which contains music files which I want to play on my tablet (but could also be a Windows desktop). The colon is perfectly valid in Linux filenames and is used a lot in MP3 filenames. On the client side it falls back to DOS mapping because the colon is not legal in Windows filenames. So it's impossible to see what the song is. Here is how you can make the displayed names more readable.

Add these lines to smb.conf, either in the global or a share section:

vfs objects = catia 

catia:mappings = 0x22:0xa8,0x2a:0xa4,0x2f:0xf8,0x3a:0xf7,0x3c:0xab,0x3e:
0xbb,0x3f:0xbf,0x5c:0xff,0x7c:0xa6

And for good measure:

mangled names = no

Those changes map problematic characters to extended characters. For example colon : is now displayed as the division sign ÷ It's not ideal but at least you can read the filenames now.

Credit to:  https://unix.stackexchange.com/questions/299130/samba-how-to-display-files-with-colon-in-their-names/381639#381639

Monday, 26 December 2022

Reducing the size of virtual machines exported from VirtualBox

VirtualBox can export virtual machine images which is handy if you want to give someone a ready to use OS for example.

I had an old scanner which I was driving under XP, using the USB forwarding feature. I wanted to give a friend the scanner and an OVA image containing the ready to go OS. The problem is that unused blocks on the NTFS filesystem will affect the compression of the image. Even though I had defragmented the virtual disk and moved the used blocks to the front using a pre-8.0 version of Ultradefrag, the unused blocks after the used area still contained old random data.

Enter sdelete. This is a free utility from Microsoft that wll zero unused blocks. You use it like this:

sdelete -z c:
I ran this once on the virtual disk, then shutdown the virtual machine and exported it to OVA format. The result was quite satisfactory; what was once a 5.1 GB dump turned out more like 2.1 GB.

Sunday, 7 August 2022

OpenVPN newer versions require new configuration to use longer Diffie-Hellman keys

I found that after an upgrade, my openvpn setup no longer worked with my client. The message in the log file was:

OpenSSL: error:1408518A:SSL routines:ssl3_ctx_ctrl:dh key too small

Long story short, you need to use dh2048 keys now. Run the command in the sample server.conf file in the sample-config-files of your OpenVPN distribution to generate a dh2048.pem, put it in the same directory where you have dh1024.pem and edit the config file for the parameter dh.

Other changes which I needed to make which you might or might not were:

I turned off comp-lzo compression as suggested. This change also needs to be made in the client config files.

I had the parameter: cipher AES-256-CBC Just let it autonegotiate to AES-256-GCM now. If you wish, turn that paramter to data-ciphers-fallback if you wish AES-256-CBC to be still considered.

Fortunately, aside from disabling comp-lzo, I didn't need to regenerate the client keys or change the rest of the config.

Saturday, 16 July 2022

Prefer IPv4 addresses to IPv6

This started when I could not refresh a repo on my Linux installation. Doing a manual wget --spider on the metadata file that it could not retrieve revealed that it was trying to connect to the IPv6 address of the mirror site.

My Internet provider does offer a dual stack IP service but I have not activated the IPv6 portion yet. So the question was how to make sure any http/https clients prefer the IPv4 address.

A web search gave the answer, you edit the file /etc/gai.conf which controls the behaviour of the getaddrinfo(3) function. In it you will find a comment to uncomment the line:

precedence ::ffff:0:0/96  100

if you want IPv6 addresses to sort lower than IPv4 addresses. I also reloaded the nscd service to flush any cached entries.

I suspect I hit this problem because I run my own DNS resolver. If you are relying on a DNS relay from your Internet provider they may have filtered out IPv6 answers for the sake of compatibility.

Monday, 30 May 2022

Alias for hexdump for canonical output

The man page for hexdump from util-linux states:

-C, --canonical

Canonical hex+ASCII display. Display the input offset in hexadecimal, followed by sixteen space-separated, two-column, hexadecimal bytes, followed by the same sixteen bytes in %_p format enclosed in '|' characters. Invoking the program as hd implies this option.

This also happens to be my preferred format for working with microprocessor code. However my Linux distro's package chose not to provide a link or alias from hd to hexdump.

No matter, I just made a symlink called hd in my private bin directory to /usr/bin/hexdump and it works as described.

Friday, 8 April 2022

Using fail2ban with systemd and firewalld

I installed fail2ban to watch over an openvpn service on a system that uses systemd (and hence journald), as well as firewalld. These are the changes I had to make. Most of it is derived from this wiki entry.

The filter /etc/fail2ban/filter.d/openvpn.local unchanged from the wiki:

# Fail2Ban filter for selected OpenVPN rejections
#
#

[Definition]

# Example messages (other matched messages not seen in the testing server's logs):
# Fri Sep 23 11:55:36 2016 TLS Error: incoming packet authentication failed from [AF_INET]59.90.146.160:51223
# Thu Aug 25 09:36:02 2016 117.207.115.143:58922 TLS Error: TLS handshake failed

failregex = ^ TLS Error: incoming packet authentication failed from \[AF_INET\]<HOST>:\d+$
           ^ <HOST>:\d+ TLS Auth Error
           ^ <HOST>:\d+ TLS Error: TLS handshake failed$
           ^ <HOST>:\d+ VERIFY ERROR
           ^ <HOST>:\d+ Connection reset, restarting

ignoreregex =

The jail file /etc/fail2ban/jail.d/openvpn.local enables the jail for openvpn and the wiki missed out the .local extension without which it will not be registered:

# Fail2Ban configuration fragment for OpenVPN

[openvpn]
enabled  = true
port     = 1194
protocol = udp
filter   = openvpn
journalmatch = _SYSTEMD_UNIT=openvpn@yourhost.service  
backend  = systemd
logpath  = /var/log/fail2ban.log
maxretry = 3

The important line is the backend = systemd. The journalmatch makes scanning the journal more efficient. Usually the openvpn service has @hostname appended as there could be more than one instance. You could also log it to the system journal but here I use a separate log file.

And finally you need to link this jail to firewalld in /etc/fail2ban/jail.local:

# Do all your modifications to the jail's configuration in jail.local!
[DEFAULT]
banaction = firewallcmd-ipset
This calls firewall-cmd to use an ipset to ban the hosts.



Wednesday, 2 March 2022

Whole array operations on bash arrays

Today I wanted to take all the elements of a bash array and append /** to each one. This was for a script that syncs selected photo albums to cloud storage using rclone. As you know, rclone syncs directories, so to limit the transfer to a subset of of the directories, I used an --include pattern. Say the albums are cny13 fiji13 misc13 under archived. Then the command required is:

rclone sync archived mydrive:Photos --include "{cny13/**,fiji13/**,misc13/**}"

So the question is how to get this from an array containing:

declare -a ALBUMS=(cny13 fiji13 misc13)

Of course, I could run a small loop where I append /** to each element and accumulate in another array. This does work and efficiency isn't really an issue. But I guess the old APL fan in me was awakened and I wondered if I could transform the whole array in one fell swoop.

I tried:

echo "${ALBUMS[@]}/**"

but this only got me:

cny13 fiji13 misc13/**

The clue was supplied in an online tutorial on bash arrays in which an example was shown of parameter substitution. This in fact works just as well on arrays element by element.

echo "${ALBUMS[@]/%/\/**}"

This substitutes the end of line with /** escaping the leading slash, so we get:

cny13/** fiji13/** misc13/**

I haven't explained how the , is inserted between items, but the entire script using another trick to create a join function shows it:

#!/bin/bash

function join { local IFS="$1"; shift; echo "$*"; }

declare -a ALBUMS=(cny13 fiji13 misc13)
declare -a includes=("${ALBUMS[@]/%/\/**}")
cd ~/Albums || exit 1
albums=$(join , ${includes[@]})
rclone sync archived mydrive:Photos --include "{$albums}"

Wednesday, 19 January 2022

Scp cannot handle file times before the Unix epoch

Native Linux filesystems have been able to store file timestamps before the Unix epoch, 1 Jan 1970, for some time now, due to use of 64-bit time_t.

Today I discovered that scp cannot transfer timestamps before the epoch. Here I copied a file which contains a scan of an old photo that I have timestamped back to the day it was sent:

-rw-r--r-- 1 me users 245768 Jan  1  1970 /tmp/1946-07-31-myrtle.pdf

By comparison, rsync does the right thing:

-rw-r--r-- 1 me users 245768 Jul 31  1946 /tmp/1946-07-31-myrtle.pdf

I suppose I could look into the scp protocol to discover why this is.

Watch out for non-breaking spaces on screen scrapes

I have been ripping DVDs I own to MP4 files for convenience of viewing on a tablet. Sometimes I need to get additional information about the episodes. For example I wanted to name the episodes of the Granada Sherlock Holmes TV series with the title of the tale for easy selection. For example 17-The_Musgrave_Ritual.mp4 is much preferred to 17.mp4.

On Wikipedia, episodes of many TV series are tabulated. You can highlight the contents of the table, and paste into a Libreoffice spreadsheet. This can then be exported as a CSV file for futher processing, e.g. with a Python program to generate a shell script that will rename the files the desired way.

This blog post is to point out that screen scraping will also capture the underlying characters in the tables, including extended characters in UTF-8 encoding. No surprise that this includes the non-breaking space: &nbsp; or 0xA0 in 8-bit encoding or \uC2A0 in UTF-8 encoding. So when processing the CSV file, this needs to be converted to a space or your shell scripts won't work. Here's an example of the conversion needed.

datestring = row[5].replace(u"\xa0", " ")

This was to generate a touch -d 'date' episode.mp4 command. Touch kept telling me the date format was invalid until I investigated the date string and found a non-breaking space in it.

Tuesday, 28 December 2021

Text formatting on a mainframe

Today I recalled that as a student in the 70s I discovered that the line printer attached to the Univac 1108 mainframe at my university's computer centre could print lower case characters. So I modified the Ratfor workalike of the Unix roff program that I had entered into the computer to treat all alphabetic characters as lower case, and implemented escape codes to raise (and lower later) the case for capitalised words and acronyms. This was to be able to use punch cards as input, as the terminals were not always available. I don't remember if I implemented auto-detection of beginning of sentences, probably not. I even typeset my undergraduate thesis this way.

At that time I was using Ratfor as a structured Fortran, having been introduced to it and the book that described it, Software Tools, at a work experience stint, and had not yet encountered Unix in person. It was only when I did my masters that I learnt Unix and C.

I have wondered if the computer operators were surprised by the appearence of lower case printout since everybody else seemed to accept that UPPER CASE was the only case available.