Using HandBrake to encode video DVDs

Now that I have a lot of spare time, I decided to re-encode some of the video DVDs I ripped years ago. I have tried various programs on Linux, such as dvd::rip, dvdbackup and HandBrake. I found some results poor, I could see TV scan lines in the result, or motion blur. This time on the recommendation of some reviews I decided to try HandBrake.

This turned out to be an excellent choice because HandBrake has come far over the years. It encodes to MP4 a.k.a M4V. My older encodes were a mix of AVI and MP4. HandBrake is very thorough. Even though I chose the Very Fast 720p30 profile the quality was great. (I didn't want to spend a lot of CPU time on higher quality as the source doesn't have enough quality anyway.) I also liked that there is a CLI version called HandBrakeCLI which I ran in scripts to extract separate chapters from titles.

Assuming you have a multi-core CPU, you may wish to use the taskset command to limit the number of cores used. When I set no limit on HandBrake, it used all the cores and I could see in the system log that the CPUs got throttled when they got too warm. There was no noticeable effect on interactive response, I could still surf the Internet and do other things. Another effect was fast encoding caused a read rate from the DVD drive so high that an external one on USB would occasionally reset and break the encoding. If scripted, I can just run it on the side and come back when it's finished. Typically my jobs ran at about 3x real time. So a 2 hour movie would be done in about 40 minutes.

Speaking of reading from the drive, it's best if you can copy the DVD to an ISO image on disk first to reduce wear. And if there are bad blocks in the DVD you might want to use ddrescue to recover as much as possible first. A tip, even though the DVD might have CSS, you want to do an image copy so that HandBrake can do the normal CSS decrypting.

Do try to find a ready made package for HandBrake. At first HandBrakeCLI didn't work for me, the loader said that it couldn't find symbols from the libdvdcss library. I decided to build from source. This took a couple of hours even on my multi-core CPU. And the problem didn't go away. Then I noticed that I had very old versions of HandBrakeCLI in my /usr/local/bin directory. There was absolutely nothing wrong with the latest package. I had been running a much older version. Duh!

bash ranges

This is not widely known, although it is adequately documented in the bash(1) man page thus:

A sequence expression takes the form {x..y[..incr]}, where x and y are either integers or single characters, and incr, an optional increment, is an integer. When integers are supplied, the expression expands to each number between x and y, inclusive. Supplied integers may be prefixed with 0 to force each term to have the same width. When either x or y begins with a zero, the shell attempts to force all generated terms to contain the same number of digits, zero-padding where necessary. When characters are supplied, the expression expands to each character lexicographically between x and y, inclusive, using the default C locale. Note that both x and y must be of the same type. When the increment is supplied, it is used as the difference between each term. The default increment is 1 or -1 as appropriate.

This means that you can expect these expressions to work:

$ echo {0..10}
0 1 2 3 4 5 6 7 8 9 10
$ echo {z..a}
z y x w v u t s r q p o n m l k j i h g f e d c b a
$ echo {01..20..2}
01 03 05 07 09 11 13 15 17 19
$ echo f{20..01..2}
f20 f18 f16 f14 f12 f10 f08 f06 f04 f02

In the last case, notice a prefix was prepended as you would expect. Suffixes work too, of course. -2 will also work. The direction is determined by the range.

Thus {01..20} is a more efficient alternative to using

$(seq -w 01 20)

Brace expansion happens early so no use storing the expression in a variable or having one of the start, end, or increment a variable.

In case you are still wondering what ranges are useful for, a typical use is in a for loop:

for month in {01..12}
do
...
done

Using a VPN can get you blacklisted

I recently helped a friend get email going out again. She uses Thunderbird and has an account with a national Internet provider. The symptom was that she could read her email with IMAPS but when she tried to reply via secure SMTP on port 465, the upload was rejected. So she had resorted to sending SMSes to friends.

I won't go through the false leads I followed but make a long story short: it was the use of a VPN that caused rejection of the outgoing email. How did this happen? The rejection message was: OB115.<ip address> blacklisted, please contact ... to resolve. When the VPN is in operation, all outgoing traffic comes from the exit IP address of the service. What had happened was that the provider had detected lots of connections from that IP address from all their customers using the VPN and concluded that it was under attack so blacklisted that address. In her former job she was required to use the VPN, which was part of an anti-virus suite. But there was no good reason to use the VPN from home, and what's more to secure TCP ports. When I disabled the VPN both immediately and at startup, outgoing mail worked again.

You might argue that the provider should have whitelisted this exit IP address, but there are so many services and exit points out there that it would be a huge task to list them all. So unfortunately the automatic blacklisting mechanisms kicked in.

If you must use a VPN, depending on the VPN software you may be able to specify that some applications or ports are exempt.

A similar thing happens when you use a VPN and websites using Cloudflare detect too many connections from the exit IP address, then you get rejected.

Moving VirtualBox VM directory in VirtualBox 6

I have some VMs on my Linux machine which run under VirtualBox 6. When I first created the VMs, I accepted the default directory "VirtualBox VMs" in $HOME. After a while this long directory name containing a space irked me and I looked into ways of renaming it. I found lots of old articles that suggested steps like dumping the old VM and then rereading it, or detaching the VM disks and reattaching.

It turns out that in VirtualBox 6 on Linux (and perhaps earlier versions, I cannot say), it's trivial. The configuration information is held in the configuration file $HOME/.config/VirtualBox/VirtualBox.xml With VirtualBox not running, I changed all instances of "VirtualBox VMs" to VMs, and moved the directory of course. Upon starting VirtualBox, everything worked as before.

If you have N VMs there will be N+1 instances of the old directory name in that XML file, one for each VM and one for the base directory of VMs that are created.

How to remove the msftdata flag on a GPT partition

You've bought this large disk for use on Linux and it's been preformatted for Windows. Usually there is a msftres partition (which you can leave alone, in case you need it later, as it takes up only a tiny fraction of the disk capacity) and a msftdata partition. You can format the filesystem as ext4 or whatever Linux filesystem you like and it will mount properly, but a list of the partitions still shows it as Microsoft. This annoys you.

The relevant documentation for parted shows that msftdata is a flag. Unfortunately you can't just set the flag off. The document states:

This flag can only be removed within parted by replacing it with a competing flag, such as boot or msftres.

So the solution is (assuming the partition is number 2):

set 2 boot on
set 2 boot off

Discrete TTL IC desk calculator

Today a memory of an old project that was a spectacular case of unfortunate timing came back to me. The British magazine Practical Electronics designed a desk calculator using TTL ICs. Unfortunately just as the 1st (or 2nd) part of 11 was published, MOS chips from Texas Instruments implementing an entire calculator came onto the market. And with better specs. The technology was obsolete even before it was published. PE decided to publish the project anyway for pedagogical benefit. Here is the cover of the July 1972 issue and the first 2 pages of that series:

The series has been archived here. It gets a mention in the EPE (Everyday Practical Electronics, two magazines having been merged sometime before) 50 year retrospective of 2014 (parts 1 and 2).

The component list is interesting. They weren't even Low power Schottky TTL packages, just standard TTL. Must have run a bit warm. 145 silicon diodes? Sounds like a microcode matrix.

I couldn't have built one anyway, it was far beyond my means at the time but it was interesting to read how the functions were implemented. I wonder if anybody other than the series author actually made one and wrote it up.

Photos got binned in Google Photos

You decide you want to free up storage on your phone so you go into Google photos and start deleting photos that you have already downloaded onto your personal computer. Some of those photos you have also shared in a particular folder on Google photos by uploading from your PC.

You notice that when you tap on the trash can icon it says that the photo will be deleted from all folders and albums you have shared it to. Surely Google would not go delete a photo that had gone through your PC?

Wrong! It turns out that Google matches up what you uploaded from your PC with what's on your phone and knows it's the same photo. When you go to your shared album, the photos you deleted are in the bin too. If you have already emptied the bin, either on the phone or on your PC, or 60 days have elapsed, those photos are gone from your album. Your only recourse is to re-upload from your PC.

I can understand the reason for this synchronised deletion. Say you shared a photo and now you regret it. It would be logical to delete from all places where it exists. However as implemented, Google is being heavy handed. There is no setting I can see to turn this behaviour off. To avoid accidental deletion from your album when you clear space on your phone, you should not use the trash can icon. Instead, use the actions Save to device, followed by Delete from device. This is far less convenient, but the latter might be done in bulk. Another way is to use a different app for deleting the photo from your device.

I have not experimented to see if the matching depends on some metadata in the JPEG that could be stripped out from your PC's copy. Maybe the device name?