Get gitlab browsing statistics for your project

Unlike github, gitlab doesn't provide a web page to see the browsing activity on your project. But there is an API to fetch this information in JSON format. I explain how it's done so you can avoid the pitfalls I encountered.

In the documentation, all the API calls are with respect to a base URL, which is currently:

https://gitlab.com/api/v4

This StackOverflow post confirms this base URL. It also shows how the access token is passed to the GET request, as an extra header.

Getting a personal access token is described here with screenshots. You can set an expiry date for the token. Make sure you save a copy of it as you cannot see it again once generated.

Now if you look at the API documentation, the relative URL to get the statistics for the last 30 days is described here and is:

/projects/:id/statistics

This is where I got fooled. The : is not to be entered literally. Instead substitute the numeric ID of your project.

Putting it all together, here is the curl command to get the statistics for a fictional project ID of 1234.

curl -H 'PRIVATE-TOKEN: yourtokenhere' https://gitlab.com/api/v4/projects/1234/statistics

And here is typical output:

{"fetches":{"total":3,"days":[{"count":3,"date":"2020-11-23"}]}}

There is no newline at the end, JSON doesn't need it. But you'll be using some other utility to parse and display the statistics in a user-friendly form.

Now you can use this information to get all the other information exposed by the API. Some relative URLs don't require authentication if the project is public.

Upgrading unetbootin to remove qt4 dependency

When my openSUSE Leap 15.2 showed a libqt4 update I wondered what application needed these old libraries. Removing libqt4 removed a bunch of library dependencies and the one application that appeared was unetbootin. A search showed that Ubuntu 20 users had encountered missing libqt4 problems too, and the official solution is to upgrade to unetbootin 681. Even though libqt4 is still supported in Leap 15.2 I decided to upgrade. This can be done as a one-click install from software.opensuse.org and the "experimental" package is from the filesystems repo. I guess the next release of Leap will make unetbootin 681 official.

As a bonus this freed up about 150 MB of disk space now that nothing needs qt4 any more.

Ripping CDs to AAC (M4A) files

Since MP3 is officially obsolete and AAC is a better format, I have decided to rerip all the CDs I own to this format.

This is quite easy with k3b,  you just have to add another external encoder specification. I got most of the info I needed from this blog post. One problem was ffmpeg was spitting out too many messages, so I did this:

ffmpeg -v 24 -i - -b:a 256k %f

The -v 24 is to suppress the normal informative output from ffmpeg which is quite verbose.

I also wanted to add tags, so after ripping I run this kid3-cli command in the directory:

kid3-cli -c 'select all' -c 'totag 3' -c save -c exit

This puts the tags implied by the filenames into the tags.

Don't forget to specify Write WAV Header as the original blog post says.

Accessing Samba 4.11 shares from old clients, e.g. XP, Android apps, media players, etc

NB: Please do not do this if you are running a work network, as SMB1 has insecure authentication, and traffic is in clear. Get your clients updated. I did this only because I run Samba on a home network serving only a few of my applications, and only for legacy clients.

When I upgraded my Linux distro I discovered that starting with Samba 4.11 the minimum protocol support level has been raised to SMB2 by default. This made the shares I export inaccessible to an XP virtual machine driving a scanner and a chip programmer, and also an app on my Android tablet.

There are lots of confusing posts on the Internet some about older versions. This relates to these changes. You can still enable SMB1 support, using these lines in smb.conf:

min protocol = NT1
ntlm auth = Yes

Restart nmb and smb and the changes should take effect.

SMB1 support may be removed in future, so hopefully my applications will be also updated before then.

Discrepancies between CRMs and mailing lists

For a while now I have been updating my email address on online accounts and have discovered how sloppy the connection between customer details held in CRMs and email addresses in mailing lists are. Also how terrible some CRMs are.

Exhibit 1: I get a newsletter. There is a link at the bottom to update my preferences so I click on it and update my email address in a form. If they did it right there is a verification step. Otherwise one risks getting locked out if one enters a typo. Later on I still get email at the old address.

1a. It eventually resolves because the address list is only refreshed from the CRM at intervals. Not ideal but at least it works.

1b. I still get email at the old address. Evidently updates not set up correctly. Now I have to contact IT. Or unsubscribe and resubscribe with a new address.

1c. I get email at both the old and new addresses. This is terrible. Are the software developers or system admins competent?

Exhibit 2: I update my email address in the account. Later I get a newsletter to the correct new address. I click on the link to update my preferences. But the page is still showing my old address. When I try to change that to the new address it says it's taken, as it should be. Again some discrepancy between the CRM and the mailing system.

Exhibit 3: I update my email address in the account. But I cannot change the login, it has to be the first address I chose, forever. How silly.

Exhibit 4: I try to change my email address (and login) in the account. But it's read-only. A query reveals that I have to create a new account and ask IT to transfer all details, in particular any purchase history, to the new account. Aren't CRMs supposed to take care of this? Or is it to keep IT in work?

Exhibit 5: I try to change my email address (and login) in the account. Again it's read-only. A query reveals that it can only be resolved by contacting IT to modify it manually. A job preservation scheme. Sometimes the contact is not easily discovered via the website. Sometimes you have to post to a forum so that a moderator will notice.

Exhibit 6: Not really an email update issue, but I used to get newsletters that were sent by sales department of a legitimate company. But they were labelled as spam. Investigation revealed that the mailout was outsourced but they didn't register the outgoing servers in SPF. I tried to tell them about it, but they're still doing it and probably losing sales.

Putting an idle WiFi router to use as a LAN-WLAN bridge

In the current situation I have time to do the little chores I promised myself to do.

One was to set up an unused WiFi router as an auxiliary access point. In parts of my abode, the WiFi signal from the main router isn't strong, and besides it only handles the older standard. I have a spare router which handles the latest WiFI standards which I decided to put to use.

You may ask at this point why not swap the unused router in? Two reasons, the old router has DD-WRT on it and I have a particular set up for it, and the new router has gigabit Ethernet ports so I can use it as an Ethernet switch at the same time, and not have to run another Cat-6 cable to that area. Finally I use the Internet more from the new location.

Note that this is not about setting up a WiFi relay. You can buy such devices off the shelf to boost the signal. This is using a spare WiFi router as a LAN-WLAN bridge so you need to have wired Ethernet already. It also doesn't matter what the front end of the router is, whether ADSL or cable, as we don't use it at all. So ignore any Ethernet ports that say WAN or Internet. We only use the ones that say LAN or Ethernet. Here's a typical back panel. We want the yellow interfaces.

These are the steps you need to do:

1. Reset the router to factory settings.
2. Connect to it with a computer, usually a laptop, at the factory IP address and login.
3. Disable the DHCP server of the router.
4. Change the LAN address of the router to a free IP address in your subnet that isn't in the dynamic range of the main DHCP server.
5. Reconnect to the router at the new address.
6. Set up the WiFi using the same SSID and passwords as your main router.
7. Check tnat WiFi clients connect to your router.
8. Backup the configuration to a file on your computer.

Let's go through each of the steps.

1. Reset the router to factory settings

Usually this is done by poking a paper clip into a recessed reset button on the back or bottom of the router.

2. Connect to it with a computer, usually a laptop, at the factory IP address and login

Connect with an Ethernet patch cable and access the web interface. You need to know the factory default IP address and login/password. The manual will tell you or it may be on a sticker on the router. Often the address iis 192.168.0.1. You could either let the laptop get an address by DHCP, but since you will need to change it in step 5, you might as well learn how to configure it to a static address. Change the admin password after you have logged in.

3. Disable the DHCP server of the router

Somewhere in the settings, probably advanced, is where you turn off the DHCP server of the router. This is so that it won't clash with the main router. In my case it wasn't sufficient to disable it. I also had to shrink the range to a small one or it would say illegal address at the next step.

4. Change the LAN address of the router

You need to assign the router a fixed address that is on your LAN but not in the DHCP range of the main router. Say your main router gives out addresses from 192.168.100.100 to 192.168.200.200. In this example .99 or .201 is available. After you have changed this, your laptop cannot reach the router, so proceed to:

5. Reconnect to the router at the new address

For this you can either change the IP address of your laptop's Ethernet interface, or switch to the main router's WiFi which will be on the same subnet. For the former you would take another address on your LAN, say 192.168.100.202. The router address you give to your browser is for the last example: http://192.168.100.201

6. Set up the WiFI using the same SSID and passwords as your main router

Otherwise your devices will be confused which one to log onto. Naturally you should use the same standard, which is usually WPA2-AES. At the point you may wish to choose the wireless channel for least overlap with your neighbours, but if you are close enough to the router, it's not as important as your signal will be stronger. For this tuning task there are various WiFi analysis apps.

7. Check that WiFi clients connect to your router

The router may have a page that shows the WiFi clients connected to it. You should see entries in it when your phone or tablet connects.

8. Backup the configuration to a file on your computer

Most routers have a web page where you can download a dump of the configuration so that you can restore it quickly if you ever do a factory reset again.

The other Ethernet ports of the router (usually 3 left after you have used one) can be used to plug in other wired devices. In this case I connected an existing desktop to a free port, using the router as an Ethernet switch.

That's it. Enjoy faster Internet access from your secondary location.

How to convert text from one charset to another within vim

Sometimes when you edit a file with vim it sees the extended characters and guesses the charset of the file. If it isn't UTF-8 it usually guesses Latin-1 or ISO8859 and displays (converted) on the message line. In the case of a file of mine that had Chinese characters encoded in GBK, when I tried to convert the contents in situ with:

:1,$! iconv -f GBK -t UTF-8

I ended up with a buffer displaying the Chinese characters correctly, but when trying to write it out, vim indicated an error. Trying to force the write with w! gave me the original file.

The trick is you have to tell vim that the charset has changed. Do:

:set fileencoding=utf-8

before you write and quit and all will be well. You can check the current encoding at any point with:

:set fileencoding

Using HandBrake to encode video DVDs

Now that I have a lot of spare time, I decided to re-encode some of the video DVDs I ripped years ago. I have tried various programs on Linux, such as dvd::rip, dvdbackup and HandBrake. I found some results poor, I could see TV scan lines in the result, or motion blur. This time on the recommendation of some reviews I decided to try HandBrake.

This turned out to be an excellent choice because HandBrake has come far over the years. It encodes to MP4 a.k.a M4V. My older encodes were a mix of AVI and MP4. HandBrake is very thorough. Even though I chose the Very Fast 720p30 profile the quality was great. (I didn't want to spend a lot of CPU time on higher quality as the source doesn't have enough quality anyway.) I also liked that there is a CLI version called HandBrakeCLI which I ran in scripts to extract separate chapters from titles.

Assuming you have a multi-core CPU, you may wish to use the taskset command to limit the number of cores used. When I set no limit on HandBrake, it used all the cores and I could see in the system log that the CPUs got throttled when they got too warm. There was no noticeable effect on interactive response, I could still surf the Internet and do other things. Another effect was fast encoding caused a read rate from the DVD drive so high that an external one on USB would occasionally reset and break the encoding. If scripted, I can just run it on the side and come back when it's finished. Typically my jobs ran at about 3x real time. So a 2 hour movie would be done in about 40 minutes.

Speaking of reading from the drive, it's best if you can copy the DVD to an ISO image on disk first to reduce wear. And if there are bad blocks in the DVD you might want to use ddrescue to recover as much as possible first. A tip, even though the DVD might have CSS, you want to do an image copy so that HandBrake can do the normal CSS decrypting.

Do try to find a ready made package for HandBrake. At first HandBrakeCLI didn't work for me, the loader said that it couldn't find symbols from the libdvdcss library. I decided to build from source. This took a couple of hours even on my multi-core CPU. And the problem didn't go away. Then I noticed that I had very old versions of HandBrakeCLI in my /usr/local/bin directory. There was absolutely nothing wrong with the latest package. I had been running a much older version. Duh!

bash ranges

This is not widely known, although it is adequately documented in the bash(1) man page thus:

A sequence expression takes the form {x..y[..incr]}, where x and y are either integers or single characters, and incr, an optional increment, is an integer. When integers are supplied, the expression expands to each number between x and y, inclusive. Supplied integers may be prefixed with 0 to force each term to have the same width. When either x or y begins with a zero, the shell attempts to force all generated terms to contain the same number of digits, zero-padding where necessary. When characters are supplied, the expression expands to each character lexicographically between x and y, inclusive, using the default C locale. Note that both x and y must be of the same type. When the increment is supplied, it is used as the difference between each term. The default increment is 1 or -1 as appropriate.

This means that you can expect these expressions to work:

$ echo {0..10}
0 1 2 3 4 5 6 7 8 9 10
$ echo {z..a}
z y x w v u t s r q p o n m l k j i h g f e d c b a
$ echo {01..20..2}
01 03 05 07 09 11 13 15 17 19
$ echo f{20..01..2}
f20 f18 f16 f14 f12 f10 f08 f06 f04 f02

In the last case, notice a prefix was prepended as you would expect. Suffixes work too, of course. -2 will also work. The direction is determined by the range.

Thus {01..20} is a more efficient alternative to using

$(seq -w 01 20)

Brace expansion happens early so no use storing the expression in a variable or having one of the start, end, or increment a variable.

In case you are still wondering what ranges are useful for, a typical use is in a for loop:

for month in {01..12}
do
...
done

Using a VPN can get you blacklisted

I recently helped a friend get email going out again. She uses Thunderbird and has an account with a national Internet provider. The symptom was that she could read her email with IMAPS but when she tried to reply via secure SMTP on port 465, the upload was rejected. So she had resorted to sending SMSes to friends.

I won't go through the false leads I followed but make a long story short: it was the use of a VPN that caused rejection of the outgoing email. How did this happen? The rejection message was: OB115.<ip address> blacklisted, please contact ... to resolve. When the VPN is in operation, all outgoing traffic comes from the exit IP address of the service. What had happened was that the provider had detected lots of connections from that IP address from all their customers using the VPN and concluded that it was under attack so blacklisted that address. In her former job she was required to use the VPN, which was part of an anti-virus suite. But there was no good reason to use the VPN from home, and what's more to secure TCP ports. When I disabled the VPN both immediately and at startup, outgoing mail worked again.

You might argue that the provider should have whitelisted this exit IP address, but there are so many services and exit points out there that it would be a huge task to list them all. So unfortunately the automatic blacklisting mechanisms kicked in.

If you must use a VPN, depending on the VPN software you may be able to specify that some applications or ports are exempt.

A similar thing happens when you use a VPN and websites using Cloudflare detect too many connections from the exit IP address, then you get rejected.

Moving VirtualBox VM directory in VirtualBox 6

I have some VMs on my Linux machine which run under VirtualBox 6. When I first created the VMs, I accepted the default directory "VirtualBox VMs" in $HOME. After a while this long directory name containing a space irked me and I looked into ways of renaming it. I found lots of old articles that suggested steps like dumping the old VM and then rereading it, or detaching the VM disks and reattaching.

It turns out that in VirtualBox 6 on Linux (and perhaps earlier versions, I cannot say), it's trivial. The configuration information is held in the configuration file $HOME/.config/VirtualBox/VirtualBox.xml With VirtualBox not running, I changed all instances of "VirtualBox VMs" to VMs, and moved the directory of course. Upon starting VirtualBox, everything worked as before.

If you have N VMs there will be N+1 instances of the old directory name in that XML file, one for each VM and one for the base directory of VMs that are created.