Squid failing to fetch IPv6 web resources and dns_v4_first

I had a strange symptom on my Chromebook on my home LAN. A particular Internet web page would not work because a jQuery file distributed by cdnjs.cloudflare.com could not be fetched. When I disabled the use of a proxy in my Chromebook, the web page worked. I have a squid caching proxy on my LAN, partly to be able to zap advertisments. So I knew it had something to do with squid.

Maybe it was a bad object in the cache? I knew there was a way to delete cached objects from the command line, and a search showed me that all I had to do was add these lines to /etc/squid/squid.conf:

acl purge method PURGE

http_access deny purge !localhost

and then use the command:

squidclient -m PURGE https://cdnjs.cloudflare.com/blah.js

to remove it. However when I ran

squidclient https://cdnjs.cloudflare.com/blah.js

to fetch it again, I saw that squid was trying to use the IPv6 address of cdnjs.cloudflare.com to get the resource and failing.

My LAN is fully IPv6 enabled because my Linux machines all support the IPv6 stack, I have a BIND server giving out IPv6 addresses, and in fact a lot of the internal traffic such as Apache goes over IPv6 transparently. I wish I had an IPv6 broadband connection but I don't so I cannot use IPv6 addresses in the outside world.

So the problem boiled down to: how can I prevent squid from trying to reach IPv6 sites. It turns out that the directive dns_v4_first is intended for this. Just adding:

dns_v4_first on

to /etc/squid/squid.conf worked and now I can view that Internet web page.

One symptom remained unexplained though, why didn't the Chrome browser on the host running squid suffer the same problem? I can only surmise that it's because in that case the proxy is specified as an IPv4 address and port so squid thinks, the request is coming in from a IPv4 host so I'll forward this request to an IPv4 origin. The Chromebook however, sends the request to the (internal) IPv6 address of squid so squid thinks it's allowed to forward the request to an IPv6 origin.

Apache won't start on CentOS 5 because self-signed certificate used by mod_nss expired

You are not likely to hit this error with the usual configuration since mod_ssl is the one normally used. However mod_nss is used by the Fedora Directory Server (aka 389 Directory Server now) in CentOS for the console for LDAP authentication (centos-idm-console). Possibly also used by RHEL.

One day I found that Apache would not start. The error log indicated that the certificate had expired. I searched and searched for how to generate a new certificate and tried various things. Too many steps and too hard.

Then I had an idea. The certificate must have been generated when the package mod_nss was installed. Let's try reinstalling it. First we move the old database directory out of the way:

mv /etc/httpd/alias /etc/httpd/alias.old

Then reinstall mod_nss:

yum reinstall mod_nss

and voila, after a while, a new database with a fresh certificate was generated and I could start Apache.

Apache2 403 error but no clue in error_log, check rewrite rules

I had a directory inside an Apache2 website that I needed to open for indexed browsing.  But it always returned 403 errors. This old but still valid document explains how to solve 403 errors. Generally the first place you should look is /var/log/apache2/error_log. The usual reason is clients' IP address not allowed or file permission problems and there will be a line in the log. However in my situation there was absolutely no diagnostic.

Finally I figured the error must be generated by a RewriteRule. But which one? This page shows how to debug rewrite rules by enabling the log. Put it in say /etc/apache2/conf.d/rewrite.conf (Debian).

RewriteEngine On
RewriteLog "/var/log/apache2/rewrite.log"
RewriteLogLevel 3

It turned out to be a rule in Joomla!'s .htaccess that was blocking indexing on the directory. I also had to re-enable Options Indexes for that directory, as well modify .htaccess to comment out IndexIgnore *, since this cannot be overridden, and Options -Indexes blocks indexing already.

Remember to disable the rewrite log afterwards.

Gotchas setting up FreeBSD as a remote syslog server

I needed to migrate an OpenBSD remote syslog server to a VM. As FreeBSD is better supported under VMWare, I switched to this.

The configuration is well documented in section 12.7.3 of the online manual but I hit a couple of gotchas which I'll share with you to save you work.

The procedure in 12.7.3.1 asks to add a couple of lines to /etc/syslog.conf for each client. This is only necessary if you want to have a separate log file for that client. Otherwise received messages will be distributed to the default log files as configured. By the way I could not find a way to specify a wildcard for the client.

You also need to edit /etc/rc.conf, and specify syslogd_flags. Here you can use a hostname, netblock or domain, see man syslogd. But there is one gotcha. FreeBSD syslogd will only accept messages originating from port 514 on the client. This may not always be the case. The symptom is that tcpdump shows that the messages are arriving on the server, but syslogd is ignoring them. To allow any origin port, specify it with an asterisk after the host or netblock specification.

syslogd_flags="-a 10.10.10.0/24:* -v -v"

And to install open-vm-tools simply do pkg install open-vm-tools. It will pull in a lot of dependencies though.

Avoiding the remote host identification changed warning ssh'ing to localhost

If you run several virtual machines under VirtualBox and use port forwarding (Settings > Network > Advanced > Port Forwarding) to map its port 22 to a localhost port so that you can ssh to it, you will end up with several remote hosts all accessed from localhost but at different ports. In such a situation, if you have an older ssh client, you might end up with this well known warning when you try to connect:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!

The reason is that older ssh clients only store the domain name or the IP address of the server in the ~/.ssh/known_hosts file. So there can only be one host key for localhost and when you connect to a different port there is a key mismatch. Newer ssh clients store the port number also and don't have this problem. You can fix this temporarily by deleting the host key entry but when you connect to another server it happens again.

I've seen solutions that suggest suppressing the host key check. There is another way. All addresses in the 127.0.0.0/8 subnet are localhost so you can use distinct IP addresses for each VM. For example in ~/.ssh/config you could have something like this:

Host centos64

Hostname localhost

        Port 2201

Host debian64

Hostname 127.0.0.2

        Port 2202

This way there are distinct host key entries for the two VMs and you will not get the dreaded clash and warning.

Once again, this is only necessary when using an older ssh client.

Setting event timezone in Lightning (Thunderbird)

This has been a solved problem since 2008 but it may not be obvious to everybody so I'm publicising it here.

You can set the timezone of individual events in Lightning. When you edit an event, there are timezone links next to the start and end times. You can see it in this page's screenshot. The link colour may not always stand out. Clicking on the link brings up a timezone chooser map where you can change the timezone of the event. If the start and end times are linked, you only need to change one. Also if you have used a foreign timezone before, a shortcut is provided.

It seems however, from limited testing, that for synced calendars such as Gcal, existing events have to be edited at both sides for the timezone to stick, even though the time is correct.

An essential feature for those of us who work or communicate across timezones, or have deadlines in other timezones, e.g. taking a MOOC.

Now if only the ICS files sent to me always had the correct timezone.

ping: icmp open socket: Operation not permitted, and capabilities

Last night I installed a SSD on my openSUSE 13.1 system, copied my root filesystem to it, and made it the boot volume. Afterwards it booted up very quickly. I went to bed a happy person.

This morning, while checking some network issues I used ping and got the error in the title. What happened!?

Checking the permissions on both the old filesystem and the new one revealed no differences.

-rwxr-xr-x 1 root root 43480 Nov 16 09:47 /usr/bin/ping

Was it supposed to be setuid? Let's see whether the package manager thinks it's installed correctly.

$ rpm -qf /usr/bin/ping
iputils-s20101006-23.4.1.x86_64
$ rpm -V iputils
/usr/bin/ping should be root:root 0755 "= cap_net_raw+ep". (wrong missing capabilities)

Ah, something was lost in the copying. Searching for capabilities revealed that they allow finer grained privileges than setuid. To fix I was supposed to run setcap. But there was no such utility installed, so I repaired the situation by reinstalling iputils instead of installing the libcap-progs package.

$ sudo zypper install -f iputils

And ping was back to normal.

Incidentally some posts claim that rsync preserves capabilities. I did use rsync to copy and yet the capability did not come across. I can find nothing in the manual page of rsync about this.

Here's a good introduction to capabilities.