Found duplicate PV: using /dev/... not /dev/...

When you mix software RAID1 (md) and LVM, in some situations you can get this message:

Found duplicate PV: using /dev/sdb1 not /dev/md0 ...

and the LVM doesn't assemble. The exact device names may differ, of course. But how does this happen?

What happened that at some point vgscan was run and read partition(s) that were later made into RAID1 members and saw a Physical Volume (PV) UUID on it. Since the PV UUID of a RAID1 array is identical to the PV UUID of the members, you get duplicate(s).

RAID1 members are usually not candidates for PVs, as vgscan  normally excludes such devices from consideration. However there is a cache: /etc/lvm/cache/.cache which may contain outdated entries. In the example above it contained an entry for /dev/sdb1 which should have been filtered out by virtue of being in a RAID array. The solution is simple: just run vgscan again to update the cache. But you may have a problem if the device is needed for booting up. If the root devices is on a different partition or you have a rescue DVD you might be able to mount the root filesystem containing /etc read-write and refresh the cache.

Some articles suggest editing the lvm.conf file to specify a filter to exclude the RAID1 members. Try refreshing the cache first before you resort to this as it should just work.

This problem occurred in the context of converting in-situ a filesystem on a single disk to reside in a RAID1.

Converting single disk to RAID1 in-situ

You have this Linux system that doesn't use RAID. You start to worry about the loss of files (from the last backup; you do backups, right?) and downtime should the disk fail. Maybe it is a good idea to have RAID. But how to retrofit RAID1 without a lot of downtime backing up, reformatting the disks and restoring the data?

I suspected there might be a way to start off with a degraded RAID1 array on the second, new disk, copy the partitions on the old disk onto it, change the type of the old disk to RAID element, add it to the array and let it resync. Sure enough it can be done, and François Marier has blogged it. In fact he goes further and shows how to reinstall the boot loader. I didn't have to do this because my partition is /home. The critical tip is the use of the keyword missing to create the degraded array without issues.

In my case the decision to go RAID1 was done after a failed disk caused loss of files. It was not a wise decision by the system builder to not use RAID1 in the first place.

I've varied the procedure a little. Instead of putting ext4 directly on the RAID partition, I put a logical volume on it, and then created an ext4 partition inside that. This allows me to migrate the content to a larger disk if expansion is needed in future, using logical volume operations, with little downtime.

There's one thing you should do if you decide to use logical volumes on the RAID. After you have assembled the RAID array, run vgscan. This will reinitialise the cache in /etc/lvm/cache/.cache. Otherwise it will contain entries for the components of the array and cause failure to assemble later on with a mysterious (to me at first) duplicate PV error because it thinks the array components are candidates for volumes. LVM is normally configured to ignore components of RAID arrays but only if the cache is up to date. See here for more details.

A couple of caveats: On other Linux systems mdadm.conf may be in /etc, not /etc/mdadm. Also the mdadm --detail --scan command to get the mdadm.conf line will contain a spares=1 directive if run while the array is resyncing. Remove it, or you will have problems next boot.