David Christensen [2024-04-08 11:28:04] wrote:

Why LVM?

Personally, I've been using LVM everywhere I can (i.e. everywhere
except on my OpenWRT router, tho I've also used LVM there back when my
router had an HDD. I also use LVM on my 2GB USB rescue image).

To me the question is rather the reverse: why not?
I basically see it as a more flexible form of partitioning.

Even in the worst cases where I have a single LV volume, I appreciate
the fact that it forces me to name things, isolating me from issue
linked to predicting the name of the device and the issues that plague
UUIDs (the fact they're hard to remember, and that they're a bit too magical/hidden for my taste, so they sometimes change when I don't want
them to and vice versa).


Stefan

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 4/8/24 14:08, Stefan Monnier wrote:

David Christensen [2024-04-08 11:28:04] wrote:

Why LVM?

Personally, I've been using LVM everywhere I can (i.e. everywhere
except on my OpenWRT router, tho I've also used LVM there back when my
router had an HDD. I also use LVM on my 2GB USB rescue image).

To me the question is rather the reverse: why not?
I basically see it as a more flexible form of partitioning.

Even in the worst cases where I have a single LV volume, I appreciate
the fact that it forces me to name things, isolating me from issue
linked to predicting the name of the device and the issues that plague
UUIDs (the fact they're hard to remember, and that they're a bit too magical/hidden for my taste, so they sometimes change when I don't want
them to and vice versa).

Stefan

If I have a hot-pluggable device (SD card, USB drive, hot-plug SATA/SAS
drive and rack, etc.), can I put LVM on it such that when the device is connected to a Debian system with a graphical desktop (I use Xfce) an
icon is displayed on the desktop that I can interact with to display the
file systems in my file manager (Thunar)?


David

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

If I have a hot-pluggable device (SD card, USB drive, hot-plug SATA/SAS
drive and rack, etc.), can I put LVM on it such that when the device is connected to a Debian system with a graphical desktop (I use Xfce) an icon
is displayed on the desktop that I can interact with to display the file systems in my file manager (Thunar)?

In the past: definitely not. Currently: no idea.
I suspect not, because I think the behavior on disconnection is still
poor (you want to be extra careful to deactivate all the volumes on the
drive *before* removing it, otherwise they tend to linger "for ever").

I guess that's one area where partitions are still significantly better
than LVM.


Stefan "who doesn't use much hot-plugging of mass storage"

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 4/8/24 16:54, Stefan Monnier wrote:

If I have a hot-pluggable device (SD card, USB drive, hot-plug SATA/SAS
drive and rack, etc.), can I put LVM on it such that when the device is
connected to a Debian system with a graphical desktop (I use Xfce) an icon >> is displayed on the desktop that I can interact with to display the file
systems in my file manager (Thunar)?

In the past: definitely not. Currently: no idea.
I suspect not, because I think the behavior on disconnection is still
poor (you want to be extra careful to deactivate all the volumes on the
drive *before* removing it, otherwise they tend to linger "for ever").

I guess that's one area where partitions are still significantly better
than LVM.

Stefan "who doesn't use much hot-plugging of mass storage"

Thank you for the clarification. :-)


David

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hello,

1. INITIAL SITUATION: WORKS (no dm-integrity at all)

I have a Debian bookwork uptodate system that boots correctly with
kernel 6.1.0-21-amd64.

It is setup like this:

- /dev/nvme1n1p1 is /boot/efi

- /dev/nvme0n1p2 and /dev/nvme1n1p2 are the two LVM physical volumes

- a volume group, vg1 is built with those PVs

vg1 has a few LVs that have been created in RAID1 LVM mode:

lvdisplay | egrep 'Path|Mirrored'

LV Path /dev/vg1/root <-- this is /
Mirrored volumes 2
LV Path /dev/vg1/swap
Mirrored volumes 2
LV Path /dev/vg1/scratch
Mirrored volumes 2
LV Path /dev/vg1/docker
Mirrored volumes 2

As said, this boots without any issue.

2. ADDING dm-integrity WHILE BOOTED: works!

Now, while booted, I can add dm-integrity to one of the volumes,
let's say /dev/vg1/docker (this LV has absolutely no link with the
boot process, except obviously it is listed in /etc/fstab -- it also
fails the same way if even the swap is dm-integrit enabled, or
/):

lvconvert --raidintegrity y --raidintegritymode bitmap vg1/docker

and wait a bit til the integrity is setup with lvs -a (100%)

Obviously, this creates and uses a few rimage/rmeta sub LVs.

Then I did this (after having boot issues):

echo dm_integrity >> /etc/initramfs-tools/modules
update-initramfs -u

This did not change the below issue:

3. grub BOOT FAILS IF ANY LV HAS dm-integrity, EVEN IF NOT LINKED TO /

if I reboot now, grub2 complains about rimage issues, clear the screen
and then I am at the grub2 prompt.

Booting is only possible with Debian rescue, disabling the dm-integrity
on the above volume and rebooting. Note that you still can see the
rimage/rmeta sub LVs (lvs -a), they are not deleted! (but no
dm-integrity is activated).

4. update-grub GIVES WARNINGS

Now, if I try to start update-grub while booted AND having enabled
dm-integrity on the vg1/docker volume, I get:

# update-grub
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-6.1.0-21-amd64
Found initrd image: /boot/initrd.img-6.1.0-21-amd64
error: unknown node 'docker_rimage_0'.
[ ... many ... ]
/usr/sbin/grub-probe: error: disk `lvmid/xLE0OV-wQy7-88H9-yKCz-4DUQ-Toce-h9rQvk/FzCf1C-95eB-7B0f-DSrF-t1pg-66qp-hmP3nZ' not found.
error: unknown node 'docker_rimage_0'.
[ ... many ... ]

[ this repeats a few times ]

Found linux image: /boot/vmlinuz-6.1.0-10-amd64
Found initrd image: /boot/initrd.img-6.1.0-10-amd64
Found memtest86+ 64bit EFI image: /boot/memtest86+x64.efi
Warning: os-prober will not be executed to detect other bootable partitions.
[ there are none ]
Systems on them will not be added to the GRUB boot configuration.
Check GRUB_DISABLE_OS_PROBER documentation entry.
Adding boot menu entry for UEFI Firmware Settings ...
done

Any idea what could be the problem? Any way to just make grub2 ignore
the rimage (sub)volumes at setup and boot time? (I could live with / aka vg1/root not using dm-integrity, as long as the data/docker/etc volumes
are integrity-protected) ? Or how to make grub 100% compatible with a
vg1/root using dm-integrity (that would be obviously the final goal!)

Thank you for any pointers!

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hello,

On Tue, May 21, 2024 at 08:41:58PM +0200, Franco Martelli wrote:

I can only recommend you to read carefully the Wiki: https://raid.wiki.kernel.org/index.php/Dm-integrity

I did, and it looks it does not seem to document anything pertaining
to my issue:

1) I don't use integritysetup (from LUKS), but LVM RAID PVs -- I don't use
LUKS encryption anyway on that system

2) the issue is not the kernel not supporting it, because when the
system is up, it works (I have done tests to destroy part of the
underlying devices, they get detected and fixed correctly)

3) the issue is not with the initrd -- I added the dm-integrity module
and rebuilt the initrd (and actually the bug happens before grub2 loads
the kernel & init) -- or at least "not yet"! maybe this will fail
later :)

4) actually the issue is just grub2, be it when the system is up
(it complains about the special subvolumes) or at boot time

Having /boot on a LVM non enabled dm-integrity logical volume does not
work either, as soon as there is ANY LVM dm-integrity enabled logical
volume anywhere (even not linked to booting), grub2 complains (at boot
time or at update-grub) about the rimage LV.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Additional info:

On Wed, May 22, 2024 at 08:49:56AM +0200, Marc SCHAEFER wrote:

Having /boot on a LVM non enabled dm-integrity logical volume does not
work either, as soon as there is ANY LVM dm-integrity enabled logical
volume anywhere (even not linked to booting), grub2 complains (at boot
time or at update-grub) about the rimage LV.

I found this [1], quoting: "I'd also like to share an issue I've
discovered: if /boot's partition is a LV, then there must not be a raidintegrity LV anywhere before that LV inside the same VG. Otherwise, update-grub will show an error (disk `lvmid/.../...' not found) and GRUB
cannot boot. So it's best if you put /boot into its own VG. (PS: Errors
like unknown node '..._rimage_0 can be ignored.)"

So, the work-around seems to be to simple have /boot not on a LVM VG where
any LV has dm-integrity enabled.

I will try this work-around and report back here. As I said, I can
live with /boot on RAID without dm-integrity, as long as the rest can be dm-integrity+raid protected.

[1] https://unix.stackexchange.com/questions/717763/lvm2-integrity-feature-breaks-lv-activation

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hello,

On Wed, May 22, 2024 at 08:57:38AM +0200, Marc SCHAEFER wrote:

I will try this work-around and report back here. As I said, I can
live with /boot on RAID without dm-integrity, as long as the rest can be dm-integrity+raid protected.

So, enable dm-integrity on all LVs, including /, /var/lib/lxc, /scratch
and swap, now boots without any issue with grub2 as long as /boot is NOT
on the same VG where the dm-integrity over LVM RAID is enabled.

This is OK for me, I don't need /boot on dm-integrity.

update-grub gives out warning for every of the rimage subvolumes, but
can still then reboot.

I would guess the bug is thus in grub2, not yet supporting boot on a
/boot not necessarily dm-integrityfied itself, but on a VG where any
of the LV is.

Are readers seconding conclusion? If yes, I could report a bug on grub2.

Have a nice day.

Details:
root@ds-03:~# lvs -a
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
docker vg1 rwi-aor--- 500.00g 100.00
[docker_rimage_0] vg1 gwi-aor--- 500.00g [docker_rimage_0_iorig] 100.00
[docker_rimage_0_imeta] vg1 ewi-ao---- <4.07g
[docker_rimage_0_iorig] vg1 -wi-ao---- 500.00g
[docker_rimage_1] vg1 gwi-aor--- 500.00g [docker_rimage_1_iorig] 100.00
[docker_rimage_1_imeta] vg1 ewi-ao---- <4.07g
[docker_rimage_1_iorig] vg1 -wi-ao---- 500.00g
[docker_rmeta_0] vg1 ewi-aor--- 4.00m
[docker_rmeta_1] vg1 ewi-aor--- 4.00m
root vg1 rwi-aor--- 10.00g 100.00
[root_rimage_0] vg1 gwi-aor--- 10.00g [root_rimage_0_iorig] 100.00
[root_rimage_0_imeta] vg1 ewi-ao---- 148.00m
[root_rimage_0_iorig] vg1 -wi-ao---- 10.00g
[root_rimage_1] vg1 gwi-aor--- 10.00g [root_rimage_1_iorig] 100.00
[root_rimage_1_imeta] vg1 ewi-ao---- 148.00m
[root_rimage_1_iorig] vg1 -wi-ao---- 10.00g
[root_rmeta_0] vg1 ewi-aor--- 4.00m
[root_rmeta_1] vg1 ewi-aor--- 4.00m
scratch vg1 rwi-aor--- 10.00g 100.00
[scratch_rimage_0] vg1 gwi-aor--- 10.00g [scratch_rimage_0_iorig] 100.00
[scratch_rimage_0_imeta] vg1 ewi-ao---- 148.00m
[scratch_rimage_0_iorig] vg1 -wi-ao---- 10.00g
[scratch_rimage_1] vg1 gwi-aor--- 10.00g [scratch_rimage_1_iorig] 100.00
[scratch_rimage_1_imeta] vg1 ewi-ao---- 148.00m
[scratch_rimage_1_iorig] vg1 -wi-ao---- 10.00g
[scratch_rmeta_0] vg1 ewi-aor--- 4.00m
[scratch_rmeta_1] vg1 ewi-aor--- 4.00m
swap vg1 rwi-aor--- 8.00g 100.00
[swap_rimage_0] vg1 gwi-aor--- 8.00g [swap_rimage_0_iorig] 100.00
[swap_rimage_0_imeta] vg1 ewi-ao---- 132.00m
[swap_rimage_0_iorig] vg1 -wi-ao---- 8.00g
[swap_rimage_1] vg1 gwi-aor--- 8.00g [swap_rimage_1_iorig] 100.00
[swap_rimage_1_imeta] vg1 ewi-ao---- 132.00m
[swap_rimage_1_iorig] vg1 -wi-ao---- 8.00g
[swap_rmeta_0] vg1 ewi-aor--- 4.00m
[swap_rmeta_1] vg1 ewi-aor--- 4.00m

root@ds-03:~# df # filtered
Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/vg1-root 10218772 2863956 6814144 30% /
/dev/nvme0n1p1 446282 109365 308450 27% /boot /dev/mapper/vg1-scratch 10218772 24 9678076 1% /scratch /dev/mapper/vg1-docker 514937088 805892 487900412 1% /var/lib/docker /dev/nvme1n1p1 486456 5972 480484 2% /boot/efi

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hello,

On Wed, May 22, 2024 at 10:13:06AM +0000, Andy Smith wrote:

metadata tags to some PVs prevented grub from assembling them,

grub is indeed very fragile if you use dm-integrity anywhere on any of
your LVs on the same VG where /boot is (or at least if in the list
of LVs, the dm-integrity protected ones come first).

I guess it's a general problem how grub2 parses LVM, yes,
as soon as their are special things going on, it somehow breaks.

However, if you don't have /boot on LVM, hand-fixing grub2 can be
trivial, e.g. here on another system with /boot/efi on 1st disk's first partition and /boot on 2nd disk's first partition.

linux (hd1,1)vmlinuz-5.10.0-29-amd64 root=/dev/mapper/vg1-root ro quiet
initrd (hd1,1)initrd.img-5.10.0-29-amd64
boot

(you even have completions in grub's interactive boot system)

and it boots. Next step: I am going to make me a USB boot key for that
system, in case (first using a simple mount of two partitions of the
USB key on /boot, respectively /boot/efi (vfat), then update-grub,
or if it breaks, completely by hand like above -- I have been using
syslinux for the last 20 years or so for that purpose, but it gets
apparently too complicated with Secure Boot and stuff).

PS: I have from now on decided I will always use a /boot no longer
on LVM but on a separate partition, like the /boot/efi, it
seems, indeed, much less fragile. Aka, back to what I
was doing a few years ago before my confidence in grub2
got apparently too high :)

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hello,

On Wed, May 22, 2024 at 08:57:38AM +0200, Marc SCHAEFER wrote:

I will try this work-around and report back here. As I said, I can
live with /boot on RAID without dm-integrity, as long as the rest can be dm-integrity+raid protected.

I'm interested in how you get on.

I don't (yet) use dm-integrity, but I have seen extreme fragility in
grub with regard to LVM. For example, a colleague of mine recently
lost 5 hours of their life (and their SLA budget) when simply adding
metadata tags to some PVs prevented grub from assembling them,
resulting in a hard to debug failed boot at next boot.

Anything that involves grub having to interact with LVM just seems
really fragile.

Thanks,
Andy

--
https://bitfolk.com/ -- No-nonsense VPS hosting

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

I found this [1], quoting: "I'd also like to share an issue I've
discovered: if /boot's partition is a LV, then there must not be a raidintegrity LV anywhere before that LV inside the same VG. Otherwise, update-grub will show an error (disk `lvmid/.../...' not found) and GRUB cannot boot. So it's best if you put /boot into its own VG. (PS: Errors
like unknown node '..._rimage_0 can be ignored.)"

Hmm... I've been using a "plain old partition" for /boot (with
everything else in LVM) for "ever", originally because the boot loader
was not able to read LVM, and later out of habit. I was thinking of
finally moving /boot into an LV to make things simpler, but I see that
it'd still be playing with fire (AFAICT booting off of LVM was still not supported by U-Boot either last time I checked). 🙁


Stefan

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hello,

On Wed, May 22, 2024 at 05:03:34PM -0400, Stefan Monnier wrote:

Hmm... I've been using a "plain old partition" for /boot (with
everything else in LVM) for "ever", originally because the boot loader
was not able to read LVM, and later out of habit. I was thinking of
finally moving /boot into an LV to make things simpler, but I see that
it'd still be playing with fire

grub supports, for a long time:

- / on LVM, with /boot within that filesystem
- /boot on LVM, separately

(it also worked with LILO, because LILO would record the exact address
where the kernel & initrd was, regardless of abstractions layers :->)

Recently, I have been playing with RAID-on-LVM (I was mostly using LVM
on md before, which worked with grub), and it works too.

Where grub fails, is if you have /boot on the same LVM volume group
where any of the LVs "before him in order" have:

- dm-integrity
- specific metadata

So yes, any advanced setup might break grub, and so the easiest is to
have /boot on its separate partition again for the time being.

Which makes two partitions of you also have an UEFI.

(AFAICT booting off of LVM was still not
supported by U-Boot either last time I checked). ????

No idea about that one, sorry.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)