• sysadmin tip 101: versions matter.

    From Jonathan N. Little@21:1/5 to All on Fri Aug 25 16:15:37 2023
    Situation: Server uses to mirrored drives to save backup data. One drive
    was throwing bad sectors. Dreaded 3TB Constellation drives so replaced
    with 4TB IronWolf. After transferring 2.2TB data from the one good drive
    to the two new drives installed drives in server. Server would only boot
    into emergency mode. In emergency mode a manual mount command would
    mount the two drives and you can exit to regular login.

    Diagnosis: Replacing the new drives triggered fsck upon boot, but systemd.fsck@dev-disk-by...service failed for each new drive. UUIDs were correctly updated in fstab. Went and tried manually fsck drives and
    e2fsck failed with incompatible version.

    Conclusion: (Or the DOH-Moment) That server is still running 18.04 LTS
    and I formatted the new drives with a usb drive dock and I am running
    23.04! The files are currently accessible and writable so I temporarily disabled fsck for those two drives. In a live 18.04 session formatted up
    a new drive with correct version. Luckily I have a 3rd spare drive to
    expedite this. Have another system coping files from old good drive to
    this new drive so it can latter be swapped to the server and repeat the processes for the other drive...Just have to get it done before Sunday
    when the backup occurs.

    Penalty: Humiliation and time. Just passing on the info and maybe save
    someone else of the same mistake.

    BTW: have not found what they did to change the format of ext4 from the different versions OS. Both drives were gpt and same format. One would
    think that would not matter...


    --
    Take care,

    Jonathan
    -------------------
    LITTLE WORKS STUDIO
    http://www.LittleWorksStudio.com

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Mark Bourne@21:1/5 to Jonathan N. Little on Sat Aug 26 13:02:56 2023
    Jonathan N. Little wrote:
    Situation: Server uses to mirrored drives to save backup data. One drive
    was throwing bad sectors. Dreaded 3TB Constellation drives so replaced
    with 4TB IronWolf. After transferring 2.2TB data from the one good drive
    to the two new drives installed drives in server. Server would only boot
    into emergency mode. In emergency mode a manual mount command would
    mount the two drives and you can exit to regular login.

    Diagnosis: Replacing the new drives triggered fsck upon boot, but systemd.fsck@dev-disk-by...service failed for each new drive. UUIDs were correctly updated in fstab. Went and tried manually fsck drives and
    e2fsck failed with incompatible version.

    Conclusion: (Or the DOH-Moment) That server is still running 18.04 LTS
    and I formatted the new drives with a usb drive dock and I am running
    23.04! The files are currently accessible and writable so I temporarily disabled fsck for those two drives. In a live 18.04 session formatted up
    a new drive with correct version. Luckily I have a 3rd spare drive to expedite this. Have another system coping files from old good drive to
    this new drive so it can latter be swapped to the server and repeat the processes for the other drive...Just have to get it done before Sunday
    when the backup occurs.

    Penalty: Humiliation and time. Just passing on the info and maybe save someone else of the same mistake.

    BTW: have not found what they did to change the format of ext4 from the different versions OS. Both drives were gpt and same format. One would
    think that would not matter...

    It is surprising that something that significant would have changed in
    the formatting. Perhaps the older fsck is just being cautious, and
    refusing to try fixing a partition formatted with a newer version of
    ext4, which might use features the older fsck doesn't know about. And
    then the rest of the system refuses to mount it because fsck failed.

    However, a couple of years ago I did have an issue with a couple of
    disks that I'd been using with a USB-SATA adapter, which then couldn't
    be read when attached directly to the PC's SATA bus. It turns out that
    some USB-SATA adapters misreport the logical block size used by the disk
    (seems to be an issue with a commonly used chipset). When I attached a
    disk to the USB adapter and formatted it, the adapter reported 4096 byte logical blocks. So the GPT partition table was placed at block 1, 4096
    bytes into the disk, and partition addresses given in 4096 byte blocks.
    But when I attached the disk directly to the PC's SATA bus, it reported
    512 byte logical blocks. The partition table couldn't be found at block
    1 (now 512 bytes into the disk) and, even if could have been found, the partition addresses would all have been misinterpreted.

    Some relevant information I came across while trying to figure this one out: <https://askubuntu.com/questions/909041/harddrive-on-usb-to-sata-adapter-not-showing-full-size>
    <https://bugzilla.redhat.com/show_bug.cgi?id=734015> <https://www.spinics.net/lists/linux-usb/msg70029.html>

    That doesn't seem like the same issue you had, since your problems were
    with fsck, so in your case it sounds like the partitions were read
    correctly and the filesystem was found.

    --
    Mark.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Paul@21:1/5 to Mark Bourne on Sat Aug 26 10:16:08 2023
    On 8/26/2023 8:02 AM, Mark Bourne wrote:
    Jonathan N. Little wrote:
    Situation: Server uses to mirrored drives to save backup data. One drive
    was throwing bad sectors. Dreaded 3TB Constellation drives so replaced
    with 4TB IronWolf. After transferring 2.2TB data from the one good drive
    to the two new drives installed drives in server. Server would only boot
    into emergency mode. In emergency mode a manual mount command would
    mount the two drives  and you can exit to regular login.

    Diagnosis: Replacing the new drives triggered fsck upon boot, but
    systemd.fsck@dev-disk-by...service failed for each new drive. UUIDs were
    correctly updated in fstab. Went and tried manually fsck drives and
    e2fsck failed with incompatible version.

    Conclusion: (Or the DOH-Moment) That server is still running 18.04 LTS
    and I formatted the new drives with a usb drive dock and I am running
    23.04! The files are currently accessible and writable so I temporarily
    disabled fsck for those two drives. In a live 18.04 session formatted up
    a new drive with correct version. Luckily I have a 3rd spare drive to
    expedite this. Have another system coping files from old good drive to
    this new drive so it can latter be swapped to the server and repeat the
    processes for the other drive...Just have to get it done before Sunday
    when the backup occurs.

    Penalty: Humiliation and time. Just passing on the info and maybe save
    someone else of the same mistake.

    BTW: have not found what they did to change the format of ext4 from the
    different versions OS. Both drives were gpt and same format. One would
    think that would not matter...

    It is surprising that something that significant would have changed in the formatting.  Perhaps the older fsck is just being cautious, and refusing to try fixing a partition formatted with a newer version of ext4, which might use features the older
    fsck doesn't know about.  And then the rest of the system refuses to mount it because fsck failed.

    However, a couple of years ago I did have an issue with a couple of disks that I'd been using with a USB-SATA adapter, which then couldn't be read when attached directly to the PC's SATA bus.  It turns out that some USB-SATA adapters misreport the
    logical block size used by the disk (seems to be an issue with a commonly used chipset).  When I attached a disk to the USB adapter and formatted it, the adapter reported 4096 byte logical blocks.  So the GPT partition table was placed at block 1, 4096
    bytes into the disk, and partition addresses given in 4096 byte blocks. But when I attached the disk directly to the PC's SATA bus, it reported 512 byte logical blocks.  The partition table couldn't be found at block 1 (now 512 bytes into the disk) and,
    even if could have been found, the partition addresses would all have been misinterpreted.

    Some relevant information I came across while trying to figure this one out: <https://askubuntu.com/questions/909041/harddrive-on-usb-to-sata-adapter-not-showing-full-size>
    <https://bugzilla.redhat.com/show_bug.cgi?id=734015> <https://www.spinics.net/lists/linux-usb/msg70029.html>

    That doesn't seem like the same issue you had, since your problems were with fsck, so in your case it sounds like the partitions were read correctly and the filesystem was found.


    That's done on purpose, and the product would have
    been <cough> popular during a certain epoch (WinXP).
    I'm not really sure such a shenanigan is really necessary
    on USB, as the addressing there just might be a SCSI CDB
    and capable of addressing large objects without a problem.
    But I suppose MSDOS partition table support would be worth
    something, and that would be the incentive to screw around.

    https://superuser.com/questions/1271871/4k-emulation-sata-usb-controllers

    There were some RAID controllers with a similar capability.
    Maybe an Areca. Instead of a 2TB max drive size, the Areca
    could host a 16TB drive/array and the user could use MSDOS partitioning.
    At the time they were doing that, maybe GPT didn't exist yet ?
    WinXP did not support GPT, so it was likely introduced after
    WinXP.

    The MSDOS partitioning table had 32 bit fields in it. And that
    may have worked out to 2.2TB (making 2TB the largest disk you
    would safely use). Various schemes were invented to handle
    larger disks, such as Acronis Capacity Manager and a couple others.

    The Areca trick existed before Acronis Capacity Manager.
    I only accidentally found out about that, while skimming through
    the user manual for an Areca (no, I'm not actually rich enough
    to afford an Areca card).

    This is a kind of feature, where the PCB should have a jumper to
    select or de-select such a feature. It should not be totally automated.
    Not everyone wants or needs that.

    *******

    On a slightly different topic...

    There are OSes now that support 4Kn drives. I do not recommend
    people search out 4Kn drives on purpose. A hyperstorage site may like
    such things, but most people don't own enough good software to handle
    4Kn drives. If there's a problem, you could be left holding a bag of
    useless bits. The OS information here is likely incorrect, because
    there was a patch part way through the W10/W11 era, intended for actual support. Real support may have been later, than any documentation claims. Earlier OSes may not have had working support (boot from 4Kn drive).

    https://en.wikipedia.org/wiki/Advanced_Format

    4Kn drives were showing up at retail at one time. I would expect
    so many consumers bought those by mistake and returned for a refund,
    the vendors figured out they should hide those. Maybe a site that sells
    server accessories would list those SKUs. But if your OSes do not support
    them, they would be a bad deal. I like to buy hardware when I can, that
    every computer in the house can use, instead of just one computer barely supporting it. I would not consider a 4Kn, even for experiments.

    The highest capacity drive I own, I think it's a 512e, as it needs
    no pampering. The only problem that drive has, is the holes are drilled
    in the wrong place on it. The drive mounting holes are different
    at perhaps 8TB or larger. The 6TB drives still have conventional holes.
    At the 8TB level, some could have old-holes, some could have new-holes.
    I only own one big drive, and I can tell ya, that hole issue is a
    fucking disaster :-/ None of my trays attach properly.

    Paul

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jonathan N. Little@21:1/5 to Mark Bourne on Sun Aug 27 15:56:46 2023
    Mark Bourne wrote:

    However, a couple of years ago I did have an issue with a couple of
    disks that I'd been using with a USB-SATA adapter, which then couldn't
    be read when attached directly to the PC's SATA bus.  It turns out that
    some USB-SATA adapters misreport the logical block size used by the disk (seems to be an issue with a commonly used chipset).

    It can depend on the size of the drive. I have external drive docks for
    coping and moving data on drives. My older dual Orico will handle up to
    2TB drives but for larger drives I had to get a newer one. I have some
    little USB-SATA dongles, one I use with a little PI server, but these
    seem to have the older drive limit. All is dependent in the controller
    chip used.

    --
    Take care,

    Jonathan
    -------------------
    LITTLE WORKS STUDIO
    http://www.LittleWorksStudio.com

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jonathan N. Little@21:1/5 to Mark Bourne on Sun Aug 27 16:03:47 2023
    Mark Bourne wrote:
    That doesn't seem like the same issue you had, since your problems were
    with fsck, so in your case it sounds like the partitions were read
    correctly and the filesystem was found.

    Yes, the drives were fully accessible and read|writable by the 18.04
    server. Just auto mounting failed because the fschk flag was set and why
    I temporarily disabled in fstab. Final note is I reformatted drives with
    18.04 and now no issue. Also noted that this server desperately needs to
    be updated.

    --
    Take care,

    Jonathan
    -------------------
    LITTLE WORKS STUDIO
    http://www.LittleWorksStudio.com

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)