Hi!
It looks like isorecorder not available since a while ago. May I remove it from the list in w.d.o/faq/CD index page?
At the same time, rufus (https://rufus.ie/) should be mentioned as a great app on Windows to make bootable usb stick from iso (GNU licensing, source code available on GitLab).
Please cc me on the answer, I'm not a member of this list, but you can find me on debian-www.
Zibi
Hi!
It looks like isorecorder not available since a while ago. May I remove it from the list in w.d.o/faq/CD index page?
At the same time, rufus (https://rufus.ie/) should be mentioned as a great app on Windows to make bootable usb stick from iso (GNU licensing, source code available on GitLab).
Please cc me on the answer, I'm not a member of this list, but you can find me on debian-www.
Zibi
If you want to recommend rufus, please do so ONLY with a prominent, un-ignoreable warning explaining that the user MUST select rufus'
"DD mode" at the prompt immediately before writing the media.
Hi,
Lou Poppler wrote:
If you want to recommend rufus, please do so ONLY with a prominent, un-ignoreable warning explaining that the user MUST select rufus'
"DD mode" at the prompt immediately before writing the media.
I understand that manual selection of this mode has been removed in
favor of automatic detection of dd candidates:
https://github.com/pbatard/rufus/issues/1148#issuecomment-395562137
I don't use Rufus myself but it is my answer to users of MS-Windows when
they ask what to do with an ISO that is bootable from USB-stick.
Have a nice day :)
Thomas
... > It looks like isorecorder not available since a while ago. May I
... > remove it from the list in w.d.o/faq/CD index page?
... > At the same time, rufus (https://rufus.ie/ [rufus.ie]) should be
... > mentioned as a great app on Windows to make bootable usb stick from
... > iso (GNU licensing, source code available on GitLab)
If you want to recommend rufus, please do so ONLY with a prominent, un-ignoreable warning explaining that the user MUST select rufus'
"DD mode" at the prompt immediately before writing the media.
I understand that manual selection of this mode has been removed in
favor of automatic detection of dd candidates:
https://github.com/pbatard/rufus/issues/1148#issuecomment-395562137
I corresponded with rufus's author Pete Batard, including some logs from #debian discussing my testing of rufus, and he kindly returned a lengthy response to my email.
Now the one thing that might have thrown this user off is that Rufus always prompts to download these components BEFORE it asks to choose between ISO and DD mode,
DD-mode image writing does often throw
Windows people off, as they think that their drive has been reduced in
size to just the ESP since Windows will not mount the main, larger installation partition.
On Sun, 2021-01-10 at 18:13 +0100, Thomas Schmitt wrote:
Hi,
Lou Poppler wrote:
If you want to recommend rufus, please do so ONLY with a prominent,
un-ignoreable warning explaining that the user MUST select rufus'
"DD mode" at the prompt immediately before writing the media.
I understand that manual selection of this mode has been removed in
favor of automatic detection of dd candidates:
https://github.com/pbatard/rufus/issues/1148#issuecomment-395562137
I don't use Rufus myself but it is my answer to users of MS-Windows when
they ask what to do with an ISO that is bootable from USB-stick.
Hello All,
I am the main developer of Rufus and since I am now being involved in
this discussion, let me reply to a few points:
On 2021.01.10 18:56, Lou Poppler wrote:
On Sun, 2021-01-10 at 18:13 +0100, Thomas Schmitt wrote:
Hi,
Lou Poppler wrote:
If you want to recommend rufus, please do so ONLY with a prominent,
un-ignoreable warning explaining that the user MUST select rufus'
"DD mode" at the prompt immediately before writing the media.
For people who seem to be under the impression that "(Rufus) will
replace parts of the debian image with its own recipe of loaders and configurations", I have to vehemently dispute that it is doing any such thing.
The goal of Rufus is to create a bootable USB with content that is as
close as possible to the ISO content *AND* is bootable as USB media,
period.
As such, when booting in UEFI mode, the *only* element that Rufus may
modify are the labels used in the GRUB/Syslinux config files (kernel
option) for source media lookup, to accommodate FAT32 limitations on
that matter (since FAT limits labels to 11 uppercase characters, and ISO volumes can and usually do use labels with much longer names).
Apart from this, everything else is a binary identical copy of the ISO content, which you can easily validate if needed.
Then, for BIOS mode, we of course need to install GRUB or Syslinux bootloaders, which can't come from the ISO but which we produce from
vanilla official releases, and rely on the distro not to have deviated
too much in terms of patches from vanilla GRUB/Syslinux. However, if
such deviation occurs, you usually get an early boot error that is easy
to diagnose, and we also have provision, through the bootloader download facility of Rufus, to produce GRUB/Syslinux bootloaders that include
specific fixes, such as the ones required for BootHole, since GRUB has
yet to produce a mainline release that includes those.
So I have to be adamant that if the media doesn't boot, especially in
UEFI mode, then it's not a matter of Rufus introducing "its own recipe
of loaders and configuration" but rather a problem of distro maintainers
not having properly tested UEFI/FAT32 boot mode by extracting the ISO
content onto a FAT32 partition, and trying to boot said media. And I am pretty positive that you will find that the users who reported issues
after creating their media through Rufus, will report the exact same
issues if creating the media themselves without using a third party
utility.
Which brings me nicely to the point that the current Debian Bullseye
test images have broken said mode of operation (UEFI install from ISO
content extracted on FAT32) early in December, which is causing problems
for people trying to install Debian on Pi 4 for instance ([1] but my
testing shows that this applies to more than arm64. And yes, I've been meaning to file a bug for this, but I haven't had a chance to do that yet).
So le me reiterate this plea:
PLEASE, PLEASE, PLEASE, do not treat ISOHybrid as an excuse to ignore
non DD mode of installation, because, when you look at it objectively,
you should find that DD is *NOT* a panacea (for instance, DD mode is completely useless for installation of vanilla Bullseye on a Pi 4,
whereas FAT32 extraction, *when* not broken, gives you the same
experience as if you were installing Debian on a UEFI x86 PC) and users
do and will rely on a non DD mode of creating their install media.
If you actively expect that everybody should use DD-mode only when
creating a bootable media from an ISOHybrid, you *WILL* diminish the
user experience.
I understand that manual selection of this mode has been removed in
favor of automatic detection of dd candidates:
https://github.com/pbatard/rufus/issues/1148#issuecomment-395562137
I'm afraid Thomas has been interpreting the item to which this pertains wrong.
Rufus used to have a separate entries in one of it's UI dropdown for the
type of image the user wanted to select. One of these entries was for
"ISO Image", which would trigger a default filter of "*.iso" when
selecting an image, and another entry was for "DD image", which would
set a default filter of "*.img, *.disk, *.vhd, ...".
But considering that, behind the scenes Rufus was ultimately performing
its own detection of the properties of an image, and that there was no
real difference between these 2 options if you applied a filter of
"*.*", it of course made sense to consolidate the 2 dropdown elements
into a single "Disk or ISO Image".
However, this threw of people who were used to seeing a separate option
for "DD image" and led them to think that this option had been removed altogether.
So I'm afraid that Thomas' interpretation is incorrect. Automatic
detection was always being enacted, and the only thing the DD or ISO
"mode" did was apply a different filter for the file selection dialog.
I don't use Rufus myself but it is my answer to users of MS-Windows when >>> they ask what to do with an ISO that is bootable from USB-stick.
As Lou pointed out from a previous e-mail I sent (and as the plea above should make clear) I see the matter very differently, on account of the feedback I receive from Windows users, as well as what I am seeing on
social media, with Windows users really being thrown off by not being
able to "see" the full content that has been copied to their USB.
For instance, I can literally point to reddit post after reddit post
where a Windows user is utterly confused after they created a media in
DD mode, by the fact that Windows only mounts the ESP and that there
only seems to be a couple files there. Most of these users end up
believing that their media was improperly created and do not try to boot
it at all.
This is why I made the decision to promote ISO/File copy over DD copy in Rufus for ISO Hybrid.
I will however point out that, when doing so, Rufus produces the
following *explicit* warning (which appears every single time a user
creates media from an ISOHybrid):
----------------------------------------------------------------------- ISOHybrid image detected
The image you have selected is an 'ISOHybrid' image. This means it can
be written either in ISO (file copy) mode or DD (disk image) mode.
Rufus recommends using ISO mode, so that you always have full access to
the drive after writing it.
However, if you encounter issues during boot, you can try writing this
image again in DD mode.
Please select the mode that you want to use to write this image:
o Write in ISO mode (Recommended)"
o Write in DD mode -----------------------------------------------------------------------
In other words, Rufus prominently advises users to try recreating the
media in DD mode if they think they encountered an issue in DD mode.
As such, I can't see myself agreeing with proponents of "If you use
Rufus, only use DD" as we are literally doing everything we can to help users, and I will maintain that most of the responsibility of ensuring
that ISO mode works is with the distro maintainers, since, once again,
all Rufus does (apart from changing the label in config files, *IF* necessary) is perform a 1:1 copy of all the ISO content onto a FAT32 media.
Moreover, since this mode of operation is very convenient, on account
that it really doesn't even require the user to use a third party
application (including Rufus, but also including a dd imager) and is
cross platform, I can't help be feel dismayed at seeing the apparent
trend that some distro maintainers appear to have of disregarding simple
ISO extraction on FAT32 media for UEFI boot, on account that "users
should just use DD anyway".
From where I stand, doing so only leads to a worse overall experience
for users trying to install your distro.
So I hope we can all agree that there is a place for both DD and
ISO/file copy mode, and that the promotion of one should not relegate
the other to a feature that is no longer being properly tested and left
to wither (as we can see with the >1 month breakage of UEFI-FAT32 installation from Bullseye testing netinstall ISOs).
Regards,
/Pete
The goal of Rufus is to create a bootable USB with content that is as[...]
close as possible to the ISO content *AND* is bootable as USB media, period.
As such, when booting in UEFI mode, the *only* element that Rufus may
modify are the labels used in the GRUB/Syslinux config files (kernel
option) for source media lookup, to accommodate FAT32 limitations on
that matter (since FAT limits labels to 11 uppercase characters, and ISO volumes can and usually do use labels with much longer names).
Apart from this, everything else is a binary identical copy of the ISO content, which you can easily validate if needed.
Then, for BIOS mode, we of course need to install GRUB or Syslinux bootloaders, which can't come from the ISO but which we produce from
vanilla official releases,
On Sun, 2021-01-10 at 20:19 +0000, Pete Batard wrote:
[...]
The goal of Rufus is to create a bootable USB with content that is as
close as possible to the ISO content *AND* is bootable as USB media, period. >>
As such, when booting in UEFI mode, the *only* element that Rufus may
modify are the labels used in the GRUB/Syslinux config files (kernel
option) for source media lookup, to accommodate FAT32 limitations on
that matter (since FAT limits labels to 11 uppercase characters, and ISO
volumes can and usually do use labels with much longer names).
Apart from this, everything else is a binary identical copy of the ISO
content, which you can easily validate if needed.
Then, for BIOS mode, we of course need to install GRUB or Syslinux
bootloaders, which can't come from the ISO but which we produce from
vanilla official releases,
Let me emphasize that the unmodified debian installer images, when copied to a
USB stick via (unix) dd or cp, boot just fine in both UEFI mode and BIOS mode.
They also work just fine when copied unmodified to a CD or DVD.
"../pool/main/f/firmware-free/firmware-linux-free_20200122-1_all.deb"failed: Operation not permitted (1)
Hi,
the advantages of putting an unmodified ISO image plainly onto a USB are:
- The boot and installation process is as near to DVD as possible.
- If anything goes wrong during that process, it is clear that debian-cd
is in charge of diagnosing and hopefully managing the effort to provide
a correction.
Nevertheless, the goals of Pete are valid too:
- A USB stick with large FAT partition is what many users expect.
- The (unspecified by UEFI) way of copying EFI boot equipment and data
payload into a FAT filesystem seems to be indeed the way how experienced
users of MS-Windows prepare a bootable USB stick.
- Copying from filesystem to filesystem is at least one order of magnitude
less prone to fatal mishaps than is copying onto a device.
Both approaches are combinable:
- Have all EFI equipment as directory tree in the ISO filesystem. I.e.
not only as image file or as appended partition.
- Make sure that the installation works without name conflicts from a
filesystem with sloppy naming rules. I.e. from FAT.
And to promote my own song:
Debian ISOs should strive for a specs compliant partition table with as
few borderline hacks as possible.
It would be fully compliant if bearing a MBR partition table with disjoint ISO and EFI partitions. But during the Ubuntu ISO reform there were
problems with new Lenovos which booted only with GPT
https://bugs.launchpad.net/ubuntu-cdimage/+bug/1886148
and with old HPs which are well known to boot only if some MBR partition bears the boot flag
https://bugs.launchpad.net/ubuntu-cdimage/+bug/1899308
The Lenovo case is probably not fully explored, because it seems that
the same machines boot from our current abominable MBR-with-invalid-GPT layout. The HP case needed a small hack by adding an MBR partition of
type 0x00 and size 1 which bears the boot flag.
I am not aware of failure reports with the Ubuntu 20.10 ISOs which could
be blamed on their partition layout.
Yes, as we've discussed elsewhere, that is pretty much my point.
symlinks [...] will of course not translate to FAT [...]
On 2021.01.10 18:56, Lou Poppler wrote:
On Sun, 2021-01-10 at 18:13 +0100, Thomas Schmitt wrote:
Hi,
Lou Poppler wrote:
If you want to recommend rufus, please do so ONLY with a prominent,
un-ignoreable warning explaining that the user MUST select rufus'
"DD mode" at the prompt immediately before writing the media.
For people who seem to be under the impression that "(Rufus) will replace >parts of the debian image with its own recipe of loaders and configurations", >I have to vehemently dispute that it is doing any such thing.
The goal of Rufus is to create a bootable USB with content that is as close >as possible to the ISO content *AND* is bootable as USB media, period.
As such, when booting in UEFI mode, the *only* element that Rufus may modify >are the labels used in the GRUB/Syslinux config files (kernel option) for >source media lookup, to accommodate FAT32 limitations on that matter (since >FAT limits labels to 11 uppercase characters, and ISO volumes can and usually >do use labels with much longer names).
Apart from this, everything else is a binary identical copy of the ISO >content, which you can easily validate if needed.
Then, for BIOS mode, we of course need to install GRUB or Syslinux >bootloaders, which can't come from the ISO but which we produce from vanilla >official releases, and rely on the distro not to have deviated too much in >terms of patches from vanilla GRUB/Syslinux. However, if such deviation >occurs, you usually get an early boot error that is easy to diagnose, and we >also have provision, through the bootloader download facility of Rufus, to >produce GRUB/Syslinux bootloaders that include specific fixes, such as the >ones required for BootHole, since GRUB has yet to produce a mainline release >that includes those.
So I have to be adamant that if the media doesn't boot, especially in UEFI >mode, then it's not a matter of Rufus introducing "its own recipe of loaders >and configuration" but rather a problem of distro maintainers not having >properly tested UEFI/FAT32 boot mode by extracting the ISO content onto a >FAT32 partition, and trying to boot said media. And I am pretty positive that >you will find that the users who reported issues after creating their media >through Rufus, will report the exact same issues if creating the media >themselves without using a third party utility.
Which brings me nicely to the point that the current Debian Bullseye test >images have broken said mode of operation (UEFI install from ISO content >extracted on FAT32) early in December, which is causing problems for people >trying to install Debian on Pi 4 for instance ([1] but my testing shows that >this applies to more than arm64. And yes, I've been meaning to file a bug for >this, but I haven't had a chance to do that yet).
So le me reiterate this plea:
PLEASE, PLEASE, PLEASE, do not treat ISOHybrid as an excuse to ignore non DD >mode of installation, because, when you look at it objectively, you should >find that DD is *NOT* a panacea (for instance, DD mode is completely useless >for installation of vanilla Bullseye on a Pi 4, whereas FAT32 extraction, >*when* not broken, gives you the same experience as if you were installing >Debian on a UEFI x86 PC) and users do and will rely on a non DD mode of >creating their install media.
Hey Pete,
HNY etc. Thanks for joining the discussion! :-)
On Sun, Jan 10, 2021 at 08:19:15PM +0000, Pete Batard wrote:
On 2021.01.10 18:56, Lou Poppler wrote:
On Sun, 2021-01-10 at 18:13 +0100, Thomas Schmitt wrote:
Hi,
Lou Poppler wrote:
If you want to recommend rufus, please do so ONLY with a prominent,
un-ignoreable warning explaining that the user MUST select rufus'
"DD mode" at the prompt immediately before writing the media.
For people who seem to be under the impression that "(Rufus) will replace
parts of the debian image with its own recipe of loaders and configurations",
I have to vehemently dispute that it is doing any such thing.
The goal of Rufus is to create a bootable USB with content that is as close >> as possible to the ISO content *AND* is bootable as USB media, period.
As such, when booting in UEFI mode, the *only* element that Rufus may modify >> are the labels used in the GRUB/Syslinux config files (kernel option) for
source media lookup, to accommodate FAT32 limitations on that matter (since >> FAT limits labels to 11 uppercase characters, and ISO volumes can and usually
do use labels with much longer names).
Thanks for clarifying that.
Apart from this, everything else is a binary identical copy of the ISO
content, which you can easily validate if needed.
Then, for BIOS mode, we of course need to install GRUB or Syslinux
bootloaders, which can't come from the ISO but which we produce from vanilla >> official releases, and rely on the distro not to have deviated too much in >> terms of patches from vanilla GRUB/Syslinux. However, if such deviation
occurs, you usually get an early boot error that is easy to diagnose, and we >> also have provision, through the bootloader download facility of Rufus, to >> produce GRUB/Syslinux bootloaders that include specific fixes, such as the >> ones required for BootHole, since GRUB has yet to produce a mainline release >> that includes those.
So I have to be adamant that if the media doesn't boot, especially in UEFI >> mode, then it's not a matter of Rufus introducing "its own recipe of loaders >> and configuration" but rather a problem of distro maintainers not having
properly tested UEFI/FAT32 boot mode by extracting the ISO content onto a
FAT32 partition, and trying to boot said media. And I am pretty positive that
you will find that the users who reported issues after creating their media >> through Rufus, will report the exact same issues if creating the media
themselves without using a third party utility.
Which brings me nicely to the point that the current Debian Bullseye test
images have broken said mode of operation (UEFI install from ISO content
extracted on FAT32) early in December, which is causing problems for people >> trying to install Debian on Pi 4 for instance ([1] but my testing shows that >> this applies to more than arm64. And yes, I've been meaning to file a bug for
this, but I haven't had a chance to do that yet).
Argh. Thanks for raising this now, at least. I certainly haven't
consciously made any changes to debian-cd that would cause this, so
I'm a little surprised to hear about it. I'll take a look in the d-i
code...
So le me reiterate this plea:
PLEASE, PLEASE, PLEASE, do not treat ISOHybrid as an excuse to ignore non DD >> mode of installation, because, when you look at it objectively, you should >> find that DD is *NOT* a panacea (for instance, DD mode is completely useless >> for installation of vanilla Bullseye on a Pi 4, whereas FAT32 extraction,
*when* not broken, gives you the same experience as if you were installing >> Debian on a UEFI x86 PC) and users do and will rely on a non DD mode of
creating their install media.
Hmmm. What exactly is the problem with the Pi 4, out of curiosity?
On 2021.01.12 11:56, Steve McIntyre wrote:
On Sun, Jan 10, 2021 at 08:19:15PM +0000, Pete Batard wrote:
Which brings me nicely to the point that the current Debian Bullseye test >> > images have broken said mode of operation (UEFI install from ISO content >> > extracted on FAT32) early in December, which is causing problems for people
trying to install Debian on Pi 4 for instance ([1] but my testing shows that
this applies to more than arm64. And yes, I've been meaning to file a bug for
this, but I haven't had a chance to do that yet).
Argh. Thanks for raising this now, at least. I certainly haven't
consciously made any changes to debian-cd that would cause this, so
I'm a little surprised to hear about it. I'll take a look in the d-i
code...
Further testing (which I also mentioned in this thread) seems to indicate >that this only seems to apply for UAS media, so it's possible that it wasn't >a recent change, but that install media detection has not been compatible >with UAS devices for some time, which we only started to pick it up on >account that some of us have been testing Pi 4 Debian installation with UAS.
I'm still planning to look more closely into this when I get a chance, and >create a new bug (unless someone beats me to it), but if you have a UAS >device lying around, you should be able to replicate the issue with the >current Bullseye installation ISOs, including amd64 ones, as I confirmed that >the problem was (still) present in the latest netinst testing amd64 ISOs.
FYI, the USB subsystem should tell you if a device is UAS with something like >this (from dmesg):
[549114.637578] usb 8-1: new SuperSpeed Gen 1 USB device number 19 using >xhci-hcd
[549114.658815] usb 8-1: New USB device found, idVendor=174c, idProduct=55aa, >bcdDevice= 1.00
[549114.658837] usb 8-1: New USB device strings: Mfr=2, Product=3, >SerialNumber=1
[549114.658852] usb 8-1: Product: Ventura Ultra
[549114.658867] usb 8-1: Manufacturer: Mushkin
[549114.658881] usb 8-1: SerialNumber: 000000000025
[549114.666050] scsi host0: uas
[549114.668168] scsi 0:0:0:0: Direct-Access Mushkin Ventura Ultra 0
PQ: 0 ANSI: 6
Hmmm. What exactly is the problem with the Pi 4, out of curiosity?
The problem with the Pi 4 (as with all Pi's) is that it needs to load a set >of binary files, from SD or USB media, before it can boot anything, and we >also need to load the UEFI firmware from SD or USB.
Now the nice thing, that was introduced a few months ago on the Pi 4 is that >you can load all these files from an ESP, with the caveat that the ESP has to >be at the beginning of your media (which means for instance that even if >adding files onto it wasn't an issue, the partition layout used by the >current ISOHybrid when copied in DD mode would still make that media >unbootable on a Pi 4).
This logically leads to the idea that, if you are okay with sacrificing a bit >of space for the ESP, you can place all of these Raspberry Pi specific boot >files on it, along with the Debian installation content extracted from a >netinst or mini ISO (we advise netinst as it makes it faster to get to a >minimal serviceable install than mini for people with slow connection, and >because we don't see the extra space needed as that big of an issue) and once >you have that, you can boot and go through a full default installation of >Debian and end up with a fully working system.
And what makes it even better is that you don't even need to provision two >medias for this (install media + target), as the same media can transparently >act as both the installer and target.
Considering that the Pi has no "internal" disk or flash to install a distro >to, and that one must use either an SD card or USB drive, it does make a lot >of sense in my opinion not to require users to juggle with multiple media, >especially as, and this is one of the main pain points we are trying to avoid >through this method, using two media would force users to go through the >shell during the install process to ensure that they copy all the Pi boot >files and UEFI firmware from the install media to the target before they >reboot, as, again, a Pi will not boot any media where these extra files >aren't present.
But by reusing the ESP, where these files have already been copied, no extra >step is necessary.
Furthermore, if you simply go with the defaults during Debian installation, >the disk partitioner will recognize the ESP as reusable and set it as the ESP >to use for the new system, and therefore when GRUB installs for the final >system, it simply replaces the one that was used for booting the installer, >which ensures that, once the user reboot, they get into their newly installed >system.
I've never seen a UAS device here yet, let alone tested with one. :-)
So it's not quite such a surprise that there might be problems there.
I'm still planning to look more closely into this when I get a chance, and >> create a new bug (unless someone beats me to it), but if you have a UAS
device lying around, you should be able to replicate the issue with the
current Bullseye installation ISOs, including amd64 ones, as I confirmed that
the problem was (still) present in the latest netinst testing amd64 ISOs.
FYI, the USB subsystem should tell you if a device is UAS with something like
this (from dmesg):
[549114.637578] usb 8-1: new SuperSpeed Gen 1 USB device number 19 using
xhci-hcd
[549114.658815] usb 8-1: New USB device found, idVendor=174c, idProduct=55aa,
bcdDevice= 1.00
[549114.658837] usb 8-1: New USB device strings: Mfr=2, Product=3,
SerialNumber=1
[549114.658852] usb 8-1: Product: Ventura Ultra
[549114.658867] usb 8-1: Manufacturer: Mushkin
[549114.658881] usb 8-1: SerialNumber: 000000000025
[549114.666050] scsi host0: uas
[549114.668168] scsi 0:0:0:0: Direct-Access Mushkin Ventura Ultra 0
PQ: 0 ANSI: 6
ACK. I'm just looking for one to buy right now so I can test with
it. I can't see anybody selling the Mushkin drive you show here, which
is not helpful.
Silly question - do you have any suggestions for a
make/model to look for at all please? Google searches are basucally
useless here for finding that level of detail on USB flash drives. :-(
Sigh, OK. Firmware that doesn't deal flexibly with partitioning,
i.e. by reading/parsing the partition table properly. I thought we'd
be past that by now. :-(
Is the firmware assuming a specific offset for the beginning of the
ESP?
Or will it deal with GPT,
[Dropping CC to Lou as we've wandered, re-adding Thomas for the GPT question.]
For the
sake of this kind of use-case, is it sensible when driving xorriso to
tweak the created partition table to list the ESP as partition 1? At
the moment we're just using "-append_partition 2 0xef
/path/to/ESP.img", would just changing that to be partition 1 instead
work?
On 2021.01.13 18:49, Steve McIntyre wrote:
I've never seen a UAS device here yet, let alone tested with one. :-)
They've existed for quite a few years.
I think currently have about 4 of those in my arsenal, with the first one >purchased more than 6 or 7 years ago.
So it's not quite such a surprise that there might be problems there.
I'm still planning to look more closely into this when I get a chance, and >> > create a new bug (unless someone beats me to it), but if you have a UAS
device lying around, you should be able to replicate the issue with the
current Bullseye installation ISOs, including amd64 ones, as I confirmed that
the problem was (still) present in the latest netinst testing amd64 ISOs. >> >
FYI, the USB subsystem should tell you if a device is UAS with something like
this (from dmesg):
[549114.637578] usb 8-1: new SuperSpeed Gen 1 USB device number 19 using >> > xhci-hcd
[549114.658815] usb 8-1: New USB device found, idVendor=174c, idProduct=55aa,
bcdDevice= 1.00
[549114.658837] usb 8-1: New USB device strings: Mfr=2, Product=3,
SerialNumber=1
[549114.658852] usb 8-1: Product: Ventura Ultra
[549114.658867] usb 8-1: Manufacturer: Mushkin
[549114.658881] usb 8-1: SerialNumber: 000000000025
[549114.666050] scsi host0: uas
[549114.668168] scsi 0:0:0:0: Direct-Access Mushkin Ventura Ultra 0 >> > PQ: 0 ANSI: 6
ACK. I'm just looking for one to buy right now so I can test with
it. I can't see anybody selling the Mushkin drive you show here, which
is not helpful.
Yeah, that Mushkin device is definitely something I've had for more than 5 >years, and that I don't expect to be sold any longer. And most regular flash >drives would not be UAS.
Silly question - do you have any suggestions for a
make/model to look for at all please? Google searches are basucally
useless here for finding that level of detail on USB flash drives. :-(
OK. One thing you may want to check first, if you have any lying around, are >USB <-> SATA or USB <-> NVMe adapters, because these frequently turn to be >UAS.
For instance, the following USB <-> SATA "enclosure", that I purchased more >than 5 years ago, is UAS: >https://archive.plugable.com/products/usb3-sata-uasp1/ (it even says so in >the link).
I don't think they are sold brand new any more, but you might be able to find >one of those off e-bay.
Or, for something more recent and perhaps more convenient to get, the >following USB <-> NVMe enclosure is also UAS (at least for the "NVME - >10Gbps" version, which is what I have):
https://www.aliexpress.com/item/1005001742426847.html
Obviously, these last two devices require an additional NVMe or SATA disk to >be usable. But I'd say USB <-> NVMe adapters are likely to become more and >more widespread especially as can solve the problems of using something a bit >more reliable and faster (as well as much better with random I/O) than >regular flash drives, even more so on platforms that have USB 3.x but no >NVMe.
Plus I can certainly see people wanting to use the "extract all the Debian >installation files into an ESP" procedure on regular x86 UEFI PCs, using a >USB SSD (especially now that USB speeds are starting to catch up with NVMe), >in which case there's a good chance their installation media will be UAS...
Sigh, OK. Firmware that doesn't deal flexibly with partitioning,
i.e. by reading/parsing the partition table properly. I thought we'd
be past that by now. :-(
Three words: cheapest platform possible.
That's what the Pi Foundation are pursuing, and thus, by transitive property, >Broadcom, since that's who the Foundation chose as their supplier. As a >result you're being met with tons of limitations or quirks (such as a >poor/buggy PCIe DMA implementation of the SoC, limited mask ROM space that's >likely to result in major trade-offs, and, recently in order to save another >few more cents on the platform, the complete removal of the xHCI controller >EEPROM that existed in the initial Pi 4 models, which means that, if you >happen to reset xHCI, you must now take care of loading its firmware data >back in RAM).
My understanding is that SoC mask ROM is designed to look only at the first >FAT partition to load its boot files, and that we can actually count >ourselves lucky that the Pi 4 SoC can handle GPT as well as ESPs, as the mask >ROM from the SoC on Pi 3 and earlier models could only handle MBR and >partition types of 0xc/0xe (and not 0xef)...
But of course, the counterpoint of that is that the price point of the Pi >continues to make it one of the most popular platforms out there.
Is the firmware assuming a specific offset for the beginning of the
ESP?
Not that I can tell. I've tested with different offsets and I'm also not >aware of a specific value at which the first partition is expected to start.
Or will it deal with GPT,
It deals with a GPT, but it'll look no further than the first partition.
I'm pretty sure the current code (which should mostly be part of the mask >EEPROM of the SoC) does something like this:
1. Find the offset of the first partition, be it either MBR or GPT.
2. *Always assume* that it's FAT16/FAT32 and see if trying to load the >platform files works.
But, of course, all of this is closed source and coming from a chip >manufacturer that seems very tight lipped about anything that pertains to >their hardware. So it's hard to tell for sure...
Steve McIntyre wrote:
[Dropping CC to Lou as we've wandered, re-adding Thomas for the GPT
question.]
[Lou and i are subscribed to debian-cd, obviously. :))
It seems you dropped the original poster: The7up <the7up@gmail.com>
who might indeed not be interested in partitioning.]
For the
sake of this kind of use-case, is it sensible when driving xorriso to
tweak the created partition table to list the ESP as partition 1? At
the moment we're just using "-append_partition 2 0xef
/path/to/ESP.img", would just changing that to be partition 1 instead
work?
That would still work, but leave the ISO fileystem without a partition
entry to mark it.
The man page of xorrisofs says:
partition_number may be 1 to 4. Number 1 will put the whole ISO
image into the unclaimed space before partition 1.
Minimalistic experiment:
xorriso -as mkisofs -o test.iso /bin/ls -append_partition 1 0xef /bin/cp
yields (with /bin/ls as ISO payload having ~ 120 KB and /bin/cp as
appended payload having ~150 KB):
$ sbin/fdisk -l test.iso
...
Device Boot Start End Sectors Size Id Type
test.iso1 364 659 296 148K ef EFI (FAT-12/16/32)
--------------------------------------------------------------------------
I could tweak libisofs so that the ISO filesystem can appear as partition 2 >but that would be unusual by partition 2 sitting before partition 1.
On the other hand, there is urgent need for an ISO filesystem partition
only if the reader cannot mount the base device but only its partitions.
On Wed, Jan 13, 2021 at 08:16:39PM +0000, Pete Batard wrote:
On 2021.01.13 18:49, Steve McIntyre wrote:
I've never seen a UAS device here yet, let alone tested with one. :-)
Right. I've found and ordered a Startech USB3S2SAT3CB which looks
useful. I've got a couple of small spare SSDs that I can use that
with. I was hoping to find a source for reasonably-priced USB-UAS
flash drives but the only one I can see off-hand is the "Corsair
Voyager GTX", e.g. at
https://www.scan.co.uk/products/128gb-corsair-flash-voyager-gtx-usb-31-gen-1-type-a-pendrive-black-460mb-s-read-460mb-s-write-33k-40
Obviously, these last two devices require an additional NVMe or SATA disk to >>be usable. But I'd say USB <-> NVMe adapters are likely to become more and >>more widespread especially as can solve the problems of using something a bit >>more reliable and faster (as well as much better with random I/O) than >>regular flash drives, even more so on platforms that have USB 3.x but no >>NVMe.
Plus I can certainly see people wanting to use the "extract all the Debian >>installation files into an ESP" procedure on regular x86 UEFI PCs, using a >>USB SSD (especially now that USB speeds are starting to catch up with NVMe), >>in which case there's a good chance their installation media will be UAS...
Right. Now I know it's a problem, I can have a look.
On Thu, Jan 14, 2021 at 03:19:09PM +0000, Steve McIntyre wrote:
On Wed, Jan 13, 2021 at 08:16:39PM +0000, Pete Batard wrote:
On 2021.01.13 18:49, Steve McIntyre wrote:
I've never seen a UAS device here yet, let alone tested with one. :-)
...
Right. I've found and ordered a Startech USB3S2SAT3CB which looks
useful. I've got a couple of small spare SSDs that I can use that
with. I was hoping to find a source for reasonably-priced USB-UAS
flash drives but the only one I can see off-hand is the "Corsair
Voyager GTX", e.g. at
https://www.scan.co.uk/products/128gb-corsair-flash-voyager-gtx-usb-31-gen-1-type-a-pendrive-black-460mb-s-read-460mb-s-write-33k-40
Obviously, these last two devices require an additional NVMe or SATA disk toRight. Now I know it's a problem, I can have a look.
be usable. But I'd say USB <-> NVMe adapters are likely to become more and >>> more widespread especially as can solve the problems of using something a bit
more reliable and faster (as well as much better with random I/O) than
regular flash drives, even more so on platforms that have USB 3.x but no >>> NVMe.
Plus I can certainly see people wanting to use the "extract all the Debian >>> installation files into an ESP" procedure on regular x86 UEFI PCs, using a >>> USB SSD (especially now that USB speeds are starting to catch up with NVMe),
in which case there's a good chance their installation media will be UAS... >>
OK, and I've sussed it.
I splurged on that Voyager drive too, and
tested with it. The problem we have is deep down in the way that Linux
/ udev describe UAS-attached drives. In d-i we look for CD/DVD media
and then USB drives, and for the latter we look for "ID_BUS=usb" from "udevadm info". On UAS media instead we get "ID_BUS=ata" so they don't
show up.
This isn't a recent change, it's been like that way forever.
I've just added extra fallback code in d-i now to check for "usb" in
the "ID_PATH" string too, and I've just run an installation now using
that build. All looks good here. The changes will be in the next daily
build early tomorrow.
Hilariously, it's not just d-i that doesn't deal with the UAS edge
cases here. My test PC here would only recognise UAS media in BIOS
mode (i.e. CSM enabled), not in UEFI mode.
In my initial debugging
here I ended up having to use two USB sticks - an older one to boot
off and then remove, then the Voyager one to test detection code
with. Firmware update needed! :-)
On 2021.01.16 23:00, Steve McIntyre wrote:
I splurged on that Voyager drive too, and
tested with it. The problem we have is deep down in the way that Linux
/ udev describe UAS-attached drives. In d-i we look for CD/DVD media
and then USB drives, and for the latter we look for "ID_BUS=usb" from
"udevadm info". On UAS media instead we get "ID_BUS=ata" so they don't
show up.
That makes a lot of sense.
Thanks for investing resources to figure this one out. I'm pretty sure this >will make a lot of people who want to install Debian (on Pi or on other >platforms), and that would otherwise have run into this issue, very happy.
This isn't a recent change, it's been like that way forever.
Yeah. I'm pretty sure I switched to testing with a UAS drive right at the >time someone reported they had an issue (that was also linked to them using >UAS media) which logically led me to think that there existed a regression. >But once we zeroed in on UAS being the culprit, I had a feeling that it might >be something that had existed for some time.
I've just added extra fallback code in d-i now to check for "usb" in
the "ID_PATH" string too, and I've just run an installation now using
that build. All looks good here. The changes will be in the next daily
build early tomorrow.
Since we're talking scanning limitation with regards to what d-i will pick as >install media, is there any chance you could looking into also scanning >SD/MMC media (most of which do not reside behind an USB controller though >some do)?
We have a very similar ESP/netint-based installation guide for the Pi 3 [1], >this time using SD, because it makes a more sense to use SD on Pi 3 for >varied reasons, and we have to require users to enter the "cdrom" target >manually with:
-t vfat -o rw /dev/mmcblk0p1
If SD removable media could be scanned in the same manner as USB media, it >would probably also improve the situation for people trying to install Debian >using SD, and just not Pi 3 users.
I also suspect that we are bound to find folks trying to use the ESP/netinst >installation method using M.2 NVMe drives, especially if they don't happen to >have a free USB drive lying around. In that case maybe /dev/nvme### media >should also be considered in the lookup for installation media?
All in all, it is likely that having d-i to only scan for USB and optical >devices might become limiting in the long run, especially if we have means to >identify, with relative certainty, whether a media/partition was created for >the goal of Debian installation.
But of course, you guys might have some constraints, that I am not aware of, >that also make you not want to overexpand the range of devices that should >qualify as installation media...
Hilariously, it's not just d-i that doesn't deal with the UAS edge
cases here. My test PC here would only recognise UAS media in BIOS
mode (i.e. CSM enabled), not in UEFI mode.
Interesting. Sadly, UEFI firmware bugs and limitations are far from uncommon.
| Sysop: | Keyop |
|---|---|
| Location: | Huddersfield, West Yorkshire, UK |
| Users: | 296 |
| Nodes: | 16 (2 / 14) |
| Uptime: | 37:36:22 |
| Calls: | 6,648 |
| Files: | 12,193 |
| Messages: | 5,329,135 |