I am just finishing up the implementation of Discoverable GPT
Partitions [0] in my growlight tool, and it seems a good time to
see if I can push this discussion [1] forward.
If not, I don't think it would be a viable replacement for partman.
But maybe an alternative? I find the partitioning step one of the most error-prone and hard-to-use parts of non-trivial Debian installations.And the preseeding syntax is as powerful as it is inconvenient.
Hello!
On 9/23/21 22:57, nick black wrote:
I am just finishing up the implementation of Discoverable GPT
Partitions [0] in my growlight tool, and it seems a good time to
see if I can push this discussion [1] forward.
Does it support partition tables other than GPT, in particular all
of the partition tables that parted supports?
If not, I don't think it would be a viable replacement for partman.
Adrian
--
.''`. John Paul Adrian Glaubitz
: :' : Debian Developer - glaubitz@debian.org
`. `' Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
`- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913
On Sep 24, Marc Haber <mh+debian-devel@zugschlus.de> wrote:
But maybe an alternative? I find the partitioning step one of the most error-prone and hard-to-use parts of non-trivial Debian installations.And the preseeding syntax is as powerful as it is inconvenient.
I had not heard of growlight before yesterday, but from what I have read
I think that it is very promising.
Implementing support for more partition formats, if missing, should be rather easy.
But which ones do we need for architectures which are not actually dead?
It supports MBR, GPT, and APM:
https://github.com/dankamongmen/growlight/blob/master/src/ptable.c#L51-L123
(sorry for the gmail-style top posting; i can't find your mail in my mbox
for whatever reason)
Any other partition schemes ought be trivial to add; they've not been added yet
simply because I don't have means with which to test them.
Looking at the "partition types" section of the Linux configuration, I see:
Acorn, AIX, Alpha, Amiga, Atari, Macintosh, MSDOS, BSD, Minix, Solaris
x86, Unixware,
Windows LDM, SGI, Ultrix, Sun, Karma, EFI/GPT, and SYSV68.
Looking at the disk-label code from partman, I see:
gpt, msdos, amiga, atari, mac, sun
So the only ones covered by partman and not covered by growlight would be: amiga, atari, sun,
and mac (if mac is not the same as APM). I don't see any difficulty in
adding these four, so long
as there's someone with an Amiga or ATARI who'd be willing to test them
out. If there are no such
people, is it that important?
[1] https://git.savannah.gnu.org/cgit/parted.git/commit/?id=9c266205416ec956d6205c828211480de3767d02
Marco d'Itri left as an exercise for the reader:
And the preseeding syntax is as powerful as it is inconvenient.
Implementing support for more partition formats, if missing, should be rather easy.
But which ones do we need for architectures which are not actually dead?
So, as I responded to Adrian [0], the only missing partition
types appear to be amiga, atari, and sun. Adding them ought be
simple enough, though I'd need testers with the hardware, or
access to the hardware.
My biggest worry personally (aside from the realpolitik of
getting this change through) regards the automated partitioning
language available through the preseed system. Trying to emulate
this bug-for-bug is a non-starter, I think, both from a
technical and quality-of-life standpoint. If the emulation can't
be perfectly accurate, I don't think it ought be attempted for
such a critical, delicate procedure.
I do have a different wish, though. Could you please purge any references
to drivemakers' units (stuff like MiB = million bytes, which current partitioner maliciously[1] swaps around with proper MB of 1048576B)?
Having them in the user interface is deeply harmful: people will get unoptimal alignment unless they 1. know about it, and 2. are careful enough.
From your comments before I see that you try to do proper alignment, but in too many cases no matter how you try, the installer won't align well enough because the hardware might be newer than the version of growlight, hide its inner workings for marketing reason (like stealth SMR drives), etc.On the other hand, a completely oblivious user will get good alignment if
you show numbers measured in gigabytes rather than gillionbytes.
I know of only one case of multi-GB alignment (some early versions of ipmctl wanted a multiple of 32GB because certain vendor BIOSes had problems with smaller blocks), but the required alignment there is 1GB for years.
And most importantly: thanks for this effort, it's greatly appreciated!
Does libparted support the discoverable partitions spec?
Hello Nick!
On 9/26/21 00:49, Nick Black wrote:
It supports MBR, GPT, and APM:
https://github.com/dankamongmen/growlight/blob/master/src/ptable.c#L51-L123
(sorry for the gmail-style top posting; i can't find your mail in my mbox for whatever reason)
Any other partition schemes ought be trivial to add; they've not been added yet
simply because I don't have means with which to test them.
So, you are not using libparted then?
Looking at the "partition types" section of the Linux configuration, I see:
Acorn, AIX, Alpha, Amiga, Atari, Macintosh, MSDOS, BSD, Minix, Solaris x86, Unixware,
Windows LDM, SGI, Ultrix, Sun, Karma, EFI/GPT, and SYSV68.
Looking at the disk-label code from partman, I see:
gpt, msdos, amiga, atari, mac, sun
So the only ones covered by partman and not covered by growlight would be: amiga, atari, sun,
and mac (if mac is not the same as APM). I don't see any difficulty in adding these four, so long
as there's someone with an Amiga or ATARI who'd be willing to test them out. If there are no such
people, is it that important?
Yes, it is important as we're supporting these architectures in Debian Ports and I invested quite some time to get Atari partition support added [1],
for example.
I think it makes little sense to not use libparted as it already supports
all common and less common partition types and reimplementing everything
that libparted makes little sense to me.
Adrian
My biggest worry personally (aside from the realpolitik ofI personally think that preseed is nasty enough that users who do automation on a scale that would make learning it worthwhile already have a better way to
getting this change through) regards the automated partitioning
language available through the preseed system. Trying to emulate
this bug-for-bug is a non-starter, I think, both from a
technical and quality-of-life standpoint. If the emulation can't
be perfectly accurate, I don't think it ought be attempted for
such a critical, delicate procedure.
do such automation. For me, d-i is for manual installs, scripted stuff
wants a partitioner + glorified debootstrap.
So, you are not using libparted then?
Yes, it is important as we're supporting these architectures in Debian Ports and I invested quite some time to get Atari partition support added [1],
for example.
I think it makes little sense to not use libparted as it already supports
all common and less common partition types and reimplementing everything
that libparted makes little sense to me.
I'd be delighted to support them -- as in, I am honestly eager
to add ATARI support; that sounds awesome -- I just need some
way to test the implementations, either via someone running it
on the environment, or getting access to such a machine.
I think it makes little sense to not use libparted as it already supports
all common and less common partition types and reimplementing everything
that libparted makes little sense to me.
parted did not have ZFS support when I embarked on this project
(it appears to have it now). i would not be opposed to
leveraging libparted if it presents a definite advantage;
supporting more partition types, so long as it exposes an API i
can easily work with, would be such an advantage.
i do note that libparted2 is 621K in the archive, whereas
growlight itself is only 555K. it is of course possible that
all that weight is desirable functionality.
with that said, i would *still want to test on the target
environment*, to make sure i'm using libparted correctly there.
so that necessity remains.
would this allay your concerns?
Hello!
On 9/26/21 12:40, Luca Boccassi wrote:
Does libparted support the discoverable partitions spec?
I'm not sure, this thread is the first time I have heard about discoverable partitions. I have read up first what the motivations for these are and
how useful they would be for Debian.
Also, since parted is maintained by RedHat, I would expect that this feature would land in parted soon as well.
Adrian
Also, since parted is maintained by RedHat, I would expect that this feature >> would land in parted soon as well.If the question is "why does X not use libparted", "does not support discoverable parts spec" is a good enough answer for me.
On Sun, Sep 26, 2021 at 01:41:18AM -0400, nick black wrote:
Marco d'Itri left as an exercise for the reader:
And the preseeding syntax is as powerful as it is inconvenient.
Implementing support for more partition formats, if missing, should be rather easy.
But which ones do we need for architectures which are not actually dead?
So, as I responded to Adrian [0], the only missing partition
types appear to be amiga, atari, and sun. Adding them ought be
simple enough, though I'd need testers with the hardware, or
access to the hardware.
I'd start with asking porters of m68k and sparc64 whether today's systems even run anything but Linux. I think there's little point in keeping compat with 80s' OSes.
At a risk of drawing ire of m68k/sparc64 folks, I'd also suggest not putting your tuits there until this millenium's hardware is covered well.
So the only ones covered by partman and not covered by growlight would be: amiga, atari, sun,
and mac (if mac is not the same as APM). I don't see any difficulty in
adding these four, so long
as there's someone with an Amiga or ATARI who'd be willing to test them
out. If there are no such
people, is it that important?
Hello!
On 9/27/21 12:46, Luca Boccassi wrote:
Also, since parted is maintained by RedHat, I would expect that this feature
would land in parted soon as well.
If the question is "why does X not use libparted", "does not support discoverable parts spec" is a good enough answer for me.
Not for me, though. Debian has always followed the philosophy to be a universal
operating system, which also meant that we can't (immediately) use all the new
technologies and features that other distributions or upstream projects develop.
For example, we don't use systemd-homed or systemd-firstboot either. That doesn't
mean Debian is per se worse off. It just means Debian tries to cater different use
cases and follows different concepts.
It's different for distributions like Fedora or openSUSE which focus on a very
limited set of targets and use cases. That's why they can quickly adopt to all
the new features and technologies that upstream projects like systemd develop.
I'm not saying that one philosophy is better than the other. I'm just saying that
Debian is not like these other distributions.
Adrian
Whether a tool that was developed new from scratch is automatically better is not a given. The burden of proof is on the person trying to introduce the new software, not on the people maintaining the current set of software.
And claiming that parted is in pure maintenance mode is not true either. It has a paid developer working on the project and is receiving updates and improvements.
Whether growlight is better and more suitable for Debian needs to be technically proven, not just by arguing that it’s the newer project.
On Sep 27, 2021, at 2:25 PM, Luca Boccassi <bluca@debian.org> wrote:
Even if that interpretation would work as an excuse to never do
anything, and I'm not really sure it does, this specification has been published in 2014 [0] so even by Debian standard it's old stuff.
It's
older than Debian Jessie, which was EOL'd last year. If libparted can't
keep up with 7 years old stuff that in the meantime was implemented in util-linux's (which is a truly universal tool) in 2014, gdisk in 2019,
and so on, then to me it sounds like a tool in maintenance mode:
perfectly fine and adequate for existing tools and programs, but not
quite the best choice for new tools developed from scratch.
On Sep 27, 2021, at 2:25 PM, Luca Boccassi <bluca@debian.org> wrote:
Even if that interpretation would work as an excuse to never do
anything, and I'm not really sure it does, this specification has been published in 2014 [0] so even by Debian standard it's old stuff.
That’s not what I said so. You’re trying to dismiss my opinion as completely invalid now by trying to frame it such that I am against progress. I am not.
Not for me, though. Debian has always followed the philosophy to be a universal
operating system, which also meant that we can't (immediately) use all the new
technologies and features that other distributions or upstream projects develop.
It's
older than Debian Jessie, which was EOL'd last year. If libparted can't keep up with 7 years old stuff that in the meantime was implemented in util-linux's (which is a truly universal tool) in 2014, gdisk in 2019,
and so on, then to me it sounds like a tool in maintenance mode:
perfectly fine and adequate for existing tools and programs, but not
quite the best choice for new tools developed from scratch.
Whether a tool that was developed new from scratch is automatically better is not a given. The burden of proof is on the person trying to introduce the new software, not on the people maintaining the current set of software.
And claiming that parted is in pure maintenance mode is not true either. It has a paid developer working on the project and is receiving updates and improvements.
Whether growlight is better and more suitable for Debian needs to be technically proven, not just by arguing that it’s the newer project.
Adrian
Not for me, though. Debian has always followed the philosophy to be a universalIt never meant that.
operating system, which also meant that we can't (immediately) use all the new
technologies and features that other distributions or upstream projects develop.
On Sep 27, 2021, at 3:41 PM, Luca Boccassi <bluca@debian.org> wrote:
On Mon, 2021-09-27 at 15:18 +0200, John Paul Adrian Glaubitz wrote:
On Sep 27, 2021, at 2:25 PM, Luca Boccassi <bluca@debian.org> wrote:
Even if that interpretation would work as an excuse to never do
anything, and I'm not really sure it does, this specification has been
published in 2014 [0] so even by Debian standard it's old stuff.
That’s not what I said so. You’re trying to dismiss my opinion as completely invalid now by trying to frame it such that I am against progress. I am not.
You dismissed it because it's "new technology":
Not for me, though. Debian has always followed the philosophy to be a universal
operating system, which also meant that we can't (immediately) use all the new
technologies and features that other distributions or upstream projects develop.
I simply pointed out that it's a 7 years old spec that saw an entire
LTS Debian version (8, we are now at 11) being released and EOL'ed
since the time it was published. If this is too new to consider, then
so are all Debian releases newer than Wheezy.
It's
older than Debian Jessie, which was EOL'd last year. If libparted can't
keep up with 7 years old stuff that in the meantime was implemented in
util-linux's (which is a truly universal tool) in 2014, gdisk in 2019,
and so on, then to me it sounds like a tool in maintenance mode:
perfectly fine and adequate for existing tools and programs, but not
quite the best choice for new tools developed from scratch.
Whether a tool that was developed new from scratch is automatically better is not a given. The burden of proof is on the person trying to introduce the new software, not on the people maintaining the current set of software.
And claiming that parted is in pure maintenance mode is not true either. It has a paid developer working on the project and is receiving updates and improvements.
Whether growlight is better and more suitable for Debian needs to be technically proven, not just by arguing that it’s the newer project.
Adrian
Of course. But jumping in and saying "you should use X instead of Y",
you can't pretend that nobody asks questions such as "ok, but does libX support this very much related and relevant 7 years old specification
that other comparable tools support", no matter how awkward it is for
libX.
Hello Nick!
On 9/26/21 16:29, Nick Black wrote:
I'd be delighted to support them -- as in, I am honestly eagerThere are emulators available for Atari such as Aranym. And emulators available for Amiga such as fs-uae. FWIW, parted should contain everything needed to be able to implement your own support for these partition tables.
to add ATARI support; that sounds awesome -- I just need some
way to test the implementations, either via someone running it
on the environment, or getting access to such a machine.
Well, using a missing feature in the past against parted that is present there is not such a good argument against using it, to be honest.I think it makes little sense to not use libparted as it already supports >>> all common and less common partition types and reimplementing everything >>> that libparted makes little sense to me.parted did not have ZFS support when I embarked on this project
(it appears to have it now). i would not be opposed to
leveraging libparted if it presents a definite advantage;
supporting more partition types, so long as it exposes an API i
can easily work with, would be such an advantage.
i do note that libparted2 is 621K in the archive, whereasI think 70k more disk space is not really a concern.
growlight itself is only 555K. it is of course possible that
all that weight is desirable functionality.
with that said, i would *still want to test on the targetI thought you didn't depend on libparted?
environment*, to make sure i'm using libparted correctly there.
so that necessity remains.
would this allay your concerns?No, not really. I consider a partitioning tool to be too important
to be replaced by an unproven solution.
But maybe an alternative? I find the partitioning step one of the most error-prone and hard-to-use parts of non-trivial Debian installations.
Marc Haber left as an exercise for the reader:
But maybe an alternative? I find the partitioning step one of the most error-prone and hard-to-use parts of non-trivial Debian installations.
so overall, i've got to say the feedback i heard here was a lot
more positive than i was expecting, though there was a bit less
than i had hoped for. but perhaps something that can be touched
would see more interest?
given that no one seemed to reject the idea out of hand, i'm
going to go ahead and rebase my work from 2012 (or more likely
look at it once and redo it) in a Salsa fork of d-i, and make
some installation media available. forgive me for likely only
having x86 available at first. i'll try to have this done within
a week or so, and put it up on my server. people can then give
it a whirl.
One thing that partman does is "support plug-ins", to allow for
configuring block devices before being able to partition them, where
needed. This can be useful for iSCSI, multipath, or (the one I care most about) NBD. I wrote a "partman-nbd" a few years back to do just that: https://salsa.debian.org/installer-team/partman-nbd
IOW, chill out, nobody's going to kill off partman unless there's
something that's *actually* better than partman.
[1] https://lwn.net/Articles/859240/
[2] http://0pointer.net/blog/mkosi-a-tool-for-generating-os-images.html
[3] https://lwn.net/Articles/859769/
On 9/28/21 12:13, Wouter Verhelst wrote:
IOW, chill out, nobody's going to kill off partman unless there's
something that's *actually* better than partman.
Just some comments after reading after this [1] because I honestly find it unfair the way I am being cornered here.
Wouter Verhelst left as an exercise for the reader:
One thing that partman does is "support plug-ins", to allow for
configuring block devices before being able to partition them, where needed. This can be useful for iSCSI, multipath, or (the one I care most about) NBD. I wrote a "partman-nbd" a few years back to do just that: https://salsa.debian.org/installer-team/partman-nbd
Thanks a lot for pointing this out, Wouter -- this is *exactly*
the kind of feedback I was hoping for! I allow loopback devices
to be set up, but not in any reboot-crossing manner, and I have
no NBD nor iSCSI functionality.
https://github.com/dankamongmen/growlight/issues/150
Marc Haber left as an exercise for the reader:
But maybe an alternative? I find the partitioning step one of the
most error-prone and hard-to-use parts of non-trivial Debian
installations.
so overall, i've got to say the feedback i heard here was a lot more
positive than i was expecting, though there was a bit less than i had
hoped for. but perhaps something that can be touched would see more
interest?
given that no one seemed to reject the idea out of hand, i'm going to
go ahead and rebase my work from 2012 (or more likely look at it once
and redo it) in a Salsa fork of d-i, and make some installation media available. forgive me for likely only having x86 available at first.
i'll try to have this done within a week or so, and put it up on my
server. people can then give it a whirl.
note that there would still be some questions where i'd need input
from Project members (as noted in my Debconf [0] presentation),
particularly regarding translation (even if i can lift significant
portions from partman, i'd need it looked over) and facilitating accessibility technology.
iSCSI works very differently and is way more complex, but I remember
from when I last played with it (which is a while ago, so the details
are hazy) that it's not possible to set up in a non-persistent manner
(i.e., all iSCSI connections survive reboot unless explicitly deleted, although obviously partman-iscsi has to do some dark magic to ensure the configuration is migrated from the d-i environment to the live system).
There's also ATA-over-Ethernet, Fibrechannel-over-ethernet, multipath,
and a whole slew of other things, if you want to configure this from growlight.
I could be wrong though, haven't looked at growlight in much detail, and
in the end it's your call, not mine :-D
100% true - expect people are busy, rather than hostile! :-)
Hi nick,
[ cc += debian-boot@ ]
nick black <dankamongmen@gmail.com> (2021-09-27):
Marc Haber left as an exercise for the reader:
But maybe an alternative? I find the partitioning step one of the
most error-prone and hard-to-use parts of non-trivial Debian
installations.
so overall, i've got to say the feedback i heard here was a lot more
positive than i was expecting, though there was a bit less than i had
hoped for. but perhaps something that can be touched would see more
interest?
FWIW I've followed the answers to your mail over the last few days,
but I haven't had a chance to look at either the video or the slides
(only 4 days before your initial mail and the one I'm replying to…).
At first glance, it seems fine to be experimenting with a different >partitioner; of course all people are somewhere on the love/hate
spectrum regarding partman and the zillions of partman-* packages, but
it's indeed a shell maze and it *probably* could use some heavy lifting.
I keep hoping that simple use cases are made simple(r)… maybe growlight >could help there (e.g. “I'd like soft RAID1 with encrypted LVM, all that >with a UEFI boot — therefore ESP —, without being a storage wizard”).
I suppose some step (once you've experimented on a growlight-only
d-i) would be to have both partman and growlight, and let people
choose (maybe with a flag at boot time, or by entering expert mode or >whatever), until we have a better idea what works, what doesn't, what
can be fixed, what cannot, and until a decision is made for the next
Debian stable release.
Leaving the technical integration aside for a moment, one question
that comes to mind is whether this would be a one-shot contribution or
some kind of longer commitment to maintain that different partitioner.
I seem to remember earlier attempts at revamping the partitioning
step, which stalled eventually.
(Your recent mail to debian-newmaint@ is a hint; your apparent
steadiness of those packages maintenance-wise is another; and your
apparent interest in adding support to possibly missing features yet >another.)
Of course, one might object that the question is moot as there isn't
much active development on partman components (even if some patches have
been submitted over the last few days), but at least that's a known >(imperfect, but…) beast.
given that no one seemed to reject the idea out of hand, i'm going to
go ahead and rebase my work from 2012 (or more likely look at it once
and redo it) in a Salsa fork of d-i, and make some installation media
available. forgive me for likely only having x86 available at first.
i'll try to have this done within a week or so, and put it up on my
server. people can then give it a whirl.
Feel free to touch base with debian-boot@ and/or debian-cd@ once you
have a working proof of concept that some people have toyed with; we can >discuss how the “alternative” part could be implemented (different >images, or both partitioners ship together, with some option/selection — >people might remember GRUB vs. LILO). I cannot guarantee a timely answer >(life tends to keep me busy), but you shouldn't see a lack of answer
after just a few days as if people were not interested.
Sysop: | Keyop |
---|---|
Location: | Huddersfield, West Yorkshire, UK |
Users: | 286 |
Nodes: | 16 (3 / 13) |
Uptime: | 86:34:32 |
Calls: | 6,496 |
Calls today: | 7 |
Files: | 12,099 |
Messages: | 5,277,030 |