XPost: linux.debian.devel

Package: general

A distro upgrade of Debian needs a lot of disk space on the root partition. That partition often doesn't have a large size. That could for example be because it's on a mobile device, on a SSD drive, has lots of installed software, or because the user simply followed the advice of guides that usually recommend the root partition
to be relatively small.
I had this problem when upgrading to Debian12 and also had it during the last distro upgrade.

A problem is that the storage requirements displayed when running sudo apt-get upgrade --without-new-pkgs or sudo apt-get full-upgrade are lower than what is actually needed. Obviously, that shouldn't be the case. Solving this problem could be redundant
if there was a way to make it upgrade incrementally in several steps.

Could you please make stepwise distribution upgrades possible?

With the two commands above one can already split it up into two steps but especially the second command still requires a lot of disk space. That it displays less storage requirements than actually needed makes it more difficult to avoid the problem.
Avoiding and solving potential issues resulting from the upgrade breaking or from what the user tries to do to free up disk space requires skill and time, and makes the upgrade userunfriendly and far less smooth than it could be.

Here's what I do to free up disk space and I had to do some of the things while the upgrade was ongoing as I noticed that it takes up more disk space than I previously freed up: https://unix.stackexchange.com/q/774199/233262 (such as deleting the cached
packages after an unrelated fatal error breaking the upgrade so that it downloaded half of them again when I ran the full-upgrade command again after solving that).

This could also be an issue for apt but it could probably also be done independently from that package which is why I filed it under general for now.

One could also make it so that it dynamically adjusts depending on how much disk space is available throughout the upgrade. It could and probably should still only be one command.

For example, I think a good approach to this would be something like this (if the user is low on root partition disk space):
1. asking for *at least* 400MB to be available
2. if a parameter for stepwise is set or it detected low free disk space:
3. downloading the first 300 MB or so of packages
4. installing these
5. deleting the cached packages from /var/cache/apt/archives/
6. downloading the next batch up to the storage limit set at start
7. and so on (without exiting in-between)

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

XPost: linux.debian.devel

On Thu, Apr 11, 2024 at 04:46:03PM +0000, mYnDstrEAm wrote:

With the two commands above one can already split it up into two steps but especially the second command still requires a lot of disk space.

I am going to assume that your "a lot of disk space" stems from the
*.deb files that are downloaded. If so, you can e.g. attach an USB disk/
drive and mount it e.g. under /media/apt-archives

Tell apt to use that directory instead of /var/cache/apt/archives, e.g.:
apt upgrade -o dir::cache::archives=/media/apt-archives

(for some more free MBs you could 'apt clean' and then move dir::cache
elsewhere, but for that you need to create some directories in the
target location and the binary caches are not THAT large to make it
really worthwhile in practice. Similar for other files like
/var/lib/apt aka dir::state::lists)


Instead of an USB drive you could do the same with e.g. an SD card, drop
them into RAM (if your device has surprisingly more RAM than disk) or
even use a network share (NFS, sshfs, … you name it). The filesystem is
not usually a concern (as in: even fat32 should work given we encode
away the : in epochs).

Note that whoever has write access to the files on the storage (or in
case of unencrypted transfer, also everyone who can meddle with transfer
over the network) could use that to attack you as apt (well, apt will
casually check them first, but after that and dpkg, who actually
interacts with them the most) will assume that the files in /var/cache/apt/archives (or where ever else you stored them and told apt
to use them) are valid & trusted.


Note also that apt uses for its space check statvfs(3) f_bavail, as in, depending on how you configured your disk, it should have a couple of additional free blocks in reserve (typically 5%, see tune2fs(8) -m).
If you know what you are doing, you could decrease that value.


Note that the value apt displays is only an estimate, powered by what
the individual packages claim (via dpkg), which is an estimate. Also, if
you happen to have a 2GB installed, the upgrade will roughly take an
additional 2GB as dpkg would first extract the new files along the old
ones and then replace them in one swoop – so for a bit, you have that
package installed two times. Multiple this by group size, divide by
unchanged files and sprinkle some salt over it for flavour.
Predictions are hard, especially about the future.


I would in general not recommend to try approaches like upgrading
individual packages as that easily leads unsuspecting users into
situations that nobody else has encountered before: aka bugs in
packages that nobody else will encounter as they are either hidden
by the involved set usually being upgraded together as intended™ or
– which tends to be even worse – the breakage is known but ignored
on purpose as the solution is far worse than the problem (at least for
everyone doing upgrades the normal way – example: usrmerge). Also, but
that is just an aside, people grossly overestimate how easy it is for
packages to be upgraded individually (compare: t64 testing migration).


Best regards

David Kalnischkies

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEE5sn+Q4uCja/tn0GrMRvlz3HQeIMFAmYammsACgkQMRvlz3HQ eINsAA//TlFwynxgJXyUhnb3JjVpzltb/70jTtq11Dep6uhxTGUhokbMfroXiQvc yf03bNVD3fEdNF8DxtglFz/0mADuxGNDooYi0KaRVij7Y3PyWhCC/RLKLDVdYTeO bttk+bUVhMO8JgNQRc6OII+cMYLH27Sv/OWgCwi9RwXtnWLNOHR9+ZnTF+KHPets 1Cs+1wwpQDDolpJfdgTjhJm/SEom0tlSDrkhjLlHFPZFQNswd3JDLr3kyO8kFc1L 3lAoJvRZjs+dI7GLmtPx7yjU08EqFJzMlSiZLFceHg701DV4i00C9OixUsNRFkDf pIZAtI4rXeCwl2rIr2NA2NGoZY5+AxaHl8Y7S6TS9ZMZH6gRzTM2Ox84XvKlFEZg S60p3ZznrIQpfQY+NIrQC2gyFYHBCh2sOYOJWuScc++1hdZyfNXWZxVAFpb5WQkn sXzkgwfoPREEvKIJcZiHiCTQOwdC3sWv5kYu6nNEpxDKqp/o95OkZh70mq4z+HGY 44ivw4ptHtiSOw8I8arjVKgoi50rtlEuqSmMA1MG8EYbY7bITRg5XRDTBsLPWtAs rrDXAcOlcZ+qnQ00fPPmqDWukiIfyk2wUWUPjevJD/kgXYcfrLljvOncNaBKb7n4 9ajIxsoZKlZF/EuHOE0xCoCq/TDyeKwMXlD6Wxb1RdhfgTGgizw=
=PnoK
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

XPost: linux.debian.devel

Thanks guys, these are very useful methods and I'll mention these as alternatives to disk cleanups recommended at https://unix.stackexchange.com/q/774199/233262 (this would probably very useful to have at places about upgrades failing due to disk space
issues even though people only look these up once the problems already occurred).

However, the problem of the upgrade requiring more disk space than displayed at first remains and the command by Zeimetz can't be used with a built-in rememberable well-known command like sudo apt-get upgrade --stepwise

I don't think peripheral devices would be the common case for personal computers, rather one would simply specify a directory on a partition that is larger than the root partition.

Upgrading individual packages is not what this is about in case that wasn't clear. It's one upgrade but it's separated into several steps where one batch of packages are download and installed, the cache deleted, and then the next batch.
If the upgrade breaks in between due to disk space, because the user aborted it, a crash, or an error, then only some packages are upgraded...a stepwise upgrade could if anything be a way to *avoid* that (or at least interruptions due to disk space
problems) and to make them less problematic.
The key thing is that it would be usable with a simple command (such as by adding --stepwise)...if that command only executes a few already existing commands with no apt changes required for the basic functionality of this, that's all the better.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Personally, I don't think a machine that has that limited storage ought to be upgraded using apt from one Debian stable release to another.

It's not the machine but the partition. One can set up Debian to use a separate partition for the home directory. When the root partition on a SSD has 20 GB (about the size I think all guides I found on this suggest or use in their example) which should
be large enough and the upgrade claims it needs 5 GB and actually needs 7 GB that can obviously be a problem.

it is not supported to arbitrarily break apt updates up like that

...which is why this an issue for a new functionality rather than already existing. It doesn't have to be arbitrary.

It's common sense to break up GB-sized downloads into several steps. Last upgrade failed due to disk space issues; this upgrade I had to clean caches during the upgrade as workaround; I'm sure many have this problem and even if it's rare it would be best
to have upgrades run smoothly and reliably (such as checking remaining disk space throughout the upgrade and adjusting accordingly). Those are not several upgrade but one upgrade that downloads not once initially but several times and cleans up the
caches after each step.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)