Hi, I'm trying to figure out an sbuild failure on my laptop. The
sbuild environment from replicated from my desktop where things work
perfectly well. But in my laptop, things are failing at the
"Setup apt archive" step with

E: Failed to change to directory ‘/<<BUILDDIR>>’: Permission denied

And I'm completely at a loss trying to figure out what might be going
wrong. Can anyone give me some hints about what to look for?

Thanks,

- Ted



sbuild (Debian sbuild) 0.84.2 (08 December 2022) on letrec.thunk.org

+==============================================================================+
| f2fs-tools 1.15.0-1 (amd64) Mon, 26 Dec 2022 19:20:17 +0000 |
+==============================================================================+

Package: f2fs-tools
Version: 1.15.0-1
Source Version: 1.15.0-1
Distribution: unstable
Machine Architecture: amd64
Host Architecture: amd64
Build Architecture: amd64
Build Type: full

I: NOTICE: Log filtering will replace 'var/run/schroot/mount/unstable-amd64-sbuild-a67a3165-6688-4368-a376-66e094e41dfa' with '<<CHROOT>>'
I: NOTICE: Log filtering will replace 'build/f2fs-tools-cVTRAh/resolver-CwG6Va' with '<<RESOLVERDIR>>'

+------------------------------------------------------------------------------+
| Update chroot |
+------------------------------------------------------------------------------+

Get:1 http://httpredir.debian.org/debian unstable InRelease [161 kB]
Get:2 http://httpredir.debian.org/debian unstable/main Sources.diff/Index [63.6 kB]
Get:3 http://httpredir.debian.org/debian unstable/main amd64 Packages.diff/Index [63.6 kB]
Get:4 http://httpredir.debian.org/debian unstable/main Sources T-2022-12-26-1404.34-F-2022-12-26-0804.45.pdiff [15.0 kB]
Get:4 http://httpredir.debian.org/debian unstable/main Sources T-2022-12-26-1404.34-F-2022-12-26-0804.45.pdiff [15.0 kB]
Get:5 http://httpredir.debian.org/debian unstable/main amd64 Packages T-2022-12-26-1404.34-F-2022-12-26-0804.45.pdiff [33.2 kB]
Get:5 http://httpredir.debian.org/debian unstable/main amd64 Packages T-2022-12-26-1404.34-F-2022-12-26-0804.45.pdiff [33.2 kB]
Fetched 337 kB in 1s (238 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
Calculating upgrade...
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

+------------------------------------------------------------------------------+
| Fetch source files |
+------------------------------------------------------------------------------+


Local sources
-------------

/tmp/gbp/f2fs-tools_1.15.0-1.dsc exists in /tmp/gbp; copying to chroot
I: NOTICE: Log filtering will replace 'build/f2fs-tools-cVTRAh/f2fs-tools-1.15.0' with '<<PKGBUILDDIR>>'
I: NOTICE: Log filtering will replace 'build/f2fs-tools-cVTRAh' with '<<BUILDDIR>>'

+------------------------------------------------------------------------------+
| Install package build dependencies |
+------------------------------------------------------------------------------+


Setup apt archive
-----------------

Merged Build-Depends: debhelper-compat (= 13), libblkid-dev, libselinux1-dev, pkg-config, uuid-dev, build-essential, fakeroot
Filtered Build-Depends: debhelper-compat (= 13), libblkid-dev, libselinux1-dev, pkg-config, uuid-dev, build-essential, fakeroot
E: Failed to change to directory ‘/<<BUILDDIR>>’: Permission denied
I: The directory does not exist inside the chroot. Use the --directory option to run the command in a different directory.
Dummy package creation failed
E: Setting up apt archive failed

Setup apt archive
-----------------

Merged Build-Depends: dose-distcheck
Filtered Build-Depends: dose-distcheck
E: Failed to change to directory ‘/<<BUILDDIR>>’: Permission denied
I: The directory does not exist inside the chroot. Use the --directory option to run the command in a different directory.
Dummy package creation failed
E: Setting up apt archive failed
E: Failed to explain bd-uninstallable

+------------------------------------------------------------------------------+
| Summary |
+------------------------------------------------------------------------------+

Build Architecture: amd64
Build Type: full
Build-Space: n/a
Build-Time: 0
Distribution: unstable
Fail-Stage: explain-bd-uninstallable
Host Architecture: amd64
Install-Time: 0
Job: /tmp/gbp/f2fs-tools_1.15.0-1.dsc
Machine Architecture: amd64
Package: f2fs-tools
Package-Time: 0
Source-Version: 1.15.0-1
Space: n/a
Status: given-back
Version: 1.15.0-1 --------------------------------------------------------------------------------
Finished at 2022-12-26T19:20:17Z
Build needed 00:00:00, no disk space

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

El 26/12/22 a las 20:29, Theodore Ts'o escribió:

I: The directory does not exist inside the chroot.

This is really a problem with schroot. I guess that this will not work either:

schroot -c the-chroot-name

This usually works when you are in your $HOME because this file:

/etc/schroot/default/fstab

has a line like this:

/home /home none rw,bind 0 0

which makes a bind mount and makes /home to "exist" inside the chroot.

So the solution is to either try sbuild from a directory which is
bind-mounted like that, like $HOME, or add a new entry to fstab
with the directory you need.

Hope this helps.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Mon, Dec 26, 2022 at 08:45:53PM +0100, Santiago Vila wrote:

El 26/12/22 a las 20:29, Theodore Ts'o escribió:

I: The directory does not exist inside the chroot.

This is really a problem with schroot. I guess that this will not work either:

schroot -c the-chroot-name

This usually works when you are in your $HOME because this file:

/etc/schroot/default/fstab

Nope, that's not the issue; yes, /tmp and /home are missing form /etc/schroot/sbuild/fstab, but that was true on the *working* setup as
well, and from what I can tell; that's deliberate. It looks like the
goal is to keep things hermetic when building with sbuild, so it's a
feature that there aren't random directories leaking through from the
host to the sbuild enviroment.

I think I've figured out the issue. The problem is that the user and
group id's for sbuild are different on my desktop and my laptop, and I
did an rsync to copy the /chroot directories from my desktop to my
laptop --- and it appears that sbuild is very sensisitve about the
user id's being the same across the host and chroot environments.

Apparently sbuild copies the files for the package it is building over
a directory in /var/lib/sbuild/build, with the permissions being mode
770, and that is what sbuild bind mounts into the chroot. If my
theory is correct, if the user/group id's are different from between
the base /etc/passwd and chroot, then things go bad in a hurry.

From my working system (while gbp buildpackage was running sbuild):

% ls -al /var/lib/sbuild/build/
total 12
4 drwxrws--- 3 sbuild sbuild 4096 Dec 26 23:05 ./
4 drwxrws--- 4 sbuild sbuild 4096 Dec 19 2020 ../
4 drwxrwx--- 3 tytso sbuild 4096 Dec 26 23:05 f2fs-tools-oT4KHz/

Amd on the broken (laptop) system:

# ls -al /var/lib/sbuild/build/
total 32
4 drwxrws--- 8 fwupd-refresh Debian-exim 4096 Dec 26 22:48 ./
4 drwxrws--- 4 sbuild sbuild 4096 Dec 25 14:45 ../
4 drwxrwx--- 2 tytso Debian-exim 4096 Dec 26 14:01 f2fs-tools-9QfprK/
4 drwxrwx--- 2 tytso Debian-exim 4096 Dec 26 16:01 f2fs-tools-btkVPv/
4 drwxrwx--- 2 tytso Debian-exim 4096 Dec 26 14:20 f2fs-tools-cVTRAh/
4 drwxrwx--- 2 tytso Debian-exim 4096 Dec 26 22:48 f2fs-tools-Myld8j/
...

Each of were created from a failed sbuild invocation... And
"Debian-exim" on my laptop has the same group id as "sbuild" on my
desktop (and which is the group id in my chroots).

This also explains the error message:

E: Failed to change to directory ‘/<<BUILDDIR>>’: Permission denied

Oops.

So I guess I need to either manually juggle group id's in the chroots
(and/or my laptop's root directory --- but it's probably easier to do
it in the chroots, since there are fewer filees to chgrp in the
chroots), or I could recreate the sbuild chroots from scratch using sbuild-createchroot (but then I would need to recreate all of the
custom hacks so that ccache,eatmydata, apt-auto-proxy, etc. are working correctly in the chroot).

What fun....

- Ted

p.s. I guess if I had been planning ahead I would have made sure that
various system users and groups which are auto-created by packages at install-time (and which are therefore different depending on install
order) were pre-created on my laptop with the same numerical id's as
on my desktop, since that would have avoided all *sorts* of random
problems, especially if I was going to play games with copying chroots
around --- or trying to use NFS --- and not getting taken by surprise
by this sort of thing. Live and learn....

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi,

Quoting Theodore Ts'o (2022-12-27 05:19:45)

On Mon, Dec 26, 2022 at 08:45:53PM +0100, Santiago Vila wrote:

El 26/12/22 a las 20:29, Theodore Ts'o escribió:

I: The directory does not exist inside the chroot.

This is really a problem with schroot. I guess that this will not work either:

schroot -c the-chroot-name

This usually works when you are in your $HOME because this file:

/etc/schroot/default/fstab

Nope, that's not the issue; yes, /tmp and /home are missing form /etc/schroot/sbuild/fstab, but that was true on the *working* setup as
well, and from what I can tell; that's deliberate. It looks like the
goal is to keep things hermetic when building with sbuild, so it's a
feature that there aren't random directories leaking through from the
host to the sbuild enviroment.

that is correct. :)

I think I've figured out the issue. The problem is that the user and group id's for sbuild are different on my desktop and my laptop, and I did an rsync to copy the /chroot directories from my desktop to my laptop --- and it appears that sbuild is very sensisitve about the user id's being the same across the host and chroot environments.

Apparently sbuild copies the files for the package it is building over
a directory in /var/lib/sbuild/build, with the permissions being mode
770, and that is what sbuild bind mounts into the chroot. If my
theory is correct, if the user/group id's are different from between
the base /etc/passwd and chroot, then things go bad in a hurry.

I think your analysis makes sense.

From my working system (while gbp buildpackage was running sbuild):

% ls -al /var/lib/sbuild/build/
total 12
4 drwxrws--- 3 sbuild sbuild 4096 Dec 26 23:05 ./
4 drwxrws--- 4 sbuild sbuild 4096 Dec 19 2020 ../
4 drwxrwx--- 3 tytso sbuild 4096 Dec 26 23:05 f2fs-tools-oT4KHz/

Amd on the broken (laptop) system:

# ls -al /var/lib/sbuild/build/
total 32
4 drwxrws--- 8 fwupd-refresh Debian-exim 4096 Dec 26 22:48 ./
4 drwxrws--- 4 sbuild sbuild 4096 Dec 25 14:45 ../
4 drwxrwx--- 2 tytso Debian-exim 4096 Dec 26 14:01 f2fs-tools-9QfprK/ 4 drwxrwx--- 2 tytso Debian-exim 4096 Dec 26 16:01 f2fs-tools-btkVPv/ 4 drwxrwx--- 2 tytso Debian-exim 4096 Dec 26 14:20 f2fs-tools-cVTRAh/ 4 drwxrwx--- 2 tytso Debian-exim 4096 Dec 26 22:48 f2fs-tools-Myld8j/
...

Each of were created from a failed sbuild invocation... And
"Debian-exim" on my laptop has the same group id as "sbuild" on my
desktop (and which is the group id in my chroots).

This also explains the error message:

E: Failed to change to directory ‘/<<BUILDDIR>>’: Permission denied

Oops.

So I guess I need to either manually juggle group id's in the chroots
(and/or my laptop's root directory --- but it's probably easier to do
it in the chroots, since there are fewer filees to chgrp in the
chroots), or I could recreate the sbuild chroots from scratch using sbuild-createchroot (but then I would need to recreate all of the
custom hacks so that ccache,eatmydata, apt-auto-proxy, etc. are working correctly in the chroot).

What fun....

Note, that if you keep upgrading a Debian unstable chroot across multiple releases, it will end up looking slightly different than a freshly debootstrapped Debian unstable chroot. So I think there is value in semi-regularly re-creating the build chroot from scratch. Maybe write a script that does what you need?

That being said, I think the biggest problem is, that the main sbuild contributors these days (me and Jochen) do not use the schroot backend anymore but the unshare backend instead which doesn't suffer from this kind of problem (but of course has different quirks that one needs to be aware of). So I must admit that in the past few years, the schroot backend hasn't gotten the love that it deserves considering that it's the default backend used on the buildd machinery.

Finally, I think this is something that could be solved in sbuild. Ultimately, schroot is able to do things as the root user, so it should have sufficient permissions to fix up a chroot that carries incorrect permissions. Could you quickly note in a bug against sbuild on the Debian BTS which steps exactly you carried out so that I am able to reproduce your problem?

I'm making no promises that I'll find the time to improve the schroot backend of sbuild to survive the kind of chroot-rsync that you have performed but if this is important to you, then maybe we can make a trade and I implement this sbuild functionality and you have a look at pull requests https://github.com/tytso/e2fsprogs/pull/118 or #107 and leave some comments in return? :)

Thanks!

cheers, josch
--==============50848174563767415=MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Description: signature
Content-Type: application/pgp-signature; name="signature.asc"; charset="us-ascii"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEElFhU6KL81LF4wVq58sulx4+9g+EFAmOre3gACgkQ8sulx4+9 g+EOjg//cKuk0ByNCB33MYcQtpWxDk1M0g9zoNv8nUKjMKryla2fNhDB4yo9675S eHZK8JLpNCG0Oah8XhQJ0dGElzyDvMWQ3Dni6APpaoYWILeDn1t+lB9OwMClhcXa FTmAi3oBYkU4LQ5j2aCjMJlCYRQqGcpRGCUkQKP02lT9aZm9VILJy7dddu/k2Crm POegnDIPQucHN0f+VR+seez0eXsOZ0c4R/RE+Qc+VrW5fsBxmWvO+6bqXtw79O2K 3d70NwJu690gubP1YYEn6+aalEwcjNJ7ayhXaLfKZ6Ua5duN5Oz5zMUVtdieoA3m 2bxjT7yom1Be5CF4gCJiTEr2ZR6rhT1NyqIrZvrVE/5mc32ZulUIz0p7RCWUg71Y pdKkxLUAEh6KISlsu5kAbAMiCG5faHH0iNgXe2vlk9KhfEHhgSnAUKToK+kru/fp B0nN7prcwzCSp4CnnlrI8/HBsR7eYiOYS1FNX/GgdCey0R+eQdIqhrm/OeqR0iWU t6/Dh0c7lZBt02v9I+E35BqF0DsOcQDzfdbXoOqBltSMZ9cF+xliK5fliSqmNWXI uNzP7ZaJyAwB3xUJWfVSLhSiyOMwjkWRy+IJ1TLvOzRjj0uuPYJNc38zWi8x+Vsp UW0sfQwtco6vR/SpiZwtRQ0MP8XemNedsrJ9dqB8UefWqmH9QE0=
=S70I
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Wed, Dec 28, 2022 at 12:10:51AM +0100, Johannes Schauer Marin Rodrigues wrote:

Note, that if you keep upgrading a Debian unstable chroot across multiple releases, it will end up looking slightly different than a freshly debootstrapped Debian unstable chroot. So I think there is value in semi-regularly re-creating the build chroot from scratch. Maybe write a script
that does what you need?

That's true, but the number of user/group id's that sbuild would
actually care about is probably quite small and might very well just
be sbuild:sbuild I would think, no?

Finally, I think this is something that could be solved in sbuild. Ultimately,
schroot is able to do things as the root user, so it should have sufficient permissions to fix up a chroot that carries incorrect permissions. Could you quickly note in a bug against sbuild on the Debian BTS which steps exactly you
carried out so that I am able to reproduce your problem?

Sure, no problem.

I'm making no promises that I'll find the time to improve the schroot backend of sbuild to survive the kind of chroot-rsync that you have performed but if this is important to you, then maybe we can make a trade and I implement this sbuild functionality and you have a look at pull requests https://github.com/tytso/e2fsprogs/pull/118 or #107 and leave some comments in
return? :)

Thanks for the reminder, I'll take a look. Most of the patch
proposals for e2fsprogs end up going to linux-ext4@vger.kernel.org (so
that other ext4 developers can review them), and I sometimes forgot to
look over the github pull requests.

I'm also about to go on a Panama Canal cruise, where my internet
access may be limited (which is why I was trying to get sbuild setup
on my laptop in the first place :-), and e-mail has the advantage of
being much easily cacheable using offlineimap...

- Ted

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)