• sbuild/schroot: need to get .deb files internally downloaded as package

    From Uladzimir Bely@21:1/5 to All on Thu Oct 28 11:30:01 2021
    When building a package with sbuild, some packages (dependencies of package being built) are internally downloaded and installed by apt. After the build finished, schroot is cleaned again (while everything is done in overlay)

    I need to cache outside of schroot these .deb files that were downloaded by apt. They are supposed to be used for creaing a local partial debian repository, so that the second build will use this local repo instead of internet one.

    Currently it's done by parsing dependencies and downloading them
    independently, but I suppose there should be some more straightforward way to get them using sbuild.

    I investigated and tried several options for sbuild (like --purge-build=never, --purge-deps=never, --purge-session=never), tried to find .deb files during
    the build using at some stages (--pre-build-commands="ls -la /var/cache/apt/ archives/", --starting-build-commands="ls -la /var/cache/apt/archives/", -- finished-build-commands="ls -la /var/cache/apt/archives/") - but I couldn't
    see any .deb files inside the schroot.

    I previously asked in debian-user maillist (https://lists.debian.org/debian-user/2021/10/msg01044.html), but the tips I got (like using caching proxy) are
    not what I really need. While sbuild already does the thing (downloading/ installing dependencies), I suppose there should be a way to (at least) see
    the *.deb files downloaded in '/var/cache/apt/archives' inside schroot.

    --
    Uladzimir Bely
    Promwad Ltd.
    External service provider of ilbers GmbH
    Maria-Merian-Str. 8
    85521 Ottobrunn, Germany
    +49 (89) 122 67 24-0
    Commercial register Munich, HRB 214197
    General Manager: Baurzhan Ismagulov

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Andrey Rahmatullin@21:1/5 to Uladzimir Bely on Thu Oct 28 11:50:02 2021
    On Thu, Oct 28, 2021 at 11:58:26AM +0300, Uladzimir Bely wrote:
    When building a package with sbuild, some packages (dependencies of package being built) are internally downloaded and installed by apt. After the build finished, schroot is cleaned again (while everything is done in overlay)

    I need to cache outside of schroot these .deb files that were downloaded by apt.
    A caching proxy is indeed a good answer for this question but if for some reason you don't want to use it, you can bind-mount some directory from
    outside the chroot as /var/cache/apt/archives, e.g. via /etc/schroot/sbuild/fstab, and set $apt_keep_downloaded_packages=1.


    --
    WBR, wRAR

    -----BEGIN PGP SIGNATURE-----

    iQJhBAABCgBLFiEEolIP6gqGcKZh3YxVM2L3AxpJkuEFAmF6ccwtFIAAAAAAFQAP cGthLWFkZHJlc3NAZ251cGcub3Jnd3JhckBkZWJpYW4ub3JnAAoJEDNi9wMaSZLh MbsP/0P58P9fUAS9CWCJRwrcYnrXnkp7Lq6l+r8khaJgfFHpVhHOYtoUQ6m87puH FjWX2EIB5XoGCHlgcGd18dVqXg5PLhfzyph5JcvRrYaP9f6C8B7ASM2JxA8kLUWK 6AkjjLZlZwC2XUPVfVeMG/4Lbj7nj4JUedNAhY7KzbmvLdqoqQDcBI4F4fYPjGyD OFOk7MEs8Fo3nAlb/+W14ge7xZjgPBnIN22/7MJJUp/qwbTASDQZr99zOMx4fmF4 txC635KECXDC9moB49Ys2MmyrlkX/tuyk9lJ3UWI2Qcp+vMsNmSYlWXnFHMv5Gb8 ZkOUNxIH69HAIQ4wadUZtAPgDwUHB3dfU1o3aUAE+W/NlLsIhOlDPQSfYTiSl1uV Zw8Q4ylA6qK54sABXHv8nnvdxoG6Jw0z7h5G63/0G9E+3Iytr1OwFuYE4ERBgjJZ G37bIZu+EWmVNY00Bfndi+WxH76RgzKZCoHIKAU8WpuSU+AzTYtXRw5c4T2i2Flf nPkXsRUhFdlSaat5ZgDflEvkwNUZpYQ1A0MMGXRnU7XZh3c5IshJIq3yEtVDzmL1 aCA3MPN2sItcCwryjw5epzEtaSnkclQgLT7mETkRoIXkorXTEKBZGwiLJC92f/pj zfIeNsOFG4ryXNbpefXyJjjhtaj2fFHUPFMXe+wKMkn5/CwO
    =Exkr
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Johannes Schauer Marin Rodrigues@21:1/5 to All on Thu Oct 28 14:00:01 2021
    Hi,

    Quoting Uladzimir Bely (2021-10-28 10:58:26)
    When building a package with sbuild, some packages (dependencies of package being built) are internally downloaded and installed by apt. After the build finished, schroot is cleaned again (while everything is done in overlay)

    I need to cache outside of schroot these .deb files that were downloaded by apt. They are supposed to be used for creaing a local partial debian repository, so that the second build will use this local repo instead of internet one.

    Currently it's done by parsing dependencies and downloading them independently, but I suppose there should be some more straightforward way to
    get them using sbuild.

    I investigated and tried several options for sbuild (like --purge-build=never,
    --purge-deps=never, --purge-session=never), tried to find .deb files during the build using at some stages (--pre-build-commands="ls -la /var/cache/apt/ archives/", --starting-build-commands="ls -la /var/cache/apt/archives/", -- finished-build-commands="ls -la /var/cache/apt/archives/") - but I couldn't see any .deb files inside the schroot.

    I previously asked in debian-user maillist (https://lists.debian.org/debian-user/2021/10/msg01044.html), but the tips I got (like using caching proxy) are
    not what I really need. While sbuild already does the thing (downloading/ installing dependencies), I suppose there should be a way to (at least) see the *.deb files downloaded in '/var/cache/apt/archives' inside schroot.

    sbuild maintainer here. You have not explained why you don't want to use a caching proxy like apt-cacher-ng. But using a caching proxy is also what I would suggest you use.

    If you don't want to use that, here is how you download only the packages you need to satisfy the build depenencies of a particular source package:

    cat << END > /tmp/apt.conf
    Apt::Architecture "amd64";
    Dir "/tmp/apt";
    Dir::Etc::trusted "/etc/apt/trusted.gpg";
    Dir::Etc::trustedparts "/etc/apt/trusted.gpg.d/";
    Apt::Get::Download-Only true;
    Apt::Install-Recommends false;
    END
    mkdir -p /tmp/apt/etc/apt/apt.conf.d/ /tmp/apt/etc/apt/sources.list.d/ /tmp/apt/var/lib/dpkg /tmp/apt/var/cache/apt/archives/partial /tmp/apt/etc/apt/preferences.d/
    touch /tmp/apt/var/lib/dpkg/status
    cat << END > /tmp/apt/etc/apt/sources.list
    deb http://deb.debian.org/debian/ unstable main
    deb-src http://deb.debian.org/debian/ unstable main
    END
    APT_CONFIG=/tmp/apt.conf apt update
    APT_CONFIG=/tmp/apt.conf apt-get dist-upgrade
    APT_CONFIG=/tmp/apt.conf apt-get build-dep hello

    Now all your packages to build src:hello are in /tmp/apt/var/cache/apt/archives. No need to waste CPU cycles on a full package build if all you want is to download a certain set of binary packages.

    Thanks!

    cheers, josch
    --==============%41129245661378181=MIME-Version: 1.0
    Content-Transfer-Encoding: 7bit
    Content-Description: signature
    Content-Type: application/pgp-signature; name="signature.asc"; charset="us-ascii"

    -----BEGIN PGP SIGNATURE-----

    iQIzBAABCgAdFiEElFhU6KL81LF4wVq58sulx4+9g+EFAmF6j2gACgkQ8sulx4+9 g+HhbA//dtw8+WxuyZAwjXQZUOs1Bar0FAjr4QlTDcifhMvupEPH9TNo65RoD726 qJecbnpO5a9Qj+CxVrQLzTs+EO+iZNA1XnICT2XgP4DRJaWdHT8aVmt2KK7kUzkZ P9li7+Lv+L74H30BlnUs3zs3UZ71XvRoaTh+p9rE3QWXt60/Z53WyR3Eh/WjcPFk vC/dLabgRsXM4TMQs8m3VTxUbcQKlBOsLGjp9ijoB0V6jS+xa6PeWBTy3RS0MBOe pU6ao4QtmiaFVnZr4wwzZro0xgFX8h+n/Jdu1vXpB4tWZZs3HvTkrtrrt0vERdRC cPgFHDeh5XFm5kbFUxz5vvuDVAtrCnc8aL7pxmgE76GpA8aE77QYa64Oos1R/u2c ibT8yjch4W6KpNFDN3htqOedvKCksVkWos/93thKOv0jZe6+fAevz86HfGxDMdDF N/jBPHK2NPKn7YJZd2CH5zCiZUJ8nIIx0jhN8UghFo0oRyeNtLLTRKUzVcS+oNN3 pEdE7Yk046Qb98UEPWaBnE4rCoRAtNiAP9WrDgQfhz+TnyHLJDb5c1qQpIEAMZ1K DJePKSM4qwW/0XrOrMAfRfd3Frczrl5ooRC/cC27AVqjRQmRHleT3+GBJ8OH3H8y JwNI6c202CvhuZBLi+FGkPqLkFVYZ2B8ijrm9n31TaGPfrAbh7k=
    =6AV8
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Uladzimir Bely@21:1/5 to All on Thu Oct 28 18:50:02 2021
    Hello.

    Thanks for the idea with $apt_keep_downloaded_packages=1. I'll try to play
    with it.

    Previously I tried to mount external directory to /var/cache/apt/archives inside schroot, but finally found it almost empty (only some .lock file was there, AFAIR). As I understand, that it was really used, but later everything downloaded was deleted. I hope, the option above will help me with it.

    --
    Uladzimir Bely
    Promwad Ltd.
    External service provider of ilbers GmbH
    Maria-Merian-Str. 8
    85521 Ottobrunn, Germany
    +49 (89) 122 67 24-0
    Commercial register Munich, HRB 214197
    General Manager: Baurzhan Ismagulov

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Uladzimir Bely@21:1/5 to All on Thu Oct 28 18:30:01 2021
    Thanks for the answer

    You have not explained why you don't want to use a caching proxy like apt-
    cacher-ng.

    The question appeared while developing for Isar, a tool build create debian- based distro for embedded systems. In two words, it's similar to Buildroot, Yocto and others, but it uses official Debian packages plus allows to build
    own customer packages.

    Currently it uses some own set of scripts to download dependencies, install them is build chroot and build the package. Every .deb that was downloaded during system build is cached internally. So, for the second build it's possible to create a local repo from these stored files and do everything (e.g bootstrapping, building custom packages, rootfs generation) not from some remote repo (e.g. http://ftp.debian.org), but from this local one (file:// foo).

    I'm currently trying to switch our custom build system to sbuild one, but I faced a problem that the dependencies downloaded by sbuild scripts are not available in form of .deb files.

    Anyway, your advice uses a bit more native approach that is currently used in Isar, so I'll look at it (or something similar).

    Also, in parallel thread I was adviced to use $apt_keep_downloaded_packages=1 option that might be also useful for me.

    --
    Uladzimir Bely
    Promwad Ltd.
    External service provider of ilbers GmbH
    Maria-Merian-Str. 8
    85521 Ottobrunn, Germany
    +49 (89) 122 67 24-0
    Commercial register Munich, HRB 214197
    General Manager: Baurzhan Ismagulov

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Tobias Frost@21:1/5 to Uladzimir Bely on Fri Oct 29 08:10:01 2021
    On Thu, Oct 28, 2021 at 07:11:05PM +0300, Uladzimir Bely wrote:
    Thanks for the answer

    You have not explained why you don't want to use a caching proxy like apt-
    cacher-ng.

    The question appeared while developing for Isar, a tool build create debian- based distro for embedded systems. In two words, it's similar to Buildroot, Yocto and others, but it uses official Debian packages plus allows to build own customer packages.

    Currently it uses some own set of scripts to download dependencies, install them is build chroot and build the package. Every .deb that was downloaded during system build is cached internally. So, for the second build it's possible to create a local repo from these stored files and do everything (e.g
    bootstrapping, building custom packages, rootfs generation) not from some remote repo (e.g. http://ftp.debian.org), but from this local one (file:// foo).

    I'm currently trying to switch our custom build system to sbuild one, but I faced a problem that the dependencies downloaded by sbuild scripts are not available in form of .deb files.

    If it does not need to be sbuild, my pbuilder setup has an hook that
    allows me to have a local directory as apt repo for pbuilder. Any deb placed
    in this directory will be available during pbuilder build...
    Kind of https://wiki.debian.org/PbuilderTricks#How_to_include_local_packages_in_the_build,
    but my setup uses just one directory, not a per-project one..

    --
    tobi

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Uladzimir Bely@21:1/5 to All on Wed Nov 3 08:20:02 2021
    I investigated $apt_keep_downloaded_packages=1 and found that this feature was added in more recent version of sbuild than debian `buster` provides (https:// bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=933723 is a bug where it was solved)

    Probably, that's why I couldn't get any results under `buster` host - apt
    cache is hardcoded to be cleaned by sbuild after installing dependencies.

    So, as I understood, the only way to cache dependency packages under `buster` is to download them in some external way.

    --
    Uladzimir Bely
    Promwad Ltd.
    External service provider of ilbers GmbH
    Maria-Merian-Str. 8
    85521 Ottobrunn, Germany
    +49 (89) 122 67 24-0
    Commercial register Munich, HRB 214197
    General Manager: Baurzhan Ismagulov

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Daniel Leidert@21:1/5 to All on Wed Nov 3 09:30:03 2021
    Am Donnerstag, dem 28.10.2021 um 11:58 +0300 schrieb Uladzimir Bely:

    I need to cache outside of schroot these .deb files that were downloaded by apt. They are supposed to be used for creaing a local partial debian repository, so that the second build will use this local repo instead of internet one.

    I use a caching proxy for that (squid-deb-proxy). There is also various tools (e.g. apt-auto-proxy, squid-deb-proxy-client) to detect proxies automatically, which might require some special network setups in the chroot/container/VM though. Otherwise just set the proxy via Acquire::http::Proxy (/etc/sbuild/chroot/<chroot>/etc/apt/apt.conf.d/30proxy) or something similar.


    I use that for sbuild chroots, autopkgtest containers and VMs, and even when testing custom installer CDs.

    HTH and regards, Daniel
    --
    Regards,
    Daniel Leidert <dleidert@debian.org> | https://www.wgdd.de/
    GPG-Key RSA4096 / BEED4DED5544A4C03E283DC74BCD0567C296D05D
    GPG-Key ED25519 / BD3C132D8B3805D1808123AB7ACE00941E338C78

    https://www.fiverr.com/dleidert
    https://www.patreon.com/join/dleidert

    -----BEGIN PGP SIGNATURE-----

    iQIzBAABCgAdFiEEvu1N7VVEpMA+KD3HS80FZ8KW0F0FAmGCRxgACgkQS80FZ8KW 0F2sVQ//Zw62GoYRp/mL2pbGXHtrRELKFm3OxQBr8vXPOCbwxniez5TL1mgtMdT9 gueGwTc7h0DjmPSREMFp9W6Kyk+xljT6TrrdaNBmWPfz/nirsMpMyKy50K8MtwH4 TP1s6q3HB/Jsc5TI7vka8s9tZepuGtCLSbgiivGH//yXL8IqIZGH2Cfevc6pLwBu 1Cz+hX4vSnVXe47LmERfHO5cxzkgcqrJr0uDUJ1gBGBk+P0gqFFrHGWgon7JuKfk DwsABS/7S822mZmgsmgDYVEMs82OQKTjV/srUdP76+5rpwRnpsvQgQQOuM2jKqYi 2tTcIMMcBCaHQnL0Xhy4A79h6AWX4WU9VBwe42m3J5SBEjTVgO5JoZ/w6PoY0NGt CJw/6A4ywRQ1ufD7ThCn8VK9ZBZW8MPDJS0uc+x9uwGyXNxQiFdVKpcFFAHpvcX5 6hyYRlmi+fsM1fcx+5FBRj979Y7L26RxITg3BiWu1OYKqp12CAKPDYHlTJgp230e DvbsNuNlXbDyCCeSZTKc/4YpeDHSWAtmESox1YvOQx8YB5+2lCe4zJmDp4J3MHd+ /bvNFgJy0r8Sio90DEOjmJZ0MR51m4Hkf/zgYAfBON7NzkWo4+eHGjoH3r9