• [gentoo-dev] Experimental binary package hosting

    From Roy Bamford@21:1/5 to Andreas K. Huettel on Tue Sep 21 20:10:02 2021
    On 2021.09.21 18:22, Andreas K. Huettel wrote:
    So let's experiment with this... :) announcing: https://gentoo.osuosl.org/experimental/amd64/binpkg/default/linux/17.1/x86-64/

    More information can be found in a blog post, which will also be on planet.g.o
    soon: https://dilfridge.blogspot.com/2021/09/experimental-binary-gentoo-package.html

    Cheers -A

    --
    Andreas K. Hüttel
    dilfridge@gentoo.org
    Gentoo Linux developer
    (council, qa, toolchain, base-system, perl, libreoffice)



    Andreas,

    How is USE=bindist treated?
    Its on for stage building and off in profiles.

    --
    Regards,

    Roy Bamford
    (Neddyseagoon) a member of
    elections
    gentoo-ops
    forum-mods
    arm64
    -----BEGIN PGP SIGNATURE-----

    iQEzBAABCAAdFiEEsOrcx0gZrrCMwJzo/xJODTqpeT4FAmFKH6cACgkQ/xJODTqp eT4FYQf/VsI0LyxnV37AF9l4bXR/4Zv042+Vd7fQc4USR/sJVN9s3Om7jlYAMqEp yzY+quXeU1r4lNaNYH3ypJfuleiCbXboABO4yHyeSXFnLiTUrd8xksob6+OxJBzT C+eCH8WGQVNOBZ+7pJQVt7Vim+2VBW91eTfvgnuuY/PiBOFc1oVd+lgRThEddMip v5sRLLIR/VXTGAzzFtmS5sShgWm5oXLH7uAaA7pDzjiwEtzoMXq77qG/lIENpSrA IRdJ0c6ldRo3/8OA6Ygv+Z5TSo+5dM4ctPboWj9e4WcO3QijAVIoQ/c5t+Fq+ekb pP6daJVPiUCTnAiRVHCr7zydEIXtgA==
    =RaqV
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Andreas K. Huettel@21:1/5 to All on Tue Sep 21 20:25:09 2021
    Copy: neddyseagoon@gentoo.org (Roy Bamford)

    Andreas,

    How is USE=bindist treated?
    Its on for stage building and off in profiles.

    USE=bindist is switched on, and in addition we have
    ACCEPT_RESTRICT="* -bindist"

    --
    Andreas K. Hüttel
    dilfridge@gentoo.org
    Gentoo Linux developer
    (council, qa, toolchain, base-system, perl, libreoffice)

    -----BEGIN PGP SIGNATURE-----

    iQIzBAABCAAdFiEE6W4INB9YeKX6Qpi1TEn3nlTQogYFAmFKI4UACgkQTEn3nlTQ ogbd9A//YO5hA18A1SlKRlAmMPbINU9whvMt1c3uvU3Gd+lYWRrFEOpi09CpXTTs 5Zi++6WuvENcuQrfaGUiZ4Jd4RwYAx0mfXVcdK+mmwVbAw6noL9OAhy9csarLUTy A6uzIeGKnwoGPKSFX/6VGM7mPoW7h6X+YVBIz0EoRbJpbatEA63YsoTgDDxKHFBw ZwmVED6rkUH+Opuwx3ufGfPbK6dc1we7IUJiH/vpk8lqkpjDD+Ayut9HJUHOmdQm tZbOO7WliqpVGOMnsLTShoxi9O6cqijBoSbZmLjHpbtZGHyhGp/4ak1CCPIcMfx1 qrnhU2eaf9j4BFvcEUlXpGnTbogRdK5uC+cpGqvTrv+LtnjIefb80YFD268hYiN7 pbLWROU+VQC7iUMCbIXYR1ac0MydnBlvxodtR4Mg2ntWx2CpqZAKJyIFEtmZc5gt vPTSWMzomHPvhJZkyDxrsQWn0yiX9bbYM/8mi8szRjGGdePLkJy3RBjtQHh1Rly9 cC9U4LfOfMc35zrdPYOQhaXrgoVTRMxSsAUjJB7iTaWq6o1k7U2qhMfKWYy0Pkgz scZ5NR/SueMntQ51boWxT4robOst4SQjbyFQWY6Nw7BbUw79glztGnIb0nMXNhja D8eyZIAwt/P900B5saJM9icTDtrXHIy7LdBi/Cb1KIij/KKn69w=
    =taOA
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNe
  • From Wolfgang E. Sanyer@21:1/5 to All on Tue Sep 21 22:40:01 2021
    Nice job!

    On Tue, Sep 21, 2021 at 2:25 PM Andreas K. Huettel <dilfridge@gentoo.org> wrote:

    Andreas,

    How is USE=bindist treated?
    Its on for stage building and off in profiles.

    USE=bindist is switched on, and in addition we have
    ACCEPT_RESTRICT="* -bindist"

    --
    Andreas K. Hüttel
    dilfridge@gentoo.org
    Gentoo Linux developer
    (council, qa, toolchain, base-system, perl, libreoffice)


    <div dir="ltr">Nice job!<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Sep 21, 2021 at 2:25 PM Andreas K. Huettel &lt;<a href="mailto:dilfridge@gentoo.org">dilfridge@gentoo.org</a>&gt; wrote:<br></div><blockquote class="
    gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">&gt; Andreas,<br>
    &gt; <br>
    &gt; How is USE=bindist treated?<br>
    &gt; Its on for stage building and off in profiles.<br>

    USE=bindist is switched on, and in addition we have <br> ACCEPT_RESTRICT=&quot;* -bindist&quot;<br>

    -- <br>
    Andreas K. Hüttel<br>
    <a href="mailto:dilfridge@gentoo.org" target="_blank">dilfridge@gentoo.org</a><br>
    Gentoo Linux developer <br>
    (council, qa, toolchain, base-system, perl, libreoffice)<br> </blockquote></div>

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Andrew Ammerlaan@21:1/5 to Torokhov Sergey on Wed Sep 22 10:20:01 2021
    On 22/09/2021 09:58, Torokhov Sergey wrote:
    I have repos configs placed into /etc/portage/repos.conf with
    "rsync-type = git" fo all repos
    so I created binhost.cond file here instead of /etc/portage/ as
    mentioned in blog post.

    binhost.conf should be in binrepos.conf instead of repos.conf. See also: https://wiki.gentoo.org/wiki/Binary_package_guide#Pulling_packages_from_a_binary_package_host

    Then I tried to sync and got warnings:
    ...
    !!! Section 'binhost' in repos.conf is missing location attribute
    !!! Repository 'binhost' has sync-uri attribute, but is missing
    sync-type attribute
    ...
    So it seems it's dosn't sync with binhost index without location attribute. After setup location attribute to specific directory I got warning on
    sync trying:
    !!! Repository 'binhost' has sync-uri attribute, but is missing
    sync-type attribute
    !!! Repository 'binhost' is missing masters attribute in '/usr/local/portage/binhost/metadata/layout.conf'
    !!! Set 'masters = gentoo' in this file for future compatibility
    I tried to set up "rsync-type = rsync" or "webrsync" with "amsters =
    gentoo" but then I got error:
    !!! Repository 'binhost' is missing masters attribute in '/usr/local/portage/binhost/metadata/layout.conf'
    !!! Set 'masters = gentoo' in this file for future compatibility
    So I created /usr/local/portage/binhost/metadata/layout.conf :
    masters = gentoo
    auto-sync = false
    thin-manifests = true

    So then on sync I got:
       The current sync-type attribute of repository 'gentoo' is not set to 'rsync' or 'webrsync':
       sync-type=git
    ----
    Is it currently git sync available for binhost?

    A binhost is different from an ebuild repository, it is just a
    collection of .xpak files so there is no git sync involved.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Torokhov Sergey@21:1/5 to All on Wed Sep 22 10:30:02 2021
    Sorry for previous html message. I tried to recend it as plaintext.

    I have repos configs placed into /etc/portage/repos.conf with "rsync-type = git" fo all repos
    so I created binhost.cond file here instead of /etc/portage/ as mentioned in blog post.

    Then I tried to sync and got warnings:
    ...
    !!! Section 'binhost' in repos.conf is missing location attribute
    !!! Repository 'binhost' has sync-uri attribute, but is missing sync-type attribute
    ...

    So it seems it's dosn't sync with binhost index without location attribute.

    After setup location attribute to specific directory I got warning on sync trying:
    !!! Repository 'binhost' has sync-uri attribute, but is missing sync-type attribute
    !!! Repository 'binhost' is missing masters attribute in '/usr/local/portage/binhost/metadata/layout.conf'
    !!! Set 'masters = gentoo' in this file for future compatibility

    I tried to set up "rsync-type = rsync" or "webrsync" with "amsters = gentoo" but then I got error:
    !!! Repository 'binhost' is missing masters attribute in '/usr/local/portage/binhost/metadata/layout.conf'
    !!! Set 'masters = gentoo' in this file for future compatibility

    So I created /usr/local/portage/binhost/metadata/layout.conf :
    masters = gentoo
    auto-sync = false
    thin-manifests = true

    So then on sync I got:
    The current sync-type attribute of repository 'gentoo' is not set to 'rsync' or 'webrsync':
    sync-type=git


    Is it currently git sync available for binhost?

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Torokhov Sergey@21:1/5 to All on Wed Sep 22 11:10:02 2021
    Thanks. These settings are works. I just forgot to add USE="bindist" for tested dev-qt/qtwebengine package as mentioned early in mailing list.

    Great! Now no need to compile qtwebengine for hours next time! :)

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Andreas K. Huettel@21:1/5 to Torokhov Sergey on Wed Sep 22 10:39:03 2021
    Copy: gentoo-dev@lists.gentoo.org

    Am Mittwoch, 22. September 2021, 10:20:10 CEST schrieb Torokhov
    Sergey:
    Sorry for previous html message. I tried to recend it as plaintext.

    I have repos configs placed into /etc/portage/repos.conf with
    "rsync-type = git" fo all repos so I created binhost.cond file here
    instead of /etc/portage/ as mentioned in blog post.

    Nope. As the blog post says, you need to put that text into a *file*
    with name -->>> /etc/portage/binrepos.conf <<<--- ...

    --
    Andreas K. Hüttel
    dilfridge@gentoo.org
    Gentoo Linux developer
    (council, toolchain, base-system, perl, libreoffice)
    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v2

    iQKTBAABCgB9FiEE6W4INB9YeKX6Qpi1TEn3nlTQogYFAmFK66dfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEU5 NkUwODM0MUY1ODc4QTVGQTQyOThCNTRDNDlGNzlFNTREMEEyMDYACgkQTEn3nlTQ ogYtTA/+M1D12xUy4jPeYieSyLuOrFc8lcGZqL+4ze6Fu/dCegqOLeAheP3vbi1R FhTkZO3T8pxb6Msl1NgfnodfQxgsKkNntQnP/RpS9PRtlCOG0HMveO8ndUMjglpH VJCRDGrbP4CFY8XIOINvUsF9sh68vTQLELQDEdq3WbPViDPBIwSZ4C4YJLD3ErTb T7V8pCs9YCebXlgVzWY5sNWbx4LkC+cAeDKO+bU9q2ZeUHVU/LKaKnWTLoXIM2l+ qkamc/MHrA02VwDEEK/RIHXRD+GFzfrtUIRngf+NMBZEbbTJHvIj+0w5lMpu39a/ w9ANXPnkq7rEJHLfYh17pL+08+k56gnt368eOBcub4Ivw4nUIvboCIrZft/imS37 0KqpYnh9D9An4AxnFSgr2rYFkUiPQTtgGlnN2POHAfT42Uc3bevwOIUoWXqLCO08 zvM3z7PfZLBK8TcWzW3+jGfFiWclDwq2KlcdXW/fUBbK2UWvKd+DNH6C3vcz81SK UFkSrkuSuVd6P1jyNHzo25XDO9XN4YuO0u3shZNojpSb67VFkiDh2BmB7qkm2ZIo uyYte/rymd1hZ1p2pvYtNvcdSY/KbseyYlPXx9zjgI2ANnTT5RuOlnr+
  • From Torokhov Sergey@21:1/5 to All on Wed Sep 22 14:20:01 2021
    Thanks! I never used an external binhost repo so mislead myself.

    Now I see that it's like a remote analogue of "/usr/portage/packages"
    and tarballs from remote binhost are fetched on demand into "local binhost"
    ( /usr/portage/packages/<category> ) before installation with replacing early presented tarballs.


    binhost.conf should be in binrepos.conf instead of repos.conf. See also: https://wiki.gentoo.org/wiki/Binary_package_guide#Pulling_packages_from_a_binary_package_host


    A binhost is different from an ebuild repository, it is just a
    collection of .xpak files so there is no git sync involved.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Vadim A. Misbakh-Soloviov@21:1/5 to All on Thu Sep 23 04:04:35 2021
    Finally it happened!

    I already planned to try to ask infra/council about sponsoring few servers for build farm for "official gentoo binhosts" when I had enough time, but fortunately, you've already did that.

    It's very good news.

    Btw, do you need any help with that?

    I'd be very happy to help with that project.
    -----BEGIN PGP SIGNATURE-----

    iNUEABYKAH0WIQQoxRjgXYBWtXJOpb/NsmC0G8NikgUCYUuaY18UgAAAAAAuAChp c3N1ZXItZnByQG5vdGF0aW9ucy5vcGVucGdwLmZpZnRoaG9yc2VtYW4ubmV0MjhD NTE4RTA1RDgwNTZCNTcyNEVBNUJGQ0RCMjYwQjQxQkMzNjI5MgAKCRDNsmC0G8Ni kkwNAP0TNX0Co06r1/se0SG/Lh8ijGmPhMgY80OS4jKf6zw7pAEAiubGB6oNS6EU ALWfYbkAErLYg8cLsTX9lO6gA/ROBgc=
    =Xz20
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Andreas K. Huettel@21:1/5 to All on Thu Sep 23 16:11:40 2021
    Copy: mva@gentoo.org (Vadim A. Misbakh-Soloviov)
    Copy: dev-portage@gentoo.org

    Hi Vadim,

    Finally it happened!
    I already planned to try to ask infra/council about sponsoring few
    servers for build farm for "official gentoo binhosts" when I had
    enough time, but fortunately, you've already did that.
    It's very good news.

    Thanks! Nice to see that this is appreciated :)

    So far I'm only using "spare time" on the machine that builds the
    releng stages (amd64, x86, m68k, riscv). So no need for a big server
    farm.

    Btw, do you need any help with that?
    I'd be very happy to help with that project.

    Sure! Feel free to add yourself to the Project:Binhost wiki page. I'll
    ask for an alias and a channel soon.

    The most useful steps now are only half related to actual building. I
    barely know any python and am not very familiar with portage
    internals... this is what in my opinion we'd need next:

    1) a tool to manage and manipulate a binpkg/ directory tree
    The main functions that I see needed are
    * delete packages/versions that are not in the gentoo repository
    anymore (xpak and in index file), maybe with some grace time
    * merge xpak files built elsewhere into the directory (also in the
    index file)
    (imagine you have a second container that builds with same CFLAGS, but
    with use settings for gnome, not plasma... or with updated
    dependencies because of changes in gentoo.git... you want to merge the
    trees for distribution without having duplicate builds)

    2) binary package cryptographic signing and verification
    Essentially we need to finish support for GLEP78; this is being worked
    on in RinCat's pull request
    https://github.com/gentoo/portage/pull/562
    See also https://www.gentoo.org/glep/glep-0078.html

    3) an easy way to figure out if a binary package repo is suitable for
    a profile / arch / ... or not, and a standard for path names
    This is not so important right now, and partially also already present
    I guess.

    The actual builder right now is very simple and wired up with a single
    daily cron job; the mirrors are only updated manually by me until bug
    813528 is handled.

    Cheers
    Andreas

    --
    Andreas K. Hüttel
    dilfridge@gentoo.org
    Gentoo Linux developer
    (council, toolchain, base-system, perl, libreoffice)
    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v2

    iQKTBAABCgB9FiEE6W4INB9YeKX6Qpi1TEn3nlTQogYFAmFMix1fFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEU5 NkUwODM0MUY1ODc4QTVGQTQyOThCNTRDNDlGNzlFNTREMEEyMDYACgkQTEn3nlTQ ogZt3hAAoeKRSoBg1kvCKIWNLndq62mrnhefZKD1MZQDqOpR5j/8UsULwDzZp2P0 9RW+4zQvdfcnXkSrOJYKRt9RSGWkj4DCmEocicwb+zPQzV0LyW7faWGw6f6ZHJin 9R7gnNLLoS4vcwRqRpxtWNnaoqM9hc9G4xoi0+I/2bOBKhhJKcMQNUHefKWiMIlP UB7RSHkcHS9PqhQW34TLQxcd+wZAvdZ9TInR9srmsboQwONcj/AlI/Z1T8qrXVm6 p5YluiuOXzemIBNaTluUlPsdbYNp4cy7iEa40Ui29T38ROQCTuivP0ziSwsz8I1p xerMtaWlNYdMKg2vs/GRo0lVCbty1eEXLucx3jYtjzd35wKdolk+augc8Rb6O+Cr ovO5PJ1oRVROCqrvVOMKVQ0eN7NnDP1LPupAW7/RIT8fHWblseAGh/NsdLkfPUfK cI3y+lRD549zCtXDRpbYnX3VnsyJc6JYrtnp1nQX7pDMcdnoOnjn2sqeOYvECBzD hYuWKtgwe1Dm9sPBlJyxRB6skbpNF46JERTQvieY3Cs4/Z9jnS6z23qG+KH7A+bG EaGXx9M3DVAjgcoT3Mosbhu/kDDdocdufXunw1iF3yJR94cADblBKda1
  • From Sheng Yu@21:1/5 to Andreas K. Huettel on Thu Sep 23 19:00:01 2021
    On Thursday, September 23rd, 2021 at 10:11, Andreas K. Huettel <dilfridge@gentoo.org> wrote:
    3. an easy way to figure out if a binary package repo is suitable for
    a profile / arch / ... or not, and a standard for path names
    This is not so important right now, and partially also already present


    There is a PROFILE data in Packages index, but it is not useful if we are
    going to mix multiple profiles together, like KDE/GNOME.

    Also currently package manager does not care about the CFLAGS / microarchitecture of the binary package, like from x86-64 to x86-64-v4. Additional metadata and matches need to be added.

    Thanks,
    Sheng Yu

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Matt Turner@21:1/5 to dilfridge@gentoo.org on Thu Sep 23 21:30:02 2021
    On Thu, Sep 23, 2021 at 7:12 AM Andreas K. Huettel <dilfridge@gentoo.org> wrote:

    Hi Vadim,

    Finally it happened!
    I already planned to try to ask infra/council about sponsoring few
    servers for build farm for "official gentoo binhosts" when I had
    enough time, but fortunately, you've already did that.
    It's very good news.

    Thanks! Nice to see that this is appreciated :)

    So far I'm only using "spare time" on the machine that builds the
    releng stages (amd64, x86, m68k, riscv). So no need for a big server
    farm.

    Btw, do you need any help with that?
    I'd be very happy to help with that project.

    Sure! Feel free to add yourself to the Project:Binhost wiki page. I'll
    ask for an alias and a channel soon.

    The most useful steps now are only half related to actual building. I
    barely know any python and am not very familiar with portage
    internals... this is what in my opinion we'd need next:

    1) a tool to manage and manipulate a binpkg/ directory tree
    The main functions that I see needed are
    * delete packages/versions that are not in the gentoo repository
    anymore (xpak and in index file), maybe with some grace time
    * merge xpak files built elsewhere into the directory (also in the
    index file)

    eclean packages from gentoolkit does this exactly.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Marc Schiffbauer@21:1/5 to All on Fri Sep 24 10:30:01 2021
    * Matt Turner schrieb am 23.09.21 um 09:28 Uhr:
    On Thu, Sep 23, 2021 at 7:12 AM Andreas K. Huettel <dilfridge@gentoo.org> wrote:

    Hi Vadim,

    Finally it happened!
    I already planned to try to ask infra/council about sponsoring few servers for build farm for "official gentoo binhosts" when I had
    enough time, but fortunately, you've already did that.
    It's very good news.

    Thanks! Nice to see that this is appreciated :)

    So far I'm only using "spare time" on the machine that builds the
    releng stages (amd64, x86, m68k, riscv). So no need for a big server
    farm.

    Btw, do you need any help with that?
    I'd be very happy to help with that project.

    Sure! Feel free to add yourself to the Project:Binhost wiki page. I'll
    ask for an alias and a channel soon.

    The most useful steps now are only half related to actual building. I barely know any python and am not very familiar with portage
    internals... this is what in my opinion we'd need next:

    1) a tool to manage and manipulate a binpkg/ directory tree
    The main functions that I see needed are
    * delete packages/versions that are not in the gentoo repository
    anymore (xpak and in index file), maybe with some grace time
    * merge xpak files built elsewhere into the directory (also in the
    index file)

    eclean packages from gentoolkit does this exactly.

    plus 'emaint binhost' for rebuilding the index after adding packages
    manually :)

    -Marc

    --
    0xCA3E7BF67F979BE5 - F7FB 78F7 7CC3 79F6 DF07
    6E9E CA3E 7BF6 7F97 9BE5

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)