1. Forum
  2. Usenet
  3. LINUX.GENTOO.DEV

  • Re: [gentoo-dev] mirror storage growth rate

    From =?UTF-8?Q?Micha=C5=82_G=C3=B3rny?=@21:1/5 to Jaco Kroon on Fri Mar 15 13:10:01 2024
    On Fri, 2024-03-15 at 10:06 +0200, Jaco Kroon wrote:
    Hi All,

    I was messing with some storage related caching on some of our hosts
    this morning when I wondered about how much storage the gentoo mirrors
    were consuming.  I'm not too worried about the current storage, but I am noticing that the storage requirements are creeping quite a bit (as per attached), and if that growth rate continues it may become a problem *eventually*.

    Can this growth be explained?


    I guess the simplest explanation is that software is growing larger,
    and in the end we should be expecting to adding new packages faster than removing dead ones. Add to that the grotesque inefficiency of modern programming languages such as Go and Rust.

    --
    Best regards,
    Michał Górny


    -----BEGIN PGP SIGNATURE-----

    iQFGBAABCgAwFiEEx2qEUJQJjSjMiybFY5ra4jKeJA4FAmX0OYoSHG1nb3JueUBn ZW50b28ub3JnAAoJEGOa2uIyniQOah8H/RBntHJq3ziGiXGXfhtU1R2+CgV2zj9E 5liH2fQ+yhnFwvItP5jC22JluaiInokfVeC/1Ht5ISDusLgcIBCv/EitoalyBOZT G9O2K+jZIewsBtNbvplX7yTTTQ3Knz+vteHw1Kc9d5VginGVEYTLjQoEYho2W/u1 en5Qpqh9v6xI04zAPat414cy/khaHC1/FcejjlkIH2pQbi9fRjBX6+ju0397mpt7 PXkSFA0/jDMPkAvVdwGuOyLeBaowtiDVq3p3rJfxOvmfk19JhL4RAOhUCQrHzo1m EvUcBtpT6svya598TrM2INrYSkAF9FFpU7prEfE9Sq6PgbR2YGS0ne4=
    =gXFn
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From =?iso-8859-1?Q?Ho=EBl_B=E9zier?=@21:1/5 to All on Fri Mar 15 16:40:59 2024
    I guess the simplest explanation is that software is growing larger,
    and in the end we should be expecting to adding new packages faster than >removing dead ones. Add to that the grotesque inefficiency of modern >programming languages such as Go and Rust.

    Wouldn’t initiatives like rust-dev[0] help with that? I know that Debian is also packaging Rust this way[1].

    [0]: https://wiki.gentoo.org/wiki/Project:Rust/rust-dev
    [1]: https://wiki.debian.org/Rust

    Hoël

    -----BEGIN PGP SIGNATURE-----

    iQIzBAABCAAdFiEElZjiR4iQkYjVEGdt6jPCPrF3Nk0FAmX0bAsACgkQ6jPCPrF3 Nk2bUhAAr+XaWfPL3xRjvSqAM8zt4Gj/8KHNNb22inXILwGYBH+4Vb6N4vQPi8GZ VTlmZdr9ecXujLY10l6uUnWhhfEdKlyjQmpz6u4WahjzvK01PAMIh70EmuD1syl0 gu+NUGXrwXlJsaR9TlXExp5PlKurqeAhIdKLkOfGP04bJARnUKcuKgY17UDPbqUP 7ujwjbxO7zFJ0zpBhEZfxgwTK3m9anAL6BS9Sx5NbCkuKp9FUomBu4jIrLky/jAg vP2W/P22vdc0uCrD4KBrP+R7Kz4nYYRZLCOrrmosbmJC0apLTEv3d/XJmcykkrED MM+OpxYHG6aSrAKicVAV+dBd2nXTp0rdX9hTjLeAU0omTa1y3g1V3tMhuZbEETpf lKI6sEmXG0LsT8tXDRaegGPUwg/7OcNhUcR9Jbjx+KvD1lqaZMmSJDVFjhLpJaMe RPabprl6ZkWmno/3W1tl/DrceVyBDHjaGVAanmcmD9Uj+NiKir15kX/Ud7DicA3I WudNSOuvHX84Btj8m1HmtM0Ua1MUzulX3RUSvCwLcYuaUfcVGaLgrFtTfAknYCWX sNL7ABT/7TEf92BKVtRnRVenyhVIUzJFsdsmQl5hJvNePRYTsCN6bQGFXq+gcEPQ 9VSKhf+Us36iwjD8uDd8WoXvfqhhQ4Ecl5MMM0XnDXPGg9819/o=
    =0V4a
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Andreas K. Huettel@21:1/5 to gentoo development on Fri Mar 15 19:21:37 2024
    Copy: jaco@uls.co.za (Jaco Kroon)

    Hi Jaco,

    * we have more stages
    * the binary packages have to go somewhere
    * and, temporarily, things are duplicated due to the 17.x / 23.0 profile transition

    The third point will eventually go away. However, I'm not sure how much it actually
    contributes.

    https://www.akhuettel.de/~huettel/plots/mirrors.php

    If you look at the plots, the distfiles part is surprisingly large.
    Binary packages (17.x and 23.0) and 17.x stages are under "releases".
    The 23.0 stages for testing are under "experimental".

    Lastly, I'm still working on an automated cleanup for outdated "small arches" binary
    packages (i.e. not arm64 and amd64, these are cleaned automatically already). This just wasn't a priority so far.

    Hope this helps.
    -a

    Am Freitag, 15. März 2024, 09:06:36 CET schrieb Jaco Kroon:
    Hi All,

    I was messing with some storage related caching on some of our hosts
    this morning when I wondered about how much storage the gentoo mirrors
    were consuming. I'm not too worried about the current storage, but I am noticing that the storage requirements are creeping quite a bit (as per attached), and if that growth rate continues it may become a problem *eventually*.

    Can this growth be explained?

    Is it expected to continue at this rate?

    Kind regards,
    Jaco



    --
    Andreas K. Hüttel
    dilfridge@gentoo.org
    Gentoo Linux developer
    (council, comrel, toolchain, base-system, perl, libreoffice) https://wiki.gentoo.org/wiki/User:Dilfridge
    -----BEGIN PGP SIGNATURE-----

    iQIzBAABCAAdFiEE/Rnm0xsZLuTcY+rT3CsWIV7VQSoFAmX0kbEACgkQ3CsWIV7V QSrSoQ//ZtbE90kC1KFJp1nYg9AqJvlGJkmMZHsTWVPnbHTIKuDbRKBfK8xMSiIv oJlDXy+Gm/sJk7yTsI7sEsVP+UZexkGh5cW7eSGU2EOgB72Zbm5YdrHwaEfOLKvG GVmh1tdhxS93PvKLuq73TQDLyej2UXBcuAGiWNC+hOEMs5WOoGQM4EbmXUT8fhbJ yosOHCtsX0CdD0AYaq7jJ6kroqzBzaqHVpJvKjQaEgFd4tr4kOtIUMkBSVuZEVcK wWemLO0CekBrysqRz/C7pCtKpvSUsXv9O2WJR45hRhk5enYCsBFcwt3Z6np1O5Jn 5R3f/LZYvBw41lrz+Bi1svgrsJQYMkj4y+1nk3oG1xxh58p6jF+pgjXM4xYBAVDv UoE+euwQAyk4gUzXt6Q7ZQmy9Zn0CnzQnAeVPLCND0HFVM6cCOGLbZnpeTLd3Ora DIQH20sHNyQN3gdhN0uS8adL/eAiZmgCuKqdCdXOmx1XSp0pJYpnFjPjyxUXhDRE /N38+4r0w66EUCY+RHPc2z6SszlAXxu8LNrekRBhckMdepiOfdkXIRQxwCQgNJ/O ZKiS0H2Eg1QFxL459+n4KrXWVjuHrPBFk/EijDv9W3sUkbI8g/fAuw3cGAfrXscl j8ecT67F9X2hdZMWLI8oc9mD6NReIpX1Jz29B5vETHTrBf42vPM=
    =AlTu
    -----END PGP SIGNATURE----