• DPT repositories checks/"violations" report

    From Sandro Tosi@21:1/5 to All on Sat Nov 27 07:10:01 2021
    Hello,
    while working on something else[1], i noticed how many of the
    repositories in the DPT salsa group are in poor shape:

    * missing branches
    * changes not pushed to salsa
    * general misalignment in configuration/setup/organization
    * many other small nuances

    [1] https://github.com/sandrotosi/debian-python-team-tracker

    That makes it hard to make mass work as in [1], so I thought it would
    be interesting to have a tool to evaluate the amount of issues our
    repos have, and identify such "violations" so that they can be
    addressed.

    That information is now available at [2].

    [2] https://github.com/sandrotosi/dpt-repos-check/blob/main/violations.txt

    please take the content with caution, as it's still an early, raw
    draft (i spot-checked some of the reported issues, but there could be bugs/issues) and it contains data that can be outdated (see below re
    caching); the fact that the report indicates only 43 repos are without violations is a bit disarming, but i think the tool tries to err on
    the side of verbosity and pedantry, with 2 level of violations (ERROR
    and WARNING) to mark which ones are the most important that require
    immediate attention vs the nice-to-have ones.

    The checks are under-documented, although they should be somehow self-explanatory. While the current format is just a text file, I plan
    on getting it converted to markdown, although I'm still not sure how
    to convey that amount of information effectively.

    The report is extremely intensive to generate, as it needs to make 10+
    API calls to salsa.d.o for each repository, and it gets throttled
    quite early in the run (i got away in dev by installing a cache, so
    that i could test it without hitting salsa every time, but that makes
    some info stale); for that reason, and if we decide is valuable to
    generate it periodically, i don't expect to be able to run it more
    than once a month (maybe once a week? TBD).

    Comments, critics and improvements are welcome.

    Thanks,
    --
    Sandro "morph" Tosi
    My website: http://sandrotosi.me/
    Me at Debian: http://wiki.debian.org/SandroTosi
    Twitter: https://twitter.com/sandrotosi

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Scott Kitterman@21:1/5 to Sandro Tosi on Sat Nov 27 10:50:01 2021
    On November 27, 2021 6:01:08 AM UTC, Sandro Tosi <morph@debian.org> wrote: >Hello,
    while working on something else[1], i noticed how many of the
    repositories in the DPT salsa group are in poor shape:

    * missing branches
    * changes not pushed to salsa
    * general misalignment in configuration/setup/organization
    * many other small nuances

    [1] https://github.com/sandrotosi/debian-python-team-tracker

    That makes it hard to make mass work as in [1], so I thought it would
    be interesting to have a tool to evaluate the amount of issues our
    repos have, and identify such "violations" so that they can be
    addressed.

    That information is now available at [2].

    [2] https://github.com/sandrotosi/dpt-repos-check/blob/main/violations.txt

    please take the content with caution, as it's still an early, raw
    draft (i spot-checked some of the reported issues, but there could be >bugs/issues) and it contains data that can be outdated (see below re >caching); the fact that the report indicates only 43 repos are without >violations is a bit disarming, but i think the tool tries to err on
    the side of verbosity and pedantry, with 2 level of violations (ERROR
    and WARNING) to mark which ones are the most important that require
    immediate attention vs the nice-to-have ones.

    The checks are under-documented, although they should be somehow >self-explanatory. While the current format is just a text file, I plan
    on getting it converted to markdown, although I'm still not sure how
    to convey that amount of information effectively.

    The report is extremely intensive to generate, as it needs to make 10+
    API calls to salsa.d.o for each repository, and it gets throttled
    quite early in the run (i got away in dev by installing a cache, so
    that i could test it without hitting salsa every time, but that makes
    some info stale); for that reason, and if we decide is valuable to
    generate it periodically, i don't expect to be able to run it more
    than once a month (maybe once a week? TBD).

    Comments, critics and improvements are welcome.

    I don't think the pypi tarball "issue" should be presumed to be a problem at all. I wasn't paying attention to Debian when that discussion happened, but in my experience there was a lot wrong with the idea. A properly constructed sdist is exactly what
    we want to build a package from. That's almost never found on GitHub.

    Scott K

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Simon McVittie@21:1/5 to Scott Kitterman on Sat Nov 27 12:20:01 2021
    On Sat, 27 Nov 2021 at 09:38:41 +0000, Scott Kitterman wrote:
    I don't think the pypi tarball "issue" should be presumed to be a
    problem at all. I wasn't paying attention to Debian when that discussion happened, but in my experience there was a lot wrong with the idea.
    A properly constructed sdist is exactly what we want to build a package
    from. That's almost never found on GitHub.

    I think the closest we got to a conclusion was "it depends": if your
    upstream reliably produces a properly constructed sdist (or at least is
    happy to accept pull requests to make their sdist properly constructed)
    then it makes an ideal source package, but if your upstream treats sdists
    in closer to the same way a C programmer would treat a prebuilt binary
    release (omitting source and including content generated from that source instead), then a git clone is probably more appropriate.

    To me, at least, it makes sense for this to be a case-by-case decision
    made by someone who is familiar with this specific upstream - and wanting
    to have someone familiar with this specific upstream is why we have named maintainers, rather than having everything collectively-maintained like
    some distributions do.

    (For what it's worth, the GNOME team uses a mixture of `meson dist` and
    git clones, and that's with an upstream that is a single project that is
    in principle meant to be team-maintained with a single cohesive policy -
    so if we can't standardize on one source format being "always the right
    one" for GNOME, I would be very surprised if the Python team was able to standardize on one source format for a large number of separate upstreams linked only by their implementation language.)

    smcv

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Stefano Rivera@21:1/5 to All on Sat Nov 27 15:00:01 2021
    Hi Sandro (2021.11.27_06:01:08_+0000)

    Hello,
    while working on something else[1], i noticed how many of the
    repositories in the DPT salsa group are in poor shape:

    * missing branches
    * changes not pushed to salsa
    * general misalignment in configuration/setup/organization
    * many other small nuances

    [1] https://github.com/sandrotosi/debian-python-team-tracker

    +1 this is great!

    I've been doing some 3.10 NMUs recently, and found myself re-creating
    several upstream and pristine-tar branches...

    please take the content with caution, as it's still an early, raw
    draft (i spot-checked some of the reported issues, but there could be bugs/issues) and it contains data that can be outdated (see below re caching); the fact that the report indicates only 43 repos are without violations is a bit disarming, but i think the tool tries to err on
    the side of verbosity and pedantry, with 2 level of violations (ERROR
    and WARNING) to mark which ones are the most important that require
    immediate attention vs the nice-to-have ones.

    When we did the migration to git, there weren't good tools for managing
    the setup of the salsa repos (hooks, etc.) yet. I'd assume those exist
    now, we should check in with what other teams are doing. That stuff can
    all be fixed in one run of a tool, I'd assume.

    SR

    --
    Stefano Rivera
    http://tumbleweed.org.za/
    +1 415 683 3272

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From =?UTF-8?Q?Louis-Philippe_V=c3=a9ron@21:1/5 to Stefano Rivera on Sat Nov 27 19:20:02 2021
    This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------1tMdbtI8O39Ij0cSwMztUnFF
    Content-Type: text/plain; charset=UTF-8
    Content-Transfer-Encoding: quoted-printable

    On 2021-11-27 08 h 54, Stefano Rivera wrote:
    Hi Sandro (2021.11.27_06:01:08_+0000)

    Hello,
    while working on something else[1], i noticed how many of the
    repositories in the DPT salsa group are in poor shape:

    * missing branches
    * changes not pushed to salsa
    * general misalignment in configuration/setup/organization
    * many other small nuances

    [1] https://github.com/sandrotosi/debian-python-team-tracker

    +1 this is great!

    \0/

    I've been wanting something for QA like that for a while, but never had
    the time / energy to look into it further. All in all, it's too easy to
    forget to push something to Salsa and never realise it.

    please take the content with caution, as it's still an early, raw
    draft (i spot-checked some of the reported issues, but there could be
    bugs/issues) and it contains data that can be outdated (see below re
    caching); the fact that the report indicates only 43 repos are without
    violations is a bit disarming, but i think the tool tries to err on
    the side of verbosity and pedantry, with 2 level of violations (ERROR
    and WARNING) to mark which ones are the most important that require
    immediate attention vs the nice-to-have ones.

    When we did the migration to git, there weren't good tools for managing
    the setup of the salsa repos (hooks, etc.) yet. I'd assume those exist
    now, we should check in with what other teams are doing. That stuff can
    all be fixed in one run of a tool, I'd assume.

    Could this become part of the Debian Janitor at some point?

    I could see teams adding a per-team config file to check things like
    what branch names should be expected, etc. and the Janitor fixing all
    this if it has commit access.

    --
    ⢀⣴⠾⠻⢶⣦⠀
    ⣾⠁⢠⠒⠀⣿⡁ Louis-Philippe Véronneau
    ⢿⡄⠘⠷⠚⠋ pollo@debian.org / veronneau.org
    ⠈⠳⣄

    --------------1tMdbtI8O39Ij0cSwMztUnFF--

    -----BEGIN PGP SIGNATURE-----

    iHUEARYKAB0WIQTKp0AHB6gWsCAvw830JXpQshz6hQUCYaJ1bgAKCRD0JXpQshz6 hVr2AP0c48GdTNBRTjJf9JgFlwTGaZ9jvbYBT96gCuB+CZhXHgEApQ0unRM0DpzG 3fuNsPGEVnKNgBLDTnLlLAR3LGAAwAM=
    =T3K0
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Sandro Tosi@21:1/5 to All on Fri Dec 10 07:40:02 2021
    When we did the migration to git, there weren't good tools for managing
    the setup of the salsa repos (hooks, etc.) yet. I'd assume those exist
    now, we should check in with what other teams are doing. That stuff can
    all be fixed in one run of a tool, I'd assume.

    yeah i figured that much, and that made sense at that time.

    I modified [1] to automatically create the salsa repo enabling both
    KGB and tagpending webhooks, but the `salsa` tool doesnt support
    setting integrations, so that will need to be fixed later on.

    [1] https://github.com/sandrotosi/pypi2deb/tree/morph

    I'm in the process of writing a tool to uniform the repo configuration
    in python-team/package

    - add integration: Emails on push
    - remove integration Irker
    - add webhook: KGB (or edit to remove all the extra parameters set,
    which are the default values anyway)
    - add webhook: tagpending

    I'll write back here when i think it's ready for review and request authorization to run it.

    I think there's still one point we need to figure out: how to make
    these remarks known to the packages maintainers, instead of all of
    them just being in a text file.

    Thanks,
    --
    Sandro "morph" Tosi
    My website: http://sandrotosi.me/
    Me at Debian: http://wiki.debian.org/SandroTosi
    Twitter: https://twitter.com/sandrotosi

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Sandro Tosi@21:1/5 to All on Sun Dec 12 07:30:01 2021
    I'm in the process of writing a tool to uniform the repo configuration
    in python-team/package

    - add integration: Emails on push
    - remove integration Irker
    - add webhook: KGB (or edit to remove all the extra parameters set,
    which are the default values anyway)
    - add webhook: tagpending

    the tool is now ready: https://github.com/sandrotosi/dpt-repos-check/blob/main/dpt-fix-integrations-webhooks.py

    I'll write back here when i think it's ready for review and request authorization to run it.

    can any admin check if the changes look sane (i've run a test run on
    ~25 repos) and that i'm allowed to run this across the whole
    `packages` subgroup on salsa?

    I think there's still one point we need to figure out: how to make
    these remarks known to the packages maintainers, instead of all of
    them just being in a text file.

    This is still an open point, and i welcome ideas

    Thanks,
    --
    Sandro "morph" Tosi
    My website: http://sandrotosi.me/
    Me at Debian: http://wiki.debian.org/SandroTosi
    Twitter: https://twitter.com/sandrotosi

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From =?UTF-8?Q?Louis-Philippe_V=c3=a9ron@21:1/5 to All on Mon Jan 3 02:00:02 2022
    This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------pY6CgXBWQeS0qpd56R7ZVyfw
    Content-Type: text/plain; charset=UTF-8
    Content-Transfer-Encoding: base64

    T24gMjAyMS0xMi0xMiAwMSBoIDIzLCBTYW5kcm8gVG9zaSB3cm90ZToNCj4+IEkgdGhpbmsg dGhlcmUncyBzdGlsbCBvbmUgcG9pbnQgd2UgbmVlZCB0byBmaWd1cmUgb3V0OiBob3cgdG8g bWFrZQ0KPj4gdGhlc2UgcmVtYXJrcyBrbm93biB0byB0aGUgcGFja2FnZXMgbWFpbnRhaW5l cnMsIGluc3RlYWQgb2YgYWxsIG9mDQo+PiB0aGVtIGp1c3QgYmVpbmcgaW4gYSB0ZXh0IGZp bGUuDQo+IA0KPiBUaGlzIGlzIHN0aWxsIGFuIG9wZW4gcG9pbnQsIGFuZCBpIHdlbGNvbWUg aWRlYXMNCg0KSXMgdGhlcmUgYSByZWFzb24gd2h5IHdlIHNob3VsZG4ndCBqdXN0IGZpbGUg YnVncyBpbiB0aGUgQlRTPyBJIGdldCBpdCdzDQpub3QgdGhlIHBlcmZlY3QgdG9vbCBmb3Ig dGhhdCwgYnV0IGl0IHdvdWxkIGNlcnRhaW5seSBoZWxwIHJlYWNoIHRoZQ0KbWFpbnRhaW5l cnMuDQoNClVzaW5nIGEgY29tbW9uIHVzZXJ0YWcgd291bGQgYWxzbyBtYWtlIGl0IGVhc2ll ciB0byBmaW5kIGFuZCBmaXggdGhlc2UNCmlzc3VlcyBlbiBtYXNzZSA7KQ0KDQotLSANCiAg 4qKA4qO04qC+4qC74qK24qOm4qCADQogIOKjvuKggeKioOKgkuKggOKjv+KhgSAgTG91aXMt UGhpbGlwcGUgVsOpcm9ubmVhdQ0KICDior/ioYTioJjioLfioJrioIsgICBwb2xsb0BkZWJp YW4ub3JnIC8gdmVyb25uZWF1Lm9yZw0KICDioIjioLPio4QNCg==

    --------------pY6CgXBWQeS0qpd56R7ZVyfw--

    -----BEGIN PGP SIGNATURE-----

    iHUEARYKAB0WIQTKp0AHB6gWsCAvw830JXpQshz6hQUCYdJIOgAKCRD0JXpQshz6 hWM/AQCnBMDqdVEX7Pq02aiz16TrzH3LJrpFOFURBJc9RyzoKgEA0ZsoaLXA1i2Q U0oqFNPYXJMwMmw5vXaIjDNBfjSLVg4=
    =fWLK
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Scott Kitterman@21:1/5 to All on Mon Jan 3 07:40:01 2022
    On Sunday, January 2, 2022 7:50:02 PM EST Louis-Philippe Vronneau wrote:
    On 2021-12-12 01 h 23, Sandro Tosi wrote:

    I think there's still one point we need to figure out: how to make
    these remarks known to the packages maintainers, instead of all of
    them just being in a text file.


    This is still an open point, and i welcome ideas


    Is there a reason why we shouldn't just file bugs in the BTS? I get it's
    not the perfect tool for that, but it would certainly help reach the maintainers.

    Using a common usertag would also make it easier to find and fix these
    issues en masse ;)

    Since this is all about team Git repositories, someone should just fix them (modulo the one about using pypi, which I think we mostly agree isn't something someone unfamiliar with the package can 'fix').

    Scott K
    -----BEGIN PGP SIGNATURE-----

    iQIzBAABCgAdFiEE53Kb/76FQA/u7iOxeNfe+5rVmvEFAmHSl/EACgkQeNfe+5rV mvG75BAAtUb6m0gFKtSFb1PeHDnP46AGnMI6U0Kd9Psfa6vDz6xZAGKkN2WN8Lb4 doj1G5psrTldoHHtG0fwpRmFlC05VBZ6e8wdbkIwspkXGdA84s7FD0Zx8zBJ4hbL LeJ19gtvj2fLxlbPZmAfzVhTRzSaxgmewXqz+RHIEqmkLo8nKzd2/vO2yD72Toxz AWNBIs+YjtFNUmcdnyvzwaAOcumK5Ju28fSUD3LPeV4wme6b10GR9zpFl+5JvwzE H8wwX0Mlq/CQMUru0QMMl+mVdSRcFNVDlfPNqYO8B3/QuDwlZ+y+UAv8wZSysp1H B/TOTUuFisM8riUOAruNHXU4vJWhulaxoxIrik/Ut56zRMfdE13mJvHq/bgL4Crn 0XD9BlbSPtRPjVbPFqtu9Z2b2gI6SOeRg6qNGsHe8ZZwTRRYDJCaHU+V8ShLyXjJ 4j9dAj2CK93FDUUJqy0CZJAzpDPHKkbyTTdw2iruqoZ6QEpo1LcdaudHiuMqwiH/ k2X7dxjlQxp39invVEaC4NdKbWdNaUnXPm7dYBreKZQYZWb3lxR6sK/eaaht95Kv RV8O7nEX1BVijmFQuZ51+Kulk57kc/+iOa7ll2QeZUjroCPklAahdYaJ/1XpEuxP NqIMgP66R4WPiZcmIQ9ik73sMbhf3g5PChE0QYgcFay5xWy2ons=
    =L6LP
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From =?UTF-8?Q?Louis-Philippe_V=c3=a9ron@21:1/5 to All on Mon Jan 3 21:20:02 2022
    This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------JE6ox0ooRUK91PoJu3ynOfrp
    Content-Type: text/plain; charset=UTF-8
    Content-Transfer-Encoding: base64

    T24gMjAyMi0wMS0wMyAwMSBoIDMwLCBTY290dCBLaXR0ZXJtYW4gd3JvdGU6DQo+IFNpbmNl IHRoaXMgaXMgYWxsIGFib3V0IHRlYW0gR2l0IHJlcG9zaXRvcmllcywgc29tZW9uZSBzaG91 bGQganVzdCBmaXggdGhlbSANCj4gKG1vZHVsbyB0aGUgb25lIGFib3V0IHVzaW5nIHB5cGks IHdoaWNoIEkgdGhpbmsgd2UgbW9zdGx5IGFncmVlIGlzbid0IA0KPiBzb21ldGhpbmcgc29t ZW9uZSB1bmZhbWlsaWFyIHdpdGggdGhlIHBhY2thZ2UgY2FuICdmaXgnKS4NCkJ1dCB0aGF0 IGRvZXNuJ3QgcHJldmVudCBmdXR1cmUgZXJyb3JzIGZyb20gcG9wcGluZyB1cCBhbmQgZG9l c24ndCBtYWtlDQptYWludGFpbmVycyBhd2FyZSBvZiB0aGVpciBlcnJvcnMgKHNvIHRoZXkg Y2FuIGxlYXJuIGZyb20gaXQpLg0KDQpJIHRoaW5rIHRoZSAicGVyZmVjdCIgc29sdXRpb24g d291bGQgYmUgdG8gaGF2ZSBhbiBhdXRvbWF0ZWQgdG9vbCAoYWthDQpqYW5pdG9yKSBmaXhp bmcgdGhlc2UgYXV0b21hdGljYWxseSwgYnV0IHRoaXMgd291bGQgcmVxdWlyZSBtb3JlIHdv cmsuDQoNCi0tIA0KICDiooDio7TioL7ioLviorbio6bioIANCiAg4qO+4qCB4qKg4qCS4qCA 4qO/4qGBICBMb3Vpcy1QaGlsaXBwZSBWw6lyb25uZWF1DQogIOKiv+KhhOKgmOKgt+KgmuKg iyAgIHBvbGxvQGRlYmlhbi5vcmcgLyB2ZXJvbm5lYXUub3JnDQogIOKgiOKgs+KjhA0K

    --------------JE6ox0ooRUK91PoJu3ynOfrp--

    -----BEGIN PGP SIGNATURE-----

    iHUEARYKAB0WIQTKp0AHB6gWsCAvw830JXpQshz6hQUCYdNY4wAKCRD0JXpQshz6 hVFUAP9crVB491As0/XE3fmyOuAbdNsMEo+oShV1pHckErpyPAEAgimby86H5c6X K93UTT884qNIX2OrIXdTQh/vnq4iBwc=
    =eW6x
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Scott Kitterman@21:1/5 to All on Mon Jan 3 21:20:01 2022
    On Monday, January 3, 2022 3:13:23 PM EST Louis-Philippe Vronneau wrote:
    On 2022-01-03 01 h 30, Scott Kitterman wrote:

    Since this is all about team Git repositories, someone should just fix
    them
    (modulo the one about using pypi, which I think we mostly agree
    isn't something someone unfamiliar with the package can 'fix').

    But that doesn't prevent future errors from popping up and doesn't make maintainers aware of their errors (so they can learn from it).

    I think the "perfect" solution would be to have an automated tool (aka janitor) fixing these automatically, but this would require more work.

    The first step if we think these things should be the standard for team repositories is to document it. Personally, I would look here:

    https://wiki.debian.org/Python/GitPackaging#Creating_new_repositories

    At least for me, a good explanation of what to do in the documentation would be much more valuable than a bug report. Approximately all of the team repositories I've created are "wrong" because I followed the documentation.

    Scott K
    -----BEGIN PGP SIGNATURE-----

    iQIzBAABCgAdFiEE53Kb/76FQA/u7iOxeNfe+5rVmvEFAmHTWfkACgkQeNfe+5rV mvFRmhAApW2k68/OWRZFLEdXjboknU6euBqUxK5QRueWLALIC76gui826ySxIaUU Fanw5s+xspjZRCLeJabpyZI1JgpDb0xtyD4R6qEb5waFzgUkhnWnwZeIlwNTE4Lg lz68/uMleZL60TqjKCgKKh/9iRx3bUvKf3jjYGcSjdj+czwTOWTKR8ejqh8GwKcb U2gJLjpmxX6lyVFrNgP2mLTUA/vvRmvzQhpzm2P0IrLble2AfU676DASFJ3dYhd8 xrOdLwecGMhS6uJX5QixNTjXsSXj6vsoC9Dj3BFrH+gsiFyA17kLlwBYT2UoHyyV faPihF48BpC6s9AycId5t+dfB+2/JfP+RuBi4rNWFkoNCRiB4mfIcmfZNUQ4csE2 oC7RTdqCM6dtMld4DyfGQAeJJP2fdAXWShOS8AReMRttKVc6O0zWZbbs+MowAYSF DMK/0A2N2COdpK86nAYAOvYew6ZkK5/o2FkG9bzUvd6AjEWDIpIbtXveWNRVVusg 8//x8yb7nRFLxVvfQxlAjJj/ils9DVDNXBvOLOKLeyy92EZWZIyaqkdGrogAw9ZF zZVjfZdsDN/07WuTUd6D6vzbHFku1VPjYFLaltXac9BYP7vQteJ924A948bSxaFX DZZhGYl3/PgTOj9Pa/BQ3fOyiX51anXIFrXR7rlyY/bGHOAgUeU=
    =968b
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)