• Re: QA Experiment: removing "stale uploaders"

    From Scott Kitterman@21:1/5 to pollo@debian.org on Fri Aug 16 00:00:02 2024
    As a first cut, I would exclude archived repositories for removed packages (e.g. ptable).

    If the team as a whole is keeping a package up to date, I think we should be happy that the package is maintained and not expend effort to make it harder for people to do that.

    I'm really not sure how you are going to sort through this. Another example is appdirs. Yes, I didn't upload it the last few times someone touched it, but it's not in need of an upload now. What it really needs is to be removed, but there's a key
    package that depends on it. I don't think it's stale at all in any meaningful sense, but I don't know how you would know that without spending more time on the package than it's worth.

    I think it would be more useful to work on finding team packages that aren't maintained at all and have issues. Those should either be fixed or removed.

    Scott K

    On August 15, 2024 8:45:24 PM UTC, "Louis-Philippe Véronneau" <pollo@debian.org> wrote:
    Hello!

    I had a chance to have a chat with doko at DC24 and one of the things that came out was this question:

    "Are there packages in the DPT that aren't maintained by their uploaders and are kept in the team because other people fix bugs and do team uploads on them?"

    Let's call these "stale uploaders".

    To get some data, I had a little fun and wrote a Python script [1] to match a package git history with the Uploaders field in d/control. If in the last 3 years, someone listed in the Uploaders field hasn't made a single commit, the package gets flagged.

    Turns out about half of the packages in our namespace get flagged one way or another :P

    =====================
    Some caveats:

    1. To save time and disk space, I did shallow clones of the DPT's git repositories, using 2021-08-14 as a cutoff date.

    This certainly creates false-positives, but it seemed like a reasonable tradeoff.

    2. There might be some discrepancies between packages' git repositories on Salsa and what's been uploaded to the archive. Some of these repos might not have gone through NEW at all.

    3. A cursory look revealed a bunch of empty repositories [2] and packages that had been moved to other namespaces, but never removed from the DPT's namespace.

    4. Packages flagged as "None" don't have an "Uploaders" field. Either:

    * the DPT is the "Maintainer" and we should make sure the package has been orphaned

    or

    * there's a human "Maintainer" and the DPT isn't listed in Uploaders and we should fix the package.

    5. Some of the packages flagged as "Debian Python Team" have the team as Uploaders. Considering the recent policy change, we should probably try to make things more uniform by having the DPT as maintainer everywhere.
    =====================

    All in all, a fair amount of manual work is probably needed to make this list useful and remove false-positive, thus the 'QA Experiment' part in the title of this mail.

    Before putting more efforts into this, I wanted to hear from other team members. Do you think this is valuable work?

    If I get a good enough list (again, the current one needs work), would you support removing "stale uploaders" from our team-maintained packages??

    Cheers,

    [1]: https://salsa.debian.org/pollo/qa-scripts/-/blob/master/dpt-stale-uploaders.py

    [2]: See empty.txt. I took the liberty of removing them, as they were all older than 6 months.


    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Emmanuel Arias@21:1/5 to Scott Kitterman on Fri Aug 16 00:20:01 2024
    On Thu, Aug 15, 2024 at 09:55:10PM +0000, Scott Kitterman wrote:
    As a first cut, I would exclude archived repositories for removed packages (e.g. ptable).

    If the team as a whole is keeping a package up to date, I think we should be happy that the package is maintained and not expend effort to make it harder for people to do that.

    I'm really not sure how you are going to sort through this. Another example is appdirs. Yes, I didn't upload it the last few times someone touched it, but it's not in need of an upload now. What it really needs is to be removed, but there's a key
    package that depends on it. I don't think it's stale at all in any meaningful sense, but I don't know how you would know that without spending more time on the package than it's worth.

    Yes, this is one of the false positives mentioned by
    pollo, I think. I have some packages in the list, but
    they don't need an upload from time ago


    I think it would be more useful to work on finding team packages that aren't maintained at all and have issues. Those should either be fixed or removed.

    Scott K

    On August 15, 2024 8:45:24 PM UTC, "Louis-Philippe Véronneau" <pollo@debian.org> wrote:
    Hello!

    I had a chance to have a chat with doko at DC24 and one of the things that came out was this question:

    "Are there packages in the DPT that aren't maintained by their uploaders and are kept in the team because other people fix bugs and do team uploads on them?"

    Let's call these "stale uploaders".

    To get some data, I had a little fun and wrote a Python script [1] to match a package git history with the Uploaders field in d/control. If in the last 3 years, someone listed in the Uploaders field hasn't made a single commit, the package gets
    flagged.

    Turns out about half of the packages in our namespace get flagged one way or another :P

    =====================
    Some caveats:

    1. To save time and disk space, I did shallow clones of the DPT's git repositories, using 2021-08-14 as a cutoff date.

    This certainly creates false-positives, but it seemed like a reasonable tradeoff.

    2. There might be some discrepancies between packages' git repositories on Salsa and what's been uploaded to the archive. Some of these repos might not have gone through NEW at all.

    3. A cursory look revealed a bunch of empty repositories [2] and packages that had been moved to other namespaces, but never removed from the DPT's namespace.

    4. Packages flagged as "None" don't have an "Uploaders" field. Either:

    * the DPT is the "Maintainer" and we should make sure the package has been orphaned

    or

    * there's a human "Maintainer" and the DPT isn't listed in Uploaders and we should fix the package.

    5. Some of the packages flagged as "Debian Python Team" have the team as Uploaders. Considering the recent policy change, we should probably try to make things more uniform by having the DPT as maintainer everywhere.
    =====================

    All in all, a fair amount of manual work is probably needed to make this list useful and remove false-positive, thus the 'QA Experiment' part in the title of this mail.

    Before putting more efforts into this, I wanted to hear from other team members. Do you think this is valuable work?

    If I get a good enough list (again, the current one needs work), would you support removing "stale uploaders" from our team-maintained packages??

    Cheers,

    [1]: https://salsa.debian.org/pollo/qa-scripts/-/blob/master/dpt-stale-uploaders.py

    [2]: See empty.txt. I took the liberty of removing them, as they were all older than 6 months.



    --
    cheers,
    Emmanuel Arias

    ⢀⣴⠾⠻⢶⣦⠀
    ⣾⠁⢠⠒⠀⣿⡁ eamanu@debian.org
    ⢿⡄⠘⠷⠚⠋⠀ OpenPGP: 13796755BBC72BB8ABE2AEB5 FA9DEC5DE11C63F1
    ⠈⠳⣄

    -----BEGIN PGP SIGNATURE-----

    iQIzBAABCgAdFiEEE3lnVbvHK7ir4q61+p3sXeEcY/EFAma+fKYACgkQ+p3sXeEc Y/FecBAAtlClmX6fqXz4Kx8E9l+jZkbECO1C8w5BKhuvaJMoFfOMe9BFXj2RHeJd opZ7cvDzBV4mVyVsIfocenaLMhG4qZbVEVi9lshTY+hCqmkwKgl1bR4gJYJnSj2Z 1/n2cAEOVAkoQE7VZKsuKeK7SJKOq0LMov41iD7+lpo+7TT4xR4ZIX9VAJpAuEZU M0zU/y/cJjMEJdPPTOOUn5mM3+bElZCXOYwQtkZmDkOXn8Py4By4Fi4R2N4OZEQf Eem4LUk1ecislWbQ8PJUypLkafVjA6WBzFaSzBlqXnMn4uyRB9yWg6tJTyhWywRq 9YhGBAK6kruchODI30gAcaItAKrEPf5N0uTc2UkWbPxSbASb6C5gSJj6PscyDZ48 my3JV7Zc6XeOK5O/MGpgVsACI6rexga/guD6G5NO+RfmDUQ4OkXUPqLlflksWrSS K2QPSvizoGseJSnj7kxf9pYRgTXNeDVs/5pb1T+YuBqiG/xBIXDCldXi2fHkf7wG CCTjwjiRcYGXbbeo3avIraup3FLP05q1wcoK14MZbnJXTHQmItB4zyPA6pk3WYrX GNeLFWef0P7nKOMjq9/+sRV3FMDCG4U86EcYlcNR3OP2J7zDw6Nc3AvjVBICfA+Z FR7bUFVldPSFYS5isM9EDnkgdfd0muKcN2/5Xooiv6Qh3w0
  • From =?UTF-8?Q?Louis-Philippe_V=C3=A9ron@21:1/5 to All on Fri Aug 16 01:20:01 2024
    T24gMjAyNC0wOC0xNSA1IGggNTUgcC5tLiwgU2NvdHQgS2l0dGVybWFuIHdyb3RlOg0KPiBB cyBhIGZpcnN0IGN1dCwgSSB3b3VsZCBleGNsdWRlIGFyY2hpdmVkIHJlcG9zaXRvcmllcyBm b3IgcmVtb3ZlZCBwYWNrYWdlcyAoZS5nLiBwdGFibGUpLg0KDQpUaGFua3MsIEkgZGlkIHNl ZSB0aGlzIGFmdGVyIHRoZSBmYWN0IChhbHRob3VnaCB0aGVyZSdzIHZlcnkgZmV3IA0KcmVw b3NpdG9yaWVzIHRoYXQgYXJlIGFyY2hpdmVkKS4NCg0KPiBJZiB0aGUgdGVhbSBhcyBhIHdo b2xlIGlzIGtlZXBpbmcgYSBwYWNrYWdlIHVwIHRvIGRhdGUsIEkgdGhpbmsgd2Ugc2hvdWxk IGJlIGhhcHB5IHRoYXQgdGhlIHBhY2thZ2UgaXMgbWFpbnRhaW5lZCBhbmQgbm90IGV4cGVu ZCBlZmZvcnQgdG8gbWFrZSBpdCBoYXJkZXIgZm9yIHBlb3BsZSB0byBkbyB0aGF0Lg0KDQpJ IHRoaW5rIHRoYXQgbWFrZXMgc2Vuc2UgZm9yIGEgc3Vic2V0IG9mIHBhY2thZ2VzLCBidXQg aW4gZ2VuZXJhbCBJIGZlZWwgDQp0aGF0IHB1dHMgYSBsb3Qgb2YgYnVyZGVuIG9uIGEgZmV3 IHBlb3BsZSwgZXNwZWNpYWxseSBkdXJpbmcgdHJhbnNpdGlvbnMuDQoNCklmIHBlb3BsZSBk byBmaW5kIHRoZXkgdGVuZCB0byB0ZWFtIHVwbG9hZCB0aGUgc2FtZSBwYWNrYWdlcyBhZ2Fp biBhbmQgDQphZ2FpbiBhbmQgd2FudCB0byB0YWtlIGNhcmUgb2YgdGhlbSwgdGhleSBzaG91 bGQgYWRvcHQgdGhlbSBhbmQgYWRkIA0KdGhlaXIgbmFtZXMgdG8gdGhlIFVwbG9hZGVycyBs aXN0Lg0KDQpJZiB3ZSBjYW4gZ2V0IGEgbGlzdCBvZiBwYWNrYWdlcyB0aGF0IHNob3VsZCBi ZSBvcnBoYW5lZCwgaXQgbWlnaHQgYWxzbyANCm1vdGl2YXRlIHBlb3BsZSB0byBhZG9wdCB0 aGVtIGFuZCB0YWtlIGJldHRlciBjYXJlIG9mIHRoZW0uDQoNCj4gSSdtIHJlYWxseSBub3Qg c3VyZSBob3cgeW91IGFyZSBnb2luZyB0byBzb3J0IHRocm91Z2ggdGhpcy4gIEFub3RoZXIg ZXhhbXBsZSBpcyBhcHBkaXJzLiAgWWVzLCBJIGRpZG4ndCB1cGxvYWQgaXQgdGhlIGxhc3Qg ZmV3IHRpbWVzIHNvbWVvbmUgdG91Y2hlZCBpdCwgYnV0IGl0J3Mgbm90IGluIG5lZWQgb2Yg YW4gdXBsb2FkIG5vdy4gIFdoYXQgaXQgcmVhbGx5IG5lZWRzIGlzIHRvIGJlIHJlbW92ZWQs IGJ1dCB0aGVyZSdzIGEga2V5IHBhY2thZ2UgdGhhdCBkZXBlbmRzIG9uIGl0LiAgSSBkb24n dCB0aGluayBpdCdzIHN0YWxlIGF0IGFsbCBpbiBhbnkgbWVhbmluZ2Z1bCBzZW5zZSwgYnV0 IEkgZG9uJ3Qga25vdyBob3cgeW91IHdvdWxkIGtub3cgdGhhdCB3aXRob3V0IHNwZW5kaW5n IG1vcmUgdGltZSBvbiB0aGUgcGFja2FnZSB0aGFuIGl0J3Mgd29ydGguDQoNCj4gSSB0aGlu ayBpdCB3b3VsZCBiZSBtb3JlIHVzZWZ1bCB0byB3b3JrIG9uIGZpbmRpbmcgdGVhbSBwYWNr YWdlcyB0aGF0IGFyZW4ndCBtYWludGFpbmVkIGF0IGFsbCBhbmQgaGF2ZSBpc3N1ZXMuICBU aG9zZSBzaG91bGQgZWl0aGVyIGJlIGZpeGVkIG9yIHJlbW92ZWQuDQoNCkNyb3NzaW5nIGRp ZmZlcmVudCAic21lbGxzIiBsaWtlIG5vIGh1bWFuIHVwbG9hZGVycyBvdXRzaWRlIG9mICJz dGFsZSANCnVwbG9hZGVycyIsIG5vIHJlY2VudCByZWxlYXNlcywgbmV3IHVwc3RyZWFtIHZl cnNpb25zIG5vdCBwYWNrYWdlZCwgZXRjLiANCndvdWxkIHByb2JhYmx5IHlpZWxkIGEgbW9y ZSB0YW1lIGxpc3Qgb2YgcGFja2FnZXMgb25lIGNvdWxkIGhhdmUgYSBsb29rIGF0Lg0KDQpJ IGFncmVlIGdvaW5nIHRocm91Z2ggdGhlIGN1cnJlbnQgbGlzdCBhcy1pcyBieSBoYW5kIGlz IG5vdCB3b3J0aCBpdCA6UA0KDQpJbiB0aGUgZW5kLCBJJ20gbm90IDEwMCUgc3VyZSB3aGF0 IEknbGwgZG8gd2l0aCBhbGwgb2YgdGhpcyB5ZXQsIGJ1dCANCnN1Z2dlc3Rpb25zIGFyZSB3 ZWxjb21lZC4NCg0KSSBkbyBoYXZlIHNvbWUgY29kZSB0byBydW4gYSBidW5jaCBvZiB0ZXN0 cyBvbiBhbGwgb2Ygb3VyIHJlcG9zaXRvcmllcyANCnRob3VnaCBhbmQgdGhhdCB3YXMgYWxz byBhIGdvYWwgb2YgbWluZS4NCg0KLS0gDQogICDiooDio7TioL7ioLviorbio6bioIANCiAg IOKjvuKggeKioOKgkuKggOKjv+KhgSAgTG91aXMtUGhpbGlwcGUgVsOpcm9ubmVhdQ0KICAg 4qK/4qGE4qCY4qC34qCa4qCLICAgcG9sbG9AZGViaWFuLm9yZyAvIHZlcm9ubmVhdS5vcmcN CiAgIOKgiOKgs+KjhA0KDQo=

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From weepingclown@21:1/5 to pollo@debian.org on Fri Aug 16 15:20:02 2024
    Hi,

    This is a very reasonable thing to do considering the policy change. I also have two packages where I mistakenly ended up as the maintaner and DPT as uploader (see python-graphene-directives and python-graphene-federation) because of tooling defaults and
    me missing to see it in the end. I have been contemplating since I realized this about how urgently I should take care of this. What I mean is that, does this call for an immediate revision of the debian package, or is it sufficient to have a git commit
    to reverse it and then later upload it with the next possible upstream release. Thoughts?

    PS: And depending on how strictly we want to enforce this, there can perhaps also be a lintian warning/error?

    Best,
    Ananthu

    On 15 August 2024 8:45:24 pm UTC, "Louis-Philippe Véronneau" <pollo@debian.org> wrote:
    5. Some of the packages flagged as "Debian Python Team" have the team as Uploaders. Considering the recent policy change, we should probably try to make things more uniform by having the DPT as maintainer everywhere.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From =?UTF-8?Q?Louis-Philippe_V=C3=A9ron@21:1/5 to All on Fri Aug 16 20:20:01 2024
    T24gMjAyNC0wOC0xNiA5IGggMDkgYS5tLiwgd2VlcGluZ2Nsb3duIHdyb3RlOg0KPiBIaSwN Cj4gDQo+IFRoaXMgaXMgYSB2ZXJ5IHJlYXNvbmFibGUgdGhpbmcgdG8gZG8gY29uc2lkZXJp bmcgdGhlIHBvbGljeSBjaGFuZ2UuIEkgYWxzbyBoYXZlIHR3byBwYWNrYWdlcyB3aGVyZSBJ IG1pc3Rha2VubHkgZW5kZWQgdXAgYXMgdGhlIG1haW50YW5lciBhbmQgRFBUIGFzIHVwbG9h ZGVyIChzZWUgcHl0aG9uLWdyYXBoZW5lLWRpcmVjdGl2ZXMgYW5kIHB5dGhvbi1ncmFwaGVu ZS1mZWRlcmF0aW9uKSBiZWNhdXNlIG9mIHRvb2xpbmcgZGVmYXVsdHMgYW5kIG1lIG1pc3Np bmcgdG8gc2VlIGl0IGluIHRoZSBlbmQuIEkgaGF2ZSBiZWVuIGNvbnRlbXBsYXRpbmcgc2lu Y2UgSSByZWFsaXplZCB0aGlzIGFib3V0IGhvdyB1cmdlbnRseSBJIHNob3VsZCB0YWtlIGNh cmUgb2YgdGhpcy4gV2hhdCBJIG1lYW4gaXMgdGhhdCwgZG9lcyB0aGlzIGNhbGwgZm9yIGFu IGltbWVkaWF0ZSByZXZpc2lvbiBvZiB0aGUgZGViaWFuIHBhY2thZ2UsIG9yIGlzIGl0IHN1 ZmZpY2llbnQgdG8gaGF2ZSBhIGdpdCBjb21taXQgdG8gcmV2ZXJzZSBpdCBhbmQgdGhlbiBs YXRlciB1cGxvYWQgaXQgd2l0aCB0aGUgbmV4dCBwb3NzaWJsZSB1cHN0cmVhbSByZWxlYXNl LiBUaG91Z2h0cz8NCg0KRmVlbCBmcmVlIHRvIGNvbW1pdCBhbmQgbm90IG1ha2UgYSBuZXcg cmVsZWFzZS4gSSBkb24ndCB0aGluayBhbnl0aGluZyANCmlzIHVyZ2VudC4NCg0KPiBQUzog QW5kIGRlcGVuZGluZyBvbiBob3cgc3RyaWN0bHkgd2Ugd2FudCB0byBlbmZvcmNlIHRoaXMs IHRoZXJlIGNhbiBwZXJoYXBzIGFsc28gYmUgYSBsaW50aWFuIHdhcm5pbmcvZXJyb3I/DQoN CkxpbnRpYW4gY2FuJ3QgYmUgdXNlZCB0byBnZW5lcmF0ZSB0aGlzIGtpbmQgb2YgaW5mb3Jt YXRpb24sIGFzIGl0IG9ubHkgDQpoYXMgYWNjZXNzIHRvIHRoZSBzb3VyY2UgYW5kIGJpbmFy eSBwYWNrYWdlcywgbm90IHRoZSBnaXQgaGlzdG9yeS4NCg0KSSBmZWVsIHJlbHlpbmcgb24g ZC9jaGFuZ2Vsb2cgZm9yIHRoaXMgZG9lc24ndCBwcm92aWRlIGVub3VnaCBpbmZvcm1hdGlv bi4NCg0KLS0gDQogICDiooDio7TioL7ioLviorbio6bioIANCiAgIOKjvuKggeKioOKgkuKg gOKjv+KhgSAgTG91aXMtUGhpbGlwcGUgVsOpcm9ubmVhdQ0KICAg4qK/4qGE4qCY4qC34qCa 4qCLICAgcG9sbG9AZGViaWFuLm9yZyAvIHZlcm9ubmVhdS5vcmcNCiAgIOKgiOKgs+KjhA0K

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)