• Uscan with gitlab and user provided tarball

    From Ole Streicher@21:1/5 to All on Tue Oct 12 15:10:01 2021
    Hi,

    the upstream of one of my packages recently moved from sourceforge to
    Gitlab:

    https://gitlab.com/aroffringa/wsclean

    He uses git submodules, and this makes the automatically created tarball incomplete. For my convenience, he created (and hopefully will continue
    so) a manual asset, which is linked on the Releases page

    https://gitlab.com/aroffringa/wsclean/-/releases/

    Unfortunately, the file URL does not have a canonical name, as seen on
    the HTML snippet:

    <a href="https://gitlab.com/aroffringa/wsclean/-/package_files/15813079/download"
    class="…">
    <svg …>…</svg>
    wsclean-v3.0.tar.bz2


    This HTML is also generated by a script, so not directly downloadable.

    Since Gitlab is one of the standard providers, there may be a good
    solution to get the correct tarball? Or is there a better way for
    upstream to provide a complete tarball than the one chosen here?

    Best regards

    Ole

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Paul Wise@21:1/5 to Ole Streicher on Tue Oct 19 07:00:02 2021
    On Tue, 2021-10-12 at 14:43 +0200, Ole Streicher wrote:

    https://gitlab.com/aroffringa/wsclean

    He uses git submodules

    These all look like embedded code copies, so I suggest packaging them separately instead of including them the wsclean source tarball.

    https://wiki.debian.org/EmbeddedCopy

    --
    bye,
    pabs

    https://wiki.debian.org/PaulWise

    -----BEGIN PGP SIGNATURE-----

    iQIzBAABCgAdFiEEYQsotVz8/kXqG1Y7MRa6Xp/6aaMFAmFuTrsACgkQMRa6Xp/6 aaOqIg//XnSqP+IVbQvydTEtHEYw5p0lWE6590wIkv4whsIZ17ufP6P3tmTaivWo 4E4ByYb9l12XS7mRLuuYWFTvKgkvuXt0UWpYQY5q4HzVzSICgb4znlLwKgMj9Msi ro1Z91kwG55es6rjcG9GsOM82hkr+VjAyOvPTeCbZuSy84cGA1gzDNmETZTLJ1YC twusRMCk7iUWzFewuBhlazHz5f2S1z54jbyiIgjcXj6jUHr/b84CxAb0+g+UMois EZ4qQDD9FmCi544R02QACeXzhjM7qZ0brdymvza7rBDpAwSUw+3wtuq4SFiSFwYv raJY3emj/pav67AIgTgZo5LoDp1uk1wdNrz3lw6A8t+wDekJazj+U6M74HwOFhns JVVzuCFzFDjkh/GyhS9xoc2qb42jwjBzsJV96z2RvQrDfU9TKyCW42sbuJOcdetg C6YayPqxWqFVtP2YWWXaF+heX98b0rMHrvgadVY1uD2SAzl5YUYvg3bomRV5P7qZ r3551UyLW6OJkAhf643j9h1qpJxTwOn7MExPx5skPEZm+v1BoLjosTD2XvIHURi/ P4DE4ywmJDrQSn6QrZD+jZB1RdbLpRxUEeUOA/mKpDDIwpknOxzAmSu73+6XZOQs cSpTkUd053Ayuf4m7VCi4xtj1Y19ZLwtn2IYyD1KUPoqZNJinCU=
    =HrmP
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Ole Streicher@21:1/5 to Paul Wise on Tue Oct 19 09:30:01 2021
    Paul Wise <pabs@debian.org> writes:
    On Tue, 2021-10-12 at 14:43 +0200, Ole Streicher wrote:

    https://gitlab.com/aroffringa/wsclean

    He uses git submodules

    These all look like embedded code copies, so I suggest packaging them separately instead of including them the wsclean source tarball.

    At least some of them are not really suitable to be packaged as separate
    public packages. The two submodules in question (AOcommon, schaapcommon)
    are personal utility functions of the upstream authors which are not
    intended for general use.

    I remember that a few years ago, I had some packages which all used a "GreatCMakeCookoff" (or so) package, a collection of (not really
    re-usable) tools to extend CMake. When trying to ITP this, the common
    wisdom here was that this should not be packaged so that others do not
    start depend on a bad-quality package.

    To me, this looks similar: These are not for the public, as their
    interface may change wildly, they are not even tagged in git. Even if
    they are re-used in another package (wsclean), there is no guarantee for
    me that both can be built on the same commit of the utility
    repositories. There is zero gain in separating them, and only causing
    lots of trouble.

    I really see these two submodules as integral part of the source and
    would like to continue packaging it this way. Which keeps the question
    how to effecitvely (with the help of upstream) detect and download the
    complete source.

    Best regards

    Ole

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Paul Wise@21:1/5 to Ole Streicher on Wed Oct 20 02:20:01 2021
    On Tue, 2021-10-19 at 09:03 +0200, Ole Streicher wrote:

    At least some of them are not really suitable to be packaged as
    separate public packages. The two submodules in question (AOcommon, schaapcommon) are personal utility functions of the upstream authors
    which are not intended for general use.

    I maintain a few packages that are in a similar situation, I've
    actually been leaning towards packaging the code copy separately but
    haven't gotten around to it yet. The code copy isn't changing very
    often though, it is fairly mature.

    how to effecitvely (with the help of upstream) detect and download
    the complete source.

    I would use the component feature of dpkg-source v3 source packages to
    add additional orig tarballs, one for each of the submodules.

    Unfortunately the component feature doesn't support subdirectories but
    you can workaround that by creating symlinks in debian/rules.

    uscan supports the component feature, but as far as I can tell the git
    mode of uscan doesn't support git submodules. I suggest you file a bug
    or patch asking for support for that if there isn't one already.

    If you don't want to use uscan, you can just create a get-orig-source
    target in debian/rules with git clone and git archive and run that to
    download releases.

    If you want to use uscan, you will have to rely on the http mode for
    downloads for now. Since you don't have proper links, the html
    searchmode will not work for you. So you will have to either use the
    plain searchmode or use a pagemangle to transform the HTML to something
    useful. Unfortunately GitLab does everything from JavaScript so neither
    of those will work. I suggest you load the releases page with your
    browser dev tools open, work out which API is used to download the
    releases information and then have uscan download that and use the
    plain searchmode and various mangle rules to find the files. Once you
    have figured it out, please document it on the wiki to help others.

    https://wiki.debian.org/debian/watch#Gitlab

    --
    bye,
    pabs

    https://wiki.debian.org/PaulWise

    -----BEGIN PGP SIGNATURE-----

    iQIzBAABCgAdFiEEYQsotVz8/kXqG1Y7MRa6Xp/6aaMFAmFvXsAACgkQMRa6Xp/6 aaPBFw//T/oVbXzYCfbppqUALDV0wm3gpFY2d/bzkjpUVi7T2l3FEGEYEzglcer5 fo1e4lD9C8EM0XqNgrIZyu/wN7aOqHkh7PY4fFP8TBWs+Qwuc1pWIiRfs0gRGFXn tTUUjOiGGkaMYc6NTD9E7AE2lWWHAJNpKrp0oq10w+nkqy3h52JXuDQsrvVgG61X h0VOgiBuxxoOotu/kOlLrWBu9HM1FpROZQtFWiAfmB3nLo/Rg6Ux+F0Xy9d6c9Hx IF/+BqLMahaG4LQGFKu6LxPPDeeJpA+H/2/YJHI90pawyw+nBQCDNa0Sa6/41bPL WmB1DW+HGBLWeB2gLVuxQST24OaCFISdwm7Q2zqhwM1mYCt/aBeXf5kU6JaaHdva uiDzpMeBxtLZUf0dn7X0By7T+Uc6sTN7X/ZOW4RR7N2mRwB7i9M7Ich32EAVXO+e JN433Cth8dipaFVyq9gv9u1gC/HF0Jwt2JsZ8oY2XuwmgmvPTGlvJtaB+HTFn0dg e9Kri4KgIndv4zpmPR7+CppyhiQVQQQCBXoCzg93M1x06LKk1WE0fu+zg8fdjuAR 2x1PYSDUXuAk3QWnAfnNi4LyAALKuY9K22W8lLCoBm0TnyTwA62D3m3pDtkhUmL6 laFE/UJAQ6+wdU90sRw3FCEVBROc1CgfKD5AAkxFD2/dEOr31Qg=
    =b2HD
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)