Summary: unhide redirectors
On Wed, Dec 01, 2021 at 09:11:17AM +0100, Yadd wrote:
Hi,....
after few discussions with some devscripts maintainers, we decided to build >> a new "version=5" format for debian/watch.
Principles:
* keep compatibility with versions 3 and 4, no need to change all
debian/watch files
* new version 5 format using the same syntax than other debian/* files
(rfc822 + "# comments")
* no option renaming (becomes case-insensitive to be compliant with
all formats)
* Version 5:
* Main (first) paragraph contains "Version: 5" and optional options
that change default values for source-paragraph
* URL and regex are separated
* Some default values change. For example, `dversionmangle` default
value will be "auto" (drop +dfsg, ~ds,...), uversionmangle=s/-/~/g,
filenamemangle=s/.*?(\d[\d\.]*@ARCHIVE_EXT@)/@PACKAGE@-$1/...
Example:
Version: 5
Of course, comments are welcome!
I think the move from v4 to v5 is an excellent opportunity
to express in the watch file that there is a dependency on a redirector.
Example
version=4
https://sf.net/<project>/ <tar-name>-(.+)\.tar\.gz debian uupdate
becomes something like
Version: 5
Source: https://qa.debian.org/watch/sourceforge/<project> <tar-name>-(.+)\.tar\.gz debian uupdate
And I think such change will allow removal of
bare
Disable all site specific special case code such as URL
redirector uses and page content alterations.
from the uscan code and uscan manual page (they are in /usr/bin/uscan )
The goal is to have documented that there are extra components being used. Avoiding nasty surprises.
Groeten
Geert Stappers
P.S.
Awareness of redirectors will get us more redirectors.
Those redirectors will help us to prevent that `uscan`
must get a javascript interpreter
Hi,
after few discussions with some devscripts maintainers, we decided to build
a new "version=5" format for debian/watch.
Principles:
* keep compatibility with versions 3 and 4, no need to change all
debian/watch files
* new version 5 format using the same syntax than other debian/* files
(rfc822 + "# comments")
* no option renaming (becomes case-insensitive to be compliant with
all formats)
* Version 5:
* Main (first) paragraph contains "Version: 5" and optional options
that change default values for source-paragraph
* URL and regex are separated
* Some default values change. For example, `dversionmangle` default
value will be "auto" (drop +dfsg, ~ds,...), uversionmangle=s/-/~/g,
filenamemangle=s/.*?(\d[\d\.]*@ARCHIVE_EXT@)/@PACKAGE@-$1/...
Example:
Version: 5
Of course, comments are welcome!
Summary: unhide redirectors
On Wed, Dec 01, 2021 at 09:11:17AM +0100, Yadd wrote:
after few discussions with some devscripts maintainers, we decided to build a new "version=5" format for debian/watch.
* URL and regex are separated
* Some default values change. For example, `dversionmangle` default
value will be "auto" (drop +dfsg, ~ds,...), uversionmangle=s/-/~/g,
filenamemangle=s/.*?(\d[\d\.]*@ARCHIVE_EXT@)/@PACKAGE@-$1/...
I think the move from v4 to v5 is an excellent opportunity[..]
to express in the watch file that there is a dependency on a redirector.
Version: 5
Source: https://qa.debian.org/watch/sourceforge/<project> <tar-name>-(.+)\.tar\.gz debian uupdate
And I think such change will allow removal of
bare
Disable all site specific special case code such as URL
redirector uses and page content alterations.
from the uscan code and uscan manual page (they are in /usr/bin/uscan )
The goal is to have documented that there are extra components being used. Avoiding nasty surprises.
Awareness of redirectors will get us more redirectors.
Those redirectors will help us to prevent that `uscan`
must get a javascript interpreter.
On Wed, Dec 01, 2021 at 12:39:41PM +0100, Geert Stappers wrote:
Summary: unhide redirectors
And not only.
On Wed, Dec 01, 2021 at 09:11:17AM +0100, Yadd wrote:
after few discussions with some devscripts maintainers, we decided to build >>> a new "version=5" format for debian/watch.
To be clear, this is a *very* non-ready proposal that we are getting
through the wider community. Nothing of this is implemented anywhere.
* URL and regex are separated
* Some default values change. For example, `dversionmangle` default
value will be "auto" (drop +dfsg, ~ds,...), uversionmangle=s/-/~/g, >>> filenamemangle=s/.*?(\d[\d\.]*@ARCHIVE_EXT@)/@PACKAGE@-$1/...
I honestly would like to add website-aware functionalities to uscan,
such as exactly this.
I think the move from v4 to v5 is an excellent opportunity[..]
to express in the watch file that there is a dependency on a redirector.
Version: 5
Source: https://qa.debian.org/watch/sourceforge/<project> <tar-name>-(.+)\.tar\.gz debian uupdate
I would like something like:
Source: qa-redirector
Source-Options:
name: sourceforge
project: <project>
Likewise, I would love if uscan could just learn how github, gitlab, launchpad, etc are made so prople won't have to bother with sticking
urls into watchfiles, such as:
Source: GitHub
Source-Options:
namespace: trendmicro
project: tlsh
match-on: tags|releases
To go either matching on https://github.com/trendmicro/tlsh/tags or https://github.com/trendmicro/tlsh/releases. using then Scheme (a name
that, tbh, I don't particularly like right now) for the tags or releases regex.
And I think such change will allow removal of
bare
Disable all site specific special case code such as URL
redirector uses and page content alterations.
from the uscan code and uscan manual page (they are in /usr/bin/uscan )
The goal is to have documented that there are extra components being used. >> Avoiding nasty surprises.
this feels like the opposite direction I'm proposing above :D
This is very useful for package with components. This is the only way toAwareness of redirectors will get us more redirectors.
Those redirectors will help us to prevent that `uscan`
must get a javascript interpreter.
Possibly, I'm indeed kind of unimpressed that we grew a parse for
nodejs' package.json and perl's META.json. Though I accepted it because
I saw some value, I'm totally in awe of universes where that is actually needed..
Quoting Yadd (2021-12-01 13:04:09)
On 01/12/2021 12:50, Mattia Rizzolo wrote:
Possibly, I'm indeed kind of unimpressed that we grew a parse forThis is very useful for package with components. This is the only way
nodejs' package.json and perl's META.json. Though I accepted it
because I saw some value, I'm totally in awe of universes where that
is actually needed..
to be able to `uscan --download-current-version`.
Speakin of components, it would be quite helpful if possible to handle versions of components - e.g. upgrade all components except
SomeComponent like this:
`uscan --download-current-version-SomeComponent`
On 01/12/2021 12:50, Mattia Rizzolo wrote:
Possibly, I'm indeed kind of unimpressed that we grew a parse forThis is very useful for package with components. This is the only way
nodejs' package.json and perl's META.json. Though I accepted it
because I saw some value, I'm totally in awe of universes where that
is actually needed..
to be able to `uscan --download-current-version`.
after few discussions with some devscripts maintainers, we decided to
build a new "version=5" format for debian/watch.
Principles:
* keep compatibility with versions 3 and 4, no need to change all
debian/watch files
* new version 5 format using the same syntax than other debian/* files
(rfc822 + "# comments")
Fix: will be
Version: 5
Source: https://qa.debian.org/watch/sourceforge/<project>
Regex: <tar-name>-(.+)\.tar\.gz
And I don't think "uupdate" is still useful.
Likewise, I would love if uscan could just learn how github, gitlab, launchpad, etc are made so prople won't have to bother with sticking
urls into watchfiles, such as:
Source: GitHub
Source-Options:
namespace: trendmicro
project: tlsh
match-on: tags|releases
Hi Yadd,
Thanks a lot for working on this. What you are proposing (ie: using a
mime thing, which is easy to parse instead of the dirty command-line
oriented thingy of version 3 and 4) feels much nicer than what we
currently have.
On 12/1/21 12:53 PM, Yadd wrote:
Fix: will be
Version: 5
Source: https://qa.debian.org/watch/sourceforge/<project>
Regex: <tar-name>-(.+)\.tar\.gz
That's much nicer than previous proposal!
And I don't think "uupdate" is still useful.
IMO, it is needed. That's what is nice with calling scripts: it can take
care with programming of things you didn't even think of. If you remove
it, the risk is that maintainers will continue to use version 3 or 4,
because they still need an update script.
How about:
Update-Script: uupdate
?
Really great!
And could the new uscan read a watch file from version 3/4/5 and output a version 5 of it by its own (in-place or stdout)?
uscan --standardize
:-)
after few discussions with some devscripts maintainers, we decided to
build a new "version=5" format for debian/watch.
Personally I dislike redirectors.
A redirector service is superior to including the redirector code
within uscan itself or within a debian/watch file, since when the
upstream website breaks the existing code, a service can be updated in
one place immediately, while uscan in Debian stable will be broken
until the next point release if it gets fixed at all and one in
debian/watch requires every package using the site to get updated.
On 02/12/2021 00:34, Paul Wise <pabs@debian.org> wrote:
On Wed, 2021-12-01 at 12:53 +0100, Yadd wrote:
Personally I dislike redirectors.
A redirector service is superior to including the redirector code
within uscan itself or within a debian/watch file, since when the
upstream website breaks the existing code, a service can be updated in
one place immediately, while uscan in Debian stable will be broken
until the next point release if it gets fixed at all and one in
debian/watch requires every package using the site to get updated.
Yes but the redirector often responded with 500 codes
I also wonder if it is time to split debian/watch out of Debian source packages, since upstream download locations generally change
independently of the Debian package and so information about upstream download locations probably should be maintained independently.
On 02/12/2021 10:16, Yadd wrote:
On 02/12/2021 00:34, Paul Wise <pabs@debian.org> wrote:
On Wed, 2021-12-01 at 12:53 +0100, Yadd wrote:
Personally I dislike redirectors.
A redirector service is superior to including the redirector code
within uscan itself or within a debian/watch file, since when the upstream website breaks the existing code, a service can be updated in one place immediately, while uscan in Debian stable will be broken
until the next point release if it gets fixed at all and one in debian/watch requires every package using the site to get updated.
Yes but the redirector often responded with 500 codes
Another idea to have a compromise:
* uscan is released with versioned schemes (GitHub.json, sf.json,...)
* when launched, it tries to download new version from a new Debian API
(static json files)
* if no response or no new version, uscan uses its own scheme or a
previously downloaded update (verifying signature)
* if a new version is available from new redirector:
* it verifies GPG signature of new scheme
* if not OK, it warns and uses cached scheme
* if OK, it stores it with signature in ~/.cache/uscan/schemes
Then:
* no more redirector with an heavy load, but just some JSON schemes
statically stored
* uscan still works if Debian website doesn't respond
What do you think about this idea?
Paul Wise <pabs@debian.org> writes:
I also wonder if it is time to split debian/watch out of Debian
source packages, since upstream download locations generally change independently of the Debian package and so information about
upstream download locations probably should be maintained
independently.
I very much agree. I don't understand the logic of tying upstream
checking to a particular version of a source package. The fact that
some version of a package once upon a time could locate and download
new upstream versions using a particular recipe is of no use if said
recipe becomes outdated at any later time.
It makes a lot more sense to provide this service in a way that allows
it to be modified/updated/improved/fixed with no regards to the actual packages that may use it. That could be as simple as a uscan service
with watch files that can be individually and independently modified.
A redirector service is superior to including the redirector code
within uscan itself or within a debian/watch file, since when the
upstream website breaks the existing code, a service can be updated in
one place immediately, while uscan in Debian stable will be broken
until the next point release if it gets fixed at all and one in
debian/watch requires every package using the site to get updated.
Yes but the redirector often responded with 500 codes
Another idea to have a compromise:
* uscan is released with versioned schemes (GitHub.json, sf.json,...)
* when launched, it tries to download new version from a new Debian API
(static json files)
* if no response or no new version, uscan uses its own scheme or a
previously downloaded update (verifying signature)
* if a new version is available from new redirector:
* it verifies GPG signature of new scheme
* if not OK, it warns and uses cached scheme
* if OK, it stores it with signature in ~/.cache/uscan/schemes
Then:
* no more redirector with an heavy load, but just some JSON schemes
statically stored
* uscan still works if Debian website doesn't respond
* GPG permits to be sure that scheme isn't corrupted (released files
are as protected as uscan itself because owned by root)
* easy update if upstream store changes its behavior: just to update
one JSON file
What do you think about this idea? Which GPG keys will be accepted?
Quoting Gard Spreemann (2021-12-02 12:31:30)
Paul Wise <pabs@debian.org> writes:
I also wonder if it is time to split debian/watch out of Debian
source packages, since upstream download locations generally change
independently of the Debian package and so information about
upstream download locations probably should be maintained
independently.
I very much agree. I don't understand the logic of tying upstream
checking to a particular version of a source package. The fact that
some version of a package once upon a time could locate and download
new upstream versions using a particular recipe is of no use if said
recipe becomes outdated at any later time.
It makes a lot more sense to provide this service in a way that allows
it to be modified/updated/improved/fixed with no regards to the actual
packages that may use it. That could be as simple as a uscan service
with watch files that can be individually and independently modified.
I find it helpful for our packages to include information about where
and how (at the time of its release) that package was monitoring for upstream releases. It helps working decentralized - both for preparing packages for Debian and for working on Debian-derived packages, both
without needing access to somewhere central for this "watch"
information.
Therefore I like the proposal to have Debian project scanners
aggressively look for _newest_ watch file for a packaging project,
including looking up declared Vcs-* hints for not-yet-released work.
Jonas Smedegaard <jonas@jones.dk> writes:
Quoting Gard Spreemann (2021-12-02 12:31:30)
Paul Wise <pabs@debian.org> writes:
I also wonder if it is time to split debian/watch out of Debian
source packages, since upstream download locations generally
change independently of the Debian package and so information
about upstream download locations probably should be maintained
independently.
I very much agree. I don't understand the logic of tying upstream
checking to a particular version of a source package. The fact that
some version of a package once upon a time could locate and
download new upstream versions using a particular recipe is of no
use if said recipe becomes outdated at any later time.
It makes a lot more sense to provide this service in a way that
allows it to be modified/updated/improved/fixed with no regards to
the actual packages that may use it. That could be as simple as a
uscan service with watch files that can be individually and
independently modified.
I find it helpful for our packages to include information about
where and how (at the time of its release) that package was
monitoring for upstream releases. It helps working decentralized -
both for preparing packages for Debian and for working on
Debian-derived packages, both without needing access to somewhere
central for this "watch" information.
Would it make sense for a package to contain a snapshot of the
relevant metadata in the hypothetical "centralized-and-often-updated
watch service" at the time in enters into the archives?
Therefore I like the proposal to have Debian project scanners
aggressively look for _newest_ watch file for a packaging project, including looking up declared Vcs-* hints for not-yet-released work.
Indeed, that sounds like a better idea than what I suggest above!
Alternatively, perhaps we could workaround outdated debian/watch files
by having vcswatch extract debian/watch files from the repo specified
in the Vcs-* URLs.
Another idea to have a compromise:
* uscan is released with versioned schemes (GitHub.json, sf.json,...)
* when launched, it tries to download new version from a new Debian API
(static json files)
* if no response or no new version, uscan uses its own scheme or a
previously downloaded update (verifying signature)
* if a new version is available from new redirector:
* it verifies GPG signature of new scheme
* if not OK, it warns and uses cached scheme
* if OK, it stores it with signature in ~/.cache/uscan/schemes
Quoting Gard Spreemann (2021-12-02 13:09:17)
Jonas Smedegaard <jonas@jones.dk> writes:
Quoting Gard Spreemann (2021-12-02 12:31:30)
Paul Wise <pabs@debian.org> writes:
I also wonder if it is time to split debian/watch out of Debian
source packages, since upstream download locations generally
change independently of the Debian package and so information
about upstream download locations probably should be maintained
independently.
I very much agree. I don't understand the logic of tying upstream
checking to a particular version of a source package. The fact that
some version of a package once upon a time could locate and
download new upstream versions using a particular recipe is of no
use if said recipe becomes outdated at any later time.
It makes a lot more sense to provide this service in a way that
allows it to be modified/updated/improved/fixed with no regards to
the actual packages that may use it. That could be as simple as a
uscan service with watch files that can be individually and
independently modified.
I find it helpful for our packages to include information about
where and how (at the time of its release) that package was
monitoring for upstream releases. It helps working decentralized -
both for preparing packages for Debian and for working on
Debian-derived packages, both without needing access to somewhere
central for this "watch" information.
Would it make sense for a package to contain a snapshot of the
relevant metadata in the hypothetical "centralized-and-often-updated
watch service" at the time in enters into the archives?
Not _instead_ of current location: What I find helpful is to have the
watch file available with the source package. I am unaware if there
would be some benefit of _additionally_ embedding the watch file in
binary packages (if that's what you meant).
Therefore I like the proposal to have Debian project scanners
aggressively look for _newest_ watch file for a packaging project,
including looking up declared Vcs-* hints for not-yet-released work.
Indeed, that sounds like a better idea than what I suggest above!
Not sure if you noticed, but (since I won't steal credit) I basically emphasized Pabs' suggestion in last paragraph of what you previously
quoted:
Quoting Paul Wise (2021-12-02 00:47:44)
Alternatively, perhaps we could workaround outdated debian/watch files
by having vcswatch extract debian/watch files from the repo specified
in the Vcs-* URLs.
It might be a idea to look at how other distributions do checking for
new upstream releases and adopt some of their improvements.
I note Fedora uses a service (that isn't Fedora specific) for this:
https://release-monitoring.org https://docs.fedoraproject.org/en-US/package-maintainers/Upstream_Release_Monitoring/
Another idea would be to use the Repology service to notice when other distros include a newer version of a package than Debian does.
https://repology.org/
I also wonder if it is time to split debian/watch out of Debian source packages, since upstream download locations generally change
independently of the Debian package and so information about upstream download locations probably should be maintained independently.
At minimum we would need a way to map from release-monitoring.org
package names to Debian source package names. Assuming they use Fedora
source package names, then the Repology service provides such a mapping
and we could presumably could get a periodic export of that.
I see using Repology as a complement to release-monitoring.org and
uscan, not as an alternative to them. It enables use-cases that aren't possible with the other two. We automatically get version monitoring
for packages that don't have other version monitoring mechanisms. We
get monitoring of whether or not a particular package needs updating to
a VCS snapshot instead of waiting for an official release. We get
monitoring of versions even when upstream has moved to a different
location not monitored by other mechanisms. There are probably other use-cases I can't think of right now.
Yes that makes sense, what I wonder is how much change is needed for putting watch files in a separate repo compared to going with release-monitoring.org (I don't know enough about the inner workings
of our tools to answer this question).
For the VCS idea it would be minimal, just vcswatch needs to also pull debian/watch files out of VCS repos with commits not yet pushed to
Debian and then the version checking infra (zero idea where that went)
needs to pay attention to that data.
For fully moving debian/watch (and Homepage) out of Debian source
packages there would need to be a lot more work, probably migrating to release-monitoring.org would be the way to go.
I think this would be the best path forward - it would probably be not
easy given that it changes entirely how the current system works, but
it might be well worth the effort. Working together with another
distribution would share the work for the distro. I'm sure if we are
willing to join them they would accommodate us if there are any
changes we would require (e.g. login via salsa instead of a fedora
account).
At minimum we would need a way to map from release-monitoring.org
package names to Debian source package names. Assuming they use Fedora
source package names, then the Repology service provides such a mapping
and we could presumably could get a periodic export of that.
Yes but the redirector often responded with 500 codes
I think this would be the best path forward - it would probably be not
easy given that it changes entirely how the current system works, but
it might be well worth the effort. Working together with another
distribution would share the work for the distro. I'm sure if we are
willing to join them they would accommodate us if there are any
changes we would require (e.g. login via salsa instead of a fedora
account).
This however I think is not a good idea. Repology is very nice to
check what versions other distros have, but for projects that don't
have any external language-specific package repository like e.g.
python, it would mean that we could easily miss a new release (think
small projects written in C that are not in any other distro) and
wrongly formatted version from other distros would impact us
(unlikely, but still bad in theory).
And since it requires the same infrastructure changes as going with release-monitoring.org, it would be better to just use that.
Yes that makes sense, what I wonder is how much change is needed for
putting watch files in a separate repo compared to going with release-monitoring.org (I don't know enough about the inner workings
of our tools to answer this question).
If I understand correctly, release-monitoring already offers such a
mapping [1].
Hm, I can't really think of an example where such a thing couldn't
also be implemented in release-monitoring.org.
Just one quick idea I had: what about a "fake" uscan backend? I.e.
something like `Version: release-monitoring.org` in d/watch. In that
case uscan will launch an external program that fetches the data from
there and gives it back to uscan, so that other tools stay unaffected
until a full transition is done.
On Thu, 2021-12-02 at 23:36 +0100, Stephan Lachnit wrote:
If I understand correctly, release-monitoring already offers such a
mapping [1].
It seems like the Ayanita distro mapping needs to be done manually once
per package, while using the Repology data would automatically get us
the mapping for each existing package and all future packages.
Hm, I can't really think of an example where such a thing couldn't
also be implemented in release-monitoring.org.
None of the three use-cases I listed can be done by it AFAICT.
It can't check things that it doesn't have a check for, while
individual package maintainers in various distros will update their
packages and Repology will notice the new versions.
It presumably doesn't look at the versions for all distros, so it can't
do the cross-distro VCS snapshot choice check, while individual package maintainers in various distros know their packages well and might
upgrade to a VCS snapshot in their distro, which Repology notices.
It also isn't going to check locations it doesn't check yet, while
individual package maintainers in other distros might do that after
noticing their package hasn't been updated recently and then going
searching for a new upstream and updating, which Repology notices.
Just one quick idea I had: what about a "fake" uscan backend? I.e. something like `Version: release-monitoring.org` in d/watch. In that
case uscan will launch an external program that fetches the data from
there and gives it back to uscan, so that other tools stay unaffected
until a full transition is done.
Excellent idea, that would be great to have.
The one issue I can think of with using release-monitoring.org is that
Debian becomes more reliant on an external service, while currently we
are completely independent of other distros for version checking.
Converting the release-monitoring.org check to a watch file might be an alternative to using it directly that maintains our independence.
On Thu, Dec 2, 2021 at 11:52 PM Paul Wise <pabs@debian.org> wrote:
On Thu, 2021-12-02 at 23:36 +0100, Stephan Lachnit wrote:
If I understand correctly, release-monitoring already offers such a
mapping [1].
It seems like the Ayanita distro mapping needs to be done manually once
per package, while using the Repology data would automatically get us
the mapping for each existing package and all future packages.
I mean it looks rather easy to do, just a couple of mouse clicks.
Compare that to writing a watch file at the moment (assuming one has
to do more than copy and paste the github example).
Hm, I can't really think of an example where such a thing couldn't
also be implemented in release-monitoring.org.
None of the three use-cases I listed can be done by it AFAICT.
It can't check things that it doesn't have a check for, while
individual package maintainers in various distros will update their
packages and Repology will notice the new versions.
Then the maintainer would just have to write a check, just like they
have to do now.
Also, mapping on Repology sometimes needs to be adjusted manually. And >sometimes they disagree and instead tell you to rename the source
package in the distro (happened to me once), which is not really
viable in Debian.
It presumably doesn't look at the versions for all distros, so it can't
do the cross-distro VCS snapshot choice check, while individual package
maintainers in various distros know their packages well and might
upgrade to a VCS snapshot in their distro, which Repology notices.
Yes it can't, but also I don't think this is something *release
monitoring* should do. It is definitely a good use case and that is
why there is a link to repology on the tracker (called "other
distros"), but it has IMHO nothing to do with *automatic* release
monitoring. Don't get me wrong, I actually like repology exactly for
this particular reason.
It also isn't going to check locations it doesn't check yet, while
individual package maintainers in other distros might do that after
noticing their package hasn't been updated recently and then going
searching for a new upstream and updating, which Repology notices.
Fair point, but if we would work together on release-monitoring.org
with Fedora, there are more eyes on it as well as in the current
situation.
Repology still has more eyes of course, but then again the link to
Repology is right there on the tracker already if one is curious.
Just one quick idea I had: what about a "fake" uscan backend? I.e.
something like `Version: release-monitoring.org` in d/watch. In that
case uscan will launch an external program that fetches the data from
there and gives it back to uscan, so that other tools stay unaffected
until a full transition is done.
Excellent idea, that would be great to have.
One more thought on this. If we go with version 5, maybe something like:
Version: 5
Source: release-monitoring.org
Would also work for multiple sources then and in general would fit
nicely to the current idea for v5. It also solves the problem with the >tooling, watch files and uscan would still exist, but the "searching"
portion is offloaded.
The one issue I can think of with using release-monitoring.org is that
Debian becomes more reliant on an external service, while currently we
are completely independent of other distros for version checking.
Converting the release-monitoring.org check to a watch file might be an
alternative to using it directly that maintains our independence.
Hm right, independence is a valid concern. Anitya itself is open
source [1] so we could host it easily, but of course the real problem
would be the stored data of the projects. I don't know if they are
hosted somewhere, but I'm sure the Fedora guys would be open to share
them with us, so that we could easily spin up a mirror in case there
are any problems (it's probably a good idea to host a read-only mirror
just in case).
This sounds more reasonable to me than writing a tool that converts a
new standard to the old one just as backup.
I mean it looks rather easy to do, just a couple of mouse clicks.
Compare that to writing a watch file at the moment (assuming one has
to do more than copy and paste the github example).
Then the maintainer would just have to write a check, just like they
have to do now.
Also, mapping on Repology sometimes needs to be adjusted manually. And sometimes they disagree and instead tell you to rename the source
package in the distro (happened to me once), which is not really
viable in Debian.
Yes it can't, but also I don't think this is something *release
monitoring* should do. It is definitely a good use case and that is
why there is a link to repology on the tracker (called "other
distros"), but it has IMHO nothing to do with *automatic* release
monitoring. Don't get me wrong, I actually like repology exactly for
this particular reason.
Fair point, but if we would work together on release-monitoring.org
with Fedora, there are more eyes on it as well as in the current
situation.
Repology still has more eyes of course, but then again the link to
Repology is right there on the tracker already if one is curious.
One more thought on this. If we go with version 5, maybe something
like:
Version: 5
Source: release-monitoring.org
Would also work for multiple sources then and in general would fit
nicely to the current idea for v5. It also solves the problem with the tooling, watch files and uscan would still exist, but the "searching"
portion is offloaded.
Hm right, independence is a valid concern. Anitya itself is open
source [1] so we could host it easily, but of course the real problem
would be the stored data of the projects. I don't know if they are
hosted somewhere, but I'm sure the Fedora guys would be open to share
them with us, so that we could easily spin up a mirror in case there
are any problems (it's probably a good idea to host a read-only mirror
just in case).
This sounds more reasonable to me than writing a tool that converts a
new standard to the old one just as backup.
The other issue with using Anitya is that Debian and Fedora have
different policies and culture for choosing which upstream versions to
update to. Debian strongly prefers LTS versions while Fedora are all
about the latest and greatest, which is a bit of a culture clash and is likely to mean for some packages we couldn't use Anitya.
On 01/12/2021 21:07, Patrice wrote:
Really great!
And could the new uscan read a watch file from version 3/4/5 and output a
version 5 of it by its own (in-place or stdout)?
uscan --standardize
:-)
Yes but without optimization neither scheme (except some few fields). Example:
version=4
opts=uversionmangle=s/-/~/g,pgpmode=none \
https://... .*(\d[\d\.]*)@ARCHIVE_EXT@
will be translated into:
Version: 5
Source: https://...
Regex: .*(\d[\d\.]*)@ARCHIVE_EXT@
Uversionmangle: s/-/~/g
You'll have to manually modify it into
Version: 5
Scheme: stable
Source: https://...
I think that there's a security consideration associated with all these proposals for externalizing finding upstream updates.
If one of these services were ever compromised it would provide a
vector for offering substitute upstream code (at least for the cases
where upstream releases aren't both signed by upstream and verified in Debian). I find that prospect concerning.
Currently watch files and at least the redirectors I know of all run
on Debian infrastructure or on the systems of the Debian person doing
the update.
On Sat, 2021-12-04 at 02:43 +0000, Scott Kitterman wrote:
I think that there's a security consideration associated with all these
proposals for externalizing finding upstream updates.
Good point.
If one of these services were ever compromised it would provide a
vector for offering substitute upstream code (at least for the cases
where upstream releases aren't both signed by upstream and verified in
Debian). I find that prospect concerning.
I think the same concern should also apply to centralised upstream >development infrastructure like GitHub and also individual upstream >developers themselves. There isn't really any mitigation for malicious
code being pushed out beyond commit/release signing (both unpopular)
and (distributed) downstream code review.
To mitigate the concern for upstream version monitoring we could prefer >debian/watch when it exists but fall back to release-monitoring.org
when one doesn't exist, have a tool to convert the Ayanita format into >debian/watch and have dh_make and similar try to create an initial >debian/watch by default.
We need a culture of doing change review before updating to new
upstream releases, but often that isn't necessarily feasible,
especially for large projects with rapid change or when switching to
new forks of existing tools.
Currently watch files and at least the redirectors I know of all run
on Debian infrastructure or on the systems of the Debian person doing
the update.
Some run on debian.org servers, and many run on debian.net domains.
However I don't think that that makes them immune to compromise.
* Version 5:I have a feature request regarding signature verification.
* Main (first) paragraph contains "Version: 5" and optional options
that change default values for source-paragraph
* URL and regex are separated
* Some default values change. For example, `dversionmangle` default
value will be "auto" (drop +dfsg, ~ds,...), uversionmangle=s/-/~/g, filenamemangle=s/.*?(\d[\d\.]*@ARCHIVE_EXT@)/@PACKAGE@-$1/...
[...]
Of course, comments are welcome!
Hi Yadd,
thank you very much for your work on uscan. That new version 5
format looks really promising.
* Yadd <yadd@debian.org> [2021-12-01 09:11]:
* Version 5:I have a feature request regarding signature verification. As luck would
* Main (first) paragraph contains "Version: 5" and optional options
that change default values for source-paragraph
* URL and regex are separated
* Some default values change. For example, `dversionmangle` default
value will be "auto" (drop +dfsg, ~ds,...), uversionmangle=s/-/~/g, >> filenamemangle=s/.*?(\d[\d\.]*@ARCHIVE_EXT@)/@PACKAGE@-$1/...
[...]
Of course, comments are welcome!
have it, I maintain three packages with upstream
signatures; one of them is me being my own upstream, and the other
two do not use the "standard" approach with one GnuPG signature per
source tarball:
- cmake releases its sources in multiple archive formats and signs
them indirectly (a text file with SHA256 hashes) [1].
- liblzf uses the BSD signify tool [2] and only GnuPG-signs the
signify key.
I don't know if any of these schemes are used elsewhere (more likely
for the CMake approach, less likely for liblzf, I'd guess), but it
would be nice if uscan offered some support for this; maybe a hook
to run the signature verification by an external script with
autopkgtest semantics (fail if output occurs on stderr the script
returns with a non-zero exit code).
The one issue I can think of with using release-monitoring.org is that
Debian becomes more reliant on an external service, while currently we
are completely independent of other distros for version checking.
Converting the release-monitoring.org check to a watch file might be an alternative to using it directly that maintains our independence.
I think that there's a security consideration associated with all these proposals for externalizing finding upstream updates. Currently watch files and at least the redirectors I know of all run on Debian infrastructure or on the systems of the Debian person doing the update.
If one of these services were ever compromised it would provide a
vector for offering substitute upstream code (at least for the cases
where upstream releases aren't both signed by upstream and verified in Debian). I find that prospect concerning.
Repology gets you mappings for all the source packages in Debian in one download (assuming it has an export of the mappings, that may need to
be added), while the Anitya mapping requires a human to manually add a mapping for each of the thousands of source packages in Debian. Not all maintainers are going to bother and repetitive clicking is going to get boring for the folks trying to make up for that.
Also, mapping on Repology sometimes needs to be adjusted manually. And sometimes they disagree and instead tell you to rename the source
package in the distro (happened to me once), which is not really
viable in Debian.
I wasn't aware of the renaming part, seems kind of weird.
Yes it can't, but also I don't think this is something *release
monitoring* should do. It is definitely a good use case and that is
why there is a link to repology on the tracker (called "other
distros"), but it has IMHO nothing to do with *automatic* release monitoring. Don't get me wrong, I actually like repology exactly for
this particular reason.
I was taking the thread topic to be the slightly more general area of "monitoring when a package needs updating to a new upstream release,
snapshot or fork". New VCS snapshots in other distros fits that IMO.
The other issue with using Anitya is that Debian and Fedora have
different policies and culture for choosing which upstream versions to
update to. Debian strongly prefers LTS versions while Fedora are all
about the latest and greatest, which is a bit of a culture clash and is likely to mean for some packages we couldn't use Anitya.
In addition to independence there is the issue Jonas mentioned
elsewhere in the initial uscan thread that some Debian people prefer
the info to be maintained in the source package instead of elsewhere.
This sounds more reasonable to me than writing a tool that converts a
new standard to the old one just as backup.
Given the above, perhaps a way to sync a locally stored file and the
Anitya one, and then have uscan understand the Anitya format?
I think that there's a security consideration associated with all these proposals for externalizing finding upstream updates. Currently watch files and at least the redirectors I know of all run on Debian infrastructure or on the systems of the Debianperson doing the update.
If one of these services were ever compromised it would provide a vector for offering substitute upstream code (at least for the cases where upstream releases aren't both signed by upstream and verified in Debian). I find that prospect concerning.
| Sysop: | Keyop |
|---|---|
| Location: | Huddersfield, West Yorkshire, UK |
| Users: | 296 |
| Nodes: | 16 (2 / 14) |
| Uptime: | 47:02:49 |
| Calls: | 6,648 |
| Files: | 12,198 |
| Messages: | 5,329,919 |