On Thu, Apr 09, 2020 at 10:44:18AM +0200, Paul Gevers wrote:
Antonio, what do you think? If we expose the blacklist, I can also have britney consume it.
I think it could be exposed, yes. I'm currrently working on optimizing
the generation of all of that data, so I added to my TODO list and will
sneak that in.
Hi Antonio,
greetings to Curitiba. ;-)
Hope you are fine!
On Thu, Apr 09, 2020 at 09:37:41AM -0300, Antonio Terceiro wrote:
On Thu, Apr 09, 2020 at 10:44:18AM +0200, Paul Gevers wrote:
Antonio, what do you think? If we expose the blacklist, I can also have britney consume it.
I think it could be exposed, yes. I'm currrently working on optimizing
the generation of all of that data, so I added to my TODO list and will sneak that in.
Do you expect new important fields in the json file or do you intend to change its structure in principle?
May be you consider some fields as really restricted to some
special applications and nobody would ever consider querying
UDD for it?
Yes, I wouldn't add blame (I don't think we add those anymore, it's
legacy, I only found two occurrences on amd64), created_at and
updated_at. I also don't think that worker and message are very useful,
but one never knows. duration_seconds and duration_human feel double and
the former is leaner for in a database.
valid_keys = ( 'run_id',I expected you to mostly see "" or "migration-reference/0" here, with
# 'created_at', # Paul Gevers: should be ignored
# 'updated_at', # Paul Gevers: should be ignored
'suite',
'arch',
'package', # ----> should be renamed to 'source'
'version',
'trigger', # usually package.*version
'status',
'requestor', # 'britney', 'debci' or e-mail
'pin_packages', # []
# 'worker', # Paul Gevers: should be ignored (is 'null' anyway)
'date',
'duration_seconds',
'last_pass_date',
'last_pass_version',
'message', # see below
'previous_status',
# 'duration_human', # Paul Gevers: duration_seconds and duration_human feel double and the former is leaner for in a database
# 'blame', # Paul Gevers: should be ignored
)
# message can be
# $ grep '"message"' packages*.json | sed 's/^.*\.json: *//' | sort | uniq # "message": "All tests passed" -> "status": "pass"
# "message": "Could not run tests due to a temporary testbed failure" -> "status": "tmpfail"
# "message": "elbrus" -> "status": "tmpfail"
# "message": "Erroneous package" -> "status": "fail"
# "message": null -> "status": "fail"
# "message": "No tests in this package or all skipped" -> "status": "neutral"
# "message": "Tests failed", -> "status": "fail"
# "message": "Tests failed, and at least one test skipped" -> "status": "fail"
# "message": "Tests passed, but at least one test skipped" -> "status": "pass"
# "message": "Unexpected autopkgtest exit code 20" -> "status": "tmpfail"
I agree that leaving out worker which is really always null makes sense
but I tend to leave message since leaving this out looks like loosing information. I tried to find a relation to status but it seems the same status can result in different messages. I think just a field in addition will not blow up UDD way more than it recently is - may be I consider
a normalised form, but usually UDD is not very normalised at all.
Hi Andreas,
On 09-04-2020 22:53, Andreas Tille wrote:
valid_keys = ( 'run_id',I expected you to mostly see "" or "migration-reference/0" here, with
# 'created_at', # Paul Gevers: should be ignored
# 'updated_at', # Paul Gevers: should be ignored
'suite',
'arch',
'package', # ----> should be renamed to 'source'
'version',
'trigger', # usually package.*version
some hand crafted text from random DD's.
'status',
'requestor', # 'britney', 'debci' or e-mail
Debian login to be precise, not e-mail.
'pin_packages', # []
Since a couple of months this json and other pages only show "pure"
suite runs, to pin_packages is always empty. pin_packages contains which packages are taken from another suite than the base suite.
# 'worker', # Paul Gevers: should be ignored (is 'null' anyway)
Oh, bug somewhere I guess.
'date',
'duration_seconds',
'last_pass_date',
'last_pass_version',
'message', # see below
'previous_status',
# 'duration_human', # Paul Gevers: duration_seconds and duration_human feel double and the former is leaner for in a database
# 'blame', # Paul Gevers: should be ignored
)
# message can be
# $ grep '"message"' packages*.json | sed 's/^.*\.json: *//' | sort | uniq
# "message": "All tests passed" -> "status": "pass"
# "message": "Could not run tests due to a temporary testbed failure" -> "status": "tmpfail"
# "message": "elbrus" -> "status": "tmpfail"
# "message": "Erroneous package" -> "status": "fail"
# "message": null -> "status": "fail"
# "message": "No tests in this package or all skipped" -> "status": "neutral"
# "message": "Tests failed", -> "status": "fail"
# "message": "Tests failed, and at least one test skipped" -> "status": "fail"
# "message": "Tests passed, but at least one test skipped" -> "status": "pass"
# "message": "Unexpected autopkgtest exit code 20" -> "status": "tmpfail"
I agree that leaving out worker which is really always null makes sense
but I tend to leave message since leaving this out looks like loosing information. I tried to find a relation to status but it seems the same status can result in different messages. I think just a field in addition will not blow up UDD way more than it recently is - may be I consider
a normalised form, but usually UDD is not very normalised at all.
In general this is the final output from autopkgtest. But, as you see my
name there, I had to clean up several times and to be able to find those back, I added my nick to the message. The list thus may change over time.
Paul
Hi Paul,
thanks for the clarification. This commit
https://salsa.debian.org/qa/udd/-/commit/6a874a89365671dd37a14a9bca25290dc55a1fc9
imports the current data. I will tests this a bit more and than activate
in a cron job as importer.
Thanks a lot for your contribution
Andreas.
On Fri, Apr 10, 2020 at 09:05:31PM +0200, Paul Gevers wrote:
Hi Andreas,
On 09-04-2020 22:53, Andreas Tille wrote:
valid_keys = ( 'run_id',I expected you to mostly see "" or "migration-reference/0" here, with
# 'created_at', # Paul Gevers: should be ignored
# 'updated_at', # Paul Gevers: should be ignored
'suite',
'arch',
'package', # ----> should be renamed to 'source'
'version',
'trigger', # usually package.*version
some hand crafted text from random DD's.
'status',
'requestor', # 'britney', 'debci' or e-mail
Debian login to be precise, not e-mail.
'pin_packages', # []
Since a couple of months this json and other pages only show "pure"
suite runs, to pin_packages is always empty. pin_packages contains which packages are taken from another suite than the base suite.
# 'worker', # Paul Gevers: should be ignored (is 'null' anyway)
Oh, bug somewhere I guess.
'date',
'duration_seconds',
'last_pass_date',
'last_pass_version',
'message', # see below
'previous_status',
# 'duration_human', # Paul Gevers: duration_seconds and duration_human feel double and the former is leaner for in a database
# 'blame', # Paul Gevers: should be ignored
)
# message can be
# $ grep '"message"' packages*.json | sed 's/^.*\.json: *//' | sort | uniq
# "message": "All tests passed" -> "status": "pass"
# "message": "Could not run tests due to a temporary testbed failure" -> "status": "tmpfail"
# "message": "elbrus" -> "status": "tmpfail"
# "message": "Erroneous package" -> "status": "fail"
# "message": null -> "status": "fail"
# "message": "No tests in this package or all skipped" -> "status": "neutral"
# "message": "Tests failed", -> "status": "fail"
# "message": "Tests failed, and at least one test skipped" -> "status": "fail"
# "message": "Tests passed, but at least one test skipped" -> "status": "pass"
# "message": "Unexpected autopkgtest exit code 20" -> "status": "tmpfail"
I agree that leaving out worker which is really always null makes sense but I tend to leave message since leaving this out looks like loosing information. I tried to find a relation to status but it seems the same status can result in different messages. I think just a field in addition
will not blow up UDD way more than it recently is - may be I consider
a normalised form, but usually UDD is not very normalised at all.
In general this is the final output from autopkgtest. But, as you see my name there, I had to clean up several times and to be able to find those back, I added my nick to the message. The list thus may change over time.
Paul
--
http://fam-tille.de
I'm sorry if I haven't paid enough attention. But what is the difference
with the 'ci' importer?
https://salsa.debian.org/qa/udd/-/blob/master/rimporters/ci.rb
On 11/04/20 at 07:12 +0200, Andreas Tille wrote:
Hi Paul,
thanks for the clarification. This commit
https://salsa.debian.org/qa/udd/-/commit/6a874a89365671dd37a14a9bca25290dc55a1fc9
imports the current data. I will tests this a bit more and than activate in a cron job as importer.
Thanks a lot for your contribution
Andreas.
On Fri, Apr 10, 2020 at 09:05:31PM +0200, Paul Gevers wrote:
Hi Andreas,
On 09-04-2020 22:53, Andreas Tille wrote:
valid_keys = ( 'run_id',I expected you to mostly see "" or "migration-reference/0" here, with some hand crafted text from random DD's.
# 'created_at', # Paul Gevers: should be ignored
# 'updated_at', # Paul Gevers: should be ignored
'suite',
'arch',
'package', # ----> should be renamed to 'source'
'version',
'trigger', # usually package.*version
'status',
'requestor', # 'britney', 'debci' or e-mail
Debian login to be precise, not e-mail.
'pin_packages', # []
Since a couple of months this json and other pages only show "pure"
suite runs, to pin_packages is always empty. pin_packages contains which packages are taken from another suite than the base suite.
# 'worker', # Paul Gevers: should be ignored (is 'null' anyway)
Oh, bug somewhere I guess.
'date',
'duration_seconds',
'last_pass_date',
'last_pass_version',
'message', # see below
'previous_status',
# 'duration_human', # Paul Gevers: duration_seconds and duration_human feel double and the former is leaner for in a database
# 'blame', # Paul Gevers: should be ignored
)
# message can be
# $ grep '"message"' packages*.json | sed 's/^.*\.json: *//' | sort | uniq
# "message": "All tests passed" -> "status": "pass"
# "message": "Could not run tests due to a temporary testbed failure" -> "status": "tmpfail"
# "message": "elbrus" -> "status": "tmpfail"
# "message": "Erroneous package" -> "status": "fail"
# "message": null -> "status": "fail"
# "message": "No tests in this package or all skipped" -> "status": "neutral"
# "message": "Tests failed", -> "status": "fail"
# "message": "Tests failed, and at least one test skipped" -> "status": "fail"
# "message": "Tests passed, but at least one test skipped" -> "status": "pass"
# "message": "Unexpected autopkgtest exit code 20" -> "status": "tmpfail"
I agree that leaving out worker which is really always null makes sense but I tend to leave message since leaving this out looks like loosing information. I tried to find a relation to status but it seems the same
status can result in different messages. I think just a field in addition
will not blow up UDD way more than it recently is - may be I consider
a normalised form, but usually UDD is not very normalised at all.
In general this is the final output from autopkgtest. But, as you see my name there, I had to clean up several times and to be able to find those back, I added my nick to the message. The list thus may change over time.
Paul
--
http://fam-tille.de
Hi Andreas,
On 15-04-2020 22:45, Andreas Tille wrote:
autodep8-python3 PASS (superficial)
superficial is translated to neutral. As is FAIL (flaky).
autodep8-python3 PASS (superficial)
Hmmmm, what exactly means "superficial".
Are all those
Testsuite: autopkgtest-pkg-*
superficial?
Do they qualify for early testing migration or not?
superficial is translated to neutral. As is FAIL (flaky).
Hmmmm, what exactly means "superficial".
Are all those
Testsuite: autopkgtest-pkg-*
superficial?
Do they qualify for early testing migration or not?
Wouldn't it be more informative to have a fourth category
pass
superficial
neutral
fail
My intention was to have a list with packages of our team where either a
test is missing or failing. My idea was that some autopkgtest-pkg-* is
"test is not missing". What is your opinion as debian-ci team about my
idea?
Hmmmm, what exactly means "superficial".
Please read the documentation: https://salsa.debian.org/ci-team/autopkgtest/raw/master/doc/README.package-tests.rst
Are all those
Testsuite: autopkgtest-pkg-*
superficial?
No. Only those that only have superficial tests. E.g. ruby runs the full upstream test-suite automatically.
Do they qualify for early testing migration or not?
Superficial tests *are* neutral, so no.
Wouldn't it be more informative to have a fourth category
pass
superficial
neutral
fail
No, because there are only three states. I.e. superficial and flaky and skipped tests all end up meaning the same.
My intention was to have a list with packages of our team where either a test is missing or failing. My idea was that some autopkgtest-pkg-* is "test is not missing". What is your opinion as debian-ci team about my idea?
It seems you want to be processing the message field then, but honestly
if there is an entry, "test is not missing". If there is no entry, "test
is missing". Failing is "fail". flaky tests are also "test is not
missing", skipped tests are also "test is not missing". Why would you
need all those states? You have the "message" field in your UDD schema
to check why you get the neutral state.
Are all those
Testsuite: autopkgtest-pkg-*
superficial?
Most of them yes, because if they fail, that's Very Bad, but if they
succeed, they don't give us a whole lot of confidence that the package
works as intended.
For example, if you have a package python3-dbus, the most that can be
done with knowledge of Python but no knowledge of dbus is to "import dbus" and see what happens. If that fails, obviously the dbus module is
unusable (or missing a dependency or something); but if that succeeds,
that fact doesn't tell us whether it can actually do D-Bus successfully, which is its real purpose. If I replaced all the code in python3-dbus
with print("hello"), obviously it wouldn't be implementing its intended
API any more, but it would still pass the "import" test. That's why the "import" test is considered to be superficial.
If a family of packages have a convention for how to discover and
run a real test suite with non-trivial coverage (for example GNOME-style installed-tests in /usr/share/installed-tests/**/*.test), then an autopkgtest-* helper that detected and used *that* convention would usually *not* be superficial.
Hi Lucas,
On Mon, Apr 13, 2020 at 10:37:57PM +0200, Lucas Nussbaum wrote:
I'm sorry if I haven't paid enough attention. But what is the difference with the 'ci' importer?
https://salsa.debian.org/qa/udd/-/blob/master/rimporters/ci.rb
I think the problem is that UDD is not documented and I simply did not
know about the ci table. :-(
However, when looking at it there is a difference:
udd=# select status, arch, count(*) from ci group by status, arch order by status, arch;
status | arch | count
---------+-------+-------
fail | amd64 | 925
neutral | amd64 | 1593
pass | amd64 | 10805
tmpfail | amd64 | 14
(4 Zeilen)
udd=# select status, architecture, count(*) from autopkgtest group by status, architecture order by status, architecture;
status | architecture | count
---------+--------------+-------
fail | amd64 | 1561
fail | arm64 | 1471
fail | ppc64el | 711
neutral | amd64 | 2879
neutral | arm64 | 1322
neutral | ppc64el | 298
pass | amd64 | 21373
pass | arm64 | 11114
pass | ppc64el | 2458
tmpfail | amd64 | 11
tmpfail | arm64 | 85
tmpfail | ppc64el | 7
(12 Zeilen)
I guess we should merge both and make sure that all data is imported.
I would never have written a new importer if I would have been aware
of the existing one - but I do not speak Ruby to fix the existing one.
Kind regards
Andreas.
On 11/04/20 at 07:12 +0200, Andreas Tille wrote:
Hi Paul,
thanks for the clarification. This commit
https://salsa.debian.org/qa/udd/-/commit/6a874a89365671dd37a14a9bca25290dc55a1fc9
imports the current data. I will tests this a bit more and than activate in a cron job as importer.
Thanks a lot for your contribution
Andreas.
On Fri, Apr 10, 2020 at 09:05:31PM +0200, Paul Gevers wrote:
Hi Andreas,
On 09-04-2020 22:53, Andreas Tille wrote:
valid_keys = ( 'run_id',I expected you to mostly see "" or "migration-reference/0" here, with some hand crafted text from random DD's.
# 'created_at', # Paul Gevers: should be ignored
# 'updated_at', # Paul Gevers: should be ignored
'suite',
'arch',
'package', # ----> should be renamed to 'source'
'version',
'trigger', # usually package.*version
'status',
'requestor', # 'britney', 'debci' or e-mail
Debian login to be precise, not e-mail.
'pin_packages', # []
Since a couple of months this json and other pages only show "pure" suite runs, to pin_packages is always empty. pin_packages contains which
packages are taken from another suite than the base suite.
# 'worker', # Paul Gevers: should be ignored (is 'null' anyway)
Oh, bug somewhere I guess.
'date',
'duration_seconds',
'last_pass_date',
'last_pass_version',
'message', # see below
'previous_status',
# 'duration_human', # Paul Gevers: duration_seconds and duration_human feel double and the former is leaner for in a database
# 'blame', # Paul Gevers: should be ignored
)
# message can be
# $ grep '"message"' packages*.json | sed 's/^.*\.json: *//' | sort | uniq
# "message": "All tests passed" -> "status": "pass"
# "message": "Could not run tests due to a temporary testbed failure" -> "status": "tmpfail"
# "message": "elbrus" -> "status": "tmpfail"
# "message": "Erroneous package" -> "status": "fail"
# "message": null -> "status": "fail"
# "message": "No tests in this package or all skipped" -> "status": "neutral"
# "message": "Tests failed", -> "status": "fail"
# "message": "Tests failed, and at least one test skipped" -> "status": "fail"
# "message": "Tests passed, but at least one test skipped" -> "status": "pass"
# "message": "Unexpected autopkgtest exit code 20" -> "status": "tmpfail"
I agree that leaving out worker which is really always null makes sense
but I tend to leave message since leaving this out looks like loosing information. I tried to find a relation to status but it seems the same
status can result in different messages. I think just a field in addition
will not blow up UDD way more than it recently is - may be I consider a normalised form, but usually UDD is not very normalised at all.
In general this is the final output from autopkgtest. But, as you see my
name there, I had to clean up several times and to be able to find those
back, I added my nick to the message. The list thus may change over time.
Paul
--
http://fam-tille.de
--
http://fam-tille.de
Hi again Lucas,
may be my mail slipped through but basically the difference between the two importers is that you are just parsing
https://ci.debian.net/data/status/unstable/amd64/packages.json
while the Python script is parsing
https://ci.debian.net/data/status/*/*/packages.json
You even have line 5
# FIXME there might be more suites at some point
so you was aware of that issue. Would you mind solving that FIXME? Sorry,
I do not speak Ruby.
Kind regards
Andreas.
On Tue, Apr 14, 2020 at 06:12:39AM +0200, Andreas Tille wrote:
Hi Lucas,
On Mon, Apr 13, 2020 at 10:37:57PM +0200, Lucas Nussbaum wrote:
I'm sorry if I haven't paid enough attention. But what is the difference with the 'ci' importer?
https://salsa.debian.org/qa/udd/-/blob/master/rimporters/ci.rb
I think the problem is that UDD is not documented and I simply did not
know about the ci table. :-(
However, when looking at it there is a difference:
udd=# select status, arch, count(*) from ci group by status, arch order by status, arch;
status | arch | count
---------+-------+-------
fail | amd64 | 925
neutral | amd64 | 1593
pass | amd64 | 10805
tmpfail | amd64 | 14
(4 Zeilen)
udd=# select status, architecture, count(*) from autopkgtest group by status, architecture order by status, architecture;
status | architecture | count
---------+--------------+-------
fail | amd64 | 1561
fail | arm64 | 1471
fail | ppc64el | 711
neutral | amd64 | 2879
neutral | arm64 | 1322
neutral | ppc64el | 298
pass | amd64 | 21373
pass | arm64 | 11114
pass | ppc64el | 2458
tmpfail | amd64 | 11
tmpfail | arm64 | 85
tmpfail | ppc64el | 7
(12 Zeilen)
I guess we should merge both and make sure that all data is imported.
I would never have written a new importer if I would have been aware
of the existing one - but I do not speak Ruby to fix the existing one.
Kind regards
Andreas.
On 11/04/20 at 07:12 +0200, Andreas Tille wrote:
Hi Paul,
thanks for the clarification. This commit
https://salsa.debian.org/qa/udd/-/commit/6a874a89365671dd37a14a9bca25290dc55a1fc9
imports the current data. I will tests this a bit more and than activate
in a cron job as importer.
Thanks a lot for your contribution
Andreas.
On Fri, Apr 10, 2020 at 09:05:31PM +0200, Paul Gevers wrote:
Hi Andreas,
On 09-04-2020 22:53, Andreas Tille wrote:
valid_keys = ( 'run_id',I expected you to mostly see "" or "migration-reference/0" here, with some hand crafted text from random DD's.
# 'created_at', # Paul Gevers: should be ignored
# 'updated_at', # Paul Gevers: should be ignored
'suite',
'arch',
'package', # ----> should be renamed to 'source'
'version',
'trigger', # usually package.*version
'status',
'requestor', # 'britney', 'debci' or e-mail
Debian login to be precise, not e-mail.
'pin_packages', # []
Since a couple of months this json and other pages only show "pure" suite runs, to pin_packages is always empty. pin_packages contains which
packages are taken from another suite than the base suite.
# 'worker', # Paul Gevers: should be ignored (is 'null' anyway)
Oh, bug somewhere I guess.
'date',
'duration_seconds',
'last_pass_date',
'last_pass_version',
'message', # see below
'previous_status',
# 'duration_human', # Paul Gevers: duration_seconds and duration_human feel double and the former is leaner for in a database
# 'blame', # Paul Gevers: should be ignored
)
# message can be
# $ grep '"message"' packages*.json | sed 's/^.*\.json: *//' | sort | uniq
# "message": "All tests passed" -> "status": "pass"
# "message": "Could not run tests due to a temporary testbed failure" -> "status": "tmpfail"
# "message": "elbrus" -> "status": "tmpfail"
# "message": "Erroneous package" -> "status": "fail"
# "message": null -> "status": "fail"
# "message": "No tests in this package or all skipped" -> "status": "neutral"
# "message": "Tests failed", -> "status": "fail"
# "message": "Tests failed, and at least one test skipped" -> "status": "fail"
# "message": "Tests passed, but at least one test skipped" -> "status": "pass"
# "message": "Unexpected autopkgtest exit code 20" -> "status": "tmpfail"
I agree that leaving out worker which is really always null makes sense
but I tend to leave message since leaving this out looks like loosing
information. I tried to find a relation to status but it seems the same
status can result in different messages. I think just a field in addition
will not blow up UDD way more than it recently is - may be I consider
a normalised form, but usually UDD is not very normalised at all.
In general this is the final output from autopkgtest. But, as you see my
name there, I had to clean up several times and to be able to find those
back, I added my nick to the message. The list thus may change over time.
Paul
--
http://fam-tille.de
--
http://fam-tille.de
--
http://fam-tille.de
I fixed it and added the same suites as you were, reverted your changes
and dropped the autopkgtest table. So everything should be fine.
Sorry for not following closely enough when you started to work on that.
Hi Lucas,
On Fri, May 01, 2020 at 10:57:39PM +0200, Lucas Nussbaum wrote:
I fixed it and added the same suites as you were, reverted your changes
and dropped the autopkgtest table. So everything should be fine.
Thanks for this.
Sorry for not following closely enough when you started to work on that.
No problem - I've learned something anyway. ;-)
BTW, if I just want to update my local mirror, how can I do this:
$ /srv/udd.debian.org/udd/rudd ci
Traceback (most recent call last):
/srv/udd.debian.org/udd/rudd:62:in `<main>': No option specified (RuntimeError)
Kind regards
Andreas.
--
http://fam-tille.de
Hi Lucas,
On Fri, May 01, 2020 at 10:57:39PM +0200, Lucas Nussbaum wrote:
I fixed it and added the same suites as you were, reverted your changes
and dropped the autopkgtest table. So everything should be fine.
Thanks for this.
Sorry for not following closely enough when you started to work on that.
No problem - I've learned something anyway. ;-)
BTW, if I just want to update my local mirror, how can I do this:
$ /srv/udd.debian.org/udd/rudd ci
Traceback (most recent call last):
/srv/udd.debian.org/udd/rudd:62:in `<main>': No option specified (RuntimeError)
| Sysop: | Keyop |
|---|---|
| Location: | Huddersfield, West Yorkshire, UK |
| Users: | 300 |
| Nodes: | 16 (2 / 14) |
| Uptime: | 57:00:11 |
| Calls: | 6,712 |
| Files: | 12,243 |
| Messages: | 5,355,478 |