I'm sure I'm not the only one who received a whole bunch of bugs
entitled "Fails to build source after successful build" last weekend.
There was one theme common to most of them: the presence of a
*.egg-info directory which was not cleaned by debian/rules clean.

I know the bug report said that this policy is currently under
discussion, but I did get thinking about it. I imagine that this
particular directory should be the responsibility of dh-python to
clean up, but it may not be sensible to always delete *.egg-info
directories, as they may be present in the orig.tar.gz file. One
could handle it by manually adding this directory to debian/clean in
each package, but perhaps this should be the default behaviour of
dh-python?

Any thoughts?

Best wishes,

Julian

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi,

Am Fri, Aug 18, 2023 at 01:42:53PM +0100 schrieb Julian Gilbey:

I'm sure I'm not the only one who received a whole bunch of bugs
entitled "Fails to build source after successful build" last weekend.
There was one theme common to most of them: the presence of a
*.egg-info directory which was not cleaned by debian/rules clean.

I know the bug report said that this policy is currently under
discussion, but I did get thinking about it. I imagine that this
particular directory should be the responsibility of dh-python to
clean up, but it may not be sensible to always delete *.egg-info
directories, as they may be present in the orig.tar.gz file. One
could handle it by manually adding this directory to debian/clean in
each package, but perhaps this should be the default behaviour of
dh-python?

Any thoughts?

I agree that having this a no-brainer and getting all this *.egg-info
caused bugs solved by a simple dh-python update without changing
packaging code would be extremely convenient. I could imagine creating
a backup of the affected *.egg-info files if existent and copy these
back in clean target could solve this.

Kind regards
Andreas.

--
http://fam-tille.de

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

* Julian Gilbey <julian@d-and-j.net> [2023-08-18 13:42]:

One could handle it by manually adding this directory to
debian/clean in each package, but perhaps this should be the
default behaviour of dh-python?

See https://salsa.debian.org/python-team/tools/dh-python/-/merge_requests/46


Cheers
Timo

--
⢀⣴⠾⠻⢶⣦⠀ ╭────────────────────────────────────────────────────╮
⣾⠁⢠⠒⠀⣿⡁ │ Timo Röhling │
⢿⡄⠘⠷⠚⠋⠀ │ 9B03 EBB9 8300 DF97 C2B1 23BF CC8C 6BDD 1403 F4CA │
⠈⠳⣄⠀⠀⠀⠀ ╰────────────────────────────────────────────────────╯

-----BEGIN PGP SIGNATURE-----

iQGzBAEBCgAdFiEEJvtDgpxjkjCIVtam+C8H+466LVkFAmTfcOQACgkQ+C8H+466 LVmM6gv9G8NpmzmmWCYl89PXIzooR0a8rzamMvcbJ9jh7I0oRnzDdOvoW5iMP5aj spRgclvKih3RYgJFZAWqtDzTR3HVIbtawDwiPvdTgsHwX2vkiVCp9ttQqduNkOrQ 1zLGTMGZb8HgM/DvQ3qqCjGS9/Z08ekdqbWr5RQUjoPRAfFy4Kr8AbRUO6JxxbkP yBksR+Boy3h5MHi+PE+42EJn8CLQ0qRYMi7SgZslKcY1sR8NH/YqxWeIAX0PFrM3 jLzGw/+QAUQkpkTlAHJA7e2wyanu78+mxTxxe0SCzwcJXI9h9OrdZUhDXr9XbiAl pkXO9eYvN1CkT8Gu1/UM2aEjwcBf7+AKuJRuS/msB2Q

On Fri, Aug 18, 2023 at 09:23:17AM -0400, Scott Talbert wrote:

On Fri, 18 Aug 2023, Andreas Tille wrote:

Am Fri, Aug 18, 2023 at 01:42:53PM +0100 schrieb Julian Gilbey:

I'm sure I'm not the only one who received a whole bunch of bugs
entitled "Fails to build source after successful build" last weekend. There was one theme common to most of them: the presence of a
*.egg-info directory which was not cleaned by debian/rules clean.
[...]

It is being worked on: https://salsa.debian.org/python-team/tools/dh-python/-/merge_requests/46

Amazing!

Thanks,

Julian

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Fri, 18 Aug 2023, Andreas Tille wrote:

Am Fri, Aug 18, 2023 at 01:42:53PM +0100 schrieb Julian Gilbey:

I'm sure I'm not the only one who received a whole bunch of bugs
entitled "Fails to build source after successful build" last weekend.
There was one theme common to most of them: the presence of a
*.egg-info directory which was not cleaned by debian/rules clean.

I know the bug report said that this policy is currently under
discussion, but I did get thinking about it. I imagine that this
particular directory should be the responsibility of dh-python to
clean up, but it may not be sensible to always delete *.egg-info
directories, as they may be present in the orig.tar.gz file. One
could handle it by manually adding this directory to debian/clean in
each package, but perhaps this should be the default behaviour of
dh-python?

Any thoughts?

I agree that having this a no-brainer and getting all this *.egg-info
caused bugs solved by a simple dh-python update without changing
packaging code would be extremely convenient. I could imagine creating
a backup of the affected *.egg-info files if existent and copy these
back in clean target could solve this.

It is being worked on: https://salsa.debian.org/python-team/tools/dh-python/-/merge_requests/46

Regards,
Scott

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 8/18/23 14:42, Julian Gilbey wrote:

I'm sure I'm not the only one who received a whole bunch of bugs
entitled "Fails to build source after successful build" last weekend.
There was one theme common to most of them: the presence of a
*.egg-info directory which was not cleaned by debian/rules clean.

I know the bug report said that this policy is currently under
discussion

As much as I know, there's no controversy: this really is a bug in
packages. However, the discussion in d-devel has shown it's a minor
issue, as everyone has switch to use Git for packaging.

, but I did get thinking about it. I imagine that this
particular directory should be the responsibility of dh-python to
clean up, but it may not be sensible to always delete *.egg-info
directories, as they may be present in the orig.tar.gz file.

They should not. If you're packaging from pypi, you should switch to
using upstream VCS (most Python modules are in github). Otherwise, one possibility is to manually (well, through debian/watch) remove the
egg-info from the orig.tar.gz.

One
could handle it by manually adding this directory to debian/clean in
each package

I used to add this to all of my packages:

$ cat debian/source/options
extend-diff-ignore = "^[^/]*[.]egg-info/"

This works, and actually fixes the above mentioned bugs. However, as I
didn't want to have a single remaining instance of this bad clean-up, I
have setup my sbuild to check on it. The config is over here:

https://wiki.debian.org/zigo/mysbuild#A.2BAH4-.2F.sbuildrc

See the $external_commands thingy. This does a diff between the source
package, and the result after build + clean. It does report files from
the egg-info. Therefore, I did add rm -rf *.egg-info in all affected
packages, so I can continue to use the $external_commands checks. I
would strongly recommend every DD to do the same thing.

Yes, we can have dh-python to do the work, but IMO, the only thing it
should be doing, is rm -rf *.egg-info, and error out if the egg-info is
within the orig tarball, as this should not happen, IMO.

Now, I still think this is a minor issue... :)

Cheers,

Thomas Goirand (zigo)

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 2023-09-05 14:16:55 +0200 (+0200), Thomas Goirand wrote:
[...]

Yes, we can have dh-python to do the work, but IMO, the only thing
it should be doing, is rm -rf *.egg-info, and error out if the
egg-info is within the orig tarball, as this should not happen,
IMO.

[...]

See
https://salsa.debian.org/python-team/tools/dh-python/-/commit/31eff8f
which merged last week.
--
Jeremy Stanley

-----BEGIN PGP SIGNATURE-----

iQKTBAABCgB9FiEEl65Jb8At7J/DU7LnSPmWEUNJWCkFAmT4dfxfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDk3 QUU0OTZGQzAyREVDOUZDMzUzQjJFNzQ4Rjk5NjExNDM0OTU4MjkACgkQSPmWEUNJ WCnHcw//YWLyyNQzMCo1InbCaz4kn2SZQDwFt6L4gf++qZZWAn5GzqP6rojDIOV2 JQPUJnJ/iwru+5nMWdbeuUqWpjyU6eR2isinx/rYflceDK+Mxsht6oB/AtWT6me2 9LW06+KKaoukQMd9Z+DAaLyJvzYdGmSIyHBCiYuX/h/MQkYMuc3FIpNVqEJDUhWb tXmEX9vBWLN4B5JtumEskEUrkB36xqOhkAoDwXsFbViByiEi7ZE41zU5W7GPGJll M5w4jLeCeWrJuqOCIXwFesZKXHtHzut66caD6qqj4BVl+HZUBOdiV4xSz/DQUhCH ypCb8i7vpRPC0fuh0MdBpWLJvS2UO6ywwuyRF7grntgnTeuZvNA2liMOO5L4eIXZ HvBHJYoixbsNuVIMfY06kmwzDL7qLGTzscmW/7n3Ljk48bW8tYaLRHyxC273AeMM Izf3Daik3rDvBsm0IKqgxrT6fbqIura9y9ZId9aj3h7KTqPE7Koy9trLQzBXc+uS gILuL8h1Z9unrHvOgX0K2RRMRxvMXUPNPRVrqCmBKsEQ3BUhieVltTknEO18kfTH 5UTQ5LpF/nojUIB5+4tFkZbvPOoffdr3EiK/aL2jyGilNv/o3X1ojRuVxMr6BCNQ NPlxf20LPYCtODEIdkis4Pft9iVXscI7ceMkd7ZwihlLfUvd+cM=
=oU82
-----END PGP SIGNATURE-----

--- SoupGate-Win32

This is a multi-part message in MIME format.

--nextPart80669739.jzhGsMfI7o
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset="UTF-8"

As a followup question, I have noticed that a lot of packages (including electrum, which I
have recently started maintaining) ship the egg-info directory. Looking through /usr/lib/
python3/dist-packages/, this is common but not universal. Is there any reason to ship this
directory or should it be removed from the binary packages?

Also, the commit linked to below states:

"if the egg-info directory is included in the upstream source, then
deleting it causes no harm as dpkg-source warns on missing files in
the source package but does not exit with error.”

It might not cause any problems for dpkg-source, but won’t it cause issues for double
builds? For reference:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1044268[1]

If so, any package that ships the egg-info directory in the tarball will get flagged with a
FTBFS bug, which might be the desired behavior if these shouldn’t be in the tarball anyway.

On Wednesday, September 6, 2023 5:52:18 AM MST Jeremy Stanley wrote:

On 2023-09-05 14:16:55 +0200 (+0200), Thomas Goirand wrote:
[...]

Yes, we can have dh-python to do the work, but IMO, the only thing
it should be doing, is rm -rf *.egg-info, and error out if the
egg-info is within the orig tarball, as this should not happen,
IMO.

[...]

See
https://salsa.debian.org/python-team/tools/dh-python/-/commit/31eff8f
which merged last week.

--
Soren Stoutner
soren@stoutner.com

--------
[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1044268

--nextPart80669739.jzhGsMfI7o
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html; charset="UTF-8"

<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
</head>
<body><p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">As a followup question, I have noticed that a lot of packages (including electrum, which I have recently started maintaining) ship the egg-info directory.&nbsp; Looking through /
usr/lib/python3/dist-packages/, this is common but not universal.&nbsp; Is there any reason to ship this directory or should it be removed from the binary packages?</p>
<br /><p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">Also, the commit linked to below states:</p>
<br /><p style="margin-top:0;margin-bottom:0;margin-left:0;margin-right:0;">&quot;if the egg-info directory is included in the upstream source,

Hi Soren

On 07/09/2023 01:05, Soren Stoutner wrote:

Also, the commit linked to below states:

"if the egg-info directory is included in the upstream source, then
  deleting it causes no harm as dpkg-source warns on missing files in
  the source package but does not exit with error.”

It might not cause any problems for dpkg-source, but won’t it cause
issues for double builds?  For reference:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1044268 <https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1044268>

If so, any package that ships the egg-info directory in the tarball will
get flagged with a FTBFS bug, which might be the desired behavior if
these shouldn’t be in the tarball anyway.

No, the opposite is true.

They _currently_ FTBFS when built twice in a row because:

* the build modifies the egg-info files
* the change to the egg-info is not undone by the clean step that is
run between the two builds
* dpkg-source doesn't want modified upstream files and so exits
with error

With the change you referred to, these packages do not FTBFS when built
twice in a row because:

* the build modifies the egg-info files
* the egg-info is removed in the clean step between that runs between
the two build attempts
* dpkg-source doesn't care about removed upstream files and so
completes its task without error (it warns about them, that's all)
* the build proceeds and regenerates the egg-info

It was pointed out on IRC today that some packages (like poetry) ship
egg-info files that are part of the test suite and so are not
regenerated... and they should not be deleted. That's the next task.

regards
Stuart



--
Stuart Prescott http://www.nanonanonano.net/ stuart@nanonanonano.net
Debian Developer http://www.debian.org/ stuart@debian.org
GPG fingerprint 90E2 D2C1 AD14 6A1B 7EBB 891D BBC1 7EBB 1396 F2F7

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Wed, Sep 06, 2023 at 08:05:45AM -0700, Soren Stoutner wrote:

As a followup question, I have noticed that a lot of packages (including electrum, which I have recently started maintaining) ship the egg-info directory. Looking through /usr/lib/python3/dist-packages/, this is common but
not universal. Is there any reason to ship this directory or should it be removed from the binary packages?

Lots of packages depend on the egg-info directory being present.

$ grep EASY-INSTALL-ENTRY-SCRIPT /usr/bin/*

will give a (probably very long) list of executables that depend on an
egg-info (or equivalent) directory being present.

Best wishes,

Julian

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)