Hi,

I'm working towards packaging pydevd (which is a recursive dependency
of Spyder via IPython), and it's a somewhat challenging package! I
have hit an issue and would appreciate some thoughts on how best to
handle it.

Background:

pydevd is a debugging package which can attach to running scripts. It
is used by PyDev, PyCharm, VSCode and Spyder, and with Spyder, it is
imported through debugpy, which in turn is imported into IPython.

In order to ensure that the libraries it is using have not been
modified by gevent, it uses vendored copies of various standard
library modules rather than just saying "import inspect" etc.

I thought I could address this issue by replacing the vendored copies
of the library modules by symlinks to /usr/lib/python3.X/, but now
I've hit another snag: some of these modules import other modules.
For example:

pydev_imps/_pydev_SimpleXMLRPCServer.py
is a very old version of /usr/lib/python3.X/xmlrpc/server.py. It
contains within it the following lines:

from _pydev_imps import _pydev_xmlrpclib as xmlrpclib
from _pydev_imps._pydev_xmlrpclib import Fault
from _pydev_imps import _pydev_SocketServer as SocketServer
from _pydev_imps import _pydev_BaseHTTPServer as BaseHTTPServer

These libraries are:
_pydev_imps._pydev_xmlrpclib -> xmlrpc.client
_pydev_imps._pydev_SocketServer -> socketserver _pydev_imps._pydev_BaseHTTPServer -> http.server

So what should I do?

One solution is just to symlink from _pydev_SimpleXMLRPCServer.py to /usr/lib/python3.X/xmlrpc/server.py and not worry about the other
modules. But that might break things in non-obvious ways, so I don't
want to do that.

Another possible solution is to update all of the vendored copies in _pydev_imps using the relevant /usr/lib/python3.X/* modules and making
the same modifications to the imports to load local copies. But then
we will have duplicate copies of standard library modules, which I
also don't want to do.

Perhaps another possibility is to have symlinks in the _pydev_imps
directory to the standard library versions and then temporarily modify
sys.path to look in _pydev_imps before looking in standard locations.
I don't know whether this will work, though.

(There is another snag, too, which is that the path will depend on the
Python version. So I will probably have
_pydev_imps/python_39/socketserver.py -> /usr/lib/python3.9/socketserver.py
and
_pydev_imps/python_310/socketserver.py -> /usr/lib/python3.10/socketserver.py But that's a simpler issue.)

Any thoughts on these ideas would be much appreciated!

Best wishes,

Julian

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Mon, Feb 07, 2022 at 09:27:28PM +0000, Julian Gilbey wrote:

[...]

I thought I could address this issue by replacing the vendored copies
of the library modules by symlinks to /usr/lib/python3.X/, but now
I've hit another snag: some of these modules import other modules.
For example:

pydev_imps/_pydev_SimpleXMLRPCServer.py
is a very old version of /usr/lib/python3.X/xmlrpc/server.py. It
contains within it the following lines:

from _pydev_imps import _pydev_xmlrpclib as xmlrpclib
from _pydev_imps._pydev_xmlrpclib import Fault
from _pydev_imps import _pydev_SocketServer as SocketServer
from _pydev_imps import _pydev_BaseHTTPServer as BaseHTTPServer
[...]
Perhaps another possibility is to have symlinks in the _pydev_imps
directory to the standard library versions and then temporarily modify sys.path to look in _pydev_imps before looking in standard locations.
I don't know whether this will work, though.

I realise now that this "nice" solution won't work, as the standard
library code says:

import socketserver

so modifying sys.path will just change the value of sys.modules["socketserver"]. However, the vendored code instead loads
this module to sys.modules["_pydev_imps._pydev_SocketServer"] or
something like that, deliberately avoiding interfering with sys.modules["socketserver"].

Ho hum.

Julian

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

I realise now that this "nice" solution won't work, as the standard
library code says:

import socketserver

so modifying sys.path will just change the value of sys.modules["socketserver"]. However, the vendored code instead loads
this module to sys.modules["_pydev_imps._pydev_SocketServer"] or
something like that, deliberately avoiding interfering with sys.modules["socketserver"].

It seems to me that the "correct" solution would be to motivate upstream
not to vendor anything in their source tree. If they really need
vendoring to avoid compatibility issues with various environments, they
should do so only when building releases. It still wouldn't solve the
problem of incompatible system modules, but at least it would make it
clearer which versions they require and why.

Perhaps they have a maintenance script for updating the vendored
dependencies? You could use that to find out how to reverse the changes,
or start from a clean slate?

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Wed, Feb 09, 2022 at 07:39:40PM +0100, Gregor Riepl wrote:

I realise now that this "nice" solution won't work, as the standard
library code says:

import socketserver

so modifying sys.path will just change the value of sys.modules["socketserver"]. However, the vendored code instead loads
this module to sys.modules["_pydev_imps._pydev_SocketServer"] or
something like that, deliberately avoiding interfering with sys.modules["socketserver"].

It seems to me that the "correct" solution would be to motivate upstream
not to vendor anything in their source tree. If they really need
vendoring to avoid compatibility issues with various environments, they should do so only when building releases. It still wouldn't solve the
problem of incompatible system modules, but at least it would make it
clearer which versions they require and why.

Hi Gregor,

Thanks!

And indeed, that would be a good idea, but I doubt it's going to
happen :( They don't need "vendoring" as far as I can tell, but they
just need a pristine copy of the system library that can be loaded independently of the system library, so that if gevent patches the
system library at runtime, this package still has access to an
unpatched copy.

So the solution I'm currently in the process of trying is to copy the
version from the oldest supported Python version at Debian package
build time. We'll see how this goes....

Perhaps they have a maintenance script for updating the vendored dependencies? You could use that to find out how to reverse the changes,
or start from a clean slate?

Unlikely; some of their vendored dependencies date back to Python 2.5!

Best wishes,

Julian

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

So the solution I'm currently in the process of trying is to copy the
version from the oldest supported Python version at Debian package
build time. We'll see how this goes....

Perhaps they have a maintenance script for updating the vendored
dependencies? You could use that to find out how to reverse the changes,
or start from a clean slate?

Unlikely; some of their vendored dependencies date back to Python 2.5!

In that case, I think this is the issue that must be solved first:
Ensuring that their code is compatible with the *latest* published
version, and vendoring the system version at build time.

Not a good solution, since it will still leave the system vulnerable
when one of the dependencies gets a security update, but better than
shipping a static version that might have numerous issues and will no
longer receive any patches.

The alternative would be that they take full responsibility for their
vendored code, but then it will be much harder to detect when they're
affected by a vulnerability.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Sun, Feb 13, 2022 at 12:07:44PM +0100, Gregor Riepl wrote:

So the solution I'm currently in the process of trying is to copy the version from the oldest supported Python version at Debian package
build time. We'll see how this goes....

Perhaps they have a maintenance script for updating the vendored
dependencies? You could use that to find out how to reverse the changes, >> or start from a clean slate?

Unlikely; some of their vendored dependencies date back to Python 2.5!

In that case, I think this is the issue that must be solved first:
Ensuring that their code is compatible with the *latest* published
version, and vendoring the system version at build time.

Hi Gregor,

Thanks for your thoughts.

Yes, and that is what I'm going with. Unfortunately, upstream are
trying to keep pydevd compatible with almost every version of Python
that exists (Python 2.7, Python 3.x, PyPy, IronPython and Jython), so
they cannot necessarily use the latest versions of the library. But
I'm only going to aim to support the supported versions of CPython in
Debian, so I can use the latest versions of the library. (Actually, I
am using the version in the lowest Debian-supported version of Python,
in case a newer version of the library uses newly introduced syntax.)

Not a good solution, since it will still leave the system vulnerable
when one of the dependencies gets a security update, but better than
shipping a static version that might have numerous issues and will no
longer receive any patches.

Indeed.

The alternative would be that they take full responsibility for their vendored code, but then it will be much harder to detect when they're affected by a vulnerability.

Quite.

Best wishes,

Julian

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)