On Wed, 17 Aug 2022 at 07:05, Dan Stromberg <drsalists@gmail.com> wrote:

Hi folks.

I'm attempting to package up a python package that uses Cython.

Rather than build binaries for everything under the sun, I've been focusing on including the .pyx file and running cython on it at install time. This requires a C compiler, but I'm OK with that.

Is keeping the cythonized file an option? That would still require a C compiler, but wouldn't require Cython.

ChrisA

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi folks.

I'm attempting to package up a python package that uses Cython.

Rather than build binaries for everything under the sun, I've been focusing
on including the .pyx file and running cython on it at install time. This requires a C compiler, but I'm OK with that.

However, when I try to install the package from test.pypi.org, I get:
$ python3 -m pip install -i https://test.pypi.org/simple/ pyx-treap
below cmd output started 2022 Tue Aug 16 01:55:16 PM PDT
Looking in indexes: https://test.pypi.org/simple/
Collecting pyx-treap
Downloading https://test-files.pythonhosted.org/packages/3a/41/af5360934adccfc086a39e1f720323895144b53454ff6dacc0f06267db55/pyx_treap-2.0.15.tar.gz
(125 kB)

????????????????????????????????????????????????????????????????????????????????
125.9/125.9 kB 1.9 MB/s eta 0:00:00
Installing build dependencies ... error
error: subprocess-exited-with-error

×? pip subprocess to install build dependencies did not run successfully.
?? exit code: 1
????> [3 lines of output]
Looking in indexes: https://test.pypi.org/simple/
ERROR: Could not find a version that satisfies the requirement
setuptools (from versions: none)
ERROR: No matching distribution found for setuptools
[end of output]

note: This error originates from a subprocess, and is likely not a
problem with pip.
error: subprocess-exited-with-error

×? pip subprocess to install build dependencies did not run successfully.
?? exit code: 1
????> See above for output.

note: This error originates from a subprocess, and is likely not a problem
with pip.

But I analyzed the pip install with strace, and found no interesting
exec's, and no interesting "= E" patterns.

I've tried quite an assortment of things to get past this, including most
of those at: https://stackoverflow.com/questions/4996589/in-setup-py-or-pip-requirements-file-how-to-control-order-of-installing-package
Except the one at:
https://stackoverflow.com/a/54269307/1084684
...because I was having a "there must be a better way" reaction to it.

/Is/ there a better way?

There're all these newer ways of doing packaging - surely one of them
addresses this problem?

The code I'm trying to package is at: https://stromberg.dnsalias.org/svn/treap/trunk/cython

BTW, the pure python version works fine, and the cython version works too
as long as you preinstall cython - but I don't want users to have to know
that :)

Thanks!

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Tue, Aug 16, 2022 at 2:08 PM Chris Angelico <rosuav@gmail.com> wrote:

On Wed, 17 Aug 2022 at 07:05, Dan Stromberg <drsalists@gmail.com> wrote:

Hi folks.

I'm attempting to package up a python package that uses Cython.

Rather than build binaries for everything under the sun, I've been

focusing

on including the .pyx file and running cython on it at install time.

This

requires a C compiler, but I'm OK with that.

Is keeping the cythonized file an option? That would still require a C compiler, but wouldn't require Cython.

ChrisA

That comes under "one of the many things I tried"; it seemed like some of
the code in the .c was tailored to the specific version of python it was
being installed on or something.

But if someone has seen this work well, I'm more than open to trying it
again.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Am 16.08.22 um 23:03 schrieb Dan Stromberg:

I'm attempting to package up a python package that uses Cython.

Rather than build binaries for everything under the sun, I've been focusing on including the .pyx file and running cython on it at install time. This requires a C compiler, but I'm OK with that.

However, when I try to install the package from test.pypi.org, I get:
$ python3 -m pip install -i https://test.pypi.org/simple/ pyx-treap
below cmd output started 2022 Tue Aug 16 01:55:16 PM PDT
Looking in indexes: https://test.pypi.org/simple/
Collecting pyx-treap
Downloading https://test-files.pythonhosted.org/packages/3a/41/af5360934adccfc086a39e1f720323895144b53454ff6dacc0f06267db55/pyx_treap-2.0.15.tar.gz
(125 kB)

????????????????????????????????????????????????????????????????????????????????
125.9/125.9 kB 1.9 MB/s eta 0:00:00
Installing build dependencies ... error
error: subprocess-exited-with-error

×? pip subprocess to install build dependencies did not run successfully.
?? exit code: 1
????> [3 lines of output]
Looking in indexes: https://test.pypi.org/simple/
ERROR: Could not find a version that satisfies the requirement setuptools (from versions: none)
ERROR: No matching distribution found for setuptools
[end of output]

note: This error originates from a subprocess, and is likely not a
problem with pip.
error: subprocess-exited-with-error

I looked at your code and I think you are trying too hard. As far as I understand, you need Cython to be installed before the build process
begins. Your entry in pyproject.toml should take care of that.
But you also have these lines in your setup.py

subprocess.check_call('%s -m pip install cython' % (sys.executable, ), shell=True)
subprocess.check_call('%s -m cython pyx_treap.pyx' % (sys.executable, ), shell=True)

The first one calls out to pip while pip is already running, I'm not
sure that this will work, but judging from the error message it is
looking for the requirements also from test.pypi. Maybe this is the
reason that it fails (the error message says that it can't find
setuptools). So jut delete this line and it might already work

The second line, which compiles the Cython code, also runs *at every
invocation of setup.py*, even if you'd do just

python3 setup.py --help

It may still work, but the correct way to do it is to create a build
extension for setuptools. In my project you can see this here:

https://github.com/j-from-b/CDEF/blob/main/setup.py#L88

OTOH, I would be surprised if Cython did not have this already, indeed
you imported cythonize from Cython.Build. So maybe just deleting these
two lines and it might work?

Christian

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Dan Stromberg wrote at 2022-8-16 14:03 -0700:

...
I'm attempting to package up a python package that uses Cython.
...
Installing build dependencies ... error
error: subprocess-exited-with-error

×? pip subprocess to install build dependencies did not run successfully.
?? exit code: 1
????> [3 lines of output]
Looking in indexes: https://test.pypi.org/simple/
ERROR: Could not find a version that satisfies the requirement
setuptools (from versions: none)
ERROR: No matching distribution found for setuptools

The message tells you that there is a `setuptools` problem.
I would start to locate all `setuptools` requirement locations.


I am using `cython` for the package `dm.xmlsec.binding`.
I have not observed nor heard of a problem similar to yours
(but I have never tried `test.pypi.org`).

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Wed, Aug 17, 2022 at 10:20 AM Christian Gollwitzer <auriocus@gmx.de>
wrote:

Am 16.08.22 um 23:03 schrieb Dan Stromberg:

I'm attempting to package up a python package that uses Cython.

Rather than build binaries for everything under the sun, I've been

focusing

on including the .pyx file and running cython on it at install time.

This

requires a C compiler, but I'm OK with that.

However, when I try to install the package from test.pypi.org, I get:
$ python3 -m pip install -i https://test.pypi.org/simple/ pyx-treap
below cmd output started 2022 Tue Aug 16 01:55:16 PM PDT
Looking in indexes: https://test.pypi.org/simple/
Collecting pyx-treap
Downloading

https://test-files.pythonhosted.org/packages/3a/41/af5360934adccfc086a39e1f720323895144b53454ff6dacc0f06267db55/pyx_treap-2.0.15.tar.gz

(125 kB)

????????????????????????????????????????????????????????????????????????????????

125.9/125.9 kB 1.9 MB/s eta 0:00:00
Installing build dependencies ... error
error: subprocess-exited-with-error

×? pip subprocess to install build dependencies did not run

successfully.

?? exit code: 1
????> [3 lines of output]
Looking in indexes: https://test.pypi.org/simple/
ERROR: Could not find a version that satisfies the requirement setuptools (from versions: none)
ERROR: No matching distribution found for setuptools
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

I looked at your code and I think you are trying too hard. As far as I understand, you need Cython to be installed before the build process
begins. Your entry in pyproject.toml should take care of that.
But you also have these lines in your setup.py

subprocess.check_call('%s -m pip install cython' % (sys.executable, ), shell=True)
subprocess.check_call('%s -m cython pyx_treap.pyx' % (sys.executable, ), shell=True)

The first one calls out to pip while pip is already running, I'm not
sure that this will work, but judging from the error message it is
looking for the requirements also from test.pypi. Maybe this is the
reason that it fails (the error message says that it can't find
setuptools). So jut delete this line and it might already work

The second line, which compiles the Cython code, also runs *at every invocation of setup.py*, even if you'd do just

python3 setup.py --help

It may still work, but the correct way to do it is to create a build extension for setuptools. In my project you can see this here:

https://github.com/j-from-b/CDEF/blob/main/setup.py#L88

OTOH, I would be surprised if Cython did not have this already, indeed
you imported cythonize from Cython.Build. So maybe just deleting these
two lines and it might work?

I commented out those too lines, but I'm still getting errors. They seem
to stem from:
$ "/home/dstromberg/venv/pyx-treap-testing/bin/python3", ["/home/dstromberg/venv/pyx-treap-testing/bin/python3", "/home/dstromberg/venv/pyx-treap-testing/lib/python3.9/site-packages/pip/__pip-runner__.py",
"install", "--ignore-installed", "--no-user", "--prefix", "/tmp/pip-build-env-9_ivrsb6/overlay", "--no-warn-script-location", "--no-binary", ":none:", "--only-binary", ":none:", "-i", " https://test.pypi.org/simple/", "--", "setuptools >= 44.1.1", "wheel", "Cython"]
"/home/dstromberg/venv/pyx-treap-testing/bin/python3" "/home/dstromberg/venv/pyx-treap-testing/lib/python3.9/site-packages/pip/__pip-runner__.py"
"install" "--ignore-installed" "--no-user" "--prefix" "/tmp/pip-build-env-9_ivrsb6/overlay" "--no-warn-script-location"
"--no-binary" ":none:" "--only-binary" ":none:" "-i" " https://test.pypi.org/simple/" "--" "setuptools >= 44.1.1" "wheel" "Cython" Looking in indexes: https://test.pypi.org/simple/
ERROR: Could not find a version that satisfies the requirement setuptools>=44.1.1 (from versions: none)
ERROR: No matching distribution found for setuptools>=44.1.1

I copied that out of an strace.

That's likely related to my pyproject.toml:
$ cat pyproject.toml
below cmd output started 2022 Wed Aug 17 01:57:09 PM PDT
[build-system]
requires = ["setuptools >= 44.1.1", "wheel", "Cython"]
build-backend = "setuptools.build_meta"

Any other suggestions folks?

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Wed, Aug 17, 2022 at 1:58 PM Dan Stromberg <drsalists@gmail.com> wrote:

On Wed, Aug 17, 2022 at 10:20 AM Christian Gollwitzer <auriocus@gmx.de> wrote:

Am 16.08.22 um 23:03 schrieb Dan Stromberg:

I'm attempting to package up a python package that uses Cython.

Rather than build binaries for everything under the sun, I've been

focusing

on including the .pyx file and running cython on it at install time.

This

requires a C compiler, but I'm OK with that.

However, when I try to install the package from test.pypi.org, I get:
$ python3 -m pip install -i https://test.pypi.org/simple/ pyx-treap
below cmd output started 2022 Tue Aug 16 01:55:16 PM PDT
Looking in indexes: https://test.pypi.org/simple/
Collecting pyx-treap
Downloading

https://test-files.pythonhosted.org/packages/3a/41/af5360934adccfc086a39e1f720323895144b53454ff6dacc0f06267db55/pyx_treap-2.0.15.tar.gz

(125 kB)

????????????????????????????????????????????????????????????????????????????????

125.9/125.9 kB 1.9 MB/s eta 0:00:00
Installing build dependencies ... error
error: subprocess-exited-with-error

×? pip subprocess to install build dependencies did not run

successfully.

?? exit code: 1
????> [3 lines of output]
Looking in indexes: https://test.pypi.org/simple/
ERROR: Could not find a version that satisfies the requirement
setuptools (from versions: none)
ERROR: No matching distribution found for setuptools
[end of output]

note: This error originates from a subprocess, and is likely not a
problem with pip.
error: subprocess-exited-with-error

I looked at your code and I think you are trying too hard. As far as I
understand, you need Cython to be installed before the build process
begins. Your entry in pyproject.toml should take care of that.
But you also have these lines in your setup.py

subprocess.check_call('%s -m pip install cython' % (sys.executable, ),
shell=True)
subprocess.check_call('%s -m cython pyx_treap.pyx' % (sys.executable, ),
shell=True)

The first one calls out to pip while pip is already running, I'm not
sure that this will work, but judging from the error message it is
looking for the requirements also from test.pypi. Maybe this is the
reason that it fails (the error message says that it can't find
setuptools). So jut delete this line and it might already work

The second line, which compiles the Cython code, also runs *at every
invocation of setup.py*, even if you'd do just

python3 setup.py --help

It may still work, but the correct way to do it is to create a build
extension for setuptools. In my project you can see this here:

https://github.com/j-from-b/CDEF/blob/main/setup.py#L88

OTOH, I would be surprised if Cython did not have this already, indeed
you imported cythonize from Cython.Build. So maybe just deleting these
two lines and it might work?

I commented out those too lines, but I'm still getting errors. They seem
to stem from:
$ "/home/dstromberg/venv/pyx-treap-testing/bin/python3", ["/home/dstromberg/venv/pyx-treap-testing/bin/python3", "/home/dstromberg/venv/pyx-treap-testing/lib/python3.9/site-packages/pip/__pip-runner__.py",
"install", "--ignore-installed", "--no-user", "--prefix", "/tmp/pip-build-env-9_ivrsb6/overlay", "--no-warn-script-location", "--no-binary", ":none:", "--only-binary", ":none:", "-i", " https://test.pypi.org/simple/", "--", "setuptools >= 44.1.1", "wheel", "Cython"]
"/home/dstromberg/venv/pyx-treap-testing/bin/python3" "/home/dstromberg/venv/pyx-treap-testing/lib/python3.9/site-packages/pip/__pip-runner__.py"
"install" "--ignore-installed" "--no-user" "--prefix" "/tmp/pip-build-env-9_ivrsb6/overlay" "--no-warn-script-location" "--no-binary" ":none:" "--only-binary" ":none:" "-i" " https://test.pypi.org/simple/" "--" "setuptools >= 44.1.1" "wheel"
"Cython"
Looking in indexes: https://test.pypi.org/simple/
ERROR: Could not find a version that satisfies the requirement setuptools>=44.1.1 (from versions: none)
ERROR: No matching distribution found for setuptools>=44.1.1

I copied that out of an strace.

That's likely related to my pyproject.toml:
$ cat pyproject.toml
below cmd output started 2022 Wed Aug 17 01:57:09 PM PDT
[build-system]
requires = ["setuptools >= 44.1.1", "wheel", "Cython"]
build-backend = "setuptools.build_meta"

Any other suggestions folks?

I don't know why, but if I delete the --ignore-installed option, I don't
get the error:
$ "/home/dstromberg/venv/pyx-treap-testing/bin/python3" "/home/dstromberg/venv/pyx-treap-testing/lib/python3.9/site-packages/pip/__pip-runner__.py"
"install" "--ignore-installed" "--prefix"
"/tmp/pip-build-env-9_ivrsb6/overlay" "-i" "https://test.pypi.org/simple/"
"--" "setuptools>=44.1.1"
below cmd output started 2022 Wed Aug 17 02:56:24 PM PDT
Looking in indexes: https://test.pypi.org/simple/
ERROR: Could not find a version that satisfies the requirement setuptools>=44.1.1 (from versions: none)
ERROR: No matching distribution found for setuptools>=44.1.1 (setuptools-investigation) above cmd output done 2022 Wed Aug 17
02:56:24 PM PDT
dstromberg@tp-mini-c:~ x86_64-pc-linux-gnu 2995

$ "/home/dstromberg/venv/pyx-treap-testing/bin/python3" "/home/dstromberg/venv/pyx-treap-testing/lib/python3.9/site-packages/pip/__pip-runner__.py"
"install" "--prefix" "/tmp/pip-build-env-9_ivrsb6/overlay" "-i" " https://test.pypi.org/simple/" "--" "setuptools>=44.1.1"
below cmd output started 2022 Wed Aug 17 02:56:35 PM PDT
Looking in indexes: https://test.pypi.org/simple/
Requirement already satisfied: setuptools>=44.1.1 in ./venv/pyx-treap-testing/lib/python3.9/site-packages (63.4.1)

If I search for foo on pypi and testpypi, shouldn't I get foo before
foo-bar and bar-foo?

Because if I search for setuptools on pypi, I get setuptools as my first
hit, but on testpypi, I don't see setuptools anywhere in the first page.
That's perhaps significant if the search ordering is working as described above.

?

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Wed, Aug 17, 2022 at 3:05 PM Dan Stromberg <drsalists@gmail.com> wrote:

I commented out those too lines, but I'm still getting errors. They seem

to stem from:
$ "/home/dstromberg/venv/pyx-treap-testing/bin/python3",
["/home/dstromberg/venv/pyx-treap-testing/bin/python3",
"/home/dstromberg/venv/pyx-treap-testing/lib/python3.9/site-packages/pip/__pip-runner__.py",
"install", "--ignore-installed", "--no-user", "--prefix",
"/tmp/pip-build-env-9_ivrsb6/overlay", "--no-warn-script-location",
"--no-binary", ":none:", "--only-binary", ":none:", "-i", "
https://test.pypi.org/simple/", "--", "setuptools >= 44.1.1", "wheel",
"Cython"]
"/home/dstromberg/venv/pyx-treap-testing/bin/python3"
"/home/dstromberg/venv/pyx-treap-testing/lib/python3.9/site-packages/pip/__pip-runner__.py"
"install" "--ignore-installed" "--no-user" "--prefix"
"/tmp/pip-build-env-9_ivrsb6/overlay" "--no-warn-script-location"
"--no-binary" ":none:" "--only-binary" ":none:" "-i" "
https://test.pypi.org/simple/" "--" "setuptools >= 44.1.1" "wheel"
"Cython"
Looking in indexes: https://test.pypi.org/simple/
ERROR: Could not find a version that satisfies the requirement
setuptools>=44.1.1 (from versions: none)
ERROR: No matching distribution found for setuptools>=44.1.1

I copied that out of an strace.

That's likely related to my pyproject.toml:
$ cat pyproject.toml
below cmd output started 2022 Wed Aug 17 01:57:09 PM PDT
[build-system]
requires = ["setuptools >= 44.1.1", "wheel", "Cython"]
build-backend = "setuptools.build_meta"

Any other suggestions folks?

I don't know why, but if I delete the --ignore-installed option, I don't
get the error:
$ "/home/dstromberg/venv/pyx-treap-testing/bin/python3" "/home/dstromberg/venv/pyx-treap-testing/lib/python3.9/site-packages/pip/__pip-runner__.py"
"install" "--ignore-installed" "--prefix" "/tmp/pip-build-env-9_ivrsb6/overlay" "-i" "https://test.pypi.org/simple/" "--" "setuptools>=44.1.1"
below cmd output started 2022 Wed Aug 17 02:56:24 PM PDT
Looking in indexes: https://test.pypi.org/simple/
ERROR: Could not find a version that satisfies the requirement setuptools>=44.1.1 (from versions: none)
ERROR: No matching distribution found for setuptools>=44.1.1 (setuptools-investigation) above cmd output done 2022 Wed Aug 17
02:56:24 PM PDT
dstromberg@tp-mini-c:~ x86_64-pc-linux-gnu 2995

$ "/home/dstromberg/venv/pyx-treap-testing/bin/python3" "/home/dstromberg/venv/pyx-treap-testing/lib/python3.9/site-packages/pip/__pip-runner__.py"
"install" "--prefix" "/tmp/pip-build-env-9_ivrsb6/overlay" "-i" " https://test.pypi.org/simple/" "--" "setuptools>=44.1.1"
below cmd output started 2022 Wed Aug 17 02:56:35 PM PDT
Looking in indexes: https://test.pypi.org/simple/
Requirement already satisfied: setuptools>=44.1.1 in ./venv/pyx-treap-testing/lib/python3.9/site-packages (63.4.1)

If I search for foo on pypi and testpypi, shouldn't I get foo before
foo-bar and bar-foo?

Because if I search for setuptools on pypi, I get setuptools as my first
hit, but on testpypi, I don't see setuptools anywhere in the first page. That's perhaps significant if the search ordering is working as described above.

?

IOW, maybe testpypi doesn't have setuptools. Indeed, it doesn't appear
to. Publishing and installing from the real pypi seems to be working.

Thanks folks.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Tue, Aug 16, 2022 at 2:03 PM Dan Stromberg <drsalists@gmail.com> wrote:

Hi folks.

I'm attempting to package up a python package that uses Cython.

Rather than build binaries for everything under the sun, I've been
focusing on including the .pyx file and running cython on it at install
time. This requires a C compiler, but I'm OK with that.

BTW, the pure python version works fine, and the cython version works too
as long as you preinstall cython - but I don't want users to have to know that :)

For the actual chicken-and-egg problem, I'd needed to include my
pyproject.toml in my MANIFEST.in, like:
include pyx_treap.pyx pyx_treap.c pyproject.toml

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)