Hello, I am packaging a python extension which use cmake as build system

here the repo

https://salsa.debian.org/science-team/dxtbx

I try to activate the test but this cause this kind of trouble

During handling of the above exception, another exception occurred: /usr/lib/python3.10/importlib/__init__.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level) tests/serialize/test_xds.py:8: in <module>
from dxtbx.imageset import ImageSetFactory
dxtbx/imageset.py:5: in <module>
import dxtbx.format.image # noqa: F401, import dependency for unpickling dxtbx/format/image.py:8: in <module>
from dxtbx_format_image_ext import * # type: ignore # isort:skip # noqa: F403
E ModuleNotFoundError: No module named 'dxtbx_format_image_ext'

Indeed the extension are build here.

cd /<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/src/dxtbx && /usr/bin/cmake -E cmake_link_script CMakeFiles/dxtbx_flumpy.dir/link.txt --verbose=1
/usr/bin/c++ -fPIC -g -O2 -ffile-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -flto -Wl,-z,relro -shared -o ../../lib/dxtbx_flumpy.cpython-310-x86_64-linux-gnu.so CMakeFiles/
dxtbx_flumpy.dir/boost_python/flumpy.cc.o /usr/lib/x86_64-linux-gnu/libboost_python310.so.1.74.0
lto-wrapper: warning: using serial compilation of 4 LTRANS jobs
lto-wrapper: note: see the ‘-flto’ option documentation for more information
[ 90%] Linking CXX shared module ../../lib/dxtbx_format_image_ext.so

so I need to add the /<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/lib directory to the PYTHONPATH.

do you know how to do this ?

thanks for your help

Fred

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Tue, Aug 30, 2022 at 03:58:26PM +0200, PICCA Frederic-Emmanuel wrote:

Hello, I am packaging a python extension which use cmake as build system

here the repo

https://salsa.debian.org/science-team/dxtbx

I try to activate the test but this cause this kind of trouble

During handling of the above exception, another exception occurred: /usr/lib/python3.10/importlib/__init__.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level) tests/serialize/test_xds.py:8: in <module>
from dxtbx.imageset import ImageSetFactory
dxtbx/imageset.py:5: in <module>
import dxtbx.format.image # noqa: F401, import dependency for unpickling dxtbx/format/image.py:8: in <module>
from dxtbx_format_image_ext import * # type: ignore # isort:skip # noqa: F403
E ModuleNotFoundError: No module named 'dxtbx_format_image_ext'

Indeed the extension are build here.

cd /<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/src/dxtbx && /usr/bin/cmake -E cmake_link_script CMakeFiles/dxtbx_flumpy.dir/link.txt --verbose=1
/usr/bin/c++ -fPIC -g -O2 -ffile-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -flto -Wl,-z,relro -shared -o ../../lib/dxtbx_flumpy.cpython-310-x86_64-linux-gnu.so CMakeFiles/

dxtbx_flumpy.dir/boost_python/flumpy.cc.o /usr/lib/x86_64-linux-gnu/libboost_python310.so.1.74.0

lto-wrapper: warning: using serial compilation of 4 LTRANS jobs
lto-wrapper: note: see the ‘-flto’ option documentation for more information
[ 90%] Linking CXX shared module ../../lib/dxtbx_format_image_ext.so

so I need to add the /<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/lib directory to the PYTHONPATH.

do you know how to do this ?

Does the same happen when you run the test in the source tree manually?

--
WBR, wRAR

-----BEGIN PGP SIGNATURE-----

iQJhBAABCgBLFiEEolIP6gqGcKZh3YxVM2L3AxpJkuEFAmMOH7EtFIAAAAAAFQAP cGthLWFkZHJlc3NAZ251cGcub3Jnd3JhckBkZWJpYW4ub3JnAAoJEDNi9wMaSZLh 5rEQAJy57PNbqGyiV5LwxyY7joDyJvpgvHwZU4plNyt10kob7Fj7hvSE97nePEoH TqgoFpC0+ca+VwDq9ApI/Z+dWpKyufx5zG6kGGqEG57Znue5U6Wve79mBVeMu8zh eau0VLU/BNWcF7bd5vua1rSyLcZ4D9G//e4Rp1Z+8/1Xm9Q0O+fYHcw7nl7FxapB bczkZ35akeSqMqVCOGXAzbxrhOBTzwRRhdQx98Z5+3AeGoqRML/jWLPackAX5Emp Av/r2zaJ+JOQcxXFigK7gusukk3Z+ldu3ZGxK1eq9hPoHVKDhzg2ht9rUbxchlOp WmbA1v2K/ULC+/SH18q5aoCG2/JOGp4Zx0KsABx6I91g8aB/reif3bjTaHd29azX XvAp7e10QLPNS3W7SJl4fd84OZrvh28iml+yNhd9jaFOEfHrBPk/NUngWTAGKwYR tbjo+1sag77E1aDc1vbnQ7q88534Xi/CG462CqBrtAY4Ox7yshnY5nwTJYzEJxky dKbScWW60ENXcWuVNq8VS2+HatP/z2ejDsYRBTB8wBcWSaXCtr3oRpg+LSArtdy2 iCjTK87uUcpRen2modvj47i3N3dwA587YGxv/2koZIR3agYXnsY+kLlpV6rhwb1i xqxxuV5QsooX/c0L+++4CRT1SVi/id9yYDRQCx/zXaIexRqN
=SC3w
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hello Andrey

Does the same happen when you run the test in the source tree manually?

I do not know, I am in the process to build the package in sbuild so I just try to fix the build process.
If I have hard code the lib path the test are running failling for other missing modules, but this is ok.

so my question is more how can I obtain the cmake default build path from the rules file ?

cheers

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Tue, Aug 30, 2022 at 05:26:01PM +0200, PICCA Frederic-Emmanuel wrote:

Hello Andrey

Does the same happen when you run the test in the source tree manually?

I do not know

When trying to run tests you should look how does the upstream intend to
run them.


--
WBR, wRAR

-----BEGIN PGP SIGNATURE-----

iQJhBAABCgBLFiEEolIP6gqGcKZh3YxVM2L3AxpJkuEFAmMOMwktFIAAAAAAFQAP cGthLWFkZHJlc3NAZ251cGcub3Jnd3JhckBkZWJpYW4ub3JnAAoJEDNi9wMaSZLh 9WQP/j09iRzSXxX6+/ClxXq4obFOWBVg2rBretPsiLzj9SQXVvNFLB3HQ8Rop9Vi diPKxQvf0yJS6yorOxIBZgHa/huzhKyIF5okKNJ4QnfdwiE3XrIzaYC7jXO96QOh 97cMGgny1n+9EZeyL0ddCQA9kuUbF9/55njt7IZC05N/m5iC8M2vmjY/prssevlX unnK0AK92mDZVjILtbn0JoFr9mXSnxIAtLy+cpNrLQCaEr2LqHMB7HueaaD4ZsRl tKiU/UvRahs6y4C19igPamlTRVjcVPRlsrJNtBtpwmcMFrj3CSv7m9x/z+bTvf45 843wDIT2Tmjm88dg5IuF4iMtGrAO/4XERWk69K+fw3IvqkfvVvo44M1742JfOHCM OW+n2rToXq5jolJpKfl119dM6LKbblgvIBypjC3fUOIzaBKhAo2z9pYsB1ORvsKG c9gAbkFl8rzhxixXCew2cGb1gj4n4Jii1R77d3FQe6+/KA26erQrgSFoky1x9VWz sh8RNYDan7az+qxIPQ+yG9fM2CMtiS2fEMeI5tGHyQBjMUp6bb3L/CEazn3E9keb 5bJylolkRAqNl6Io6R4Ml63pl0fbFWrCYXSAFIcJKO0izZAru6n3a5Keohectxbm LwEm+lDNxfNLGEuPKRu9xUMK6SKDIgDWo1vBxWjth+edTnM1
=skxr
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

When trying to run tests you should look how does the upstream intend to

run them.


Yes they are building inplace the module like this

from https://github.com/dials/dxtbx/blob/main/.azure-pipelines/unix-build.yml


# Build dxtbx
- bash: |
set -e
. conda_base/bin/activate
set -ux
mkdir build
cd build
cmake ../modules/dxtbx
cmake --build . --target install
pip install ../modules/dxtbx
displayName: Build dxtbx
workingDirectory: $(Pipeline.Workspace)

then

# Finally, run the full regression test suite
- bash: |
set -e
. conda_base/bin/activate
set -ux
export DIALS_DATA=${PWD}/data
cd modules/dxtbx
export PYTHONDEVMODE=1
pytest -v -ra -n auto --basetemp="$(Pipeline.Workspace)/tests" --durations=10 \
--cov=dxtbx --cov-report=html --cov-report=xml --cov-branch \
--timeout=5400 --regression || echo "##vso[task.complete result=Failed;]Some tests failed"
displayName: Run tests
workingDirectory: $(Pipeline.Workspace)


--
WBR, wRAR

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Tue, Aug 30, 2022 at 06:02:01PM +0200, PICCA Frederic-Emmanuel wrote:

When trying to run tests you should look how does the upstream intend to

run them.

Yes they are building inplace the module like this

from https://github.com/dials/dxtbx/blob/main/.azure-pipelines/unix-build.yml

# Build dxtbx
- bash: |
set -e
. conda_base/bin/activate
set -ux
mkdir build
cd build
cmake ../modules/dxtbx
cmake --build . --target install
pip install ../modules/dxtbx

Oh. In this case setting PYTHONPATH (if it works, and I'm not 100% sure it will) sounds like a better option.
So, the cmake build path is, as far as I know, defined in /usr/share/perl5/Debian/Debhelper/Buildsystem.pm to be just "obj-$DEB_HOST_GNU_TYPE".


--
WBR, wRAR

-----BEGIN PGP SIGNATURE-----

iQJhBAABCgBLFiEEolIP6gqGcKZh3YxVM2L3AxpJkuEFAmMOQUEtFIAAAAAAFQAP cGthLWFkZHJlc3NAZ251cGcub3Jnd3JhckBkZWJpYW4ub3JnAAoJEDNi9wMaSZLh 9v0P/1GvBxvg1S525JPcdkpdkwKF/P/l4W0u9zosLxUyNLz4UJl9HbggZ+zIAztk eB0MW19vmQc4RwbQnVOr2JjjtrLULhNxPfm2Q8iiP3XvobCYC0erd+1y2bFQ01QR Qsddbd3X+KswyVxYYs3YXEz7CIzGDkeCuLzPy6/rm3xS/jw2oLSzPkLAS3C6FaPy PeOjQRWaR0ieYZ+P7VO4L44bBqyyotoXAyO09O2ZkQ72CU7h/RMgSy4M8OH463G8 vbQsLLWQ/STPsnK53aNpOy1YU7KSDsNY4ejKSjEKiICtGQH1wL0EVIyL6J1T4GGG fQ6omsS6j75Z06ppYZkRRSn4H4/W0BQ/tDv/XJKWRud/rC7eS/R9tTAzaksSnU/g yAhJqfVCFNTkt+w8pcXQBlJCPgwH+ql6VEEnDgbF9feXK34oRk4DIEFPzCDZP9v8 zwUa15/ojnf+Tw85lmZk3cX9LGDdbmEddFWXF7Dy7GySq+v7k730+VnFQy5Mlh0/ 9pfWjE7MyAbeKd2bOGPm/x7vuGhE4aNicw4gyXPVTBDP6YlBpgkztLc876jeetAC zb/Uv1WA78kaJgZQu+7cZUdtLoJSLCAHTSKRtGN4U3muLlYKIsnYImi1/pvCTKf8 dkl66pTYCgNvNjGwEDINuUAmkejEuSummRK4KO3PVNmzPRYv
=74pP
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

/usr/share/perl5/Debian/Debhelper/Buildsystem.pm to be just &#034;obj-$DEB_HOST_GNU_TYPE&#034;.

Thanks for the info, if I an not wrong during the build process we can setup a new builddir.
So is is possible to obtain the real builddir during the build ?

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Oh. In this case setting PYTHONPATH (if it works, and I&#039;m not 100% sure it
will) sounds like a better option.
So, the cmake build path is, as far as I know, defined in /usr/share/perl5/Debian/Debhelper/Buildsystem.pm to be just obj-$DEB_HOST_GNU_TYPE&#034;.

thanks for the info. It seems to me that the builddir can be set directly in the ruels file with --builddir=<something>
So do you know a way to extract the real builddir during the build process ?

Cheers

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Tue, 30 Aug 2022 19:02:24 +0200 (CEST)
PICCA Frederic-Emmanuel <frederic-emmanuel.picca@synchrotron-soleil.fr>
wrote:

/usr/share/perl5/Debian/Debhelper/Buildsystem.pm to be just &#034;obj-$DEB_HOST_GNU_TYPE&#034;.

Thanks for the info, if I an not wrong during the build process we
can setup a new builddir. So is is possible to obtain the real
builddir during the build ?

PYTHONPATH can be a relative path, there is no need to try to identify
the full build path (that can introduce reproducibility issues). No
need to force a change to the build dir either.

So just obj-$DEB_HOST_GNU_TYPE

--
Neil Williams
linux@codehelp.co.uk
https://linux.codehelp.co.uk

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEf3HB6ceOc10DYMbM8WfkPIFDtoIFAmMOUNcACgkQ8WfkPIFD toKPhA//XnN7Kr6QIyDjKc+0tfyRXzKjY1uCLWp/I9BguDtR6oUa3ulHqbEGLkiy pNTGW4KcG2b/UYcOShu91EbNbot6y/wZEXswy1RrKAbgt/Cs4j76GyQAC87G6+9T 5o+RZcohORxClMF76hV65R5lIQ1sa9l3548z/6Vtg0S38U7xdUC5hoMrP+28gR/D rLu3VdPKRYr2UD9f00s56XvPoNvV0ewt0DYYl0BSMIUxSY8zfPSNU3yFYgyb3twX es7KNmBo5ytlg9aluUC7Mfntufkh1nw1DNLunssQJ+N9hno0KHS1+IbtQPZZr5q/ 2ETORNH0Jh5C6LPVg6CAUFRWOdMuVSg+S/IOLQZn6ACvY2dzZwnhqLUKS5w7GC3A Ltyo5+K2CMpazoLJobrv2g/KWP003wMZ/HlEBWYKDm2ukfRReb4VkBrU0AcYolbp JPt/0Yg4NQwA9eKdrEFb+W/HhAaQi0cwDTVG1FQ/OqIddrNEGxW7X/THDJTyuIuU 1xv98cuZa2jUML6HbWg5M+fGPfo9kUxfD+BmI8pycsRwwjUXU9Nzw/SDxnI2Va7q TwkbuEEQr+6B4hVKa1QYrYWdnAHoRtmcbMbCxojb90doP8HmfUJtWlyaCHKqkmG4 l1QjE4SIqtRUXbyzZMGFNAnDaagFrsfK0aMtiE8GnBrsphDBQCI=
=TDN4
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi PICCA (2022.08.30_16:02:01_+0000)

cd build
cmake ../modules/dxtbx
cmake --build . --target install
pip install ../modules/dxtbx

This package essentially has 2 build systems, camke for the C++ stuff
and setuptools for the pure-python. I don't see any integration between
the two (which is quite annoying, the upstream really should fix that).

pybuild knows how to drive distutils, and knows how to drive cmake, but
not both together. Ideally, you do want to use pybuild when building
Python extensions, so you can do them for all supported Python versions.

So, can I suggest something like this?

#! /usr/bin/make -f

export PYBUILD_NAME=dxtbx
export PYBUILD_SYSTEM=distutils
export PYBUILD_AFTER_CONFIGURE=cmake -DPython_EXECUTABLE=/usr/bin/{interpreter} -S . -B {build_dir}/lib
export PYBUILD_AFTER_BUILD=make -C {build_dir}/lib
export PYBUILD_BEFORE_TEST=cp {build_dir}/lib/lib/*.so {build_dir}

%:
dh $@ --with python3 --buildsystem=pybuild

You'll need something for the install too, but I didn't get that far, as
the tests fail (due to missing dependencies in Debian).

SR

--
Stefano Rivera
http://tumbleweed.org.za/
+1 415 683 3272

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hello Stephano

I end up with this

#! /usr/bin/make -f

export DH_VERSOBE=1

export PYBUILD_NAME=dxtbx
export PYBUILD_SYSTEM=distutils
export PYBUILD_AFTER_CONFIGURE=cmake -DPython_EXECUTABLE=/usr/bin/{interpreter} -S . -B {build_dir}/lib
export PYBUILD_AFTER_BUILD=make -C {build_dir}/lib
export PYBUILD_BEFORE_TEST=cp {build_dir}/lib/lib/*.so {build_dir}
export PYBUILD_BEFORE_INSTALL=cp {build_dir}/lib/lib/*.so {build_dir}; rm -rf {build_dir}/lib

%:
dh $@ --with python3 --buildsystem=pybuild

override_dh_auto_test:

it seem to work but I am am not really confident with the 'rm -rf {build_dir}/lib' during the install
If I do not remove these files, they got installed by pybuild.

no more magical obj-$blabla path :)


I now can concentrate on the unit test failures...

Cheers

Fred

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi PICCA (2022.08.31_09:05:43_+0000)

export PYBUILD_AFTER_BUILD=make -C {build_dir}/lib

I didn't do anything to support parallel, there. It may be nice to
(makes a huge difference to this package's build time).

export PYBUILD_BEFORE_TEST=cp {build_dir}/lib/lib/*.so {build_dir}
export PYBUILD_BEFORE_INSTALL=cp {build_dir}/lib/lib/*.so {build_dir}; rm -rf {build_dir}/lib
it seem to work but I am am not really confident with the 'rm -rf {build_dir}/lib' during the install

I'd suggest an && instead of ; of course :)

And maybe logically, that should all happen in AFTER_BUILD...

If I do not remove these files, they got installed by pybuild.

What's actually happening there is that they're getting installed by setuptools. If we could install them into the right spot with cmake,
maybe we could reduce the copying and deleting. I didn't experiment with
that very much, just got to the point it worked...

SR

--
Stefano Rivera
http://tumbleweed.org.za/
+1 415 683 3272

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)