1. Forum
  2. Usenet
  3. LINUX.DEBIAN.MAINT.PYTHON

  • Re: Installing data files with pybuild

    From Andrey Rahmatullin@21:1/5 to Julian Gilbey on Wed Dec 1 18:30:02 2021
    On Wed, Dec 01, 2021 at 04:57:44PM +0000, Julian Gilbey wrote:
    Hello!

    pybuild is magic. It knows where the build directory is, despite it seemingly calling setup.py with no command line arguments specifying
    it!

    But anyway, my actual question is this. How do I ensure that the data
    files are also copied into the package?
    Why do you need this? Does setup.py not install them?

    Only the .py files are currently included in the build; what is the
    best way to include all of the data files after the build step and
    before the test step, and then to ensure they are included in the
    final package?
    Apart from fixing setup.py? execute_after_dh_auto_install I guess.

    --
    WBR, wRAR

    -----BEGIN PGP SIGNATURE-----

    iQJhBAABCgBLFiEEolIP6gqGcKZh3YxVM2L3AxpJkuEFAmGnsLctFIAAAAAAFQAP cGthLWFkZHJlc3NAZ251cGcub3Jnd3JhckBkZWJpYW4ub3JnAAoJEDNi9wMaSZLh QrYP/AxzEup8iHYlL7cCcn+g+OidM+uYQqBnMGAElGUgScJ3IK9BmLkzkUYOawAk n2+6iWDYOUapcPQnGi56OIkhjwtfbql8XYxOhHpe4z0RCFkeLyjUkNYVnCcHQG9m 3nAM7ZRXrTUx8+dc40QNhmTxqXcyKBeW7HPvKewh2pGq4twOTyTkYrHtDJx9OQ2q /XzMEgPKuJl9P3BGVs6l0EBbZa7v0JN33AhqEhzFG643z5ruBud928QyySnfr8O/ 5c+CGhE6EI68q7bVWsT/8aJqeAVhg3oWmAOu9tTnyimEHIw3dn1uZZArmKp+Qlnx 0pzfpqapeYrUODyXQ99F3ejwm55wll/c/rg78s5j9rrWqYHn6HoIZCOAKksi1HM1 DzG3vqOkvtQW7WkXU/gPsM1/iVmvtbbKcl1PffJRW5LS91a64QGtyfRoZHN2CJUe W1QVi52cg6Fj0/0caB4nkqwD8jplPck3TqCTnxp+n8BGLwVaKADBtbRlFHTH2Xjw SwcefV2REFxCrLTwHFtD/0yyq8cZzVzHbnOoMT3po21UEm2DF6QNepGBQmTxD5Be Pkf5pu05RRwNv9k0mnC8MN8KsQpmkrl5vLLWcVFp/BJufQufIoZiRNCU9lEDCKy7 mpsiWgIzsqfMD5JqJo8/Uw0D+yY1+hsfRqLIg6/EnYcWCszz
    =jH3s
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Julian Gilbey@21:1/5 to All on Wed Dec 1 18:30:02 2021
    Hello!

    pybuild is magic. It knows where the build directory is, despite it
    seemingly calling setup.py with no command line arguments specifying
    it!

    But anyway, my actual question is this. How do I ensure that the data
    files are also copied into the package? My package has a source tree
    that looks roughly like this:

    setup.py
    setup.cfg
    MANIFEST.in
    ...
    debian/
    mypkg/
    __init__.py
    module1.py
    imgs/
    img1.png
    img2.png
    data/
    data.csv
    data2.txt
    module2/
    __init__.py
    foo.py
    module2_data.txt

    Only the .py files are currently included in the build; what is the
    best way to include all of the data files after the build step and
    before the test step, and then to ensure they are included in the
    final package?

    Thanks!

    Julian

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From =?UTF-8?Q?Louis-Philippe_V=c3=a9ron@21:1/5 to Andrey Rahmatullin on Wed Dec 1 19:40:03 2021
    This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------gPHvoWf0iPUgOScfhJmn4SpR
    Content-Type: text/plain; charset=UTF-8
    Content-Transfer-Encoding: quoted-printable

    On 2021-12-01 12 h 28, Andrey Rahmatullin wrote:

    Only the .py files are currently included in the build; what is the
    best way to include all of the data files after the build step and
    before the test step, and then to ensure they are included in the
    final package?
    Apart from fixing setup.py? execute_after_dh_auto_install I guess.

    Or you can use debian/foobar.install to install the missing files in the
    right location, and keep your d/rules file cleaner :)

    ex. (man dh_install):

    https://sources.debian.org/src/sublime-music/0.11.16-1/debian/sublime-music.install/

    --
    ⢀⣴⠾⠻⢶⣦⠀
    ⣾⠁⢠⠒⠀⣿⡁ Louis-Philippe Véronneau
    ⢿⡄⠘⠷⠚⠋ pollo@debian.org / veronneau.org
    ⠈⠳⣄

    --------------gPHvoWf0iPUgOScfhJmn4SpR--

    -----BEGIN PGP SIGNATURE-----

    iHUEARYKAB0WIQTKp0AHB6gWsCAvw830JXpQshz6hQUCYae/xgAKCRD0JXpQshz6 hUVtAPwNRdbwl9p3ZKG+vcSQhw7YGAwPZS17rfh87vHB1oPAVwD+LgiOVxUOwmgT D1uR/8yoLgzvkwRZqTL3Qma1BYf7HwU=
    =hWqo
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Julian Gilbey@21:1/5 to All on Wed Dec 1 22:50:01 2021
    On Wed, Dec 01, 2021 at 01:32:38PM -0500, Louis-Philippe Vronneau wrote:
    On 2021-12-01 12 h 28, Andrey Rahmatullin wrote:

    Only the .py files are currently included in the build; what is the
    best way to include all of the data files after the build step and
    before the test step, and then to ensure they are included in the
    final package?
    Apart from fixing setup.py? execute_after_dh_auto_install I guess.

    Or you can use debian/foobar.install to install the missing files in the right location, and keep your d/rules file cleaner :)

    ex. (man dh_install):

    https://sources.debian.org/src/sublime-music/0.11.16-1/debian/sublime-music.install/

    Thanks Andrey and Louis-Philippe!

    Yes, that might work, though the build-time tests might still fail as
    I think they expect these files to be present at test time. Since
    pybuild uses the library pre-installation to perform the build-time
    tests, there might be a problem here.

    I don't understand why the files are not correctly installed by
    setup.py; it has set include_package_data=True, which includes the
    files (and file patterns) listed in MANIFEST.in, but it seems that
    this is only used for the bdist and sdist targets of setup.py, but not
    for the install target.

    There is another possibility, which is to use the --after-build
    pybuild option or the PYBUILD_AFTER_BUILD environment variable. There
    is a discussion of this in the scikit-image package here: https://github.com/scikit-image/scikit-image/issues/4366 which links
    to the actual code in debian/rules: https://salsa.debian.org/science-team/scikit-learn/blob/37a7b4eda8b15f514c6d0f6f801c2d67e35e36c7/debian/rules#L77-84
    But I was wondering whether there is a "nicer" way to do it.

    Thanks!

    Julian

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Stefano Rivera@21:1/5 to All on Thu Dec 2 02:30:01 2021
    Hi Julian (2021.12.01_21:47:38_+0000)
    I don't understand why the files are not correctly installed by
    setup.py; it has set include_package_data=True, which includes the
    files (and file patterns) listed in MANIFEST.in, but it seems that
    this is only used for the bdist and sdist targets of setup.py, but not
    for the install target.

    Correct, the package_data key needs to be set in setup.py / cfg to
    install data in bdist_wheels and install: https://setuptools.pypa.io/en/latest/userguide/datafiles.html

    I would suggest forwarding a patch upstream that does that.

    SR

    --
    Stefano Rivera
    http://tumbleweed.org.za/
    +1 415 683 3272

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)