Hi,

The following is a little bit involved, but I hope can make the problem clear.

Using poetry I have written a dummy application which just uses to typer
to illustrate a possible interface design. The directory structure is a follows:

$ tree -P *.py
.
|-- dist
|-- stoat
| |-- hpc
| | |-- database.py
| | |-- group.py
| | |-- __init__.py
| | |-- main.py
| | |-- owner.py
| | `-- user.py
| |-- __init__.py
| |-- main.py
| `-- storage
| |-- database.py
| |-- group.py
| |-- __init__.py
| |-- main.py
| |-- owner.py
| |-- share.py
| `-- user.py
`-- tests
|-- __init__.py
`-- test_stoat.py

With in the poetry shell I can run the application successfully:

$ python stoat/main.py hpc user --help
Usage: main.py hpc user [OPTIONS] COMMAND [ARGS]...

manage HPC users

Options:
--help Show this message and exit.

Commands:
add add a user
remove remove a user

I then install this in a non-standard path (because the OS Python3 is
3.6.8) and can run the installed version successfully:

$ PYTHONPATH=/trinity/shared/zedat/lib/python3.9/site-packages python /trinity/shared/zedat/lib/python3.9/site-packages/stoat/main.py hpc user --help
Usage: main.py hpc user [OPTIONS] COMMAND [ARGS]...

manage HPC users

Options:
--help Show this message and exit.

Commands:
add add a user
remove remove a user

However, poetry creates a script 'stoat' from the entry

[tool.poetry.scripts]
stoat = "stoat.main:main"

in pyproject.toml, which looks like

#!/trinity/shared/easybuild/software/Python/3.9.6-GCCcore-11.2.0/bin/python3.9
# -*- coding: utf-8 -*-
import re
import sys
from stoat.main import main
if __name__ == '__main__':
sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
sys.exit(main())

If I run that I get

$ PYTHONPATH=/trinity/shared/zedat/lib/python3.9/site-packages stoat hpc user --help
Traceback (most recent call last):
File "/trinity/shared/zedat/bin/stoat", line 5, in <module>
from stoat.main import main
File "/trinity/shared/zedat/lib/python3.9/site-packages/stoat/main.py", line 3, in <module>
import hpc.main
ModuleNotFoundError: No module named 'hpc'

Why is the module 'hpc' not found by the poetry script?

Cheers,

Loris

--
This signature is currently under construction.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

"Loris Bennett" <loris.bennett@fu-berlin.de> writes:

Hi,

The following is a little bit involved, but I hope can make the problem clear.

Using poetry I have written a dummy application which just uses to typer
to illustrate a possible interface design. The directory structure is a follows:

$ tree -P *.py
.
|-- dist
|-- stoat
| |-- hpc
| | |-- database.py
| | |-- group.py
| | |-- __init__.py
| | |-- main.py
| | |-- owner.py
| | `-- user.py
| |-- __init__.py
| |-- main.py
| `-- storage
| |-- database.py
| |-- group.py
| |-- __init__.py
| |-- main.py
| |-- owner.py
| |-- share.py
| `-- user.py
`-- tests
|-- __init__.py
`-- test_stoat.py

With in the poetry shell I can run the application successfully:

$ python stoat/main.py hpc user --help
Usage: main.py hpc user [OPTIONS] COMMAND [ARGS]...

manage HPC users

Options:
--help Show this message and exit.

Commands:
add add a user
remove remove a user

I then install this in a non-standard path (because the OS Python3 is
3.6.8) and can run the installed version successfully:

$ PYTHONPATH=/trinity/shared/zedat/lib/python3.9/site-packages python /trinity/shared/zedat/lib/python3.9/site-packages/stoat/main.py hpc user --help
Usage: main.py hpc user [OPTIONS] COMMAND [ARGS]...

manage HPC users

Options:
--help Show this message and exit.

Commands:
add add a user
remove remove a user

However, poetry creates a script 'stoat' from the entry

[tool.poetry.scripts]
stoat = "stoat.main:main"

in pyproject.toml, which looks like

#!/trinity/shared/easybuild/software/Python/3.9.6-GCCcore-11.2.0/bin/python3.9
# -*- coding: utf-8 -*-
import re
import sys
from stoat.main import main
if __name__ == '__main__':
sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
sys.exit(main())

If I run that I get

$ PYTHONPATH=/trinity/shared/zedat/lib/python3.9/site-packages stoat hpc user --help
Traceback (most recent call last):
File "/trinity/shared/zedat/bin/stoat", line 5, in <module>
from stoat.main import main
File "/trinity/shared/zedat/lib/python3.9/site-packages/stoat/main.py", line 3, in <module>
import hpc.main
ModuleNotFoundError: No module named 'hpc'

Why is the module 'hpc' not found by the poetry script?

Never mind, I worked it out. I had to replace

import hpc.main

with

import stoat.hpc.main

However, this raises the question of why it worked in the first place
in the poetry shell.

Cheers,

Loris

--
This signature is currently under construction.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

"Loris Bennett" <loris.bennett@fu-berlin.de> writes:

However, this raises the question of why it worked in the first place
in the poetry shell.

It might have had a different or extended sys.path.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

ram@zedat.fu-berlin.de (Stefan Ram) writes:

"Loris Bennett" <loris.bennett@fu-berlin.de> writes:

However, this raises the question of why it worked in the first place
in the poetry shell.

It might have had a different or extended sys.path.

In the poetry shell sys.path has this additional path

/home/loris/gitlab/stoat

The module not being found was

/home/gitlab/stoat/stoat/hpc/main.py

But if I run

[~/gitlab/stoat] $ python stoat/main.py hpc user --help

wouldn't

import hpc.main

still fail? Or is it because I am calling

stoat/main.py

and so Python looks for 'hpc' relative to the second 'stoat' directory?

Confused,

loris

--
This signature is currently under construction.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

"Loris Bennett" <loris.bennett@fu-berlin.de> writes:

Why is the module 'hpc' not found by the poetry script?

I have tried to execute the following sequence of shell
commands to understand your problem. Here they all worked
without error messages. Warning: Some of these commands might
alter your directories or data, so only execute them if you
are aware of their meaning/consequences and agree with them!

mkdir stoat
mkdir stoat/hpc
echo import hpc.main >stoat/main.py
echo >stoat/hpc/main.py
python3 stoat/main.py
cd stoat
python3 main.py

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

"Loris Bennett" <loris.bennett@fu-berlin.de> writes:

However, in the development environment, if I run
python stoat/main.py hpc user --help
then is
stoat/hpc/main.py
being found via
import hpc.main
because Python looks in
stoat
as the parent directory of
stoat/main.py
rather than the current working directory?

When you run "stoat/main.py", the directory of that file
"main.py" is automatically added at the front of sys.path.
(This also would happen after "cd stoat; python main.py".)

Then, when "import hpc.main" is being executed, the system
will search for "hpc/main.py" in every entry of sys.path
and will use the first entry wherein it finds "hpc/main.py".
So it will use the directory of the file "stoat/main.py",
i.e., the directory "stoat". It finds "stoat/hpc/main.py".

You can call "python" with "-v", and it will show some lines
with information about the imports executed, including the
directories used (albeit hidden in a lot of other lines).

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi Stefan,

ram@zedat.fu-berlin.de (Stefan Ram) writes:

"Loris Bennett" <loris.bennett@fu-berlin.de> writes:

Why is the module 'hpc' not found by the poetry script?

I have tried to execute the following sequence of shell
commands to understand your problem. Here they all worked
without error messages. Warning: Some of these commands might
alter your directories or data, so only execute them if you
are aware of their meaning/consequences and agree with them!

I do know what the commands do. However, just to be on the safe side,
but mainly in honour of a classic dad-joke, I changed 'stoat' to
'weasel'.

mkdir stoat
mkdir stoat/hpc
echo import hpc.main >stoat/main.py
echo >stoat/hpc/main.py
python3 stoat/main.py
cd stoat
python3 main.py

I guess with the production version, the package

stoat

is visible in sys.path and thus the subpackages have to be referred to
via the main package, e.g.

import stoat.hpc.main

However, in the development environment, if I run

python stoat/main.py hpc user --help

then is

stoat/hpc/main.py

being found via

import hpc.main

because Python looks in

stoat

as the parent directory of

stoat/main.py

rather than the current working directory? That doesn't seem likely to
me, but I am already confused.

Cheers,

Loris

--
This signature is currently under construction.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

ram@zedat.fu-berlin.de (Stefan Ram) writes:

"Loris Bennett" <loris.bennett@fu-berlin.de> writes:

However, in the development environment, if I run
python stoat/main.py hpc user --help
then is
stoat/hpc/main.py
being found via
import hpc.main
because Python looks in
stoat
as the parent directory of
stoat/main.py
rather than the current working directory?

When you run "stoat/main.py", the directory of that file
"main.py" is automatically added at the front of sys.path.
(This also would happen after "cd stoat; python main.py".)

OK, that explains it. I initially found that a bit odd, but thinking
about it I see that "the directory containing the file being run" is a
more sensible reference point than the current working directory, which
is totally arbitrary.

Then, when "import hpc.main" is being executed, the system
will search for "hpc/main.py" in every entry of sys.path
and will use the first entry wherein it finds "hpc/main.py".
So it will use the directory of the file "stoat/main.py",
i.e., the directory "stoat". It finds "stoat/hpc/main.py".

You can call "python" with "-v", and it will show some lines
with information about the imports executed, including the
directories used (albeit hidden in a lot of other lines).

That's useful, although in the production case I would have to tweak the
script generated by poetry.

Cheers,

Loris

--
This signature is currently under construction.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)