• Re: load after package require

    From Conor Williams@21:1/5 to Oleg O. Nemanov on Thu Oct 14 10:53:36 2021
    comrade -- a basic kestion...
    dot is not in your LD_LIBRARY_PATH
    export LD_LIBRARY_PATH=.:$LD_LIBRARY_PATH
    .profile .bashrc .tclsh etc etc almost ad infinium mate
    for most shells
    or in your autoexec.bat on dos
    or maybe graphically...
    rc shell on plan 9 is similar
    /c:2021141018:02002
    On Thursday, October 14, 2021 at 5:35:52 PM UTC, Oleg O. Nemanov wrote:
    Hi, all.

    load right after tclsh is started(libmy.so is in current directory):
    % load libmy.so
    %

    load after package require:
    % package require test
    can't find package test
    % load libmy.so
    couldn't load file "libmy.so": libmy.so: cannot open shared object file: No such file or directory
    % load ./libmy.so
    %

    Why this is happen?

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Oleg O. Nemanov@21:1/5 to All on Thu Oct 14 10:35:50 2021
    Hi, all.

    load right after tclsh is started(libmy.so is in current directory):
    % load libmy.so
    %

    load after package require:
    % package require test
    can't find package test
    % load libmy.so
    couldn't load file "libmy.so": libmy.so: cannot open shared object file: No such file or directory
    % load ./libmy.so
    %

    Why this is happen?

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Mike Griffiths@21:1/5 to Oleg O. Nemanov on Thu Oct 14 22:28:26 2021
    On Thursday, 14 October 2021 at 18:35:52 UTC+1, Oleg O. Nemanov wrote:
    Hi, all.

    load right after tclsh is started(libmy.so is in current directory):
    % load libmy.so
    %

    load after package require:
    % package require test
    can't find package test
    % load libmy.so
    couldn't load file "libmy.so": libmy.so: cannot open shared object file: No such file or directory
    % load ./libmy.so
    %

    Why this is happen?
    Check your current working directory (via [pwd]) at each stage; it may be that a package is changing it?

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Oleg O. Nemanov@21:1/5 to All on Fri Oct 15 00:28:25 2021
    пятница, 15 октября 2021 г. в 08:28:28 UTC+3, Mike Griffiths:
    Check your current working directory (via [pwd]) at each stage; it may be that a package is changing it?

    I didn't show [pwd] results in my example, but i done it - a current working directory is not changed.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Uwe Klein@21:1/5 to All on Fri Oct 15 09:50:43 2021
    Am 15.10.21 um 09:26 schrieb Oleg O. Nemanov:
    четверг, 14 октября 2021 г. в 20:53:38 UTC+3, conor.w...@gmail.com:
    comrade -- a basic kestion...
    dot is not in your LD_LIBRARY_PATH
    export LD_LIBRARY_PATH=.:$LD_LIBRARY_PATH

    This helps, but why loading works without manipulation with
    LD_LIBRARY_PATH before package require command?

    IMU:
    loading a lib from inside the interpreter does not look at LD_LIBRARY_PATH.

    man n load ( or Windows equivalent. M$ seems to use some heuristics ..)

    best practice probably is handing a fully resolved path/filename
    to [load].

    semi OT:
    there is a search path defined for the [package require ... ] mechanism.

    Uwe

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Oleg O. Nemanov@21:1/5 to All on Fri Oct 15 00:26:28 2021
    четверг, 14 октября 2021 г. в 20:53:38 UTC+3, conor.w...@gmail.com:
    comrade -- a basic kestion...
    dot is not in your LD_LIBRARY_PATH
    export LD_LIBRARY_PATH=.:$LD_LIBRARY_PATH

    This helps, but why loading works without manipulation with
    LD_LIBRARY_PATH before package require command?

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Harald Oehlmann@21:1/5 to All on Fri Oct 15 10:18:30 2021
    Am 14.10.2021 um 19:35 schrieb Oleg O. Nemanov:
    Hi, all.

    load right after tclsh is started(libmy.so is in current directory):
    % load libmy.so
    %

    load after package require:
    % package require test
    can't find package test
    % load libmy.so
    couldn't load file "libmy.so": libmy.so: cannot open shared object file: No such file or directory
    % load ./libmy.so
    %

    Why this is happen?


    Dear Oleg,

    may I allow to give my opinion? I am not sure of the search path on
    Linux to find the library.
    I can only tell that in the TCL world, it is common practice to load a
    DLL with a full path.
    This path is normally generated within the pckIndex.tcl mecahnism.

    Typically, you have a pckIndex.tcl file in a folder or subfolder listed
    by the TCL variable "auto_path".

    This file typically is in the same folder as your library and contains
    the line:
    package ifneeded my 1.0.0 "[list load [file join $dir libmy.so]]"

    Remark that the "dir variable contains the path of the current folder.
    So, you can relocate that and it still works.

    This works on the harddisk, in starkits and hopefully also within the
    TCL 8.7 zip file.

    If you don't like the pckIndex mechanism, you may load it in the
    following way from a TCL script in the same folder:
    source [file join [file dirname [info script]] libmy.so]

    Take care,
    Harald

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Oleg O. Nemanov@21:1/5 to All on Fri Oct 15 03:03:33 2021
    пятница, 15 октября 2021 г. в 11:18:33 UTC+3, Harald Oehlmann:
    If you don't like the pckIndex mechanism, you may load it in the
    following way from a TCL script in the same folder:
    source [file join [file dirname [info script]] libmy.so]

    No-no. I like pkgIndex mechanism. I only noted an inconsistency
    in load behaviour before and after package require call.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Oleg O. Nemanov@21:1/5 to All on Fri Oct 15 02:55:41 2021
    пятница, 15 октября 2021 г. в 10:50:47 UTC+3, Uwe Klein:
    IMU:
    loading a lib from inside the interpreter does not look at LD_LIBRARY_PATH.

    If so why then LD_LIBRARY_PATH manipulation helps?

    man n load ( or Windows equivalent. M$ seems to use some heuristics ..)

    I can't found any info related to a lib search path in load man page.

    semi OT:
    there is a search path defined for the [package require ... ] mechanism.

    Do you talk about $auto_path?

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Ralf Fassel@21:1/5 to All on Fri Oct 15 11:15:44 2021
    * "Oleg O. Nemanov" <oleg.o.nemanov@gmail.com>
    | четверг, 14 октября 2021 г. в 20:53:38 UTC+3, conor.w...@gmail.com:
    | > comrade -- a basic kestion...
    | > dot is not in your LD_LIBRARY_PATH
    | > export LD_LIBRARY_PATH=.:$LD_LIBRARY_PATH

    | This helps, but why loading works without manipulation with
    | LD_LIBRARY_PATH before package require command?

    Never ever put "." in LD_LIBRARY_PATH, this opens up all kinds of
    security and other problems.

    As others have said, use the full path name for loading. Also, on
    Linux, the .so needs to have proper execute permissions, but I think
    you've checked that.

    HTH
    R'

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Oleg O. Nemanov@21:1/5 to All on Fri Oct 15 03:04:47 2021
    пятница, 15 октября 2021 г. в 12:15:48 UTC+3, Ralf Fassel:
    Never ever put "." in LD_LIBRARY_PATH, this opens up all kinds of
    security and other problems.

    Yeah. I know this. Thanks.

    As others have said, use the full path name for loading. Also, on
    Linux, the .so needs to have proper execute permissions, but I think
    you've checked that.

    Yes. The permissions are right.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Oleg O. Nemanov@21:1/5 to All on Fri Oct 15 03:54:22 2021
    пятница, 15 октября 2021 г. в 13:24:02 UTC+3, Ralf Fassel:
    * "Oleg O. Nemanov" <oleg.o....@gmail.com>
    | No-no. I like pkgIndex mechanism. I only noted an inconsistency
    | in load behaviour before and after package require call.
    I would suggest to recompile TCL with debugging information and
    trace the calls to TclpDlopen() (unix/tclLoadDl.c) in a debugger.
    That way you could check which pathnames are tried in each situation.
    I can't see any obvious reason in the code for the behaviour you
    describe...

    Hm. It's funny, but in 8.6.11 version of tcl i can't reproduce this. Before i tried 8.6.9.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Ralf Fassel@21:1/5 to All on Fri Oct 15 12:23:57 2021
    * "Oleg O. Nemanov" <oleg.o.nemanov@gmail.com>
    | No-no. I like pkgIndex mechanism. I only noted an inconsistency
    | in load behaviour before and after package require call.

    I would suggest to recompile TCL with debugging information and
    trace the calls to TclpDlopen() (unix/tclLoadDl.c) in a debugger.
    That way you could check which pathnames are tried in each situation.
    I can't see any obvious reason in the code for the behaviour you
    describe...

    R'

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Oleg O. Nemanov@21:1/5 to All on Fri Oct 15 04:25:47 2021
    пятница, 15 октября 2021 г. в 13:54:24 UTC+3, Oleg O. Nemanov:
    пятница, 15 октября 2021 г. в 13:24:02 UTC+3, Ralf Fassel:
    * "Oleg O. Nemanov" <oleg.o....@gmail.com>
    | No-no. I like pkgIndex mechanism. I only noted an inconsistency
    | in load behaviour before and after package require call.
    I would suggest to recompile TCL with debugging information and
    trace the calls to TclpDlopen() (unix/tclLoadDl.c) in a debugger.
    That way you could check which pathnames are tried in each situation.
    I can't see any obvious reason in the code for the behaviour you describe...
    Hm. It's funny, but in 8.6.11 version of tcl i can't reproduce this. Before i
    tried 8.6.9.

    Sorry. The wrong info :-). In 8.6.11 the same stranges.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Schelte@21:1/5 to Oleg O. Nemanov on Fri Oct 15 14:38:49 2021
    On 15/10/2021 13:25, Oleg O. Nemanov wrote:
    Sorry. The wrong info :-). In 8.6.11 the same stranges.

    There must be some misbehaving package on your system. Executing
    [package require test] causes all pkgIndex.tcl files found under the directories in $auto_path to be executed. These files are not supposed
    to have any side-effects, other than updating the list of available
    packages via [package provide] commands. But apparently one of the files
    does something more.

    You can investigate by running [package names] before and after you do
    [package require test]. For each of the packages that newly appear, run [package versions <pkg>]. Then for each of the available versions of a suspected package, find the pkgIndex.tcl file and examine it.

    Or easier, run: exec find {*}$auto_path -name pkgIndex.tcl

    You can narrow down the culprit by removing directories from $auto_path
    before doing the [package require test]. If that solves the problem, the
    rogue package is below one of the directories you removed.


    Schelte.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Oleg Nemanov@21:1/5 to All on Fri Oct 15 07:38:59 2021
    пятница, 15 октября 2021 г. в 15:40:05 UTC+3, Schelte:
    On 15/10/2021 13:25, Oleg O. Nemanov wrote:
    Sorry. The wrong info :-). In 8.6.11 the same stranges.

    There must be some misbehaving package on your system. Executing
    [package require test] causes all pkgIndex.tcl files found under the directories in $auto_path to be executed. These files are not supposed
    to have any side-effects, other than updating the list of available
    packages via [package provide] commands. But apparently one of the files does something more.

    You can investigate by running [package names] before and after you do [package require test]. For each of the packages that newly appear, run [package versions <pkg>]. Then for each of the available versions of a suspected package, find the pkgIndex.tcl file and examine it.

    Before ```llength [package names]``` returns 4, after - 448 :-).

    You can narrow down the culprit by removing directories from $auto_path before doing the [package require test]. If that solves the problem, the rogue package is below one of the directories you removed.

    I set auto_path to only one empty dir and load behaviour isn't changed(before package require and after it load successfully load a lib in current dir).

    After that, i set auto_path to only /usr/lib/tcl8.6 and load behaviour is changed.
    But here only 2 pkgIndex.tcl files:

    /usr/lib/tcl8.6/http1.0/pkgIndex.tcl
    /usr/lib/tcl8.6/opt0.4/pkgIndex.tcl

    ~$ cat /usr/lib/tcl8.6/http1.0/pkgIndex.tcl
    # Tcl package index file, version 1.0
    # This file is generated by the "pkg_mkIndex" command
    # and sourced either when an application starts up or
    # by a "package unknown" script. It invokes the
    # "package ifneeded" command to set up package-related
    # information so that packages will be loaded automatically
    # in response to "package require" commands. When this
    # script is sourced, the variable $dir must contain the
    # full path name of this file's directory.

    package ifneeded http 1.0 [list tclPkgSetup $dir http 1.0 {{http.tcl source {httpCopyDone httpCopyStart httpEof httpEvent httpFinish httpMapReply httpProxyRequired http_code http_config http_data http_formatQuery http_get http_reset http_size http_status
    http_wait}}}]
    ~$ cat /usr/lib/tcl8.6/opt0.4/pkgIndex.tcl
    # Tcl package index file, version 1.1
    # This file is generated by the "pkg_mkIndex -direct" command
    # and sourced either when an application starts up or
    # by a "package unknown" script. It invokes the
    # "package ifneeded" command to set up package-related
    # information so that packages will be loaded automatically
    # in response to "package require" commands. When this
    # script is sourced, the variable $dir must contain the
    # full path name of this file's directory.

    if {![package vsatisfies [package provide Tcl] 8.2]} {return}
    package ifneeded opt 0.4.6 [list source [file join $dir optparse.tcl]]

    In http pkgIndex.tcl file i placed right after comments:

    puts "1"
    load libsetval.so

    In opt pkgIndex.tcl file i placed right after comments:

    puts "2"
    load libsetval.so

    After that i do:

    ~$ tclsh
    % set auto_path /usr/lib/tcl8.6
    /usr/lib/tcl8.6
    % package require test
    1
    error reading package index file /usr/lib/tcl8.6/http1.0/pkgIndex.tcl: couldn't load file "libsetval.so": libsetval.so: cannot open shared object file: No such file or directory
    2
    error reading package index file /usr/lib/tcl8.6/opt0.4/pkgIndex.tcl: couldn't load file "libsetval.so": libsetval.so: cannot open shared object file: No such file or directory
    can't find package test
    %

    I.e. load behaviour is changed before any statements in pkgIndex.tcl.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Ralf Fassel@21:1/5 to All on Fri Oct 15 17:37:56 2021
    * Oleg Nemanov <oleg.o.nemanov@gmail.com>
    | In http pkgIndex.tcl file i placed right after comments:

    | puts "1"
    | load libsetval.so

    Make that:

    puts "1: pwd {[pwd]} glob {[glob -nocomplain libsetval.so]}"
    load libsetval.so

    and also in 2, verify that pwd and the glob indeed indicate the expected
    dir and file existence.

    | I.e. load behaviour is changed before any statements in pkgIndex.tcl.

    Another possibility is that the 'load' command is replaced by something
    else by the 'package require'. So insert a printing version of load:

    rename load load_orig
    proc load {args} {
    puts stderr "load: $args"
    load_orig {*}$args
    }

    If after the 'package require' the printout is not there, 'load' has
    been replaced.

    R'

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Oleg Nemanov@21:1/5 to All on Fri Oct 15 08:55:05 2021
    пятница, 15 октября 2021 г. в 18:37:59 UTC+3, Ralf Fassel:
    ...[CUT]...
    Done:

    ~$ tclsh
    % set auto_path /usr/lib/tcl8.6
    /usr/lib/tcl8.6
    % rename load load_orig
    % proc load {args} { puts stderr "load: $args"; load_orig {*}$args; }
    % package require test
    1: pwd {/home/lego/work/libs/tcl/setval} glob {libsetval.so}
    load: libsetval.so
    error reading package index file /usr/lib/tcl8.6/http1.0/pkgIndex.tcl: couldn't load file "libsetval.so": libsetval.so: cannot open shared object file: No such file or directory
    2: pwd {/home/lego/work/libs/tcl/setval} glob {libsetval.so}
    load: libsetval.so
    error reading package index file /usr/lib/tcl8.6/opt0.4/pkgIndex.tcl: couldn't load file "libsetval.so": libsetval.so: cannot open shared object file: No such file or directory
    can't find package test
    % load libsetval.so
    load: libsetval.so
    couldn't load file "libsetval.so": libsetval.so: cannot open shared object file: No such file or directory

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Ralf Fassel@21:1/5 to All on Fri Oct 15 18:40:20 2021
    * Oleg Nemanov <oleg.o.nemanov@gmail.com>
    | ~$ tclsh
    | % set auto_path /usr/lib/tcl8.6
    | /usr/lib/tcl8.6
    | % rename load load_orig
    | % proc load {args} { puts stderr "load: $args"; load_orig {*}$args; }

    Obviously what's missing here is the

    load libsetval.so

    right before the

    package require test

    ;-)

    | % package require test
    | 1: pwd {/home/lego/work/libs/tcl/setval} glob {libsetval.so}
    | load: libsetval.so
    | error reading package index file /usr/lib/tcl8.6/http1.0/pkgIndex.tcl:
    | couldn't load file "libsetval.so": libsetval.so: cannot open shared
    | object file: No such file or directory
    | 2: pwd {/home/lego/work/libs/tcl/setval} glob {libsetval.so}
    | load: libsetval.so
    | error reading package index file /usr/lib/tcl8.6/opt0.4/pkgIndex.tcl:
    | couldn't load file "libsetval.so": libsetval.so: cannot open shared
    | object file: No such file or directory
    | can't find package test

    Ok, looks good, 'load' has obviously not been replaced and gives
    consistent results.

    | % load libsetval.so
    | load: libsetval.so
    | couldn't load file "libsetval.so": libsetval.so: cannot open shared object file: No such file or directory

    This is what I would have expected from the manpage of dlopen(): the
    filename does not contain a "/", so the default search path is applied
    which only includes the system defaults (and notably not "."), thus the
    library is not found. If you do "load ./libsetval.so" instead, it
    contains a "/" and so the name is used as-is.

    R'

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Schelte@21:1/5 to Oleg Nemanov on Fri Oct 15 20:01:19 2021
    On 15/10/2021 16:38, Oleg Nemanov wrote:
    I.e. load behaviour is changed before any statements in pkgIndex.tcl.

    I tried to reproduce your scenario and managed to easily do so.

    First observation: The initial load only works in an interactive
    session. Running the same commands from a file immediately throws the
    "couldn't load file" error.

    Executing [package require] for a package that is not yet known is
    handled by procs registered to [package unknown]. On my system, [package unknown] returns "::tcl::tm::UnknownHandler ::tclPkgUnknown". The first
    proc is for modules, the second for packages. Getting rid of the command
    for modules didn't change anything. Then I made my own simplified
    version of tclPkgUnknown. Reducing that step by step, I found that the
    changed behavior happens after running a [glob] command.

    So then I just executed a glob myself, instead of the package require.
    That also triggered the change in behavior:

    % glob -nocomplain foo
    % load libfswatch2.0.1.so
    couldn't load file "libfswatch2.0.1.so": libfswatch2.0.1.so: cannot open
    shared object file: No such file or directory

    The same happens with [pwd] and [cd]. It seems to me those commands
    initialize some filesystem access, after which [load] no longer looks in
    the current directory.

    This is curious. But since it already never works in a script, I don't
    think it is really anything to worry about.


    Schelte.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Oleg Nemanov@21:1/5 to All on Mon Oct 18 00:43:18 2021
    пятница, 15 октября 2021 г. в 21:02:04 UTC+3, Schelte:
    % glob -nocomplain foo
    % load libfswatch2.0.1.so
    couldn't load file "libfswatch2.0.1.so": libfswatch2.0.1.so: cannot open shared object file: No such file or directory
    The same happens with [pwd] and [cd]. It seems to me those commands initialize some filesystem access, after which [load] no longer looks in
    the current directory.

    This is curious. But since it already never works in a script, I don't
    think it is really anything to worry about.

    Ok :-). Thanks for your research and explanation!

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)