• ANNOUNCE: www 2.0 released

    From Schelte@21:1/5 to All on Tue Sep 7 21:22:41 2021
    www is a package for performing http requests. It is intended to be used
    in situations where the http package is too cumbersome to use (which is
    pretty much always) and tclcurl is overkill.

    The basic way to obtain a web resource using the www package is as follows:

    if {[catch {www get http://www.tcl.tk/} data info]} {
    puts stderr "Failed to get the web page"
    } else {
    puts $data
    }

    The package takes care of redirects, retries, cookies, and proxies and
    only uses the standard way of throwing errors. The command returns the
    document body. Metadata is available through return options.

    The package also provides extensions to handle WebSockets and HTTP/2.
    For HTTP/2 over TLS a patch is needed for TclTLS. I have filed a ticket
    with the patch to the TclTLS tracker (https://core.tcl-lang.org/tcltls/tktview?name=e1f9a21c67). Hopefully it
    will be picked up soon.

    For more information on www, please visit the chiselapp repo at https://chiselapp.com/user/schelte/repository/www


    Enjoy,
    Schelte.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Dave@21:1/5 to All on Tue Sep 7 16:17:42 2021
    All the symbolic links are playing havoc on windows.

    Using msys2 with the TAR file, I get a bunch of errors when doing 'tar
    -zxf':
    tar: www/www-2.0.tm: Cannot create symlink to 'www.tcl': No such file or directory

    Using 7z with the TAR file I get the errors:
    Can not create symbolic link : Acess is denied. : C:\users\<user>\AppData\Local\Temp\7zE069A5F1\www\www-2.0.tm

    Using 7z with the ZIP file creates a normal file with the symlink target
    as its contents.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Dave@21:1/5 to Schelte on Tue Sep 7 17:45:13 2021
    On 9/7/2021 5:23 PM, Schelte wrote:
    On 07/09/2021 23:17, Dave wrote:
    All the symbolic links are playing havoc on windows.

    Well, that's annoying. Then I'm unsure how to handle Tcl modules in a
    VCS. Each version of a TM will have a different file name. If that means
    you also have to add a new copy of the file to the repository for each version, then there's not much version control going on. What's the
    suggested way to handle this?

    Windows does support both hard (must be on same drive letter) or
    symlinks (win7 requires admin privilege, don't know about win10, etc.).

    Possibly an install script of some sort? With that you could avoid
    multiple copies of the same file in the fossil repository. The script
    could be very simple since, for www, there are not that many files
    affected and the extra disk space is irrelevant.


    --
    computerjock AT mail DOT com

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Schelte@21:1/5 to Dave on Wed Sep 8 00:23:07 2021
    On 07/09/2021 23:17, Dave wrote:
    All the symbolic links are playing havoc on windows.

    Well, that's annoying. Then I'm unsure how to handle Tcl modules in a
    VCS. Each version of a TM will have a different file name. If that means
    you also have to add a new copy of the file to the repository for each
    version, then there's not much version control going on. What's the
    suggested way to handle this?


    Schelte.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Rolf Ade@21:1/5 to Schelte on Wed Sep 8 01:43:05 2021
    Schelte <nospam@wanadoo.nl> writes:
    www is a package for performing http requests. It is intended to be
    used in situations where the http package is too cumbersome to use
    (which is pretty much always) and tclcurl is overkill.

    I wholeheartedly agree with "the http package is cumbersome to use".

    [...]
    For more information on www, please visit the chiselapp repo at https://chiselapp.com/user/schelte/repository/www

    Great read. Will give it a try. It may be the answer.

    Enjoy,

    I hope so.

    Schelte.

    rolf

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From EL@21:1/5 to Schelte on Wed Sep 8 04:52:47 2021
    Schelte <nospam@wanadoo.nl> wrote:
    On 07/09/2021 23:17, Dave wrote:
    All the symbolic links are playing havoc on windows.

    Well, that's annoying. Then I'm unsure how to handle Tcl modules in a
    VCS. Each version of a TM will have a different file name. If that means
    you also have to add a new copy of the file to the repository for each version, then there's not much version control going on. What's the
    suggested way to handle this?

    That’s the very reason because I avoid .tm modules under these
    circumstances and use normal packages.

    Tm modules are a great way to organize the code in a larger application, though. I usually have a modules/ directory in the application source directory, which I add to the modules path in the main.tcl file, and I use modules in this directory to organize my application code.


    --
    EL

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Harald Oehlmann@21:1/5 to All on Thu Oct 7 15:42:15 2021
    Am 07.09.2021 um 21:22 schrieb Schelte:
    www is a package for performing http requests. It is intended to be used
    in situations where the http package is too cumbersome to use (which is pretty much always) and tclcurl is overkill.

    The basic way to obtain a web resource using the www package is as follows:

        if {[catch {www get http://www.tcl.tk/} data info]} {
            puts stderr "Failed to get the web page"
        } else {
            puts $data
        }

    The package takes care of redirects, retries, cookies, and proxies and
    only uses the standard way of throwing errors. The command returns the document body. Metadata is available through return options.

    The package also provides extensions to handle WebSockets and HTTP/2.
    For HTTP/2 over TLS a patch is needed for TclTLS. I have filed a ticket
    with the patch to the TclTLS tracker (https://core.tcl-lang.org/tcltls/tktview?name=e1f9a21c67). Hopefully it
    will be picked up soon.

    For more information on www, please visit the chiselapp repo at https://chiselapp.com/user/schelte/repository/www


    Enjoy,
    Schelte.

    Hi Schelte,

    I appreciate, thank you !
    Great that you even care about DNS block by delegating it to a different thread.
    It was always an aim by Reinhard and myself make this non-blocking in
    the core... So much to do.... Do you support iocp by Ashok ?

    Thanks again,
    Harald

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Harald Oehlmann@21:1/5 to All on Thu Oct 7 15:44:58 2021
    Am 07.10.2021 um 15:42 schrieb Harald Oehlmann:

    Am 07.09.2021 um 21:22 schrieb Schelte:
    www is a package for performing http requests. It is intended to be
    used in situations where the http package is too cumbersome to use
    (which is pretty much always) and tclcurl is overkill.

    The basic way to obtain a web resource using the www package is as
    follows:

         if {[catch {www get http://www.tcl.tk/} data info]} {
             puts stderr "Failed to get the web page"
         } else {
             puts $data
         }

    The package takes care of redirects, retries, cookies, and proxies and
    only uses the standard way of throwing errors. The command returns the
    document body. Metadata is available through return options.

    The package also provides extensions to handle WebSockets and HTTP/2.
    For HTTP/2 over TLS a patch is needed for TclTLS. I have filed a
    ticket with the patch to the TclTLS tracker
    (https://core.tcl-lang.org/tcltls/tktview?name=e1f9a21c67). Hopefully
    it will be picked up soon.

    For more information on www, please visit the chiselapp repo at
    https://chiselapp.com/user/schelte/repository/www


    Enjoy,
    Schelte.

    Hi Schelte,

    I appreciate, thank you !
    Great that you even care about DNS block by delegating it to a different thread.
    It was always an aim by Reinhard and myself make this non-blocking in
    the core... So much to do.... Do you support iocp by Ashok ?

    Thanks again,
    Harald

    Hi Schelte,
    I would also appreciate, if Ashoks TWAPI would be usable for the TLS part. Thanks,
    Harald

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Schelte@21:1/5 to Harald Oehlmann on Thu Oct 7 17:50:01 2021
    On 07/10/2021 15:44, Harald Oehlmann wrote:
    Do you support iocp by Ashok ?

    I would also appreciate, if Ashoks TWAPI would be usable for the TLS part.

    Both of the packages you inquire about are Windows only. I don't use
    Windows myself, so I can't say for sure if they will work in combination
    with the www package.

    The www package provides a way to register a command for implementing a
    scheme. That command is expected to apply a channel transformation on
    top of a socket after it has been opened. Reading the documentation, it
    seems that the iocp_inet package replaces the standard Tcl socket
    command. That seems like it could work quite seamlessly. The first
    problem is loading the package in the helper thread. There currently is
    no support for doing something like that. I also wonder if transferring
    an iocp channel between threads has ever been tested.

    The TWAPI documentation makes me believe that the twapi_crypto starttls
    command should be usable for doing https with TWAPI. That should only
    require a short proc that is installed using `www register`.


    Schelte.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Harald Oehlmann@21:1/5 to All on Thu Oct 7 18:52:19 2021
    Am 07.10.2021 um 17:50 schrieb Schelte:
    On 07/10/2021 15:44, Harald Oehlmann wrote:
    Do you support iocp by Ashok ?

    I would also appreciate, if Ashoks TWAPI would be usable for the TLS
    part.

    Both of the packages you inquire about are Windows only. I don't use
    Windows myself, so I can't say for sure if they will work in combination
    with the www package.

    The www package provides a way to register a command for implementing a scheme. That command is expected to apply a channel transformation on
    top of a socket after it has been opened. Reading the documentation, it
    seems that the iocp_inet package replaces the standard Tcl socket
    command. That seems like it could work quite seamlessly. The first
    problem is loading the package in the helper thread. There currently is
    no support for doing something like that. I also wonder if transferring
    an iocp channel between threads has ever been tested.

    The TWAPI documentation makes me believe that the twapi_crypto starttls command should be usable for doing https with TWAPI. That should only
    require a short proc that is installed using `www register`.

    Sorry for the questions. I was not aware that you don't use Windows.

    By the way, I don't understand the usage as backround command.
    It means, that a:
    catch {www get ...}
    will call vwait and events are executed, right ?

    Great package and thank you,
    Harald

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Schelte@21:1/5 to Harald Oehlmann on Thu Oct 7 20:01:34 2021
    On 07/10/2021 18:52, Harald Oehlmann wrote:
    By the way, I don't understand the usage as backround command.
    It means, that a:
    catch {www get ...}
    will call vwait and events are executed, right ?

    That is correct. But that is really just a safety net for when the
    command is invoked from direct code. The suggested way is to perform www
    calls in a coroutine.


    Schelte.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Harald Oehlmann@21:1/5 to All on Fri Oct 8 08:41:11 2021
    Am 07.10.2021 um 20:01 schrieb Schelte:
    On 07/10/2021 18:52, Harald Oehlmann wrote:
    By the way, I don't understand the usage as backround command.
    It means, that a:
    catch {www get ...}
    will call vwait and events are executed, right ?

    That is correct. But that is really just a safety net for when the
    command is invoked from direct code. The suggested way is to perform www calls in a coroutine.


    Schelte.

    Dear Schelte,
    thank you for the valuable answer.

    Thanks,
    Harald

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Ashok@21:1/5 to Schelte on Fri Oct 8 13:14:41 2021
    On 10/7/2021 9:20 PM, Schelte wrote:
    The www package provides a way to register a command for implementing a scheme. That command is expected to apply a channel transformation on
    top of a socket after it has been opened. Reading the documentation, it
    seems that the iocp_inet package replaces the standard Tcl socket
    command. That seems like it could work quite seamlessly. The first
    problem is loading the package in the helper thread. There currently is
    no support for doing something like that. I also wonder if transferring
    an iocp channel between threads has ever been tested.

    iocp passes the entire socket test suite, the http test suite as well as
    the tcltls tests. However, the only thread transfer test in those is socket-13.1 so it is a bit light (but no lighter than ::socket :-) to
    say the least. If I get some time, I'll add the www test suite to the
    iocp tests as well.


    The TWAPI documentation makes me believe that the twapi_crypto starttls command should be usable for doing https with TWAPI. That should only
    require a short proc that is installed using `www register`.


    Schelte.

    I think that should work "in principle" but there is a difference
    between twapi::tls, which validates against the Windows certificate
    store versus tcltls which uses a specified file/dir path. The
    corresponding options may need some tweaking. Again, I'll try if I get
    some time on the weekend.

    BTW, are you the same Schelte as sbron on the wiki?

    /Ashok

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Schelte@21:1/5 to Ashok on Fri Oct 8 12:56:19 2021
    On 08/10/2021 09:44, Ashok wrote:
    I'll try if I get some time on the weekend.

    Thanks. It will be interesting to learn what you find.

    BTW, are you the same Schelte as sbron on the wiki?

    Yes, that's me.


    Schelte.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Ashok@21:1/5 to Rolf Ade on Mon Oct 18 21:15:37 2021
    On 9/8/2021 5:13 AM, Rolf Ade wrote:

    Schelte <nospam@wanadoo.nl> writes:
    www is a package for performing http requests. It is intended to be
    used in situations where the http package is too cumbersome to use
    (which is pretty much always) and tclcurl is overkill.

    I wholeheartedly agree with "the http package is cumbersome to use".


    I understand the comment about the built-in http package being awkward / cumbersome for many use cases, mostly historical hold overs, but I have
    to point out a commonly overlooked strength - its test suite in its
    current incarnation is orders of magnitude more comprehensive than any
    of the competing http clients. I *think* it was Keith Nash (apologies if
    I'm wrong) who updated it in the recent past and clearly a lot of effort
    went into it. The quality of the test suite is a major consideration for me.

    /Ashok

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Ashok@21:1/5 to Schelte on Mon Oct 18 21:08:29 2021
    On 10/8/2021 4:26 PM, Schelte wrote:
    On 08/10/2021 09:44, Ashok wrote:
    I'll try if I get some time on the weekend.

    Thanks. It will be interesting to learn what you find.


    Well, did not get very far I'm afraid (using iocp in lieu of Tcl sockets
    as well as TLS). Aside from the file link issues on Windows, I was not
    able to get the register command to work at all. I think it is not being
    passed to the helper thread but did not have time to look closely.

    BTW, are you the same Schelte as sbron on the wiki?

    Yes, that's me.

    I was asking because coincidentally I ran into your version of
    getopt-like packages on the Wiki while looking for that class of
    functionality. Very nicely done, full featured, expected behaviors
    supported and a very intuitive switch-like Tcl-ish interface. My pick
    for Editor's choice :-)

    /Ashok

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Schelte@21:1/5 to Ashok on Tue Oct 19 00:12:09 2021
    On 18/10/2021 17:45, Ashok wrote:
    I understand the comment about the built-in http package being awkward / cumbersome for many use cases, mostly historical hold overs, but I have
    to point out a commonly overlooked strength - its test suite in its
    current incarnation is orders of magnitude more comprehensive than any
    of the competing http clients.

    As far as I could quickly tell, there are about twice as many tests for
    the http package as there are for the www package. Not exactly "orders
    of magnitude". But please let me know if there is anything you feel is
    missing from the test suite of the www package.

    Despite the more extensive test suite of the http package, it still has
    a number of bugs/issues. Which is one of the reasons I chose to write
    the www package from scratch, rather than build it on top of the http
    package.


    Schelte.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Schelte@21:1/5 to Ashok on Tue Oct 19 00:13:35 2021
    On 18/10/2021 17:38, Ashok wrote:
    Well, did not get very far I'm afraid (using iocp in lieu of Tcl sockets
    as well as TLS). Aside from the file link issues on Windows, I was not
    able to get the register command to work at all. I think it is not being passed to the helper thread but did not have time to look closely.

    As I mentioned in an earlier post: "The www package provides a way to
    register a command for implementing a scheme. That command is expected
    to apply a channel transformation on top of a socket after it has been
    opened." So this mechanism cannot be used in its current form to open a
    socket using a command other than [socket]. Which is why I also
    remarked: "The first problem is loading the package in the helper
    thread. There currently is no support for doing something like that."

    So for now, probably the only way to test the package with iocp would be
    to modify the source code to load the package in the helper thread.

    I hope I have addressed the file link issue for the base www package.
    The same still needs to be done for the sub-packages.

    I was asking because coincidentally I ran into your version of
    getopt-like packages on the Wiki while looking for that class of functionality. Very nicely done, full featured, expected behaviors
    supported and a very intuitive switch-like Tcl-ish interface. My pick
    for Editor's choice :-)

    Thanks. I use it myself in most applications I make.


    Schelte.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Ashok@21:1/5 to Schelte on Tue Oct 19 14:06:56 2021
    On 10/19/2021 3:42 AM, Schelte wrote:
    As far as I could quickly tell, there are about twice as many tests for
    the http package as there are for the www package. Not exactly "orders
    of magnitude". But please let me know if there is anything you feel is missing from the test suite of the www package.

    Just to be clear, I certainly did not mean to denigrate other http implementations with respect to testing or quality. That was not my
    intent. Rather I meant to highlight at least one positive aspect of the
    http package.

    The Tcl http suite contains over 2000 tests. You might not have noticed
    because they are spread over multiple test files and will not run
    without multiple constraints enabled (and will take more than an hour to
    run). But more than just the number of tests my use of "comprehensive"
    related not just to tests but to number of scenarios covered given the
    number of HTTP option *combinations* possible - methods(get,post etc.)*keepalive*pipelining*encoding*contentlength*deflate|gzip|uncompressed*chunking
    variations*mime types*HTTP codes*error conditions. That's an impossible
    number to test! Perhaps a look at http.test http11.test and
    httpPipeline.test might help identify missing (if any) test cases for
    www and whether they are worth adding/spending time on.


    Despite the more extensive test suite of the http package, it still has
    a number of bugs/issues. Which is one of the reasons I chose to write
    the www package from scratch, rather than build it on top of the http package.


    Quite possible though in my usage I've found the http package has
    greatly improved between 2.7 and the current version.

    Again, I'm not knocking the other http packages which I also use. They
    are undoubtedly more convenient and featureful and much appreciated. I'm
    merely drawing attention to what I see as a strength in the Tcl http one.

    /Ashok

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Schelte@21:1/5 to Schelte on Wed Oct 20 14:59:39 2021
    On 19/10/2021 00:13, Schelte wrote:
    So for now, probably the only way to test the package with iocp would be
    to modify the source code to load the package in the helper thread.

    I have added a new configuration option to the www package. You should
    now (hopefully) be able to use the iocp socket command by doing
    something like this at the start of your script:

    package require www
    www configure -socketcmd [list apply [list {path args} {
    set ::auto_path $path
    package require iocp_inet
    tailcall socket {*}$args
    }] $auto_path]

    Clearly, you can simplify this a bit if the iocp_inet package is in the
    default auto_path.

    Probably needles to say: The socket command replacement is executed in a separate thread, so it has to be self-contained. You can't use any procs defined in the main thread.


    Schelte

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Ashok@21:1/5 to Schelte on Tue Oct 26 08:21:26 2021
    On 10/20/2021 6:29 PM, Schelte wrote:
    On 19/10/2021 00:13, Schelte wrote:
    I have added a new configuration option to the www package. You should
    now (hopefully) be able to use the iocp socket command by doing
    something like this at the start of your script:

    package require www
    www configure -socketcmd [list apply [list {path args} {
        set ::auto_path $path
        package require iocp_inet
        tailcall socket {*}$args
    }] $auto_path]

    Clearly, you can simplify this a bit if the iocp_inet package is in the default auto_path.

    Probably needles to say: The socket command replacement is executed in a separate thread, so it has to be self-contained. You can't use any procs defined in the main thread.


    Schelte

    Great. I'll give it a shot when I have some time.

    /Ashok

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From apn@21:1/5 to Schelte on Wed Nov 10 22:25:39 2021
    On 10/20/2021 6:29 PM, Schelte wrote:
    I have added a new configuration option to the www package. You should
    now (hopefully) be able to use the iocp socket command by doing
    something like this at the start of your script:

    package require www
    www configure -socketcmd [list apply [list {path args} {
        set ::auto_path $path
        package require iocp_inet
        tailcall socket {*}$args
    }] $auto_path]


    Finally got around to this and it works as advertised.

    /Ashok

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Schelte@21:1/5 to apn on Wed Nov 10 23:46:04 2021
    On 10/11/2021 17:55, apn wrote:
    Finally got around to this and it works as advertised.

    Excellent. Thanks for checking.


    Schelte

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)