• ANNOUNCE: NaviServer 4.99.23 available

    From gustafn@21:1/5 to All on Wed Dec 29 23:26:55 2021
    Dear all,

    I am pleased to announce the availability of NaviServer 4.99.23 [1,2]. This release contains various fixes and enhancements. The summary of changes is listed below. This release was tested with Ubuntu 20.04, Rocky Linux 8.5, OpenBSD 6.9 (clang), FreeBSD
    13.0, macOS 11.6.2 (Intel and M1).

    The following people have contributed to this release:

    Antonio Pisano
    Gustaf Neumann
    Maksym Zinchenko
    Oleg Oleinick
    Russell Sorensen
    Wolfgang Winkler
    Zoran Vasiljevic

    Many thanks to all contributors!

    The upgrade to 4.99.23 is recommended. The installation script install-ns is updated to point to the new version.

    -gustaf neumann
    [1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.23/
    [2] https://bitbucket.org/naviserver/naviserver/

    =======================================
    NaviServer 4.99.23, released 2021-12-29
    =======================================

    70 files changed, 1431 insertions(+), 566 deletions(-)

    New Features:

    - Improved hash algorithms for improved security

    The new version supports SCRAM (Salted Challenge Response
    Authentication Mechanism), which is one of the newer recommended
    password hash algorithms to replace the classic salted SHA1
    approaches. The classical hash algorithms become easier to attack
    via high-performance hashing hardware (GPUs). NaviServer supports
    now SCRYPT and SCRAM (when compiled with more recent versions of
    OpenSSL; SCRYPT requires at least OpenSSL 3.0. SCRAM requires OpenSSL 1.1)

    The actual hash function of SCRAM is PBKDF2 [RFC2898] with HMAC as
    the pseudorandom function (PRF) and with dkLen == output length of
    HMAC == output length of the digest function.

    Here is an example of using pbkdf2_hmac for the hash function of
    SCRAM-sha-256. RFC 7677 recommends to use 15K iterations for PBKDF2:

    ::ns_crypto::pbkdf2_hmac \
    -digest sha256 \
    -iterations 15000 \
    -secret $password \
    -salt $salt]

    OpenACS supports already switching to from salted SHA1 to SCAM (or
    SCRYPT) via configuration variable.


    - Better Unicode support, including emojis requiring 4-byte UTF-8 characters.

    Earlier versions of NaviServer and the nsdb* database drivers
    assumed in a few places that Tcl-internal UTF-8 is also valid UTF-8
    for external sources, which is often, but not always true. Now, the
    proper export functions are everywhere called.

    The new code was tested with Emojis up to Unicode 13 (many thanks
    to Wolfgang Winkler)

    This change affects as well the database driver module "nsdbpg".

    - ns_trim enhancements:
    The new option "-prefix ..." can be used to strip a string
    (such as ">> ") from every line starting with it.

    - extended time unit support (added "w" for weeks and "y" for years)

    - Added an experimental global parameter "nocache" to ease to
    experiment with horizontal scaling. As a consequence, "ns_cache
    eval" becomes a dummy operation.

    - Added an experimental command "ns_baseunit"

    ns_baseunit ?-size size? ?-time time?

    Convert from memory units or from time units to its base value
    using the NaviServer internal converters, which are used the same
    way for various commands. The base unit for a memory size is a
    byte, for a time value a second). This command is necessary to
    provide Tcl-level commands calculating with these units to support
    uniform interfaces (e.g., calculating cache partition sizes base on
    values such as 2MB).

    Either "-size" or "-time" has to be specified.

    % ns_baseunit -size 10KB
    10240

    ns_baseunit -time 2.5h
    9000

    - Added an experimental command "ns_strcoll"

    ns_strcoll ?-locale locale? string1 string2"

    This command compares lexicographically string1 with string2
    according to the current locale collation and returns an integer
    greater than, equal to, or less than 0, depending on whether
    string1 is greater than, equal to, or less than string2. The
    command is necessary in cases, where e.g., the sorting order from
    the database (normally based on local collation) is different from
    default Tcl sorting order to provide a uniform interface with the same
    sorting orders. The name is derived from the baseline POSIX
    function call.

    The command is suitable for usage in the lsort command:

    % set l {Bor Bar Bär}
    % lsort -command ns_strcoll $l
    Bar Bär Bor

    % lsort $l
    Bar Bor Bär

    Note that the output of the command depends on the correct
    installation of locales on the host. Furthermore, the output of the
    command varies depending on the C library's implementation, which
    might differ for some locales/charsets between platforms.

    Performance Improvements:

    - Increase scalability on DB operations by reducing DB pool locks

    On high load servers, the total number of locks and busy locks on
    the DB pools might become quite high. The new code caches these
    statistics in the handles (which are per thread, requiring no
    locks) and transfers the aggregated values on handle closes or
    statistics calls.

    - Set default for "concurrentinterpcreate" to "true" for Tcl 8.6 or
    newer. Versions up to at least Tcl 8.5 are known to crash in case
    two threads create interpreters at the same time. These crashes
    were hard to reproduce, but serializing interpreter creation
    helped. Since all our major servers are running for several years
    without problems with this parameter turned on the default is now
    set to "true", when NaviServer is compiled with Tcl 8.6 or newer.


    Bug Fixes:
    - Fixed memory leak in "nsv_dict get" operations.


    Documentation improvements:
    ---------------------------

    - Improved the following man pages:

    doc/src/manual/admin-tuning.man
    doc/src/naviserver/commandlist.man
    doc/src/naviserver/ns_adp_include.man
    doc/src/naviserver/ns_baseunit.man
    doc/src/naviserver/ns_cache.man
    doc/src/naviserver/ns_charsets.man
    doc/src/naviserver/ns_choosecharset.man
    doc/src/naviserver/ns_connchan.man
    doc/src/naviserver/ns_cookie.man
    doc/src/naviserver/ns_cookiecharset.man
    doc/src/naviserver/ns_crypto.man
    doc/src/naviserver/ns_encodingforcharset.man
    doc/src/naviserver/ns_encodingfortype.man
    doc/src/naviserver/ns_formfieldcharset.man
    doc/src/naviserver/ns_ictl.man
    doc/src/naviserver/ns_job.man
    doc/src/naviserver/ns_register.man
    doc/src/naviserver/ns_schedule.man
    doc/src/naviserver/ns_setformencoding.man
    doc/src/naviserver/ns_shutdown.man
    doc/src/naviserver/ns_sleep.man
    doc/src/naviserver/ns_sockcallback.man
    doc/src/naviserver/ns_sockopen.man
    doc/src/naviserver/ns_strcoll.man
    doc/src/naviserver/ns_time.man
    doc/src/naviserver/ns_urlcharset.man
    doc/src/naviserver/ns_valid_utf8.man
    doc/src/naviserver/textutil-cmds.man


    Configuration Changes:
    ----------------------

    - Ease configuration via environment variables

    This feature is useful to manage many NaviServer instances with
    mostly identical configurations without having to provide multiple
    configuration files (e.g. for Docker setups, or clusters).

    The sample configuration for of OpenACS (openacs-config.tcl)
    contains now a Tcl dictionary with default values:

    set defaultConfig {
    hostname localhost
    ipaddress 127.0.0.1
    httpport 8000
    httpsport ""
    server "openacs"
    serverroot /var/www/$server
    logroot $serverroot/log/
    homedir /usr/local/ns
    bindir $homedir/bin
    db_name $server
    db_user $server
    db_host localhost
    db_port ""
    }

    These configuration values (keys of the dict) can be overridden by
    environment variables prefixed with "oacs_" followed by the
    parameter name. One can change the default port specified in the
    configuration file for plain HTTP connections by e.g., providing it
    via environment variables:

    oacs_httpport=8101 /usr/local/ns/bin/nsd -i -t ....



    Code Changes:
    -------------
    - Extended regression test

    - Code Cleanup
    . Do not declare reserved C identifiers

    - Improved comments, fixed typos

    - Marked "ns_set_precision" as deprecated, since there is no
    reason why not setting the Tcl variable ::tcl_precision directly.


    Changes in NaviServer Modules:
    ==============================

    8 files changed, 697 insertions(+), 283 deletions(-)


    nsdbpg:
    -------

    - Extended added support for UTF-8 including emojis (see above)

    - Added "ns_pg pid /handle/" to return the current backend PID
    without SQL parsing. This command is about 100x faster than using
    "select pg_backend_pid()" and can be used e.g. for deploying
    prepared statements.

    - Some code cleanup and modernization

    - Added minimal documentation of ns_pg* API in README file

    - Bumped version number to 2.6

    nsstats:
    --------

    - improved handling of fractional seconds
    - added summary line with total savings to cache statistics
    (to estimate consequences of eliminating this cache)
    - improve bread-crums for "mapped" server URLs
    - added user interface for unmapping URLs from connection pools

    nsodbc:
    -------

    - adjust code with current prototypes
    - fix warnings from static checkers

    nsshell:
    --------

    - Improved source code documentation
    - Updated PRISM.JS libraries links, to prevent X-Content-Type-Options errors

    letsencrypt:
    ------------

    - Removed obtaining the intermediate cross-signed certificate, since this does not seem
    to be required anymore
    - Improved source code documentation

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Ted Nolan @21:1/5 to neumann@wu-wien.ac.at on Thu Dec 30 17:32:18 2021
    In article <32fba293-717b-4fd5-ae6d-b7a50dfc143an@googlegroups.com>,
    gustafn <neumann@wu-wien.ac.at> wrote:


    Earlier versions of NaviServer and the nsdb* database drivers
    assumed in a few places that Tcl-internal UTF-8 is also valid UTF-8
    for external sources, which is often, but not always true. Now, the
    proper export functions are everywhere called.


    Could someone expand on this a bit? I've never looked at the code, but
    my impression from what I've read is that Tcl would store strings internally
    as pure Unicode, presumably in 32-bit units, and that when you wanted a
    string representation (assuming utf-8 encoding) it would produce it
    according to well defined UTF-8 rules. What is Tcl-internal UTF-8,
    when would you encounter it, and what is wrong with it?
    --
    columbiaclosings.com
    What's not in Columbia anymore..

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Rich@21:1/5 to ted@loft.tnolan.com on Thu Dec 30 18:41:55 2021
    Ted Nolan <tednolan> <ted@loft.tnolan.com> wrote:
    In article <32fba293-717b-4fd5-ae6d-b7a50dfc143an@googlegroups.com>,
    gustafn <neumann@wu-wien.ac.at> wrote:


    Earlier versions of NaviServer and the nsdb* database drivers
    assumed in a few places that Tcl-internal UTF-8 is also valid UTF-8
    for external sources, which is often, but not always true. Now, the
    proper export functions are everywhere called.


    Could someone expand on this a bit? I've never looked at the code, but
    my impression from what I've read is that Tcl would store strings internally as pure Unicode, presumably in 32-bit units,

    Nope. Tcl stores strings internally using 'modifed utf-8': https://en.wikipedia.org/wiki/UTF-8#Modified_UTF-8

    and that when you wanted a string representation (assuming utf-8
    encoding) it would produce it according to well defined UTF-8 rules.

    If you set your channel encodings properly, and use puts, yes.

    But if you extract the raw bytes (using the 'binary' command, or
    puts'ing into a 'binary' channel) you get the raw internal encoding.

    What is Tcl-internal UTF-8, when would you encounter it, and what is
    wrong with it?

    ASCII null is stored as an overlong (two byte) encoding of null, so
    that an actual null byte (0x00) never appears in the string. That way
    the encoding is compatiable with C null terminated strings, while also
    allowing storage of ASCII null characters. The wikipedia article has
    an example.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Ted Nolan @21:1/5 to rich@example.invalid on Thu Dec 30 18:46:57 2021
    In article <sqkuhi$f7r$1@dont-email.me>, Rich <rich@example.invalid> wrote: >Ted Nolan <tednolan> <ted@loft.tnolan.com> wrote:
    In article <32fba293-717b-4fd5-ae6d-b7a50dfc143an@googlegroups.com>,
    gustafn <neumann@wu-wien.ac.at> wrote:


    Earlier versions of NaviServer and the nsdb* database drivers
    assumed in a few places that Tcl-internal UTF-8 is also valid UTF-8
    for external sources, which is often, but not always true. Now, the
    proper export functions are everywhere called.


    Could someone expand on this a bit? I've never looked at the code, but
    my impression from what I've read is that Tcl would store strings internally >> as pure Unicode, presumably in 32-bit units,

    Nope. Tcl stores strings internally using 'modifed utf-8': >https://en.wikipedia.org/wiki/UTF-8#Modified_UTF-8

    and that when you wanted a string representation (assuming utf-8
    encoding) it would produce it according to well defined UTF-8 rules.

    If you set your channel encodings properly, and use puts, yes.

    But if you extract the raw bytes (using the 'binary' command, or
    puts'ing into a 'binary' channel) you get the raw internal encoding.

    What is Tcl-internal UTF-8, when would you encounter it, and what is
    wrong with it?

    ASCII null is stored as an overlong (two byte) encoding of null, so
    that an actual null byte (0x00) never appears in the string. That way
    the encoding is compatiable with C null terminated strings, while also >allowing storage of ASCII null characters. The wikipedia article has
    an example.


    Ah. Interesting -- thanks!
    --
    columbiaclosings.com
    What's not in Columbia anymore..

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)