• thread::send -async with varname and potential race condition

    From ted brown@21:1/5 to All on Fri Sep 3 13:20:11 2021
    The thread::send manual page includes an example to show the use of the
    varname parameter for returning and waiting on an -async call. Here's
    the example with some extra commented out statements:

    set t1 [thread::create]
    set t2 [thread::create]
    thread::send -async $t1 "set a 1" result
    thread::send -async $t2 "set b 2" result
    # uncomment these 2 and it will hang
    # after 1 {set timervar 1}
    # vwait timervar
    for {set i 0} {$i < 2} {incr i} {
    vwait result
    }

    Doesn't the above work, *only* because this code doesn't enter the event
    loop via a call to vwait (or update)? If I uncomment the after and
    vwait, it hangs. So, isn't there an undocumented trap here that should
    at least be mentioned in the documentation?

    My conjecture is the threads return the result via a thread::send back
    to the main thread, and so that goes into the event queue. Anything that activates the event queue before the [vwait result] will set result
    before the main thread can vwait on it, and vwait doesn't "queue up"
    these signals.

    Or am I wrong about how it actually works?

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From ted brown@21:1/5 to All on Sat Sep 4 11:14:26 2021
    I've resolved the problem, though not optimally. I'm unsetting the
    variable before the send and spin waiting (1 ms) after the send until
    the variable is created testing with [info exist].

    I think an optimal solution might be to do the unset in the send, and
    enhancing the [vwait] command as such:

    vwait varname ?-exist?

    which would wait until the variable exists and has a value, or if it
    already exists return immediately (and has a value, in case there's any
    way that's not atomic).

    Then the example, which is waiting for the variable to be set twice,
    might use 2 variables instead. Then this could work I think (but I never
    say never with these sticky timing problems):

    thread::send -async $t1 "set a 1" result1
    thread::send -async $t2 "set b 2" result2

    vwait result1 -exist
    vwait result2 -exist

    and it shouldn't matter which thread completed first either.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Schelte@21:1/5 to ted brown on Sat Sep 4 21:06:06 2021
    On 04/09/2021 20:14, ted brown wrote:
    Then the example, which is waiting for the variable to be set twice,
    might use 2 variables instead. Then this could work I think (but I never
    say never with these sticky timing problems):

    You are overcomplicating things. The variables will only be updated when
    the event loop is running. So you can just check if the variable is
    already set before you are about to run the vwait. There is no need for
    a vwait option.

    Even easier may be to use array elements for the result variables and
    then vwait on the array:

    thread::send -async $t1 {after 4321; set a 1} result(a)
    thread::send -async $t2 {after 1234; set b 2} result(b)

    while {[array size result] < 2} {
    vwait result
    puts [array get result]
    }

    Strangely, the vwait manual page doesn't mention that you can use vwait
    on an array and setting any array element will then cause the vwait to
    return. But in practice that works.

    If you don't feel comfortable with this solution for that reason, you
    can also set up variable traces on the result variables. Those can fire
    in any order.


    Schelte.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Uwe Klein@21:1/5 to All on Sun Sep 5 11:28:23 2021
    Am 03.09.21 um 22:20 schrieb ted brown:
    The thread::send manual page includes an example to show the use of the varname parameter for returning and waiting on an -async call. Here's
    the example with some extra commented out statements:

        set t1 [thread::create]
        set t2 [thread::create]
        thread::send -async $t1 "set a 1" result
        thread::send -async $t2 "set b 2" result
    #   uncomment these 2 and it will hang
    #   after 1 {set timervar 1}
    #   vwait timervar
        for {set i 0} {$i < 2} {incr i} {
            vwait result
        }

    Doesn't the above work, *only* because this code doesn't enter the event
    loop via a call to vwait (or update)? If I uncomment the after and
    vwait, it hangs. So, isn't there an undocumented trap here that should
    at least be mentioned in the documentation?

    My conjecture is the threads return the result via a thread::send back
    to the main thread, and so that goes into the event queue. Anything that activates the event queue before the [vwait result] will set result
    before the main thread can vwait on it, and vwait doesn't "queue up"
    these signals.

    Or am I wrong about how it actually works?

    it should hang when the var "result" is no longer written to
    after the "vwait timervar" returns.

    What I'd do is
    trace the "result" and "timervar" with a proc that increments another
    variable "touchcount" :-)

    Then:
    while {$touchcount < 3} {
    vwait touchcount
    }
    you can saveguard this costruct with

    after $longtime {incr touchcount 100 }
    # test on touchcount >= 100 for determining a timeout condition.

    Uwe

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From ted brown@21:1/5 to Schelte on Mon Sep 6 13:10:16 2021
    On 9/4/2021 12:06 PM, Schelte wrote:
    You are overcomplicating things. The variables will only be updated when
    the event loop is running. So you can just check if the variable is
    already set before you are about to run the vwait. There is no need for
    a vwait option.

    I don't know how to check for a variable being set to a value unless
    that variable didn't yet exist. So, that would mean one should (or must)
    unset the variable before doing the send.

    AFIK [vwait] doesn't care if the variable already exists, just that it
    has been just set to some value (and it need not be a different value
    than it currently has).



    Even easier may be to use array elements for the result variables and
    then vwait on the array:

    thread::send -async $t1 {after 4321; set a 1} result(a)
    thread::send -async $t2 {after 1234; set b 2} result(b)

    while {[array size result] < 2} {
        vwait result
        puts [array get result]
    }

    Strangely, the vwait manual page doesn't mention that you can use vwait
    on an array and setting any array element will then cause the vwait to return. But in practice that works.

    Interesting, I'll have to think about the array count. However, once
    again, this would only work if the array didn't already exist with 2 or
    more elements.


    If you don't feel comfortable with this solution for that reason, you
    can also set up variable traces on the result variables. Those can fire
    in any order.


    Schelte.

    My concern is that there are many things I might be doing between the
    send and later needing to wait for that variable to be set. I'm looking
    for a general solution, not just for one case since I'm writing a thread wrapper package.

    For example, I have code where I need to do delays, such as when I'm
    sending http requests to some lan device like a Roku Tv or a Tivo. These devices tend to drop requests that come in too quickly. They don't
    provide any "ready" feedback. So, I have be mindful about timing.

    Also my "wait" proc uses an after with a variable to vwait on. So, one
    way or the other, I end up in the event loop.

    Traces might be a good approach, I'll have to give that some thought.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From ted brown@21:1/5 to Uwe Klein on Mon Sep 6 13:22:27 2021
    On 9/5/2021 2:28 AM, Uwe Klein wrote:
    Am 03.09.21 um 22:20 schrieb ted brown:
    The thread::send manual page includes an example to show the use of the
    varname parameter for returning and waiting on an -async call. Here's
    the example with some extra commented out statements:

        set t1 [thread::create]
        set t2 [thread::create]
        thread::send -async $t1 "set a 1" result
        thread::send -async $t2 "set b 2" result
    #   uncomment these 2 and it will hang
    #   after 1 {set timervar 1}
    #   vwait timervar
        for {set i 0} {$i < 2} {incr i} {
            vwait result
        }

    Doesn't the above work, *only* because this code doesn't enter the event
    loop via a call to vwait (or update)? If I uncomment the after and
    vwait, it hangs. So, isn't there an undocumented trap here that should
    at least be mentioned in the documentation?

    My conjecture is the threads return the result via a thread::send back
    to the main thread, and so that goes into the event queue. Anything that
    activates the event queue before the [vwait result] will set result
    before the main thread can vwait on it, and vwait doesn't "queue up"
    these signals.

    Or am I wrong about how it actually works?

    it should hang when the var "result" is no longer written to
    after the "vwait timervar" returns.

    What I'd do is
    trace the "result" and "timervar" with a proc that increments another variable "touchcount" :-)

    Then:
    while {$touchcount < 3} {
    vwait touchcount
    }
    you can saveguard this costruct with

    after $longtime {incr touchcount 100 }
    # test on touchcount >= 100 for determining a timeout condition.

    Uwe


    I'll have to think about using [trace]. And I'll have to think about
    using another variable, though I recall from long ago that often one
    just pushes the race condition from one to the other :)

    I guess I could also consider using the thread mutexes and condition
    variables, but I'm already using them and I was hoping for a simpler
    solution.

    For now, I'm ok with a spin check. I see no appreciable cpu time if I
    wake up every millisecond. Since the variable is set in an event loop,
    if I don't respond for up to 1 ms, it probably won't matter too much.

    Polling loops have come to my rescue before, and so I'd don't always
    consider them bad coding practice.

    Thanks for the response.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Ralf Fassel@21:1/5 to All on Tue Sep 7 10:03:28 2021
    * ted brown <tedbrown888@gmail.com>
    | On 9/4/2021 12:06 PM, Schelte wrote:
    | > You are overcomplicating things. The variables will only be updated
    | > when the event loop is running. So you can just check if the
    | > variable is already set before you are about to run the vwait. There
    | > is no need for a vwait option.

    | I don't know how to check for a variable being set to a value unless
    | that variable didn't yet exist. So, that would mean one should (or
    | must) unset the variable before doing the send.

    This sounds much like the classical cond-wait-deadlock...
    If there is a value which the variable will definitely not be set to,
    you could:

    set var ""
    thread::send -async $t1 {after 4321; set a 1} var
    thread::send -async $t2 {after 1234; set b 2} var
    # if var was not yet set, wait for it
    if {$var eq ""} {
    vwait var
    }

    R'

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From ted brown@21:1/5 to Ralf Fassel on Tue Sep 7 04:21:41 2021
    On 9/7/2021 1:03 AM, Ralf Fassel wrote:
    * ted brown <tedbrown888@gmail.com>
    | On 9/4/2021 12:06 PM, Schelte wrote:
    | > You are overcomplicating things. The variables will only be updated
    | > when the event loop is running. So you can just check if the
    | > variable is already set before you are about to run the vwait. There
    | > is no need for a vwait option.

    | I don't know how to check for a variable being set to a value unless
    | that variable didn't yet exist. So, that would mean one should (or
    | must) unset the variable before doing the send.

    This sounds much like the classical cond-wait-deadlock...
    If there is a value which the variable will definitely not be set to,
    you could:

    set var ""
    thread::send -async $t1 {after 4321; set a 1} var
    thread::send -async $t2 {after 1234; set b 2} var
    # if var was not yet set, wait for it
    if {$var eq ""} {
    vwait var
    }

    R'


    Hmmmm, I can't predict the values but you've shook something loose in my
    mind.

    However, I should be able to test it's existence instead of using a
    unique value. This assumes that [info exist var] does *not* ever enter
    the event loop, which I think is a good bet, and there's nothing else
    between the test and the vwait. So,

    unset var
    do the thread::send
    if {![info exist var]} {
    vwait var
    }

    Something tells me one of the above posters was trying to tell me this
    but I was too block-headed to see it.

    Thanks for the suggestion.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From ted brown@21:1/5 to ted brown on Tue Sep 7 10:24:53 2021
    On 9/7/2021 4:21 AM, ted brown wrote:

    unset var
    do the thread::send
    if {![info exist var]} {
       vwait var
    }


    In my actual case, I use a different var for each thread I'm waiting on,
    so I think the way to do it for the example from the manual where they
    want to continue only after both complete, would be to use 2 variables.

    set t1 [thread::create]
    set t2 [thread::create]

    unset -nocomplain var1
    unset -nocomplain var2

    thread::send -async $t1 {after 4321; set a 1} var1
    thread::send -async $t2 {after 1234; set b 2} var2

    # ok to do other vwaits or updates here
    after 500 {set var3 1}
    vwait var3
    update

    if {![info exist var1]} {
    vwait var1
    }

    if {![info exist var2]} {
    vwait var2
    }
    puts "var1 = $var1 var2 = $var2"


    The example wants to wait on both being done. It shouldn't matter which
    one finishes first with 2 variables. The above code works in my tests.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Ralf Fassel@21:1/5 to All on Wed Sep 8 11:32:43 2021
    * ted brown <tedbrown888@gmail.com>
    | set t2 [thread::create]

    | unset -nocomplain var1
    | unset -nocomplain var2

    | thread::send -async $t1 {after 4321; set a 1} var1
    | thread::send -async $t2 {after 1234; set b 2} var2

    | # ok to do other vwaits or updates here
    | after 500 {set var3 1}
    | vwait var3
    | update

    | if {![info exist var1]} {
    | vwait var1
    | }

    | if {![info exist var2]} {
    | vwait var2
    | }
    | puts "var1 = $var1 var2 = $var2"

    This indeed is a variation on using conditional vars, which is the
    'usual' way of doing this in languages where the concept of 'unset
    variables' is not available; check the thread::cond section in the
    thread(n) manpage for more.

    In TCL you can shortcut this because of the separation of interps/threads/eventloop.

    R'

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Rich@21:1/5 to ted brown on Wed Sep 8 21:31:50 2021
    ted brown <tedbrown888@gmail.com> wrote:
    This is for my "tasks" wrapper/extension of threads. Each wrapped
    thread includes all the needed code and shared variables to implement
    a single-queue multi-sever model.

    I'm still not sure exactly what it is that you are building, but have
    you looked at the 'tpool' module that is part of the threads package?

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From ted brown@21:1/5 to Ralf Fassel on Wed Sep 8 14:18:56 2021
    On 9/8/2021 2:32 AM, Ralf Fassel wrote:
    * ted brown <tedbrown888@gmail.com>
    | set t2 [thread::create]

    | unset -nocomplain var1
    | unset -nocomplain var2

    | thread::send -async $t1 {after 4321; set a 1} var1
    | thread::send -async $t2 {after 1234; set b 2} var2

    | # ok to do other vwaits or updates here
    | after 500 {set var3 1}
    | vwait var3
    | update

    | if {![info exist var1]} {
    | vwait var1
    | }

    | if {![info exist var2]} {
    | vwait var2
    | }
    | puts "var1 = $var1 var2 = $var2"

    This indeed is a variation on using conditional vars, which is the
    'usual' way of doing this in languages where the concept of 'unset
    variables' is not available; check the thread::cond section in the
    thread(n) manpage for more.

    In TCL you can shortcut this because of the separation of interps/threads/eventloop.

    R'


    I had been thinking of using a conditional variable here as a last
    resort, but I am pleased that I can do it with the unset approach, which
    I think should be more efficient, and certainly simpler.

    This is for my "tasks" wrapper/extension of threads. Each wrapped thread includes all the needed code and shared variables to implement a
    single-queue multi-sever model.

    I used the mutex/cond-var example in Ashok's wonderful TCL book.
    However, I must confess I don't really understand fully how it works,
    only that it does :)

    That's for the side that sends the message to the worker thread. Getting
    the result back wasn't in the book, so I struggled with that, but it all
    seems to be working now.

    Anyway, it's been a great learning experience, and I couldn't have done
    it without all the great help and support I got here.

    Thanks again to everyone.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From ted brown@21:1/5 to Rich on Wed Sep 8 18:40:45 2021
    On 9/8/2021 2:31 PM, Rich wrote:
    ted brown <tedbrown888@gmail.com> wrote:
    This is for my "tasks" wrapper/extension of threads. Each wrapped
    thread includes all the needed code and shared variables to implement
    a single-queue multi-sever model.

    I'm still not sure exactly what it is that you are building, but have
    you looked at the 'tpool' module that is part of the threads package?


    I'm pretty much done, just tidying up, that vwait was the last hurdle.

    I had looked at tpool. I found it a bit daunting since I then had no
    experience with threads. And no examples in the man page.

    So as a learning exercise, I decided to build my own. However, my task
    model is not tcl-threads rpc-like, but rather the more familiar
    procedure call/return.

    Here's a small example which lets one do heavy compute in a separate
    thread to keep the main thread's gui responsive:

    #--------------------------------------------------
    proc sumtwo {a1 a2} { ;# add 2 args plus simulate computing
    for {set n 0} {$n < 10000000 } {incr n} {incr m}
    return [expr {$a1+$a2}]
    }

    Task summer {sumtwo} { ;# import sumtwo proc into new thread
    twait -> a1 a2 ;# get work & "lassign" to a1 and a2

    treturn [sumtwo $a1 $a2] ;# call sumtoo, return result, then repeat

    }

    tcall $summer answer 5 10 ;# call sync and wait for answer

    puts "answer = $answer"
    #--------------------------------------------------

    I think this shows the simplicity I was aiming for. Well, it seems
    simple to me, but that's always the advantage of "building your own". I
    do think any beginner tcl programmer could learn to use this quite easily.

    This can then easily extend to the multiple server model by just
    encoding the task name as,

    Task helper1/summer .... ;# any number, all use same "summer" queue

    And we can also make async calls, using separate result array elements:

    tcall $summer -async answer(0) 5 10
    tcall $summer -async answer(1) 6 10
    ...

    foreach t {0 1 ...} { ;# and wait for all to complete
    tvwait answer($t)
    }

    So, a Task is sort of a wrapper-class to threads, and one thread or
    multiple threads look almost identical.

    The goal was to make it look and work as much like a normal procedure
    call as possible. I think it's all working, and only about 400 lines of
    actual code. Lots more too, like a debugging "puts", error catching, and
    some info like tools.

    I've always preferred building tools to normal coding. My favorite quote
    of all times was by Brian Kernighan:

    "I'd rather write programs that write programs, than write programs".

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)