• Is popen() unsafe (with untrusted string) ?

    From Nicolas George@21:1/5 to All on Wed Jan 13 18:23:10 2021
    Scott Lurndal, dans le message <OZELH.2391$ew6.163@fx11.iad>, a écrit :
    Can you elaborate on this rather odd statement? POSIX threads and
    file descriptor (or even stdio) I/O interfaces work just fine together.

    Oh? Then please tell me: how do you multiplex a mutex or condition wait with
    a socket accept?

    There's always the poll and select family of system calls to provide timeouts; I use poll extensively in threaded networking code.

    That's exactly what I mean: you have threads, and you still need to use I/O multiplexing.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Kaz Kylheku@21:1/5 to Nicolas George on Wed Jan 13 19:51:18 2021
    On 2021-01-13, Nicolas George <nicolas$george@salle-s.org> wrote:
    Scott Lurndal, dans le message <OZELH.2391$ew6.163@fx11.iad>, a écrit :
    Can you elaborate on this rather odd statement? POSIX threads and
    file descriptor (or even stdio) I/O interfaces work just fine together.

    I understand this as meaning that "to use threads with I/O effectively, we need to use multiplexing mechanisms that are also usable by single-threaded programs that don't know anything about threads".

    I.e. threads (or at least POSIX threads) do not succeed in replacing mechanisms for multiplexing events onto one thread such as timeouts, select/poll, async I/O and whatever.


    Oh? Then please tell me: how do you multiplex a mutex or condition wait with a socket accept?

    Yes, that can be a problem. I needed to do this in the kernel once,
    and wrote it!

    In the lmc-2.0 archive given here

    http://www.kylheku.com/~kaz/lmc.html

    See this function (in mutex.c):

    /**
    * Atomically give up the mutex and wait on the condition variable.
    * Wake up if the specified timeout elapses, or if a signal is delivered.
    * Additionally, also wait on the specified file descriptors to become
    * ready, combining condition waiting with poll().
    * KCOND_WAIT_SUCCESS means the condition was signaled, or one or more
    * file descriptors are ready.
    * Also, a negative value can be returned indicating an error!
    * (The poll needs to dynamically allocate some memory for the wait table).
    * The timeout is relative to the current time, specifying how long to sleep in
    * jiffies (CPU clock ticks).
    */
    int kcond_timed_wait_rel_poll(kcond_t *, kmutex_t *, long,
    kcond_poll_t *, unsigned int);

    --
    TXR Programming Language: http://nongnu.org/txr

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Scott Lurndal@21:1/5 to Nicolas George on Wed Jan 13 19:24:10 2021
    Nicolas George <nicolas$george@salle-s.org> writes:
    Scott Lurndal, dans le message <OZELH.2391$ew6.163@fx11.iad>, a écrit :
    Can you elaborate on this rather odd statement? POSIX threads and
    file descriptor (or even stdio) I/O interfaces work just fine together.

    Oh? Then please tell me: how do you multiplex a mutex or condition wait with >a socket accept?

    There's always the poll and select family of system calls to provide
    timeouts; I use poll extensively in threaded networking code.

    That's exactly what I mean: you have threads, and you still need to use I/O >multiplexing.

    No, you don't _need_ to use I/O multiplexing in most cases (e.g. disk files).

    For socket endpoints, I'll create a pipe(2) to use to notify the thread to exist and
    the thread main loop will poll the pipe and the socket fd. The main code
    will write a single byte to the pipe to terminate the poll, and the
    thread will exit.

    Once can certainly setup one or more threads to just do synchronous I/O on demand using a request
    and completion queue (similar to most modern host controller hardware)
    without using poll or select.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Nicolas George@21:1/5 to All on Wed Jan 13 20:18:37 2021
    Scott Lurndal, dans le message <uNHLH.22369$Ad1.15178@fx33.iad>, a
    écrit :
    No, you don't _need_ to use I/O multiplexing in most cases (e.g. disk files).

    Wow, that was a waste of time.

    For socket endpoints, I'll create a pipe(2) to use to notify the thread to exist and the thread main loop will poll the pipe and the socket fd. The main code will write a single byte to the pipe to terminate the poll, and
    the thread will exit.

    So you know how to force threads to work with file descriptors despite the
    fact they're not designed for. Good for you.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Philip Guenther@21:1/5 to Kenny McCormack on Thu Jan 14 20:13:55 2021
    On Thursday, January 7, 2021 at 11:20:00 AM UTC-8, Kenny McCormack wrote:
    ...
    Also, and this is related, is there a version of popen() (or some library
    or something available) that is bidirectional - i.e., you can both write
    and read from it - for example, you could run the Unix 'sort' utility this way - send it some data, then read back the sorted result (*).

    One classic (i.e., "decades old") and portable inside POSIX technique that solves a large subset of this problem space is to use a process sandwich, where you create two pipes, then fork twice with one child execing the target utility after some fd
    swizzling, with the other child serving as the writer/source and the original process being the reader/sink**. This simplifies things whenever the writer and reader operations are not so tightly coupled as to need to share state, the key idea being that
    running the writer and reader in separate processes eliminates the deadlock issues.

    (I suppose this can be 'simplified' by only doing one pipe and fork for the writer, then using popen() for the utility, but that requires more fd swizzling to set up the stdio for the popen() and then revert it afterwards, but I've never seen this idiom
    written that way.)


    This obviously doesn't work when the writer and reader share state. For example, if the process needs to run some non-trivial protocol over a TCP connection (like HTTP, or TLS) to receive the input and send back the output, possibly interleaved, then
    some way to share the necessary state across the fork would be necessary, which would probably be more complicated than just doing I/O multiplexing with poll() in one process.


    Philip Guenther

    ** or the other way around, with the original process being the writer. The cases I've seen this used have all had the full input available from the start and wanted to carry on processing with the output, so making the writer the child was correct for
    them.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Kaz Kylheku@21:1/5 to Philip Guenther on Fri Jan 15 08:24:10 2021
    On 2021-01-15, Philip Guenther <guenther@gmail.com> wrote:
    On Thursday, January 7, 2021 at 11:20:00 AM UTC-8, Kenny McCormack wrote:
    ...
    Also, and this is related, is there a version of popen() (or some library
    or something available) that is bidirectional - i.e., you can both write
    and read from it - for example, you could run the Unix 'sort' utility this >> way - send it some data, then read back the sorted result (*).

    One classic (i.e., "decades old") and portable inside POSIX technique
    that solves a large subset of this problem space is to use a process sandwich, where you create two pipes, then fork twice with one child
    execing the target utility after some fd swizzling, with the other
    child serving as the writer/source and the original process being the reader/sink**.

    E.g.

    VAR=$(function | sort)

    function runs in child process, sort in another, and the original
    process captures the output, storing it in VAR.

    --
    TXR Programming Language: http://nongnu.org/txr

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Rainer Weikusat@21:1/5 to Rainer Weikusat on Mon Jan 18 17:53:12 2021
    XPost: comp.unix.shell

    Rainer Weikusat <rweikusat@talktalk.net> writes:
    spudisnotyourbud@grumpysods.com writes:
    On Tue, 12 Jan 2021 17:53:09 +0000
    Rainer Weikusat <rweikusat@talktalk.net> wrote: >>>spudisnotyourbud@grumpysods.com writes:
    On 11 Jan 2021 16:36:37 GMT
    Casper H.S. Dik <Casper.Dik@OrSPaMcle.COM> wrote: >>>>>gazelle@shell.xmission.com (Kenny McCormack) writes:
    Also, and this is related, is there a version of popen() (or some library >>>>>>or something available) that is bidirectional - i.e., you can both write >>>>>>and read from it - for example, you could run the Unix 'sort' utility this
    way - send it some data, then read back the sorted result (*).

    In Solaris there is a p2open()/p2close() as part of libgen; I'm not sure >>>>>whether it is common.


    There is, of course, a risk: if you write to one end but you are not >>>>>reading the other end at the same time, you might be blocked by the >>>>>other program which is waiting for your to read but you are blocked >>>>>trying to write more. Using threads would fix that.

    Or alternatively you could be sensible and use select/poll multiplexing on >>>> the descriptor returned from fileno() instead of messing around with >>>threading
    and all the nonsense that goes with it.

    Something like this is much simpler with threads which can block >>>individually.

    It really isn't. But if you only know how to use a hammer...

    It is

    For illustration: Main 'working function' for a program relaying data to
    and from a AF_UNIX stream socket:

    static void forward_data(int from, int to)
    {
    char buf[1024];
    ssize_t rc_r, rc_w;

    while (rc_r = read(from, buf, sizeof(buf)), rc_r > 0) {
    rc_w = write(to, buf, rc_r);
    rc_w != -1 || sys_die("write");
    }

    rc_r != -1 || sys_die("read");
    }

    This runs twice, from a 2nd thread and from main, and that's all of the program.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)