• Re: a kind of a web-shell-api for managing long running processes

    From Scott Lurndal@21:1/5 to Meredith Montgomery on Thu Dec 30 21:55:07 2021
    Meredith Montgomery <mmontgomery@levado.to> writes:
    I'm helping someone to improve a CMS-type of web system. This web
    system sometimes needs to run some background processes --- such as send
    a batch of e-mail or other process that takes a while to finish.

    I see the challenge here as just writing something that will look like a
    very basic UNIX shell --- so I'll call it ``web-api-shell'' from now on. >(``Web'' because it will be used by a web system through some API.)

    This thing must run really well. It has to be flawless. I'm looking
    for design principles and advice.

    (*) Where will it run

    It will run on GNU systems running the Linux kernel.

    (*) My own thoughts

    The interface to shell will be through HTTP requests, so this shell will >likely be a web server of some sort. But before I get involved in the
    web at all, I need the shell working flawlessly.

    So I need a laboratory first. I could write a program that reads some
    named pipe on disk to get commands such as ``run this or that'' while I
    work. (Later I can write a web server replacing this named-pipe
    interface.)

    Just like a UNIX shell, this web-api-shell must know all every process
    it runs. I suppose the work is essentially fork(), execve() followed by >waitpid().

    One concern I have is the following. Is it possible for a process to
    simply ``get out of'' the shell? What do I mean by that? A process
    that does fork() and puts itself into background would leave the >web-api-shell's control, wouldn't it?

    Define 'get out of' in this context. Any process forked by a
    shell is by definition a child of the shell; the only way it can
    become a child of some other process (pid=1 (init)) is for the
    shell to "orphan" the child by exiting before the child exits.

    The shell itself is known as a 'session leader'. A session consists
    of one or more 'process groups' (a concept developed originally in BSD4).
    Each process group contains one or more processes. One, and only
    one, of the process groups within a session is considered the foreground process group and is allowed access to the "controlling" terminal of the session. Background process groups run in the background until they need
    to access the terminal, in which case a signal is thrown (e.g. SIGTTIN) and
    the process will be suspended until the shell makes the process group the foreground process group.

    You'll find a great deal of useful information, if you can find a
    copy, in:

    Session Management in System V Release 4
    Usenix Conference Proceedings Winter 1989 pp. 365-375.
    Tim Williams (Tim was an engineer at AT&T/USL).

    See setsid(2), getsid(2), setpgid(2), setpgrp(2), tcgetsid(3)
    and other man pages for more information.

    https://pubs.opengroup.org/onlinepubs/9699919799/functions/setsid.html# https://pubs.opengroup.org/onlinepubs/9699919799/functions/setpgid.html#

    Note that the full source is available on-line for a dozen
    or more open source shells (Korn shell, Bourne Again Shell, Bourne Shell,
    C Shell, and many others); useful to see how they handle
    sessions.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Meredith Montgomery@21:1/5 to Scott Lurndal on Fri Dec 31 08:06:06 2021
    scott@slp53.sl.home (Scott Lurndal) writes:

    Meredith Montgomery <mmontgomery@levado.to> writes:
    I'm helping someone to improve a CMS-type of web system. This web
    system sometimes needs to run some background processes --- such as send
    a batch of e-mail or other process that takes a while to finish.

    I see the challenge here as just writing something that will look like a >>very basic UNIX shell --- so I'll call it ``web-api-shell'' from now on. >>(``Web'' because it will be used by a web system through some API.)

    This thing must run really well. It has to be flawless. I'm looking
    for design principles and advice.

    (*) Where will it run

    It will run on GNU systems running the Linux kernel.

    (*) My own thoughts

    The interface to shell will be through HTTP requests, so this shell will >>likely be a web server of some sort. But before I get involved in the
    web at all, I need the shell working flawlessly.

    So I need a laboratory first. I could write a program that reads some >>named pipe on disk to get commands such as ``run this or that'' while I >>work. (Later I can write a web server replacing this named-pipe >>interface.)

    Just like a UNIX shell, this web-api-shell must know all every process
    it runs. I suppose the work is essentially fork(), execve() followed by >>waitpid().

    One concern I have is the following. Is it possible for a process to >>simply ``get out of'' the shell? What do I mean by that? A process
    that does fork() and puts itself into background would leave the >>web-api-shell's control, wouldn't it?

    Define 'get out of' in this context.

    Say the shell runs a process P and suppose P forks() --- creating Q ---
    then P exits, leaving Q orphaned. I believe Q's parent is now process
    1. I wonder if this creates difficulties to my objectives. It's not
    clear to me. (Perhaps this project should use a PID namespace provided
    by the Linux kernel, say. Perhaps I'm worried about irrelevant things.
    Maybe I should just run every process under a certain uid and search the process table by that uid and these are the processes I should manage:
    on the web, I will list out all current background processes and let the sysadmin send signals to them --- kill them, for example. I can't let
    the sysadmin send signals to other [unrelated] processes and I should
    always display every process that directly or indirectly was born by way
    of this background-process-manager, so I must book-keep every process
    spawned.)

    Any process forked by a shell is by definition a child of the shell;
    the only way it can become a child of some other process (pid=1
    (init)) is for the shell to "orphan" the child by exiting before the
    child exits.

    Written down. Thanks.

    The shell itself is known as a 'session leader'. A session consists
    of one or more 'process groups' (a concept developed originally in BSD4). Each process group contains one or more processes. One, and only
    one, of the process groups within a session is considered the foreground process group and is allowed access to the "controlling" terminal of the session. Background process groups run in the background until they need
    to access the terminal, in which case a signal is thrown (e.g. SIGTTIN) and the process will be suspended until the shell makes the process group the foreground process group.

    You'll find a great deal of useful information, if you can find a
    copy, in:

    Session Management in System V Release 4
    Usenix Conference Proceedings Winter 1989 pp. 365-375.
    Tim Williams (Tim was an engineer at AT&T/USL).

    See setsid(2), getsid(2), setpgid(2), setpgrp(2), tcgetsid(3)
    and other man pages for more information.

    https://pubs.opengroup.org/onlinepubs/9699919799/functions/setsid.html# https://pubs.opengroup.org/onlinepubs/9699919799/functions/setpgid.html#

    Note that the full source is available on-line for a dozen
    or more open source shells (Korn shell, Bourne Again Shell, Bourne Shell,
    C Shell, and many others); useful to see how they handle
    sessions.

    Thank you so much for info. Very appreciated.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Scott Lurndal@21:1/5 to Meredith Montgomery on Fri Dec 31 15:28:05 2021
    Meredith Montgomery <mmontgomery@levado.to> writes:
    scott@slp53.sl.home (Scott Lurndal) writes:


    One concern I have is the following. Is it possible for a process to >>>simply ``get out of'' the shell? What do I mean by that? A process
    that does fork() and puts itself into background would leave the >>>web-api-shell's control, wouldn't it?

    Define 'get out of' in this context.

    Say the shell runs a process P and suppose P forks() --- creating Q ---
    then P exits, leaving Q orphaned. I believe Q's parent is now process
    1. I wonder if this creates difficulties to my objectives.

    Process Q is still part of the shell "session" until it explicitly
    starts a new session (setsid). Consider the case where process Q
    is another shell.

    What you probably want is to run your application in a container.

    https://en.wikipedia.org/wiki/LXC

    For the case you enumerated (background tasks) you may want
    to consider a cooperative tasking setup where multiple independent
    processes communicate (e.g. the web system communicates through
    some form of inter process communication (pipes, unix domain sockets,
    network sockets, FIFO's or posix message queues) with a mailer or other
    process rather than forking and exec'ing each time you need to perform
    the task).

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jorgen Grahn@21:1/5 to Meredith Montgomery on Sat Jan 1 01:03:58 2022
    On Thu, 2021-12-30, Meredith Montgomery wrote:
    I'm helping someone to improve a CMS-type of web system. This web
    system sometimes needs to run some background processes --- such as send
    a batch of e-mail or other process that takes a while to finish.

    I see the challenge here as just writing something that will look like a
    very basic UNIX shell --- so I'll call it ``web-api-shell'' from now on. (``Web'' because it will be used by a web system through some API.)

    This thing must run really well. It has to be flawless. I'm looking
    for design principles and advice.

    (*) Where will it run

    It will run on GNU systems running the Linux kernel.

    (*) My own thoughts

    The interface to shell will be through HTTP requests, so this shell will likely be a web server of some sort. But before I get involved in the
    web at all, I need the shell working flawlessly.
    ...

    I don't know CMS-type web systems at all, but haven't web servers done
    this since CGI was invented, in the early 1990s? There's always been
    a need to run Unix processes as part of serving HTTP requests, so
    there must be a wealth of patterns and frameworks already. Even
    though I suspect a lot of the frameworks want you to reimplement that
    Unix command you want to run in Node.js, or whatever's fashionable
    this week.

    I'm tired, and haven't read very carefully, but I don't think you've
    explained your requirements very well. You're saying this thing is
    like the Unix shell. Then why not let it be the Unix shell? I don't
    see how writing your own with a HTTP interface instead of pipes would
    help[0]. You're just stuck with a socket to manage instead of a pipe.

    If you want to run it on some other host, there's always ssh.

    /Jorgen

    [0] Except it would be more fashionable.

    --
    // Jorgen Grahn <grahn@ Oo o. . .
    \X/ snipabacken.se> O o .

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jorgen Grahn@21:1/5 to Meredith Montgomery on Sat Jan 1 00:46:05 2022
    On Thu, 2021-12-30, Meredith Montgomery wrote:
    I'm helping someone to improve a CMS-type of web system. This web
    system sometimes needs to run some background processes --- such as send
    a batch of e-mail or other process that takes a while to finish.

    The right way to send mail is to pipe it into sendmail(8); I don't
    think that takes very long. I'm mostly bringing this up because it
    could perhaps serve as a pattern for some of the other tasks. And
    because I hate mail done badly.

    Using sendmail(8) means it's fire-and-forget. The local MTA takes responsibility for delivery, does a really good job, and most failures
    end up as bounced mails.

    [snip]

    /Jorgen

    --
    // Jorgen Grahn <grahn@ Oo o. . .
    \X/ snipabacken.se> O o .

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)