• Simple webserver

    From Janis Papanagnou@21:1/5 to All on Thu Oct 19 00:09:23 2023
    I am pondering about writing a client/server software with
    websockets as communication protocol. The clients will run
    in browser as Javascript programs and the server may be in
    any (any sensible) programming language running standalone
    to be connected remotely by the browser-based JS clients.

    I found a Python sample[*] but I am neither familiar with
    Python nor with the 'simple_websocket_server' package that
    is used in that sample code. But the code looks so simple
    that I'm considering to learn and use Python for the task.

    The requirements I have are quite simple; I want to get the
    client "address"/identifier from an incoming message, store
    it in a list, and send responses to all active clients for
    which addresses have been stored.

    Can anyone tell me whether a simple extension of that "echo
    incoming message" sample[*] would be easily possible with
    Python and with that 'simple_websocket_server' package used?

    Thanks for any hints (or search keywords, or code samples)!

    Janis

    [*] https://pypi.org/project/simple-websocket-server/

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Chris Angelico@21:1/5 to python-list@python.org on Thu Oct 19 10:23:41 2023
    On Thu, 19 Oct 2023 at 10:07, Janis Papanagnou via Python-list <python-list@python.org> wrote:

    I am pondering about writing a client/server software with
    websockets as communication protocol. The clients will run
    in browser as Javascript programs and the server may be in
    any (any sensible) programming language running standalone
    to be connected remotely by the browser-based JS clients.

    I found a Python sample[*] but I am neither familiar with
    Python nor with the 'simple_websocket_server' package that
    is used in that sample code. But the code looks so simple
    that I'm considering to learn and use Python for the task.

    The requirements I have are quite simple; I want to get the
    client "address"/identifier from an incoming message, store
    it in a list, and send responses to all active clients for
    which addresses have been stored.

    Can anyone tell me whether a simple extension of that "echo
    incoming message" sample[*] would be easily possible with
    Python and with that 'simple_websocket_server' package used?

    Thanks for any hints (or search keywords, or code samples)!

    Oooh you've touched on one of my favourite topics. I *love* networking
    and communication, and websockets are one of my well-used
    technologies. Let's do this!!

    I've never used the "simple_websocket_server" you mentioned, but I've
    used this one in a few projects:

    https://pypi.org/project/websockets/

    Be aware that it is designed with asyncio in mind, so if you prefer
    different concurrency models, you may need to look elsewhere. But I've
    had good success with this one.

    Broadly speaking, your ideas are great. Any programming language CAN
    be used for the server (and I've used several, not just Python).

    My personal preference is to build a protocol on top of websockets, for example:

    * All messages are "text", and are JSON-encoded
    * All messages represent objects (in Python, dictionaries) with a
    "cmd" attribute
    * The first message sent by the client has cmd "init" and specifies a
    "type" and "group".
    * The server tracks all connected clients by their groups, and can
    broadcast messages to everyone in a group.

    Here's one example, actually one of my brother's projects but I
    contributed to the websocket aspects:

    https://github.com/stephenangelico/BioBox/blob/master/browser.py

    And here's one that doesn't actually use Python, but uses all the same
    ideas; this is the JS end:

    https://github.com/Rosuav/StilleBot/blob/master/httpstatic/ws_sync.js

    (It has quite a bit more sophistication than you'll need to get
    started with, but shows how the protocol can expand as needed.)

    So! Tying this back in with your goals:

    The requirements I have are quite simple; I want to get the
    client "address"/identifier from an incoming message, store
    it in a list, and send responses to all active clients for
    which addresses have been stored.

    Sounds to me like the best way would be for the socket group to be the identifier of the client. You could augment the "init" message to
    include some sort of authentication, or alternatively, rely on other
    forms of authentication; a websocket established to the same origin as
    the page itself can take advantage of regular browser credentials.

    Once that's established, you can have a message from one client result
    in the server sending out that message to all clients for the
    recipient's address.

    The reason I'm talking about "groups" here instead of simply having
    one client per address is that it scales well to one person having
    multiple tabs open, or having the app on their phone as well as their
    computer, or anything like that. Messages will arrive on all of them.

    Hope that's enough to get you started! I'd be delighted to help
    further if you run into difficulties.

    ChrisA

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Janis Papanagnou@21:1/5 to Chris Angelico on Thu Oct 19 03:04:40 2023
    On 19.10.2023 01:23, Chris Angelico wrote:
    [snip]

    Hope that's enough to get you started! I'd be delighted to help
    further if you run into difficulties.

    Thanks for your quick reply, Chris! This is already great information!
    I'll dive into your resources soon, and I also appreciate your offer
    and will probably come back soon with a question... - Thanks again!

    Janis

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Janis Papanagnou@21:1/5 to Chris Angelico on Fri Oct 20 12:13:02 2023
    On 19.10.2023 01:23, Chris Angelico wrote:

    Broadly speaking, your ideas are great. Any programming language CAN
    be used for the server (and I've used several, not just Python).

    Out of curiosity; what where these languages? - If there's one I
    already know I might save some time implementing the server. :-)

    Janis

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Chris Angelico@21:1/5 to python-list@python.org on Fri Oct 20 22:34:32 2023
    On Fri, 20 Oct 2023 at 22:31, Janis Papanagnou via Python-list <python-list@python.org> wrote:

    On 19.10.2023 01:23, Chris Angelico wrote:

    Broadly speaking, your ideas are great. Any programming language CAN
    be used for the server (and I've used several, not just Python).

    Out of curiosity; what where these languages? - If there's one I
    already know I might save some time implementing the server. :-)


    I've done websocket servers in Python, Node.js, and Pike, and possibly
    others but I can't recall at the moment. Might have done one in Ruby,
    but that would have just been part of playing around and comparing
    features ("how easy is it to do <X> in Ruby").

    ChrisA

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From De ongekruisigde@21:1/5 to Chris Angelico on Fri Oct 20 11:40:24 2023
    On 2023-10-20, Chris Angelico <rosuav@gmail.com> wrote:
    On Fri, 20 Oct 2023 at 22:31, Janis Papanagnou via Python-list
    <python-list@python.org> wrote:

    On 19.10.2023 01:23, Chris Angelico wrote:

    Broadly speaking, your ideas are great. Any programming language CAN
    be used for the server (and I've used several, not just Python).

    Out of curiosity; what where these languages? - If there's one I
    already know I might save some time implementing the server. :-)


    I've done websocket servers in Python, Node.js, and Pike, and possibly
    others but I can't recall at the moment. Might have done one in Ruby,
    but that would have just been part of playing around and comparing
    features ("how easy is it to do <X> in Ruby").

    ChrisA

    *Big list of http static server one-liners*

    Each of these commands will run an ad hoc http static server
    in your current (or specified) directory, available at
    http://localhost:8000. Use this power wisely.

    <https://gist.github.com/willurd/5720255>

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Paul Rubin@21:1/5 to Janis Papanagnou on Fri Oct 20 14:05:23 2023
    Janis Papanagnou <janis_papanagnou+ng@hotmail.com> writes:
    I found a Python sample[*] but I am neither familiar with
    Python nor with the 'simple_websocket_server' package that
    is used in that sample code. But the code looks so simple
    that I'm considering to learn and use Python for the task.

    I've generally used ThreadingServer(SocketServer) for this purpose
    and I think threads are less confusing than async, and performance is
    fine if the concurrency level is not too high. But, trying to write a
    web server in Python if you don't know Python doesn't seem like a great
    idea, except as a learning project.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Janis Papanagnou@21:1/5 to Paul Rubin on Sat Oct 21 04:03:32 2023
    On 20.10.2023 23:05, Paul Rubin wrote:
    Janis Papanagnou <janis_papanagnou+ng@hotmail.com> writes:
    I found a Python sample[*] but I am neither familiar with
    Python nor with the 'simple_websocket_server' package that
    is used in that sample code. But the code looks so simple
    that I'm considering to learn and use Python for the task.

    I've generally used ThreadingServer(SocketServer) for this purpose
    and I think threads are less confusing than async, and performance is
    fine if the concurrency level is not too high. But, trying to write a
    web server in Python if you don't know Python doesn't seem like a great
    idea, except as a learning project.

    I have a couple decades experience with about a dozen programming
    languages (not counting assemblers). Asynchronous processing, IPC, multi-processing, client/server architectures, multi-threading,
    semaphores, etc. etc. are concepts that are not new to me.

    I'm not, literally, intending to write a web-server. It's a JS
    application that is running in (browser based) clients, and the
    server is just centrally coordinating the client applications.

    My expectation would be that any sophistically designed socket/
    web-socket library would not impose any risk. And the intended
    server by itself has only very limited requirements; listening to
    incoming request, storing some client information, broadcasting
    to the attached clients. Basically just (informally written):

    init server
    forever:
    wait for request(s) -> queue
    handle requests from queue (sequentially):
    store specific information from new registered clients
    broadcast some information to all registered clients

    It seems to me that multi-threading or async I/O aren't necessary.

    I'd like to ask; where do you see the specific risks with Python
    (as language per se) and it's (web-socket-)libraries here?

    If the web-socket IPC is well supported the algorithmic parts in
    Python seem trivial to learn and implement. - Or am I missing
    something?

    (A brief search gave me the impression that for JS communication
    web-sockets would be the method to use. Otherwise I'd just use
    basic Unix domain sockets for the purpose and write it, say, in
    C or C++ that I already know. But I don't know whether (or how)
    plain sockets are used from JS running in a browser. Here I'm
    lacking experience. And that lead me to have a look at Python,
    since the web-sockets/server examples that I found looked simple.)

    Janis

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Chris Angelico@21:1/5 to python-list@python.org on Sun Oct 22 05:58:17 2023
    On Sun, 22 Oct 2023 at 04:13, Janis Papanagnou via Python-list <python-list@python.org> wrote:
    I have a couple decades experience with about a dozen programming
    languages (not counting assemblers). Asynchronous processing, IPC, multi-processing, client/server architectures, multi-threading,
    semaphores, etc. etc. are concepts that are not new to me.

    Oh, sweet, sweet, then you should be fine with the library I
    suggested. It's certainly served me well (and I have similar
    experience, having learned networking mainly on OS/2 in the 1990s).

    My expectation would be that any sophistically designed socket/
    web-socket library would not impose any risk. And the intended
    server by itself has only very limited requirements; listening to
    incoming request, storing some client information, broadcasting
    to the attached clients. Basically just (informally written):

    init server
    forever:
    wait for request(s) -> queue
    handle requests from queue (sequentially):
    store specific information from new registered clients
    broadcast some information to all registered clients

    It seems to me that multi-threading or async I/O aren't necessary.

    Technically that's true, but "wait for request(s)" has to handle (a)
    new incoming sockets, (b) messages from currently-connected sockets,
    and possibly (c) sockets now being writable when previously they
    blocked. So you have most of the work of async I/O. Since the
    library's been built specifically for asyncio, that's the easiest.

    I'd like to ask; where do you see the specific risks with Python
    (as language per se) and it's (web-socket-)libraries here?

    If the web-socket IPC is well supported the algorithmic parts in
    Python seem trivial to learn and implement. - Or am I missing
    something?

    Pretty trivial, yeah. You shouldn't have too much trouble here I expect.

    (A brief search gave me the impression that for JS communication
    web-sockets would be the method to use. Otherwise I'd just use
    basic Unix domain sockets for the purpose and write it, say, in
    C or C++ that I already know. But I don't know whether (or how)
    plain sockets are used from JS running in a browser. Here I'm
    lacking experience. And that lead me to have a look at Python,
    since the web-sockets/server examples that I found looked simple.)

    Yes, that's correct. You can't use plain sockets from inside a web
    browser, mainly because they offer way way too much flexibility (JS
    code is untrusted and is now running on your computer, do you really
    want that to be able to telnet to anything on your LAN?). So
    websockets are the way to go. There are other similar technologies,
    but for this sort of "broadcast to connected clients" messaging
    system, websockets rule.

    ChrisA

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Dieter Maurer@21:1/5 to Janis Papanagnou on Sun Oct 22 19:35:13 2023
    Janis Papanagnou wrote at 2023-10-21 04:03 +0200:
    ...
    I'd like to ask; where do you see the specific risks with Python
    (as language per se) and it's (web-socket-)libraries here?

    The web server in Python's runtime library is fairly simple,
    focusing only on the HTTP requirements.

    You might want additional things for an HTTP server
    exposed on the internet which should potentially handle high trafic:
    e.g.

    * detection of and (partial) protection against denial of service attacks,
    * load balancing,
    * virtual hosting
    * proxing
    * URL rewriting
    * high throughput, low latency

    Depending on your requirements, other web servers might be preferable.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From pozz@21:1/5 to All on Tue Oct 24 18:01:39 2023
    Il 19/10/2023 00:09, Janis Papanagnou ha scritto:
    I am pondering about writing a client/server software with
    websockets as communication protocol. The clients will run
    in browser as Javascript programs and the server may be in
    any (any sensible) programming language running standalone
    to be connected remotely by the browser-based JS clients.

    I found a Python sample[*] but I am neither familiar with
    Python nor with the 'simple_websocket_server' package that
    is used in that sample code. But the code looks so simple
    that I'm considering to learn and use Python for the task.

    The requirements I have are quite simple; I want to get the
    client "address"/identifier from an incoming message, store
    it in a list, and send responses to all active clients for
    which addresses have been stored.

    Can anyone tell me whether a simple extension of that "echo
    incoming message" sample[*] would be easily possible with
    Python and with that 'simple_websocket_server' package used?

    Thanks for any hints (or search keywords, or code samples)!

    Janis

    [*] https://pypi.org/project/simple-websocket-server/

    I'm not sure, but MQTT protocol could help for this application.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Frank Millman@21:1/5 to Dieter Maurer via Python-list on Wed Oct 25 09:57:17 2023
    On 2023-10-22 7:35 PM, Dieter Maurer via Python-list wrote:

    The web server in Python's runtime library is fairly simple,
    focusing only on the HTTP requirements.

    You might want additional things for an HTTP server
    exposed on the internet which should potentially handle high trafic:
    e.g.

    * detection of and (partial) protection against denial of service attacks,
    * load balancing,
    * virtual hosting
    * proxing
    * URL rewriting
    * high throughput, low latency

    Depending on your requirements, other web servers might be preferable.

    Dieter's response was very timely for me, as it provides some answers to
    a question that I was thinking of posting. My use-case is reasonably
    on-topic for this thread, so I won't start a new one, if that is ok.

    I am writing a business/accounting application. The server uses Python
    and asyncio, the client is written in Javascript. The project is inching towards a point where I may consider releasing it. My concern was
    whether my home-grown HTTP server was too simplistic for production, and
    if so, whether I should be looking into using one of the more
    established frameworks. After some brief investigation into Dieter's
    list of additional requirements, here are my initial thoughts. Any
    comments will be welcome.

    I skimmed through the documentation for flask, Django, and FastAPI. As
    far as I can tell, none of them address the points listed above
    directly. Instead, they position themselves as one layer in a stack of technologies, and rely on other layers to provide additional
    functionality. If I read this correctly, there is nothing to stop me
    doing the same.

    Based on this, I am considering the following -

    1. Replace my HTTP handler with Uvicorn. Functionality should be the
    same, but performance should be improved.

    2. Instead of running as a stand-alone server, run my app as a
    reverse-proxy using Nginx. I tested this a few years ago using Apache,
    and it 'just worked', so I am fairly sure that it will work with Nginx
    as well. Nginx can then provide the additional functionality that Dieter
    has mentioned.

    My main concern is that, if I do release my app, I want it to be taken seriously and not dismissed as 'Mickey Mouse'. Do you think the above
    changes would assist with that?

    When I talk about releasing it, it is already available on Github here - https://github.com/FrankMillman/AccInABox.

    You are welcome to look at it, but it needs a lot of tidying up before
    it will be ready for a wider audience.

    Frank Millman

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Chris Angelico@21:1/5 to python-list@python.org on Wed Oct 25 19:56:20 2023
    On Wed, 25 Oct 2023 at 19:00, Frank Millman via Python-list <python-list@python.org> wrote:
    2. Instead of running as a stand-alone server, run my app as a
    reverse-proxy using Nginx. I tested this a few years ago using Apache,
    and it 'just worked', so I am fairly sure that it will work with Nginx
    as well. Nginx can then provide the additional functionality that Dieter
    has mentioned.

    This, I would recommend. In general, tools like Django and Flask and
    such are aimed at the creation of web apps, but not necessarily web
    servers; it's convenient to have a high-performance web server like
    Nginx or Apache, and then it passes along requests to the application
    server. This provides a number of benefits:

    1. Static files can be served directly, without involving the app
    2. Need to scale horizontally? Run two or more copies of your app and
    have the web server share requests among them.
    3. App crashed? The web server can return an HTTP 503 error so end
    users aren't left wondering what's going on.
    4. DOS protection can be done in the web server (although it could
    also be done in a firewall, or any other level that's appropriate)

    My main concern is that, if I do release my app, I want it to be taken seriously and not dismissed as 'Mickey Mouse'. Do you think the above
    changes would assist with that?

    You shouldn't be dismissed as Still-In-Copyright-Decades-Old-Rodent
    even if you don't make those changes. I've been hosting a number of
    servers, some fairly home-grade, and nobody's ever told me that it
    looks bad. If you want to be respected, the main thing is to have
    something that people find interesting - everything else is secondary.
    But there are a few points to consider:

    * Performance. People respect something that's snappy and responsive
    more than they respect something where a single HTTP request takes
    anywhere from 2.89 seconds to nearly 30 seconds. And no, I'm totally
    not still mindblown at having seen this kind of appalling performance
    from a published and very expensive API.
    * Have your own domain name. https://rosuav.github.io/AntiSocialMedia/
    is a toy; https://devicat.art/ is a professional web site. (They're
    not related, and the first one really is just a toy that I whipped up
    in like half an hour.)
    * Have a proper SSL certificate. It looks *really bad* to have a
    broken or outdated certificate (or none at all, these days).
    LetsEncrypt can do that for you, no charge.
    * Put some effort into styling. Yeah, I know, most of my web sites are
    ugly, so I shouldn't talk. But things definitely look more
    professional if you take the time to style them up a bit.

    When I talk about releasing it, it is already available on Github here - https://github.com/FrankMillman/AccInABox.

    You are welcome to look at it, but it needs a lot of tidying up before
    it will be ready for a wider audience.

    Cool! This is particularly of note to me personally. Back in the 90s,
    I was working in the family business, and we used a fairly clunky
    piece of professionally-written software (and if you want stories, ask
    me about overnight runs of report generation, or Ctrl-Alt-Shift and
    old mouse pedal importing, or 32-bit installers for 16-bit
    applications, or a bunch of other things). There was, for a while, a
    theory of us designing our own accounting system, but it turns out,
    that's a really REALLY big job, and it's less effort to keep using the
    thing you already have.

    Your README, dating from 9 years ago, says that you support/require
    Python 3.4 - that's kinda ancient now. If you want to take advantage
    of newer features, I think you should be safe bumping that up a long
    way. :)

    I wouldn't recommend sqlite3 for any production work here, but it's
    good for a demo. Postgres is a far better choice if you're going to be
    storing your vital information in this.

    You can list vital packages in a file called requirements.txt - this
    will be recognized by both people and automated tooling.

    .... huh. I'm listed as a contributor. I'll be quite honest, I do not
    remember this, but presumably you shared this previously! Looks like
    all I contributed was a minor suggestion and commit, but still, I have absolutely no memory. LOL.

    Looks pretty good there. I don't have time right now to download and
    install it for a proper test, but based on flipping through the code,
    looks like you have something decent going on.

    ChrisA

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Dieter Maurer@21:1/5 to Frank Millman on Wed Oct 25 13:26:34 2023
    Frank Millman wrote at 2023-10-25 09:57 +0200:
    ...
    Based on this, I am considering the following -

    1. Replace my HTTP handler with Uvicorn. Functionality should be the
    same, but performance should be improved.

    2. Instead of running as a stand-alone server, run my app as a
    reverse-proxy using Nginx. I tested this a few years ago using Apache,
    and it 'just worked', so I am fairly sure that it will work with Nginx
    as well. Nginx can then provide the additional functionality that Dieter
    has mentioned.

    Good ideas.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)