In an effort to better learn network programming in Ada, I've been working through the Protohacker Challenges (https://protohackers.com/), and the current challenge (number 3) is to create a chat server.connected Clients.
I am using a TCP Connections Server with Simple Components, specifically a Connection_State_Machine, but I've run into a problem. I'm trying to send a message received via "procedure Process_Packet (Client : in out Server_Connection)" to all
My (potentially incorrect) thought on how to accomplish this is to iterate through all of the clients currently connected, and use Send to send the message received to those clients. I've been struggling with how to actually do this though, since Icouldn't use "function Get_Clients_Count (Listener : Connections_Server) return Natural" from within Process_Packets.
Another thought I had could be to just place every message received in a central queue, and then once all of the packets have been received, to then process that queue and send the results to every connected client.objects anyway.
I tried overriding "procedure On_Worker_Start (Listener : in out Connections_Server)", thinking that I could use it to read such a queue, but it never seemed to be called from within my program and I'm still unsure how to iterate through the Connection
Am I approaching this the right way, or am I missing something very obvious? I've read the test files that came with Simple Components, including the data server but couldn't see a way to get each client to interact with each other. If I didn'texplain this well enough, please let me know, I'll be happy to clarify.
In an effort to better learn network programming in Ada, I've been working through the Protohacker Challenges (https://protohackers.com/), and the current challenge (number 3) is to create a chat server.
sockets (socket select).
P.S. This poses difficulties for users, who see all communication turned upside down being driven by arbitrary socket events rather than by the protocol logic. This was a reason I argued for introducing co-routines
with task interface in Ada.
I am not sure how familiar you are with Network programming in general (not just as it would be done in Ada).[...]
Using a blocking Send could actually kill your performance.
A similar approach when receiving messages from clients, by the way. The message might have sent only part of its message, so you
need to give up temporarily, and come back to it when poll tells you there is something new to read.
Well, if there is Linux kernel level support why it is not used in
socket select as it is in epoll? I would expect them do that at some
point or drop epoll... (:-))
Yes, state machine is what I want to avoid. With complex layered
protocols it imposes incredible difficulties requiring auxiliary stacks
and buffers with errors almost intractable either by testing or by
formal proofs.
I'd like to have special Ada "tasks" acting as co-routines on top of
proper tasks yielding when the socket buffer is empty or full.
sockets (socket select).
Have you taken a look at epoll(), on linux ?
It is so much more natural to use, and so much more efficient in practice. The example I mentioned above (a server with 25_000 concurrent connections) cannot work with select (which only
accepts file descriptors up to 1024), and is slow with poll (since the result of the latter is the number of events, and we
need to iterate over all registered sockets every time).
P.S. This poses difficulties for users, who see all communication turned
upside down being driven by arbitrary socket events rather than by the
protocol logic. This was a reason I argued for introducing co-routines
with task interface in Ada.
In my own code, I basically provide an epoll-based generic framework. One of the formal parameters is a `Job_Type`
with one primitive operation `Execute`.
The latter is initially called when a new connection is established, and is expected to do as much non-blocking work
as it can (Execute is run in one of the worker tasks). When it cannot make progress, it returns a tuple (file_descriptor, type_of_event_to_wait_for)
to indicate when it needs to be called again in the future, for instance some data became available to read on the
specified file_descriptor.
The intent is that the `Job_Type` is implemented as a state machine internally.
Of course, a state machine is one of the two ways I know (along with a task) to implement the equivalent of
a co-routine in Ada. So I 100% agree with you that co-routines would be very useful in simplifying user code,
in particular in the scenario we are discussing here !
On Monday, February 13, 2023 at 11:55:10 AM UTC+1, Dmitry A. Kazakov wrote:
Well, if there is Linux kernel level support why it is not used in
socket select as it is in epoll? I would expect them do that at some
point or drop epoll... (:-))
Because in practice the linux developers don't get to modify such APIs, which are mandated by Posix, or Unix, or some RFC.
So the API for select and poll will *never* change.
epoll is definitely the modern approach on linux, until of course someone finds
something even better. epoll is fully thread safe too, which is very nice when
used from Ada.
Using select() is totally outdated at this point, and means you can never handle
more than 1000 simultaneous clients, and that only if you do not have other file descriptors open (database, files,...)
The person who developed GNAT.Sockets has left AdaCore a while ago, so "they" (which
I assume is what your message was referring to) are actually unlikely to update that.
They also have strong concerns about platform-agnostic support, and epoll is linux-specific
at this point (likely also BSD).
There exists multiple libraries out there that provide an API
common to multiple platforms, and that use epoll on linux. Maybe that's what would make
sense, but nowadays with Alire, I would expect someone to build a crate there rather than
AdaCore modify GNAT.Sockets.
I'd like to have special Ada "tasks" acting as co-routines on top of
proper tasks yielding when the socket buffer is empty or full.
This is an approach we had discussed at AdaCore before I left. There are multiple drawbacks
here: the limited stack size for tasks by default (2MB), the fact that entries cannot return indefinite
types, the fact that currently those tasks are assigned to OS threads (so too many of them does
impact resource usage),...
A colleague had found an external library that would provide several stacks and thus let people
implement actual co-routines. We did not do much more work on that, but it was a nice proof
of concept, and efficient. I think things are mostly blocked now, as the ARG has been discussing
these topics for quite a few years now.
On 2023-02-13 12:07, Emmanuel Briot wrote:
On Monday, February 13, 2023 at 11:55:10 AM UTC+1, Dmitry A. Kazakov
wrote:
I'd like to have special Ada "tasks" acting as co-routines on top of
proper tasks yielding when the socket buffer is empty or full.
This is an approach we had discussed at AdaCore before I left. There
are multiple drawbacks
here: the limited stack size for tasks by default (2MB), the fact that
entries cannot return indefinite
types, the fact that currently those tasks are assigned to OS threads
(so too many of them does
impact resource usage),...
My idea is to have these as pseudo-tasks scheduled by the Ada run-time
and not mapped onto any OS threads. A proper thread would pick up such a
task and run it until it yields. The crucial point is to use the stack
of the pseudo-task in place of the thread's stack or backing it up and cleaning the portion of the stack at the point of yielding, whatever.
A colleague had found an external library that would provide several
stacks and thus let people
implement actual co-routines. We did not do much more work on that,
but it was a nice proof
of concept, and efficient. I think things are mostly blocked now, as
the ARG has been discussing
these topics for quite a few years now.
I have an impression that ARG's view on co-routines totally ignores the
use case of communication stacks and other cases state machines show
their ugly faces...
On 2023-02-13 13:57, Dmitry A. Kazakov wrote:
I have an impression that ARG's view on co-routines totally ignores
the use case of communication stacks and other cases state machines
show their ugly faces...
So your co-routines would (1) have their own stack and (2) be
independently schedulable, which implies (3) having their own execution context (register values, instruction pointer, etc.)
How is that
different from the Ada concept of a "task"?
How could the ARG separate
between a "task" and a "co-routine" in the Ada RM?
So is your problem only that using OS threads is less "efficient" than switching and scheduling threads of control in the run-time system?
If
so, that seems to be a quality-of-implementation issue that could be
solved in a compiler-specific way, and not an issue with the Ada
language itself.
The point (from Emmanuel) that task entries cannot return indefinite
types is certainly a language limitation, but seems to have little to do
with the possible differences between tasks and co-routines, and could
be addressed on its own if Ada users so desire.
The point (from Emmanuel) that task entries cannot return indefinite
types is certainly a language limitation, but seems to have little to do
with the possible differences between tasks and co-routines, and could
be addressed on its own if Ada users so desire.
epoll is definitely the modern approach on linux, until of course someone finds
something even better. epoll is fully thread safe too, which is very nice when
used from Ada.
They also have strong concerns about platform-agnostic support, and epoll is linux-specific
at this point (likely also BSD). There exists multiple libraries out there that provide an API
common to multiple platforms, and that use epoll on linux. Maybe that's what would make
sense, but nowadays with Alire, I would expect someone to build a crate there rather than
AdaCore modify GNAT.Sockets.
On 2023-02-13 14:22, Niklas Holsti wrote:
On 2023-02-13 13:57, Dmitry A. Kazakov wrote:
I have an impression that ARG's view on co-routines totally ignores
the use case of communication stacks and other cases state machines
show their ugly faces...
So your co-routines would (1) have their own stack and (2) be
independently schedulable, which implies (3) having their own
execution context (register values, instruction pointer, etc.)
Sure. You should be able to implement communication logic in a natural way:
1. Read n bytes [block until finished]
2. Do things
3. Write m bytes [block until finished]
4. Repeat
How is that different from the Ada concept of a "task"?
It is no different, that the whole point of deploying high level
abstraction: task instead of low level one: state machine.
How could the ARG separate between a "task" and a "co-routine" in the
Ada RM?
Syntax sugar does not bother me. I trust ARG to introduce a couple of reserved words in the most annoying way... (:-))
So is your problem only that using OS threads is less "efficient" than
switching and scheduling threads of control in the run-time system?
This too.
However the main purpose is control inversion caused by
callback architectures. A huge number of libraries are built on that
pattern. This OK for the library provider because it is the most natural
and efficient way. For the user implementing his own logic, be it communication protocol, GUI etc, it is a huge architectural problem as
it distorts the problem space logic. So the goal is to convert a callback/event driven architecture into plain control flow.
On 2023-02-13 17:10, Dmitry A. Kazakov wrote:
But why should the RM define two "task" concepts that have exactly the
same properties? Or should co-routines have some different properties?
That was the point of my question, apologies if I was unclear.
For example, co-routines could have the ability to form some kind of "mutually exclusive groups" such that there would be no need for mutual-exclusion controls such as protected objects between co-routines
in the same group.
What properties do you propose for co-routines that would not hold for ordinary tasks?
But you cannot ask the ARG to define co-routines as being "like tasks
but more efficient". :-)
However the main purpose is control inversion caused by callback
architectures. A huge number of libraries are built on that pattern.
This OK for the library provider because it is the most natural and
efficient way. For the user implementing his own logic, be it
communication protocol, GUI etc, it is a huge architectural problem as
it distorts the problem space logic. So the goal is to convert a
callback/event driven architecture into plain control flow.
Yes, yes. But tasks can do that. How would co-routines help, aside from perhaps being "more efficient"?
Hello everyone,
In an effort to better learn network programming in Ada, I've been
working through the Protohacker Challenges (https://protohackers.com/),
and the current challenge (number 3) is to create a chat server.
I am using a TCP Connections Server with Simple Components, specifically
a Connection_State_Machine, but I've run into a problem. I'm trying to
send a message received via "procedure Process_Packet (Client : in out Server_Connection)" to all connected Clients.
My (potentially incorrect) thought on how to accomplish this is to
iterate through all of the clients currently connected, and use Send to
send the message received to those clients. I've been struggling with
how to actually do this though, since I couldn't use "function Get_Clients_Count (Listener : Connections_Server) return Natural" from
within Process_Packets.
Another thought I had could be to just place every message received in a central queue, and then once all of the packets have been received, to
then process that queue and send the results to every connected client.
I tried overriding "procedure On_Worker_Start (Listener : in out Connections_Server)", thinking that I could use it to read such a queue,
but it never seemed to be called from within my program and I'm still
unsure how to iterate through the Connection objects anyway.
Am I approaching this the right way, or am I missing something very obvious? I've read the test files that came with Simple Components, including the data server but couldn't see a way to get each client to interact with each other. If I didn't explain this well enough, please
let me know, I'll be happy to clarify.
Kind regards,
AJ.
Hello everyone,
In an effort to better learn network programming in Ada, I've been working through the Protohacker Challenges (https://protohackers.com/), and the current challenge (number 3) is to create a chat server.
On 2023-02-13 17:26, Niklas Holsti wrote:
On 2023-02-13 17:10, Dmitry A. Kazakov wrote:
But why should the RM define two "task" concepts that have exactly the
same properties? Or should co-routines have some different properties?
That was the point of my question, apologies if I was unclear.
Ada has floating-point, fixed-point, universal real implementations of
the same concept of real number. Why?
Rendezvous is asymmetric client-server. In a callback scenario parties
are equivalent. Each can wait for another. With tasks a protected object
is usually thrown in to work it around.
The objective is twofold:
1. To simplify "pipeline" stuff. Each side need not to know what the
other side does or each other. They know the pipeline. The programmer
should use a call or a timed entry call or a selective accept. I leave
that open for now.
2. To allow sharing context of a single OS thread. So the term
"co-routine."
3. As I said above, the goal is to take plain I/O code (the client) and
reuse it with no adjustments with a server/provider/consumer in place of
the OS I/O subsystem.
On 2023-02-13 21:48, Dmitry A. Kazakov wrote:
On 2023-02-13 17:26, Niklas Holsti wrote:
On 2023-02-13 17:10, Dmitry A. Kazakov wrote:
But why should the RM define two "task" concepts that have exactly
the same properties? Or should co-routines have some different
properties? That was the point of my question, apologies if I was
unclear.
Ada has floating-point, fixed-point, universal real implementations of
the same concept of real number. Why?
Of course because these implementations have /different/ properties,
suitable for different uses. They are /not/ the same, and their
differences are defined in the Ada RM.
Perhaps you think that those differences are implicit in the term "co-routine", but I don't find it so, sorry.
Rendezvous is asymmetric client-server. In a callback scenario parties
are equivalent. Each can wait for another. With tasks a protected
object is usually thrown in to work it around.
Aha, so you are proposing that co-routines would have some other means,
not protected objects, for symmetric inter-co-routine communication.
What would that be, perhaps directly shared (unprotected) variables?
Then you have to ensure that those co-routines are implicitly mutually exclusive, right?
1. To simplify "pipeline" stuff. Each side need not to know what the
other side does or each other. They know the pipeline. The programmer
should use a call or a timed entry call or a selective accept. I leave
that open for now.
By "pipeline", do you mean an order-preserving stream of data from a
producer to a consumer? If so, a simple protected queue or buffer
between producer and consumer tasks implements it.
By leaving the details open, you leave your proposal fuzzy and hard to understand.
2. To allow sharing context of a single OS thread. So the term
"co-routine."
I don't think that property is implicit in the term "co-routine",
because co-routines can exist even without OS threads.
But most importantly, this property cannot be used to /define/ the "co-routine" concept in the Ada RM. That definition may of course be
/chosen/ so as to /allow/ this implementation.
However, this property is already satisfied for "tasks" in those Ada
systems that implement tasking in the RTS, without using multiple OS
threads. So the current definition of Ada "tasks" already allows such sharing.
3. As I said above, the goal is to take plain I/O code (the client)
and reuse it with no adjustments with a server/provider/consumer in
place of the OS I/O subsystem.
And that server/provider/consumer would be what?
Ada code in the
application? Or part of the Ada RTS? Would it have to be aware of all of
the OS I/O subsystem's services? For example, would it have to know
about the socket interface to networking?
Clearly co-routine and OS thread have different properties as
implementations of same concept of a "thread of control."
I couldn't get Send() to send more data than Available_To_Send (after calling it, Available_To_Send ended up returning 0, and continued to do so despite wrapping Send() in a loop),
but increasing the send buffer to 8kb per connection worked fine.
Creating a task in Ada, at least on linux, ends up creating a
pthread,
coroutines are managed by the go runtime (I believe in user space)
and have much less overhead to create or manage, since it's not
creating a specific thread.
Ada 202x supports the "parallel" block[3] though I understand no
runtime has utilized it yet-- would that end up being a coroutine or
is it meant for something else?
[3] http://www.ada-auth.org/standards/22rm/html/RM-5-6-1.html
| Sysop: | Keyop |
|---|---|
| Location: | Huddersfield, West Yorkshire, UK |
| Users: | 335 |
| Nodes: | 16 (2 / 14) |
| Uptime: | 150:22:10 |
| Calls: | 7,411 |
| Calls today: | 1 |
| Files: | 12,670 |
| Messages: | 5,613,280 |