• Sockets, Streams, and Element_Arrays: Much confusion

    From Mark Gardner@21:1/5 to All on Sat Dec 31 14:11:55 2022
    Hello, I've been having a bit of difficulty doing some UDP socket
    programming in Ada. As outlined in my stackoverflow question here (https://stackoverflow.com/q/74953052/7105391), I'm trying to reply to
    messages I am getting over UDP.

    GNAT.Sockets gives me a Stream_Element_Array, which I can't find any documentation on how to make use of other than "You should also be able
    to get a Stream, which you should use instead" (About ten years ago, on
    this very newsgroup, somebody said not to use streams with UDP, or at
    least not GNAT.Sockets.Stream).

    Adasockets gives me a String, which I can work with, except it throws
    away the from data recvfrom gives it, apparently making it impossible to
    reply to the querying address.

    At this point, I'm half-tempted to make my own binding, but as I've
    never done that sort of thing before, I thought I'd ask the wisdom of
    the Usenet if there is a way to convert a Stream_Element_Array into the
    exotic types of Unsigned_16 and String.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Dmitry A. Kazakov@21:1/5 to Mark Gardner on Sat Dec 31 14:11:11 2022
    On 2022-12-31 13:11, Mark Gardner wrote:

    GNAT.Sockets gives me a Stream_Element_Array, which I can't find any documentation on how to make use of other than "You should also be able
    to get a Stream, which you should use instead" (About ten years ago, on
    this very newsgroup, somebody said not to use streams with UDP, or at
    least not GNAT.Sockets.Stream).

    Stream_Element_Array is declared in Ada.Streams as

    type Stream_Element_Array is
    array(Stream_Element_Offset range <>) of
    aliased Stream_Element;

    For communication purpose it is an array of octets. Your datagram is represented as a Stream_Element_Array or a slice of.

    As for streams, yes, it does not make sense to use them for networking,
    unless you override all stream primitives. The reasons for that are

    - non-portability of predefined primitives
    - low efficiency for complex data types
    - encoding inefficiency as well

    You will need to handle some application protocol artifacts, checksums, counters, strange encodings, sequence number etc. It is easier to this
    directly on the Stream_Element_Array elements.

    And, well, do not use UDP, expect for broadcasting. There is no reason
    to use it. For multicast consider delivery-safe protocols like PGM. For
    single cast use TCP/IP. (If you need low latency see the socket NO_DELAY option)

    --
    Regards,
    Dmitry A. Kazakov
    http://www.dmitry-kazakov.de

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Mark Gardner@21:1/5 to Dmitry A. Kazakov on Sat Dec 31 15:50:29 2022
    On 31/12/2022 15:11, Dmitry A. Kazakov wrote:
    On 2022-12-31 13:11, Mark Gardner wrote:

    ...

    Stream_Element_Array is declared in Ada.Streams as

       type Stream_Element_Array is
          array(Stream_Element_Offset range <>) of
             aliased Stream_Element;

    For communication purpose it is an array of octets. Your datagram is represented as a Stream_Element_Array or a slice of.


    According to RM 13.13.1, "Stream_Element is mod implementation-defined"
    which to me says there is no guarantee that they will be octets, unless
    this is specified elsewhere?

    As for streams, yes, it does not make sense to use them for networking, unless you override all stream primitives. The reasons for that are

    - non-portability of predefined primitives
    - low efficiency for complex data types
    - encoding inefficiency as well

    You will need to handle some application protocol artifacts, checksums, counters, strange encodings, sequence number etc. It is easier to this directly on the Stream_Element_Array elements.

    So, how would I do this directly on the elements? I mean, if it is an octet-array to a string, I expect an element-to-element copy, or type conversion to work, but what about integers? Do I need to do something like My_Int:=Unsigned_8(octet(1))+2**8*Unsigned_8(octet(2));
    or whatever endianness demands? Or is this the time to learn how to use Unchecked_Conversion?

    And, well, do not use UDP, expect for broadcasting. There is no reason
    to use it. For multicast consider delivery-safe protocols like PGM. For single cast use TCP/IP. (If you need low latency see the socket NO_DELAY option)


    Well, my use case just so happens to be broadcasting, and
    re-broadcasting data across a binary-tree-like p2p network.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Dmitry A. Kazakov@21:1/5 to Mark Gardner on Sat Dec 31 15:16:05 2022
    On 2022-12-31 14:50, Mark Gardner wrote:
    On 31/12/2022 15:11, Dmitry A. Kazakov wrote:
    On 2022-12-31 13:11, Mark Gardner wrote:

    ...

    Stream_Element_Array is declared in Ada.Streams as

        type Stream_Element_Array is
           array(Stream_Element_Offset range <>) of
              aliased Stream_Element;

    For communication purpose it is an array of octets. Your datagram is
    represented as a Stream_Element_Array or a slice of.

    According to RM 13.13.1, "Stream_Element is mod implementation-defined"
    which to me says there is no guarantee that they will be octets, unless
    this is specified elsewhere?

    GNAT.Sockets is GNAT-specific. All GNAT compilers have Stream_Element 8
    bits. I can imagine some DSP implementation with Stream_Element of 32
    bits. But realistically add

    pragma Assert (Stream_Element'Size >= 8);

    and be done with that.

    So, how would I do this directly on the elements? I mean, if it is an octet-array to a string, I expect an element-to-element copy, or type conversion to work, but what about integers?

    Hmm, it cannot be string. It is a string encoded in some specific way
    (usually most peculiar (:-)). Then you will have to decode it into your
    machine type e.g. Wide_String or String.

    An UTF-8 string you could put into String ignoring Ada's Latin-1 stuff,
    as most people would do:

    function To_String (S : Stream_Element_Array) return String is
    begin
    return Result : String (1..S'Length) do
    for I in S'Range loop
    Result (Positive (I - S'First + 1) := Character'Val (S (I));
    end loop;
    end return;
    end To_String;

    Do I need to do something like My_Int:=Unsigned_8(octet(1))+2**8*Unsigned_8(octet(2));
    or whatever endianness demands? Or is this the time to learn how to use Unchecked_Conversion?

    There are many ways to convert Stream_Element_Array "in situ" to string. However, in network protocols you rarely have any strings at all.
    Usually it is some binary data you need to decode into some machine
    type. (So is String or Wide_String actually)

    --
    Regards,
    Dmitry A. Kazakov
    http://www.dmitry-kazakov.de

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jeffrey R.Carter@21:1/5 to Mark Gardner on Sat Dec 31 16:18:50 2022
    On 2022-12-31 14:50, Mark Gardner wrote:

    According to RM 13.13.1, "Stream_Element is mod implementation-defined" which to
    me says there is no guarantee that they will be octets, unless this is specified
    elsewhere?

    The ARM has always tried to ensure that the language could be implemented on any
    kind of processor. Thus you have implementation-defined separate definitions of Storage_Element and Stream_Element, which need not be the same, and no guarantee
    that Interfaces contains declarations of Integer_8 or Unsigned_8.

    But these days almost everything is byte oriented, so unless you need what you're writing to work on some unusual H/W, you can presume that both of these are bytes, and that Interfaces contains those declarations.

    --
    Jeff Carter
    "My legs are gray, my ears are gnarled, my eyes are old and bent."
    Monty Python's Life of Brian
    81

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Simon Wright@21:1/5 to Mark Gardner on Sat Dec 31 17:39:07 2022
    Mark Gardner <magardner2017@gmail.com> writes:

    GNAT.Sockets gives me a Stream_Element_Array, which I can't find any documentation on how to make use of other than "You should also be
    able to get a Stream, which you should use instead" (About ten years
    ago, on this very newsgroup, somebody said not to use streams with
    UDP, or at least not GNAT.Sockets.Stream).

    The reasoning behind the recommendation not to use streams with UDP was
    as follows (there's a faint possibility that it no longer applies!)

    If the data type you want to send is e.g.

    type Message is record
    Id : Integer;
    Val : Boolean;
    end record;

    and you create a datagram socket and from that a stream, then use
    Message'Write to the stream, GNAT will transmit each component of
    Message separately in canonical order (the order they're written in the
    type declaration). This results in two datagrams being sent, one of 4
    bytes and one of 1 byte.

    If you take the same approach at the destination, Message'Read reads one datagram of 4 bytes, and one of 1 byte, and it all looks perfect from the outside. If the destination is expecting a 5 byte record, of course,
    things won't work so well.

    The approach we adopted was to create a 'memory stream', which is a
    chunk of memory that you can treat as a stream (see for example ColdFrame.Memory_Streams at [1]). With Ada2022, you should be able to
    use Ada.Streams.Storage.Bounded[2].

    Message'Write the record into the memory stream;
    transmit the written contents as one datagram.

    To read, create a memory stream large enough for the message you expect;
    read a datagram into the memory stream;
    Message'Read (Stream => the_memory_stream, Item => a_message);

    You can use gnatbind's switch -xdr to "Use the target-independent XDR
    protocol for stream oriented attributes instead of the default
    implementation which is based on direct binary representations and is
    therefore target-and endianness-dependent".

    [1] https://github.com/simonjwright/coldframe/blob/master/lib/coldframe-memory_streams.ads
    [2] http://www.ada-auth.org/standards/22rm/html/RM-13-13-1.html#p25

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Mark Gardner@21:1/5 to Simon Wright on Sat Dec 31 21:36:40 2022
    On 31/12/2022 19:39, Simon Wright wrote:
    [...]

    The approach we adopted was to create a 'memory stream', which is a
    chunk of memory that you can treat as a stream (see for example ColdFrame.Memory_Streams at [1]). With Ada2022, you should be able to
    use Ada.Streams.Storage.Bounded[2].


    Wait, so if I know what shape my data is, and use a memory_stream (like
    the one in the Big Online Book of Linux Ada Programming chapter 11 [1]),
    I'm fine using Stream, in conjunction with Get_Address? That's
    wonderful. Not at all frustrated that I just wasted approximately three
    working days looking for a solution to a problem that didn't exist.

    Message'Write the record into the memory stream;
    transmit the written contents as one datagram.

    I'm guessing with Memory_Stream'Write(Socket_Stream, Buffer);?


    To read, create a memory stream large enough for the message you expect;
    read a datagram into the memory stream;
    Message'Read (Stream => the_memory_stream, Item => a_message);

    Does this second buffer need to be added? If the datagram arrives (UDP), shouldn't GNAT.Sockets.Stream() be able to handle it?

    You can use gnatbind's switch -xdr to "Use the target-independent XDR protocol for stream oriented attributes instead of the default
    implementation which is based on direct binary representations and is therefore target-and endianness-dependent".

    Oh fun, I didn't think of that aspect. Thanks! Would I have to pass it
    as a command line flag, or would there be some kind of pragma I could use?

    Thanks for the help so far, and happy new year!

    [1] http://www.pegasoft.ca/resources/boblap/11.html#11.12

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Dmitry A. Kazakov@21:1/5 to Mark Gardner on Sat Dec 31 21:16:18 2022
    On 2022-12-31 20:36, Mark Gardner wrote:
    On 31/12/2022 19:39, Simon Wright wrote:

    Message'Write the record into the memory stream;
    transmit the written contents as one datagram.

    I'm guessing with Memory_Stream'Write(Socket_Stream, Buffer);?

    No, you create a memory stream object. Then you write your packet into it:

    My_Message'Write (My_Memory_Stream'Access);

    Once written you use the accumulated stream contents to write it into
    the socket. An implementation of a memory-resident stream is very
    simple. E.g. see:

    http://www.dmitry-kazakov.de/ada/strings_edit.htm#Strings_Edit.Streams

    My advise would be not to do this. It is wasting resources and
    complicated being indirect when 'Write and 'Read are compiler-generated.
    If you implement 'Write and 'Read yourself, then why not calling these implementations directly. It just does not make sense to me. I always
    wonder why people always overdesign communication stuff.

    Build messages directly in a Stream_Element_Array. Use
    system-independent ways to encode packet data. E.g. chained codes for
    integers. Mantissa + exponent for real numbers. If you have Booleans and enumerations it is a good idea to pack them into one or two octets to
    shorten the packets. All this is very straightforward and easy to implement.

    You can also consider using some standard data representation format,
    e.g. ASN.1. An Ada ASN.1 implementation is here:

    http://www.dmitry-kazakov.de/ada/components.htm#ASN.1

    You describe your message in ASN.1 as an Ada tagged type derived from
    building blocks. Then you can encode and decode it directly from Stream_Element_Array. I would not recommend that either. ASN.1 is quite overblown.

    Happy New Year!

    --
    Regards,
    Dmitry A. Kazakov
    http://www.dmitry-kazakov.de

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Dmitry A. Kazakov@21:1/5 to philip...@gmail.com on Sat Dec 31 23:55:07 2022
    On 2022-12-31 23:32, philip...@gmail.com wrote:

    I have to disagree here. UDP is perfectly fine for RPC-like (Remote Procedure Call) transactions on a local area network.

    RPC and other synchronous exchange policies should be avoided as much as possible.

    Saying said that, implementation of RPC on top of streams is
    incomparable more easier than on top of UDP.

    And it is orders of magnitude easier to implement on microcontrollers than TCP.

    Not at all. You need:

    - Safe transmission and error correction on top UDP;
    - Buffering and sorting out incoming datagrams;
    - Maintaining sequence numbers;
    - Splitting messages that do not fit into a single datagram and
    reassembling them on the receiver side;
    - Buffering on the sender side to service resend requests.

    This is extremely difficult and huge load for a microcontroller.

    Getting between Stream_Element_Array and a byte array is a pain and I wound up just looping over arrays, copying one byte at a time. If somebody has a better idea, let me know.

    Use "in situ" conversion if you are concerned about copying. E.g.

    pragma Import (Ada, Y);
    for Y'Address use X'Address;

    --
    Regards,
    Dmitry A. Kazakov
    http://www.dmitry-kazakov.de

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From philip.munts@gmail.com@21:1/5 to Dmitry A. Kazakov on Sat Dec 31 14:32:17 2022
    On Saturday, December 31, 2022 at 5:11:14 AM UTC-8, Dmitry A. Kazakov wrote:

    And, well, do not use UDP, expect for broadcasting. There is no reason
    to use it. For multicast consider delivery-safe protocols like PGM. For single cast use TCP/IP. (If you need low latency see the socket NO_DELAY option)

    I have to disagree here. UDP is perfectly fine for RPC-like (Remote Procedure Call) transactions on a local area network. And it is orders of magnitude easier to implement on microcontrollers than TCP. An Ada program using UDP to communicate with data
    collecting microcontrollers makes perfect sense in some contexts. I use it for my Remote I/O Protocol.

    The only trick is that the server (or responder, as I like to call it) and client (or initiator) can't quite use the same code.

    Here is my generic package for UDP with fixed length messages:

    https://github.com/pmunts/libsimpleio/blob/master/ada/objects/messaging-fixed-gnat_udp.ads
    https://github.com/pmunts/libsimpleio/blob/master/ada/objects/messaging-fixed-gnat_udp.adb

    Getting between Stream_Element_Array and a byte array is a pain and I wound up just looping over arrays, copying one byte at a time. If somebody has a better idea, let me know.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jeffrey R.Carter@21:1/5 to philip...@gmail.com on Sat Dec 31 23:49:33 2022
    On 2022-12-31 23:32, philip...@gmail.com wrote:

    Getting between Stream_Element_Array and a byte array is a pain and I wound up just looping over arrays, copying one byte at a time. If somebody has a better idea, let me know.

    You should be able to use Unchecked_Conversion for that.

    --
    Jeff Carter
    "My legs are gray, my ears are gnarled, my eyes are old and bent."
    Monty Python's Life of Brian
    81

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Simon Wright@21:1/5 to Dmitry A. Kazakov on Sat Dec 31 23:41:11 2022
    "Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes:

    My advise would be not to do this. It is wasting resources and
    complicated being indirect when 'Write and 'Read are
    compiler-generated. If you implement 'Write and 'Read yourself, then
    why not calling these implementations directly. It just does not make
    sense to me. I always wonder why people always overdesign
    communication stuff.

    Build messages directly in a Stream_Element_Array. Use
    system-independent ways to encode packet data. E.g. chained codes for integers. Mantissa + exponent for real numbers. If you have Booleans
    and enumerations it is a good idea to pack them into one or two octets
    to shorten the packets. All this is very straightforward and easy to implement.

    It has to depend on the design criteria.

    If you need something now, and it's not performance critical, and you
    have control over both ends of the channel, why not go for a
    low-brain-power solution?

    On the other hand, when faced with e.g. SNTP, why not use Ada's
    facilities (e.g. [1]) to describe the network packet and use unchecked conversion to convert to/from the corresponding stream element array to
    be sent/received?

    I'd have thought that building messages directly in a stream element
    array would be the least desirable way to do it.

    [1] https://sourceforge.net/p/coldframe/adasntp/code/ci/default/tree/SNTP.impl/sntp_support.ads

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Dmitry A. Kazakov@21:1/5 to Simon Wright on Sun Jan 1 10:48:16 2023
    On 2023-01-01 00:41, Simon Wright wrote:
    "Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes:

    My advise would be not to do this. It is wasting resources and
    complicated being indirect when 'Write and 'Read are
    compiler-generated. If you implement 'Write and 'Read yourself, then
    why not calling these implementations directly. It just does not make
    sense to me. I always wonder why people always overdesign
    communication stuff.

    Build messages directly in a Stream_Element_Array. Use
    system-independent ways to encode packet data. E.g. chained codes for
    integers. Mantissa + exponent for real numbers. If you have Booleans
    and enumerations it is a good idea to pack them into one or two octets
    to shorten the packets. All this is very straightforward and easy to
    implement.

    It has to depend on the design criteria.

    If you need something now, and it's not performance critical, and you
    have control over both ends of the channel, why not go for a
    low-brain-power solution?

    Because it is still much simpler (and safer) than record layouts.

    On the other hand, when faced with e.g. SNTP, why not use Ada's
    facilities (e.g. [1]) to describe the network packet and use unchecked conversion to convert to/from the corresponding stream element array to
    be sent/received?

    Well decoding NTP query from Stream_Element_Array takes 2 statements (extracting two big-endian 32-bit numbers). You can check the
    implementation here:

    http://www.dmitry-kazakov.de/ada/components.htm#17.17

    which is under-zero brained (:-)) compared to dealing with definition of
    record layout and bit orders you suggest.

    --
    Regards,
    Dmitry A. Kazakov
    http://www.dmitry-kazakov.de

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Simon Wright@21:1/5 to Dmitry A. Kazakov on Sun Jan 1 16:11:22 2023
    "Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes:

    Well decoding NTP query from Stream_Element_Array takes 2 statements (extracting two big-endian 32-bit numbers). You can check the
    implementation here:

    http://www.dmitry-kazakov.de/ada/components.htm#17.17

    which is under-zero brained (:-)) compared to dealing with definition
    of record layout and bit orders you suggest.

    It's obviously OK to use just the fields of interest in the received
    packet.

    But this strikes me as something I would have questioned at review:

    Data : Stream_Element_Array (1..NTP_Packet_Size) :=
    ( 1 => 2#1110_0011#, -- LI, Version, Mode
    2 => 0, -- Stratum, or type of clock
    3 => 0, -- Polling Interval
    4 => 16#EC#, -- Peer Clock Precision
    13 => 49,
    14 => 16#4E#,
    15 => 49,
    16 => 52,
    others => 0
    );

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Niklas Holsti@21:1/5 to Simon Wright on Sun Jan 1 18:17:07 2023
    On 2023-01-01 1:41, Simon Wright wrote:
    "Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes:

    My advise would be not to do this. It is wasting resources and
    complicated being indirect when 'Write and 'Read are
    compiler-generated. If you implement 'Write and 'Read yourself, then
    why not calling these implementations directly. It just does not make
    sense to me. I always wonder why people always overdesign
    communication stuff.

    Build messages directly in a Stream_Element_Array. Use
    system-independent ways to encode packet data. E.g. chained codes for
    integers. Mantissa + exponent for real numbers. If you have Booleans
    and enumerations it is a good idea to pack them into one or two octets
    to shorten the packets. All this is very straightforward and easy to
    implement.

    It has to depend on the design criteria.

    If you need something now, and it's not performance critical, and you
    have control over both ends of the channel, why not go for a
    low-brain-power solution?

    On the other hand, when faced with e.g. SNTP, why not use Ada's
    facilities (e.g. [1]) to describe the network packet and use
    unchecked conversion to convert to/from the corresponding stream
    element array to be sent/received?

    [1]
    https://sourceforge.net/p/coldframe/adasntp/code/ci/default/tree/SNTP.impl/sntp_support.ads


    One reason is that the Scalar_Storage_Order representation aspect is GNAT-specific, so that code is not really "Ada code", but "GNAT code".
    With most processors now being little-endian, while network traffic is
    still big-endian, byte-endianness conversion is usually necessary.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Daniel Norte de Moraes@21:1/5 to All on Thu Jan 5 07:55:43 2023
    Em Sat, 31 Dec 2022 14:11:55 +0200, Mark Gardner escreveu:
    At this point, I'm half-tempted to make my own binding, but as I've
    never done that sort of thing before, I thought I'd ask the wisdom of
    the Usenet if there is a way to convert a Stream_Element_Array into the exotic types of Unsigned_16 and String.

    You are obliged to use gnat-sockets ?
    No? then use https://github.com/danieagle/adare-net

    And Be Happy! (Enjoy!!)

    p.s.: see the client and server in udp in example diretory.
    p.s.: nowdays Adare_Net has a manual in pdf, too

    Thanks All,
    Dani.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Dmitry A. Kazakov@21:1/5 to Daniel Norte de Moraes on Thu Jan 5 11:35:33 2023
    On 2023-01-05 08:55, Daniel Norte de Moraes wrote:
    Em Sat, 31 Dec 2022 14:11:55 +0200, Mark Gardner escreveu:
    At this point, I'm half-tempted to make my own binding, but as I've
    never done that sort of thing before, I thought I'd ask the wisdom of
    the Usenet if there is a way to convert a Stream_Element_Array into the
    exotic types of Unsigned_16 and String.

    You are obliged to use gnat-sockets ?
    No? then use https://github.com/danieagle/adare-net

    The OP's question was about Stream_Element_Array being used for payload.
    In that regard there is no difference between GNAT.Sockets and the
    library you suggest.

    P.S. I am not sure if the library supports socket select and setting all necessary socket flags (e.g. NODELAY).

    --
    Regards,
    Dmitry A. Kazakov
    http://www.dmitry-kazakov.de

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)