• Delphi D4: using pointers to transfer data from COMM queue to circular

    From P E Schoen@21:1/5 to All on Fri Aug 19 02:19:06 2016
    I have an application that uses a serial port component to read and write
    data through a USB serial port at 57.6 kb. Originally I used the SerialNG component and with some tweaking I was able to get it to work reliably. But
    it has problems with Win10, so I changed the code to use ComDrv32, which
    works when used in a simple TTY demo.

    The SerialNG component uses clusters or packets which can be set to a range
    of values. I am communicating with a USB device set up as a CDC serial
    emulator using Microchip PIC18F2550. It is basically a data acquisition unit that samples at 2400 samples/sec with a 10 bit ADC. I am splitting the data into two 5 bit values in 8 bit data characters, and I'm using the remaining three to identify the data as a MSB or LSB (using one evem/odd bit), and a
    two bit value to determine if the data has been received in proper sequence.

    Here is the function that receives the data from the serial port component:

    ===============================================================
    procedure TfmCommLib.CommPortDriver1ReceiveData(Sender: TObject;
    DataPtr: Pointer; DataSize: Cardinal);
    var
    RecdPtr: PChar;
    CSize, NCSize : Integer;
    n: Integer;
    begin
    RecdPtr := PChar(DataPtr);
    RxDcount := DataSize;
    if RxDcount > MaxRxDcount then
    MaxRxDcount := RxDcount;
    if CommBufferPtr + DataSize >= MAXCOMM then begin
    move( DataPtr^, CommBuffer[CommBufferPtr], MAXCOMM-CommBufferPtr );
    n := DataSize - (MAXCOMM-CommBufferPtr);
    RecdPtr := RecdPtr+n;
    DataPtr := RecdPtr;
    CommBufferPtr := 0;
    move( DataPtr^, CommBuffer[CommBufferPtr], n );
    end
    else begin
    move( DataPtr^, CommBuffer[CommBufferPtr], DataSize );
    CommBufferPtr := CommBufferPtr + DataSize; end;
    CommCharCount := CommCharCount + DataSize;
    if(CommCharCount > MAXCOMM) then
    CommCharCount := MAXCOMM+1;
    end;
    ========================================================

    It works, but I get errors after running for about 20 seconds. That
    corresponds to the point where the CommBuffer will overflow the MAXCOMM
    value of 100,000 (4800*20). My previous code for SerialNG used a cluster
    size of 2 and a loop to transfer the received data to the buffer one byte at
    a time. When I tried that, it seemed to work poorly. So I used a variation
    of the method from the ComDrv32 TTY demo using move( DataPtr^, CommBuffer[CommBufferPtr], DataSize ); But you can see that I had to split
    the transfer when near the end of the queue.

    I had to use a separate PChar pointer RecdPtr, and I also had to use an
    integer for the pointer arithmetic. Apparently Delphi does not allow RecdPtr
    := RecdPtr + (DataSize - (MAXCOMM-CommBufferPtr) ); RecdPtr + n is OK. I'm
    not sure if I could just advance the DataPtr in the same way, since it is declared as a generic Pointer type.

    I can't very well use the Delphi debugger to see what's happening, as the
    data is received continuously and there is no handshaking.

    TIA for any suggestions.

    Paul

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From P E Schoen@21:1/5 to All on Fri Aug 19 03:58:01 2016
    I mentioned elsewhere that I used TCriticalSections previously during the transfers from the ComDrv32 component input buffer to the larger CommBuffer, and also when processing the CommBuffer to detect and compensate for errors. This seemed to work for the SerialNG component, but for ComDrv32 it seemed
    to increase the errors. Removing them, I got some errors starting at 20
    seconds as before, but after 700 seconds there are only 146 errors compared
    to nearly 1000 previously. There is probably a thread running to receive the characters into the COM16 file.

    I tried various settings for the serial port component with inconsistent results. What is consistent is that errors always start when the CommBuffer
    is nearly full and the data must be split to fill the buffer and then start again at the beginning. I may need to check my arithmetic.

    Paul

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From P E Schoen@21:1/5 to All on Fri Aug 19 04:42:54 2016
    I was able to rewrite the receive procedure and now it works with no errors.
    I think I have used pointers properly here:

    ==========================================================
    procedure TfmCommLib.CommPortDriver1ReceiveData(Sender: TObject;
    DataPtr: Pointer; DataSize: Cardinal);

    var
    RecdPtr: PChar;
    n: Integer;
    begin
    RecdPtr := PChar(DataPtr);
    RxDcount := DataSize;
    if RxDcount > MaxRxDcount then
    MaxRxDcount := RxDcount;
    For n:= 1 to RxDcount do begin //2
    CommBuffer[CommBufferPtr] := RecdPtr^;
    inc(RecdPtr);
    inc(CommBufferPtr);
    if(CommBufferPtr > MAXCOMM) then
    CommBufferPtr := 0;
    inc(CommCharCount);
    if(CommCharCount > MAXCOMM) then
    fmCommLib.CommCharCount := MAXCOMM+1;
    end; //2
    end;
    ========================================================

    In the previous procedure for SerialNG, I did this:

    var
    RecdPtr: PChar;
    begin
    RecdPtr := CommSerialPort.ReadNextCluster( NCSize, RdErrors );
    ...
    FreeMem( RecdPtr, NCSize ); // Added 4/8/09
    end;
    ========================================================

    I don't know if that was a problem or not. It seems that I might need to reserve memory for the PChar before using it?

    Paul

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Hans-Peter Diettrich@21:1/5 to P E Schoen on Fri Aug 19 12:27:25 2016
    P E Schoen schrieb:
    I have an application that uses a serial port component to read and
    write data through a USB serial port at 57.6 kb. Originally I used the SerialNG component and with some tweaking I was able to get it to work reliably. But it has problems with Win10, so I changed the code to use ComDrv32, which works when used in a simple TTY demo.

    I don't remember what prevented programs from using the asynchronous I/O
    API. 57kbd also doesn't look like a dangerous transmission speed.


    I had to use a separate PChar pointer RecdPtr, and I also had to use an integer for the pointer arithmetic. Apparently Delphi does not allow
    RecdPtr := RecdPtr + (DataSize - (MAXCOMM-CommBufferPtr) ); RecdPtr + n
    is OK. I'm not sure if I could just advance the DataPtr in the same way, since it is declared as a generic Pointer type.

    Delphi allows pointer math only with PCHAR. At least I'd use Cardinal
    instead of Integer, in a workaround.

    DoDi

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Hans-Peter Diettrich@21:1/5 to P E Schoen on Fri Aug 19 12:30:04 2016
    P E Schoen schrieb:

    I tried various settings for the serial port component with inconsistent results. What is consistent is that errors always start when the
    CommBuffer is nearly full and the data must be split to fill the buffer
    and then start again at the beginning. I may need to check my arithmetic.

    Can't you use multiple buffers, to eliminate the moving of data?

    DoDi

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Hans-Peter Diettrich@21:1/5 to P E Schoen on Fri Aug 19 12:34:55 2016
    P E Schoen schrieb:

    In the previous procedure for SerialNG, I did this:

    var
    RecdPtr: PChar;
    begin
    RecdPtr := CommSerialPort.ReadNextCluster( NCSize, RdErrors );
    ...
    FreeMem( RecdPtr, NCSize ); // Added 4/8/09
    end;
    ========================================================

    I don't know if that was a problem or not. It seems that I might need to reserve memory for the PChar before using it?

    Various implementations of ReadNextCluster are possible. It may be
    required or forbidden to free the memory of the returned pointer.

    DoDi

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From P E Schoen@21:1/5 to Hans-Peter Diettrich on Sat Aug 20 19:15:48 2016
    "Hans-Peter Diettrich" wrote in message news:e1qqjiF6cpqU3@mid.individual.net...

    P E Schoen schrieb:

    You may have already "left the building" at this point, unless you find
    this sort of discussion interesting. I have been involved in the
    measurement of power line frequency current and voltage signals for a
    LONG time, close to 40 years. Technology has changed since then, from
    mostly analog methods, then ADCs with early microprocessors like the
    8085 and Z80, then signal processing boards with MSDOS, and now PICs and
    DSPs with Windows 10 or other OS.

    Currently I see similar (but hobbyist) projects in the Arduino forum. I
    never tried real time programming on Win32/64, nor did I use COM port components with Delphi. In the early PC days I did some data acquisition using GW-Basic, later with BCB on Win3. I'd no more try to make my hands dirty with real time programs for Windows, and in detail not with Delphi,
    be some old reliable (up to D7) or uncomfortable and bloated new (RAD
    Studio) versions. It's interesting to hear that you still are in business
    :-)

    Yes, I'm still in business, although at age 67 I've been semi-retired for
    about 2 years. I still support my Ortmaster products, including the original ORTM-1 and ORTM-2 that use MSDOS and the LPT parallel port to interface to
    the data acquisition device I designed around 1994 and produced until about 2004 when I designed the first ORTM-3 (serial interface) and later the
    ORTM-4 (USB interface). Some customers still have and use the ORTM-2 but it
    is becoming more difficult for them to find working computers with MSDOS (or Win95) and native parallel ports. That's good for me, because I offer the
    only viable upgrade option for modern Windows machines.

    I have considered upgrading to a newer version of Delphi. I have heard that
    the even numbered releases (D4 and D6) were buggy and their odd-numbered successors (D5 and D7) were much better. I found some Delphi Enterprise 8 products on eBay for under $200 and CodeGear Delphi 2007 Enterprise for
    $430. Embarcadero does not offer an upgrade deal from D4, and their lowest level (Pro) version that offers database support is $1405. https://www.embarcadero.com/app-development-tools-store/delphi

    Perhaps even better is the open source Free Pascal and the Lazarus IDE that
    has a look and feel very similar to the Borland Delphi IDE that I am used to with D4-Pro.
    http://wiki.freepascal.org/Lazarus_Documentation https://www.youtube.com/playlist?list=PLB24C56953A79987A

    Thanks for the information and discussion.

    Paul

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Hans-Peter Diettrich@21:1/5 to P E Schoen on Sat Aug 20 12:43:43 2016
    P E Schoen schrieb:

    You may have already "left the building" at this point, unless you find
    this sort of discussion interesting. I have been involved in the
    measurement of power line frequency current and voltage signals for a
    LONG time, close to 40 years. Technology has changed since then, from
    mostly analog methods, then ADCs with early microprocessors like the
    8085 and Z80, then signal processing boards with MSDOS, and now PICs and
    DSPs with Windows 10 or other OS.

    Currently I see similar (but hobbyist) projects in the Arduino forum. I
    never tried real time programming on Win32/64, nor did I use COM port components with Delphi. In the early PC days I did some data acquisition
    using GW-Basic, later with BCB on Win3. I'd no more try to make my hands
    dirty with real time programs for Windows, and in detail not with
    Delphi, be some old reliable (up to D7) or uncomfortable and bloated new
    (RAD Studio) versions. It's interesting to hear that you still are in
    business :-)

    DoDi

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From P E Schoen@21:1/5 to Hans-Peter Diettrich on Fri Aug 19 20:54:54 2016
    "Hans-Peter Diettrich" wrote in message news:e1o5msFhoooU2@mid.individual.net...

    P E Schoen schrieb:

    I tried various settings for the serial port component with inconsistent
    results. What is consistent is that errors always start when the
    CommBuffer is nearly full and the data must be split to fill the buffer
    and then start again at the beginning. I may need to check my arithmetic.

    Can't you use multiple buffers, to eliminate the moving of data?

    Much of this code dates back to around 1996-1999 when I first tried making
    an application for Windows using Borland C++, and my first use of Borland Delphi 4 seems to be around 2003, when a consultant for another project
    showed me how to use it. I think I purchased Delphi and BC++ around 1998.

    I'm not sure how multiple buffers would help. The serial port component has
    its own inBuffer which I have set to 48,000 bytes, or enough for 10 seconds
    of data. There is probably another small buffer at a lower level (there is
    an FTempInBuffer which is a pointer to data space allotted to
    FInBufferSize). The actual reading of data is accomplished by using:

    nRead := 0;
    nToRead := comStat.cbInQue;
    if (nToRead > 0) and ReadFile( FHandle, FTempInBuffer^, nToRead, nRead,
    nil ) then
    if (nRead <> 0) and Assigned(FOnReceiveData) then
    FOnReceiveData( Self, FTempInBuffer, nRead );

    So it does not appear that the FTempInBuffer is a circular queue, and thus I need to implement that in my own code using my CommBuffer[]. I could
    probably use much smaller values for the buffer sizes, but that's not really
    an issue these days when most computers have at least 2g of memory. The RxDataCount displayed in my status dialog is usually 280-390 bytes, which
    may depend on the PollingDelay which I have set at 50 mSec. At 4800 char/sec that is 240 characters. The extra characters likely occur during the buffer processing procedures.

    I am doing additional processing in a procedure called by a timer set at
    50-250 mSec. So that may account for more delays and accumulation of data in the buffers. My MaxRxdCount after 10 minutes is now 1758. I have found that other Windows processes can increase this number, but so far it does not
    seem to lose data.

    Quite a lot of real time processing occurs during these update intervals. Character pairs are read from the CommBuffer circular queue, and are parsed
    to verify their validity according to the extra two bits in each character, while the remaining 6 bits in each are concatenated to form a 12 bit
    integer. The ADC is only 10 bits but I designed the system to use 12 bits if needed. The value is shifted by 2048 to get a signed integer.

    The absolute value is compared to a threshold value and a number of samples
    to determine if the value (of current) is enough to indicate that a test is
    in progress. If so, a small circular queue is used for pre-trigger data and added to a Waveform data buffer (like a digital storage scope). When the
    value drops below threshold for a number of samples, the actual start and
    stop points of the waveform are calculated, and a true RMS computation is performed to give the value of current in the pulse. This may be repeated
    for up to five pulses of 60 Hz current in a single test, with off times of
    as long as 20 seconds (although usually 1 or 2).

    Meanwhile, the true RMS value of the samples in each update interval is computed and displayed on the screen, giving a continuous RMS value at all times. The sampling intervals of 50, 100, 150, 200, and 250 mSec are chosen
    to use an integral number of zero crossings for either 50 or 60 Hz. At 2400 samples per second, 50 mSec is 120 samples, and corresponds to 3 cycles at
    60 Hz or 2.5 cycles at 50 Hz. This measuring technique is used in DMMs to cancel the effects of line voltage noise. It also produces consistent true
    RMS readings when the start and end of the sampling period are not
    synchronized to the zero crossings.

    You may have already "left the building" at this point, unless you find this sort of discussion interesting. I have been involved in the measurement of power line frequency current and voltage signals for a LONG time, close to
    40 years. Technology has changed since then, from mostly analog methods,
    then ADCs with early microprocessors like the 8085 and Z80, then signal processing boards with MSDOS, and now PICs and DSPs with Windows 10 or other OS.

    Thanks for your valued suggestions and time.

    Paul

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Hans-Peter Diettrich@21:1/5 to P E Schoen on Sun Aug 21 12:50:38 2016
    P E Schoen schrieb:

    I have considered upgrading to a newer version of Delphi. I have heard
    that the even numbered releases (D4 and D6) were buggy and their
    odd-numbered successors (D5 and D7) were much better.

    That's correct for the Borland products, up to D7. Afterwards bad goals
    had been chosen, e.g. a competitor for Microsoft .NET, which failed
    miserably. Then the adaptation to the MS (WinAPI) help files took
    several years, and online help still is at <50% compared to D4 or D7.
    Next came the introduction of Unicode support and Firemonkey, and way
    too late a 64 bit compiler.

    I found some
    Delphi Enterprise 8 products on eBay for under $200 and CodeGear Delphi
    2007 Enterprise for $430. Embarcadero does not offer an upgrade deal
    from D4, and their lowest level (Pro) version that offers database
    support is $1405. https://www.embarcadero.com/app-development-tools-store/delphi

    Upgrading existing projects seems to be quite easy, at least I could run
    a D5 program very soon on XE and XE4, with only a few missed detail
    adaptations found later. The prices, well, I think they are due to a
    shrinking user base. New projects mostly use VS, but that's not an
    option for your existing Delphi projects.

    Perhaps even better is the open source Free Pascal and the Lazarus IDE
    that has a look and feel very similar to the Borland Delphi IDE that I
    am used to with D4-Pro.
    http://wiki.freepascal.org/Lazarus_Documentation https://www.youtube.com/playlist?list=PLB24C56953A79987A

    I discontinued watching FPC and Lazarus development. Some Delphi
    projects may port easily, others can require much work.

    DoDi

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From P E Schoen@21:1/5 to Hans-Peter Diettrich on Mon Aug 22 07:12:46 2016
    "Hans-Peter Diettrich" wrote in message news:e1thpaFq4gbU1@mid.individual.net...

    P E Schoen schrieb:

    Perhaps even better is the open source Free Pascal and the Lazarus IDE
    that has a look and feel very similar to the Borland Delphi IDE that I
    am used to with D4-Pro.
    http://wiki.freepascal.org/Lazarus_Documentation
    https://www.youtube.com/playlist?list=PLB24C56953A79987A

    I discontinued watching FPC and Lazarus development. Some Delphi projects
    may port easily, others can require much work.

    I installed them on my Win10 computer and I have had some success with
    simpler projects. I'm having trouble installing the ComDrv32 package, but I don't fully understand how they work in Delphi, although I was able to do it somehow. I was able to get around the problem by removing the component from the form and then manually creating the component.

    Now that I think about it, I probably have two instances of the component in
    my project, which might cause problems.

    Thanks,

    Paul

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)