Fix the source of the problem. If it can't be fixed don't allow the
broken to enter. That's straight, clear and simple.

Maybe we should block all PKTs generated by Mystic BBS then? And also PKTs from hpt itself? (which doesn't set the JAM OADDRESS and rewrites the message date for Squish messages.


* Origin: kakistocracy (2:280/464.47)

Hello Zhenja!

11 Jan 20, Zhenja Kaliuta wrote to Mark lewis:

But software may have bugs and it's sad to lose otherwise valid data because of that.

I don't want to accept that argument because there is no clear line what is "a goog bug" that needs to be corrected by others and "a bad bug" that causes more harm than being useful. And even "good bugs" would create a bunch of workarounds with the risk of interferring each other by time.

It's like a water bucket with a small hole. You can workaround by adding water up to full and run faster to your destination. Or you can place your hand on the hole. With more holes you will have less water at the destination or you need more hands to cover the holes. What do you do if there is no water in the bucket when you arrived at the destination? Throw the bucket away and get a new one? Or at last start fixing the holes?

BTW, in RFC world it's pretty different. Ex.: rfc2045

If there is a better bucket throw away the old one and use the new one.

Regards

Kai

--- GoldED+/LNX 1.1.4.7
* Origin: Monobox (2:240/77)

Hello Zhenja!

11 Jan 20, Zhenja Kaliuta wrote to Kai Richter:

Well, in general in the world there are other opinions:

https://tools.ietf.org/html/rfc1122

Other opinions of another world.

I do understand why people would like to "improve" fidonet or "advance" FTS. But hey, that job is already done. We already got rid of the numeric address system and use letters and numbers together, we can display pictures and videos next to text, we reduced the transfer time and many new standards where invented, even the name was refreshed to "The internet".

Regards

Kai

--- GoldED+/LNX 1.1.4.7
* Origin: Monobox (2:240/77)

Hello mark!

11 Jan 20, mark lewis wrote to Zhenja Kaliuta:

- is it ok to handle the situation without failing the pkt?
- is it ok to make the pkt standard complied?

in my personal and professional opinion, the answer to both questions
is a resounding "yes" but with at least the following caveat...

Sorry, i didn't noticed your next mail where you highly agreed with me before writing. Please ignore this mail. I send it because i added some details of the consequences how works arounds can block each other.

1. HPT is the first FTN processor to handle the defective pkt
after the flawed (gating?) software has created it...

The first FTN processor is the flawed software that generates non FTS pkts.
No matter what that software normally does - it has a module that exports FTN pkt and that job should be done in accordance to FTS.

once the defective pkts are already in the distribution stream,
meaning another FTN tosser has processed it, then no... mark them as
bad and set them aside with a proper logged reason...

The result is that only direct link systems are flooded with broken content. That could be a nice idea, but because of the actual practice for network stability improvement called multi-link routing, there are many "first HPT tosser" that would receive the broken pkt.

This is a good example that work arounds do interfere with other work arounds. Your idea would work for a straight star network but not for a mesh type network.

or some such... possibly also record the identity of the processor
that created the defective packet if that's not already being done...
one might also want to log the above when the processor repairs such
a defective message and log an additional message stating the repair decision... logging bad pkts is generally already done and adding
logging that the repair of message #xxx is being done is trivial...

others may or may not feel the same way...

Instead of all other network members waste their resources for logging, fixing and accepting broken subjects within the network only one system needs to find a better solution how to transfer too long subject lines into the standard pkt format. A short and simple way is to copy the full subject line into the message text and cut the subject short plus adding a timestamp. The timestamp is required if the subject line is edited in the too long part only. This change must be visible in the short line somehow to keep reply linking searching and sorting by subject line working as usual.

i fully agree that this is a technical problem which is allowed for
by current fidonet policy

I don't think so. I'm not in details but it doesn't make sense. The FTS are the standard valid for any FTS based networks to keep the networks running without problems. The FTS sets the frame. If the subject line is limited any FTN software have to be compliance to that limit - or it is not a FTS software.

The purpose of the "other technical problems" in the policy can't be used to transform the network into a trashcan and leave the responsibility to find useful stuff in the trash to us.

The problem of non FTS conform subject lines is a problem out of the FTN/FTS responsibility because it's not a valid FTS pkt. Period. The decision of HPT to move that non standard content to bad is correct. The decision not to modifiy and not to route the broken content is the only way to tell the source of the problem that his software has a serious bug that needs to be fixed.

basically mean a setting per FTN... possibly something like
Repair defective pkts: yes/no

That would make it more complex. Keep it simple. It's not your/our job to fix incoming trash. HPT should detect a broken pkt, log it and move it to bad. That's how the broken pkt was found so hpt already works as designed.

Regards

Kai

--- GoldED+/LNX 1.1.4.7
* Origin: Monobox (2:240/77)

Hello mark!

11 Jan 20, mark lewis wrote to Kai Richter:

ew (TINW) really need to know the name of this defective software
creating pkts with too long subject lines... it is possible there's a newer/older version that doesn't have this defect and the originating system can be pointed to this version to replace their defective
one...

Good point. Additionally we all know about the development status of very old FTN software. If that software can't be fixed then there may be an alternate software that does the same job.

Regards

Kai

--- GoldED+/LNX 1.1.4.7
* Origin: Monobox (2:240/77)

Hello Zhenja!

11 Jan 20, Zhenja Kaliuta wrote to Mark lewis:

There is no problem there (anymore). It was some newly written
software and the author is aware of it and has fixed it AFAIU.

That's a positiv surprise. Wonders will never cease. ;-)

Regards

Kai

--- GoldED+/LNX 1.1.4.7
* Origin: Monobox (2:240/77)

Hello Oli!

11 Jan 20, Oli wrote to Kai Richter:

Fix the source of the problem. If it can't be fixed don't allow
the broken to enter. That's straight, clear and simple.

Maybe we should block all PKTs generated by Mystic BBS then?

Well, i'm emotionally affected so i'm the wrong person for judgement. If you would start a seriously vote i would mark the YES field. But thats because some mails from that software crash my terminal output to unreadable. I have to quit my editor then, do a terminal reset and start again carefully skipping the broken mail. That is very annoying sometimes.

And also PKTs from hpt itself? (which doesn't set the JAM OADDRESS and rewrites the message date for Squish messages.

Actually i have no clue if msgbase formats are part of the FTS. I never noticed the msgbase format of incoming mails. I can't see if you use JAM or squish. Anyway your mail is stored in squish on my system. That's why i can't see a reason to drop a bug report.

Regards

Kai

--- GoldED+/LNX 1.1.4.7
* Origin: Monobox (2:240/77)

13 Jan 20 12:27, you wrote to me:

Fix the source of the problem. If it can't be fixed don't allow
the broken to enter. That's straight, clear and simple.

Maybe we should block all PKTs generated by Mystic BBS then?

Well, i'm emotionally affected so i'm the wrong person for judgement.
If you would start a seriously vote i would mark the YES field. But
thats because some mails from that software crash my terminal output
to unreadable. I have to quit my editor then, do a terminal reset and start again carefully skipping the broken mail. That is very annoying sometimes.

I'm emotionally affected too. Most of the time I'm reading fido mails on my 4.8" phone (ssh session to my headless Raspberry). The screen is a bit small to display 80 columns in a comfortable font size. Mystic's hard wrapping of all mail that passes through to 80 chars/line is very annoying.

And also PKTs from hpt itself? (which doesn't set the JAM OADDRESS
and rewrites the message date for Squish messages.

Actually i have no clue if msgbase formats are part of the FTS. I
never noticed the msgbase format of incoming mails. I can't see if you
use JAM or squish. Anyway your mail is stored in squish on my system. That's why i can't see a reason to drop a bug report.

As far as I understand it from the documentation hpt always does import/export in one pass, is that correct? But What about echoareas that get rescanned? Around 50% of the mails (or a little bit less) will have a modified timestamp (DOS time format with 2 second granularity). To be clear, Squish message base format has a field for storing the original FTN time field, it is just not used by hpt (if I understand the C sources correctly).

* Origin: kakistocracy (2:280/464.47)

Hello Oli,

Tuesday January 14 2020, Oli wrote to Kai Richter:

And also PKTs from hpt itself? (which doesn't set the JAM
OADDRESS and rewrites the message date for Squish messages.

As far as I understand it from the documentation hpt always does import/export in one pass, is that correct? But What about echoareas
that get rescanned? Around 50% of the mails (or a little bit less)
will have a modified timestamp (DOS time format with 2 second granularity). To be clear, Squish message base format has a field for storing the original FTN time field, it is just not used by hpt (if I understand the C sources correctly).

Could you please send me a bug report with full description of actions to reproduce the error with an example of the message base and pkt taking part in the actions. I am very busy at the moment but I'll try to investigate it when time permits.

Michael

... node (at) f1042 (dot) ru
--- GoldED+/LNX 1.1.5-b20170303
* Origin: Moscow, Russia (2:5020/1042)

And also PKTs from hpt itself? (which doesn't set the
JAM OADDRESS and rewrites the message date for Squish
messages.

As far as I understand it from the documentation hpt
always does import/export in one pass, is that correct?
But What about echoareas that get rescanned? Around 50%
of the mails (or a little bit less) will have a modified
timestamp (DOS time format with 2 second granularity). To
be clear, Squish message base format has a field for
storing the original FTN time field, it is just not used
by hpt (if I understand the C sources correctly).

Could you please send me a bug report with full
description of actions to reproduce the error with an
example of the message base and pkt taking part in the
actions. I am very busy at the moment but I'll try to
investigate it when time permits.

I'm not using hpt and I cannot provide example files, but the problem is simple:

a squish tosser should write the date string to the __ftsc_date field when importing mails and should use the __ftsc_date field (if it is not empty) on export. see the squish spec. smapi seems to support that field, but hpt doesn't use it.

the JAM problem:
it was mentioned in some other echoarea that hpt doesn't set the OADDRESS for echomail and that the origin address is missing on rescans. i also couldn't find a place in the sources where the OADDRESS is set.

jammnntpd even has a workaround for missing OADDRESS fields.

please correct me if i'm wrong. if i had the time, i would do some testing with hpt.

and thank you for your hpt support and bug fixing, michael.


* Origin: kakistocracy (2:280/464.47)

Hello Oli,

Tuesday January 14 2020, Oli wrote to Michael Dukelsky:

a squish tosser should write the date string to the __ftsc_date field
when importing mails and should use the __ftsc_date field (if it is
not empty) on export. see the squish spec. smapi seems to support that field, but hpt doesn't use it.

the JAM problem:
it was mentioned in some other echoarea that hpt doesn't set the
OADDRESS for echomail and that the origin address is missing on
rescans. i also couldn't find a place in the sources where the
OADDRESS is set.

jammnntpd even has a workaround for missing OADDRESS fields.

OK, thank you, I'll look at it when I have time.

Michael

... node (at) f1042 (dot) ru
--- GoldED+/LNX 1.1.5-b20170303
* Origin: Moscow, Russia (2:5020/1042)

14 Jan 20 15:12, Oli wrote to Michael Dukelsky:

I'm not using hpt and I cannot provide example files, but the problem is simple:

a squish tosser should write the date string to the __ftsc_date field

when

importing mails and should use the __ftsc_date field (if it is not empty)

on

export. see the squish spec. smapi seems to support that field, but hpt

doesn't

use it.

Which field is that? Can I see it in GoldED hexdump pressing "i" ?

These date's are found in your message in my squish base, tossed by hpt.

=== Cut ===
DateTime : 14 Jan 20 15:12:47
OrigAddr : 2:280/464.47
DestAddr : 2:221/1.0
See : 92, 0, 0, 0, 0, 0, 0, 0, 0
Attr : 00020000h (00000000000000100000000000000000b)
DateWritten : 2020-01-14 15:12:46 (7997502Eh)
DateArrived : 2020-01-14 16:13:52 (81BA502Eh)
UTC-Offset : 0
=== Cut ===

'Tommi

---
* Origin: 2001:470:1f15:cb0:f1d0:2:221:1 (2:221/1)

a squish tosser should write the date string to the
__ftsc_date field when importing mails and should use the
__ftsc_date field (if it is not empty) on export. see the
squish spec. smapi seems to support that field, but hpt
doesn't use it.

Which field is that? Can I see it in GoldED hexdump
pressing "i" ?

Good question. I only discovered "i" recently by hitting the wrong key and don't know much about it ...

These date's are found in your message in my squish base,
tossed by hpt.

... but DateTime looks exactly like the value __ftsc_date should have. So I guess I was at least partly wrong. hpt does write __ftsc_date.

=== Cut ===
DateTime : 14 Jan 20 15:12:47
OrigAddr : 2:280/464.47
DestAddr : 2:221/1.0
See : 92, 0, 0, 0, 0, 0, 0, 0, 0
Attr : 00020000h
(00000000000000100000000000000000b)
DateWritten : 2020-01-14 15:12:46 (7997502Eh)
DateArrived : 2020-01-14 16:13:52 (81BA502Eh)
UTC-Offset : 0
=== Cut ===

* Origin: kakistocracy (2:280/464.47)

Hello Oli!

14 Jan 20, Oli wrote to Kai Richter:

if you use JAM or squish. Anyway your mail is stored in squish on
my system. That's why i can't see a reason to drop a bug report.

As far as I understand it from the documentation hpt always does import/export in one pass, is that correct?

I don't know. There is control for import by "hpt toss" and control for export via "hpt scan". But for routing i don't know if it's possible to import pkt to local msgbase without exporting the mail to all linked systems. That option doesn't make sense to me. If the user deletes mails before they are forwarded to the linked systems that would be censorship and against the "no modification of in transit mail" of the policy.

But i found another function that reveals that the date field is common for modification:

***
@item Syntax:
@code{processPkt <string>}
@item Example:
@code{processPkt pktdate}

Space char and name of the current file "*.pkt" will be append into end of <string> and <string> willl be executed before tossing each pkt file. You
are able to fix your pkts using pktdate or any other tool before tossing
them. Note that pkt file may be renamed depending of @code{tossingExt}
token value.
***

In the past when i used OS/2 there was a tool called pktsort that could be hook at the same place to sort the mails by date. That fixed the situation that the answer was visible before the question in the msgbase.

For hpt scan there is an option to bypass netmail:

'When PackNetMailOnScan is "on" (default) hpt packs netmail when doing
"hpt scan" and netmail area is found in EchoTossLog file. When it is
"off" hpt leaves netmail area(s) in EchoTossLog file until "hpt pack" is invoked.'

But What about echoareas that get rescanned? Around 50% of the mails
(or a little bit less) will have a modified timestamp (DOS time format
with 2 second granularity).

That "adjusted" timestamp is common to me since i use fidonet. I don't know the decision for that, maybe it's due to 8-bit limitation of the date format on early fidonet computers back in the 80's?

Regards

Kai

--- GoldED+/LNX 1.1.4.7
* Origin: Monobox (2:240/77)