While browsing around for Ada-related docs some years ago, I came across
this site <https://iment.com/maida/computer/redref/index.htm> which
collects info on the DoD’s “Strawman”, “Woodenman”, “Tinman”, “Ironman”
and “Steelman” series of RFPs, and the specs for the “Red” language that
didn’t become Ada.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 06/09/2024 02:55, Lawrence D'Oliveiro wrote:

While browsing around for Ada-related docs some years ago, I came across
this site <https://iment.com/maida/computer/redref/index.htm> which
collects info on the DoD’s “Strawman”, “Woodenman”, “Tinman”, “Ironman”
and “Steelman” series of RFPs, and the specs for the “Red” language that
didn’t become Ada.

We have all the colours now https://www.reddit.com/r/ada/comments/165f5zg/common_hol_phase_1_reports/

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Sat, 7 Sep 2024 17:43:26 +0100, Luke A. Guest wrote:

We have all the colours now https://www.reddit.com/r/ada/comments/165f5zg/common_hol_phase_1_reports/

Hey, terrific. Don’t you wonder why people insist on returning “403 Forbidden” for those using a command-line tool like wget?

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 08/09/2024 00:06, Lawrence D'Oliveiro wrote:

On Sat, 7 Sep 2024 17:43:26 +0100, Luke A. Guest wrote:

We have all the colours now
https://www.reddit.com/r/ada/comments/165f5zg/common_hol_phase_1_reports/

Hey, terrific. Don’t you wonder why people insist on returning “403 Forbidden” for those using a command-line tool like wget?

Because it's a massive doc?

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

"Lawrence D'Oliveiro" <ldo@nz.invalid> wrote in message news:vbimad$1j26j$6@dont-email.me...
...

Hey, terrific. Don't you wonder why people insist on returning "403 Forbidden" for those using a command-line tool like wget?

No (as the operator of a web site). Most SEO tools and other
useless/criminal scrapers like to fake their identification, and WGet is a favorite for that task. There are many owners that block that and many other abused user-agents. (The Ada-Auth.org blocks about 20 user-agents, but not WGet. See https://support.tigertech.net/error-blocked-user-agents for a
better explanation than I can give as to why. I don't block all of these agents, but I do throttle some of them and block their access to parts of
the site.)

Additionally, some sites want to provide access to their documents, but not really to allow people to copy them (in order to get advertising revenue
from reading them). I know the owners of the Red specification are in this category, as I offered to host a copy in the AdaIC archives, and they turned that down.

Randy.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Wed, 11 Sep 2024 23:57:33 -0500, Randy Brukardt wrote:

"Lawrence D'Oliveiro" <ldo@nz.invalid> wrote in message news:vbimad$1j26j$6@dont-email.me...

Don't you wonder why people insist on returning "403
Forbidden" for those using a command-line tool like wget?

Most SEO tools and other
useless/criminal scrapers like to fake their identification, and WGet is
a favorite for that task. There are many owners that block that and many other abused user-agents.

That doesn’t make any sense, because anybody who knows how to use wget
would know about its “--user-agent” option. So if they really were using wget to conduct their site abuse, you wouldn’t know, and blocking wget’s default user-agent setting wouldn’t help.

(The Ada-Auth.org blocks about 20 user-agents, but not WGet.

Which kind of proves my point.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 13 Sep 2024, Lawrence D'Oliveiro wrote
(in article <vbvpta$esm6$8@dont-email.me>):

On Wed, 11 Sep 2024 23:57:33 -0500, Randy Brukardt wrote:

"Lawrence D'Oliveiro"<ldo@nz.invalid> wrote in message news:vbimad$1j26j$6@dont-email.me...

Don't you wonder why people insist on returning "403
Forbidden" for those using a command-line tool like wget?

Most SEO tools and other
useless/criminal scrapers like to fake their identification, and WGet is
a favorite for that task. There are many owners that block that and many other abused user-agents.

That doesn�t make any sense, because anybody who knows how to use wget
would know about its "--user-agent" option. So if they really were using
wget to conduct their site abuse, you wouldn�t know, and blocking wget�s default user-agent setting wouldn�t help.

Wrong.
--
Bill Findlay

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

geodandw <geodandw@gmail.com> writes:

Why is this wrong?

I run into sites all the time that block the wget user agent, but that I
can retrieve with curl.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Thu, 12 Sep 2024 19:16:50 -0700, Paul Rubin wrote:

I run into sites all the time that block the wget user agent, but that I
can retrieve with curl.

And I run into sites all the time that block the default wget user agent,
but that I can retrieve with wget.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

"Lawrence D'Oliveiro" <ldo@nz.invalid> wrote in message news:vc091i$ljiq$2@dont-email.me...

On Thu, 12 Sep 2024 19:16:50 -0700, Paul Rubin wrote:

I run into sites all the time that block the wget user agent, but that I
can retrieve with curl.

And I run into sites all the time that block the default wget user agent,
but that I can retrieve with wget.

You're confused. The attackers aren't using Wget, but they are *claiming* to
be WGet. As you point out, real WGet users tend to claim to be other things.
So blocking WGet would be more likely to block the attackers than real
users. (As you state, real users know how to get around the blocks, so the inconvinience for them is minor. Usually, the attackers don't change their attacks often, there's plenty of sites that don't protect themselves at all.
So they are more effective against attackers.)

And anyone that thinks that ad revenue is important is probably blocking all grabbers, and probably throttling everything else so that grabbing multiple pages is very slow (at human reading speeds). (At least 90% of the browser
hits I see are obviously fake, and if I cared enough I would block all of
them - it would just take a bit of programming to check if the behavior is similar to that of a live human. But I only block when something is causing performance problems, and generally by IP.)

Randy.


Randy.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Sat, 14 Sep 2024 01:27:22 -0500, Randy Brukardt wrote:

"Lawrence D'Oliveiro" <ldo@nz.invalid> wrote in message news:vc091i$ljiq$2@dont-email.me...

On Thu, 12 Sep 2024 19:16:50 -0700, Paul Rubin wrote:

I run into sites all the time that block the wget user agent, but that
I can retrieve with curl.

And I run into sites all the time that block the default wget user
agent, but that I can retrieve with wget.

You're confused. The attackers aren't using Wget, but they are
*claiming* to be WGet.

But that long list of user agents being blocked that you previously
mentioned did not include wget.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

"Lawrence D'Oliveiro" <ldo@nz.invalid> wrote in message news:vc3di3$1bmhk$3@dont-email.me...

On Sat, 14 Sep 2024 01:27:22 -0500, Randy Brukardt wrote:

...

You're confused. The attackers aren't using Wget, but they are
*claiming* to be WGet.

But that long list of user agents being blocked that you previously
mentioned did not include wget.

I don't block WGet myself, I haven't seen enough abuse. There's more from
Meta and Amazon, and I probably shouldn't block them, either. (Meta and
Amazon both have crawlers that ignore requests for throttling, I know some webmasters block them for that reason.)

But I'm sure that other webmasters have different experiences. I don't find
it surprising when anything is blocked anymore (although blocking Firefox
seems a bit much).

Randy.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)