Hi Nicholas,

Nicholas Hubbard, on 2022-12-19:

Hey, I wrote a script to download the .debian.tar.xz file for every
package on packages.debian.org.

You sound like you would like to build a mirror (or sort of) of
Debian source code. Please kindly lookup the documentation on
setting up a mirror[1], you will find information as of whether
this makes sense for your use case, and methods to get archives
without hammering web front-ends.

[1]: https://www.debian.org/mirror/ftpmirror

I was running the script and all of a
sudden it stopped working, and I am now unable to access
packages.debian.org from my home IP. It seems that I got (rightfully)
flagged as a bot. I was wondering if there was anyway somebody could
make an exception for me so I can run my script without getting blocked
out. I only need to run the script once. I would be happy to share the
exact code I am running, and would be fine scheduling the script to run
at a low-traffic time of day.

There's always some place where the sun shines. But if your
concerned about using low-traffic time of day, you can get the
.debian.tar.xz files from your nearest public repository, which
may have low-ish traffic early in your morning.

Have a nice day, :)
--
Étienne Mollier <emollier@emlwks999.eu>
Fingerprint: 8f91 b227 c7d6 f2b1 948c 8236 793c f67e 8f0d 11da
Sent from /dev/pts/2, please excuse my verbosity.
On air: Kaipa - Screwed-Upness

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEj5GyJ8fW8rGUjII2eTz2fo8NEdoFAmOgw4QACgkQeTz2fo8N Edomnw/+NsjUJn43orj7zqQgwwyuR7k+VbGWXZiNvrolf5+emHWRiLk8+pMw+TKv zKqYrIS8kYvAejz00ezAuxSZuEwtCh5UIi/j96JK9SyDLtKqaUsEtBkYAk4AnKI0 WZJkOQcmWqjMu837IMOvY58vp0LjkYc50Gb0tdn6BFt9g0QoTGrv+wfWg3AAZTCs hizsMthhhK0OJRYzFJVs4EJWOO8mKAhnq/r4pkOkc3gi7Xex03yZiJcxrz4aftUC r89jDXv/e1ql6CJSAUYOz66XUG0M8prEFRk8jk1Uvi+IyV94052Yf1i+7hExYzrF S+M/Nf8zsEibH9ZO6Vd+u1ullt/q3ONGvv/JBmZ415RipFzNpwiBL/1GJEYT+EiI aNc8eowlxxTOoVQPr9T1gI+iL49UtL2zJv8ISl7LVxuPG0DaNr2VS5FJ2RGyP7um 8HdvovNSaLfF2EZHDY5TZGcQBJpLMDLMBLXold4C0PAKSeIIe3/XAChkBWg4lyW9 jTwmUhPuZVt6TjxmS2dX+CpIzh+NbVwZ3EXj026lLNft6QQpVsvaAH27TFUiQRyX 2xnwvGEbI/77aTcOSMV1WrM+OdEKV4JLVr7aP5G4LNUtz1C4ncey2FBWd+KuPiRV gDWbtOaoxGmwDtU35463dgte9zs5gKDMB04LXRoStMI0PHfff0o=
=ZExy
-----E

Hey, I wrote a script to download the .debian.tar.xz file for every
package on packages.debian.org. I was running the script and all of a
sudden it stopped working, and I am now unable to access
packages.debian.org from my home IP. It seems that I got (rightfully)
flagged as a bot. I was wondering if there was anyway somebody could
make an exception for me so I can run my script without getting blocked
out. I only need to run the script once. I would be happy to share the
exact code I am running, and would be fine scheduling the script to run
at a low-traffic time of day.

Thanks,
Nicholas

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Il Lun 19 Dic 2022, 20:48 Nicholas Hubbard <nicholashubbard@posteo.net> ha scritto:

Hey, I wrote a script to download the .debian.tar.xz file for every
package on packages.debian.org. I was running the script and all of a
sudden it stopped working, and I am now unable to access
packages.debian.org from my home IP. It seems that I got (rightfully)
flagged as a bot. I was wondering if there was anyway somebody could
make an exception for me so I can run my script without getting blocked
out. I only need to run the script once. I would be happy to share the
exact code I am running, and would be fine scheduling the script to run
at a low-traffic time of day.

Hi Nicholas,

wait that the block ends and then limit the number of pararell connections this will not block you or at least the block will last less than one
minute, supposing that is server is configured in a reasonable way.

In this file, you will find an example of limitations on 8 downloading processes in parallel

https://github.com/robang74/tinycore-editor/blob/main/tinycore/provides/tcmkdepsdb.sh

Best regards, R-

<div dir="auto"><div class="gmail_quote" dir="auto"><div dir="ltr" class="gmail_attr">Il Lun 19 Dic 2022, 20:48 Nicholas Hubbard &lt;<a href="mailto:nicholashubbard@posteo.net" target="_blank" rel="noreferrer">nicholashubbard@posteo.net</a>&gt; ha
scritto:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hey, I wrote a script to download the .debian.tar.xz file for every<br>
package on <a href="http://packages.debian.org" rel="noreferrer noreferrer noreferrer" target="_blank">packages.debian.org</a>. I was running the script and all of a<br>
sudden it stopped working, and I am now unable to access<br>
<a href="http://packages.debian.org" rel="noreferrer noreferrer noreferrer" target="_blank">packages.debian.org</a> from my home IP. It seems that I got (rightfully)<br>
flagged as a bot. I was wondering if there was anyway somebody could<br>
make an exception for me so I can run my script without getting blocked<br> out. I only need to run the script once. I would be happy to share the<br> exact code I am running, and would be fine scheduling the script to run<br>
at a low-traffic time of day.<br></blockquote></div><div dir="auto"><br></div><div dir="auto">Hi Nicholas,</div><div dir="auto"><br></div><div dir="auto"> wait that the block ends and then limit the number of pararell connections this will not block you
or at least the block will last less than one minute, supposing that is server is configured in a reasonable way.</div><div dir="auto"><br></div><div dir="auto"> In this file, you will find an example of limitations on 8 downloading processes in
parallel </div><div dir="auto"><br></div><div dir="auto"><a href="https://github.com/robang74/tinycore-editor/blob/main/tinycore/provides/tcmkdepsdb.sh">https://github.com/robang74/tinycore-editor/blob/main/tinycore/provides/tcmkdepsdb.sh</a></div><div
dir="auto"><br></div><div dir="auto">Best regards, R-</div></div>

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

I see, a mirror should work fine for me! Thanks for the help!

Nicholas
Étienne Mollier <emollier@emlwks999.eu> writes:

[[PGP Signed Part:Undecided]]
Hi Nicholas,

Nicholas Hubbard, on 2022-12-19:

Hey, I wrote a script to download the .debian.tar.xz file for every
package on packages.debian.org.

You sound like you would like to build a mirror (or sort of) of
Debian source code. Please kindly lookup the documentation on
setting up a mirror[1], you will find information as of whether
this makes sense for your use case, and methods to get archives
without hammering web front-ends.

[1]: https://www.debian.org/mirror/ftpmirror

I was running the script and all of a
sudden it stopped working, and I am now unable to access
packages.debian.org from my home IP. It seems that I got (rightfully)
flagged as a bot. I was wondering if there was anyway somebody could
make an exception for me so I can run my script without getting blocked
out. I only need to run the script once. I would be happy to share the
exact code I am running, and would be fine scheduling the script to run
at a low-traffic time of day.

There's always some place where the sun shines. But if your
concerned about using low-traffic time of day, you can get the
.debian.tar.xz files from your nearest public repository, which
may have low-ish traffic early in your morning.

Have a nice day, :)

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

El 19/12/22 a las 20:31, Nicholas Hubbard escribió:

Hey, I wrote a script to download the .debian.tar.xz file for every
package on packages.debian.org.

You might want to download and parse this file:

https://deb.debian.org/debian/dists/sid/main/source/Sources.xz

and maybe you don't need to use packages.debian.org at all.

Thanks.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)