Yesterday I switched my gentoo repo from rsync to git, and the initial
--sync with an empty directory did a git clone successfully.

Today, when I try to sync, it always fails:

$ sudo emerge --sync >>> Syncing repository 'gentoo' into '/var/db/repos/gentoo'...
/usr/bin/git fetch origin
error: RPC failed; HTTP 504 curl 22 The requested URL returned error: 504 fatal: the remote end hung up unexpectedly
!!! git fetch error in /var/db/repos/gentoo

Action: sync for repo: gentoo, returned code = 128

The SSL connection to anongit.gentoo.org happens, the handshake goes
OK, a few frames of data go back and forth, then the nothing happens
for 60s and the client end sends a FIN.

After the initial sync which does a git clone, how do you do
subsequent sync operations?

--
Grant

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Wed, 13 Oct 2021 18:25:12 -0000 (UTC), Grant Edwards wrote:

Yesterday I switched my gentoo repo from rsync to git, and the initial
--sync with an empty directory did a git clone successfully.

Today, when I try to sync, it always fails:

$ sudo emerge --sync >>> Syncing repository 'gentoo' into '/var/db/repos/gentoo'... /usr/bin/git fetch origin
error: RPC failed; HTTP 504 curl 22 The requested URL returned error:
504 fatal: the remote end hung up unexpectedly !!! git fetch error in /var/db/repos/gentoo

Action: sync for repo: gentoo, returned code = 128

The SSL connection to anongit.gentoo.org happens, the handshake goes
OK, a few frames of data go back and forth, then the nothing happens
for 60s and the client end sends a FIN.

After the initial sync which does a git clone, how do you do
subsequent sync operations?

emerge --sync works here. What do you have in repos.conf?


--
Neil Bothwick

There's too much blood in my caffeine system.

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEE8k9T/rX16EJxEKG692eFu0QSMJgFAmFnJ1wACgkQ92eFu0QS MJjwIhAAkCABUMXCO6QkjtL3xs1Uuuyj4J0EMwwk0SKAs/Ln+oy8T8xCqV8llFVE zPUpfbDoe72K38kTn4vk7UbwtGGAvyrCBekkzxYGnSjZin5CTvODGh4COKtSmVSG xqrnl6tjqAVHVzlLNHiaI/u++5gcDinEdjgJnKcRwiqXlxfsV52TzOmg/j+MfPpJ F5QsS1QvAIM1BlaYBMZQ6b2KUk6WNy65stz7vUHN3gLaDTqw4LmCATtN0gtoQU/j J6zoXhupjNPmogFpgHLeJyCMgUAFcnLlSAdNYkuzDHAsbtKcjjf1c6PbG8KdnQrK xHoK4+9tqhOgd+tMHrnpQnLXug3qieAmSqiDHw+Q08MYrZPmvUP8Aj/YCqY1uE92 7WhGbJvPqxbSO3v5yJlgM+ZAJFF40l+0cDlJpAzFKAVebU48tbgF5UOQ1FOYFhHe UqdSAZ8web5ZzENPnKS2uLzAJr1SHaoINKNOm+JDXL/NTfPuFiv+nNqPPbQ3E8nx mYT9QhwX64wkFW/JPt6wUQvrqkzbQfiMNm4l9Oeg4x+k3vqGI9uMCL7jUlxDievx XpTWNWCL6llngorQ1Jf89AFn4ddpvnnuq5Ub9Xw15r9+pw/lEvP5ep/yKwGfWYMe wZloidIFcGuolhrYU8LLgXCfi2pR3Vab0UQ3rC6s6GPqNE8aJgo=
=GLis
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 2021-10-13, Neil Bothwick <neil@digimed.co.uk> wrote:

On Wed, 13 Oct 2021 18:25:12 -0000 (UTC), Grant Edwards wrote:

Yesterday I switched my gentoo repo from rsync to git, and the initial
--sync with an empty directory did a git clone successfully.

Today, when I try to sync, it always fails:

$ sudo emerge --sync >>> Syncing repository 'gentoo' into
'/var/db/repos/gentoo'... /usr/bin/git fetch origin
error: RPC failed; HTTP 504 curl 22 The requested URL returned error:
504 fatal: the remote end hung up unexpectedly

[...]

After the initial sync which does a git clone, how do you do
subsequent sync operations?

emerge --sync works here. What do you have in repos.conf?

I didn't have a sync-depth setting. Setting that to 1 fixed the
problem. It was apparently timing out because it takes too long for
the server to respond to a fetch with an unlimited depth. Seems like
maybe it ought to default to 1? Is there some reason it should default
to doing unlimited depth fetch operations?

--
Grant

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Wed, 13 Oct 2021 18:50:10 -0000 (UTC), Grant Edwards wrote:

emerge --sync works here. What do you have in repos.conf?

I didn't have a sync-depth setting. Setting that to 1 fixed the
problem. It was apparently timing out because it takes too long for
the server to respond to a fetch with an unlimited depth. Seems like
maybe it ought to default to 1? Is there some reason it should default
to doing unlimited depth fetch operations?

I don't have that either.


--
Neil Bothwick

IRQs? We don't need no stinking IRQs!

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEE8k9T/rX16EJxEKG692eFu0QSMJgFAmFnNQoACgkQ92eFu0QS MJg9MQ/9E5YRt7XtvA5QNnOtwRj2CeiLFEIzNWXn/cMilDCm+41UA/AVSSjYh0rF fn58CdOWIW3k0nZuGwmWvgnnUxXmhgzHtYvYJplYXX32d6kIDMnhn2zAtNn5sFSX PJ8H8/H1ZY+LXgIVywOFKrM7NfMnByCWfm+XCZXmvj8EvBVZo5pMe0SsrC5to0V7 w+Ljyvj0m2xTDILdk5pjtVMF3wFaSq1ciqPOPDs+NsaYgQ2ZbCEyuAzR3LmRw5m+ nRNjhfL+HaO0qJElIJmP0mwYzXT5PstJ+axljdzAbXk1h7hyw7ylOLi/1UI+8zuJ 4jsuBcEr+wFhGZ2K4F1i8/B8j0Jja7JbuJrVTZ3nqH3vMprsqhTEoZkRUNT8wKkX BCxMhywNI58jZ1g+vgcJSUs+fOWY7ljY9brogCM/9KKf2tLlL6vgrOiL3uM2cZou lmnn04lzoXfyVS2D/Z0ew2pdpfCJLn+z6VVgMzYjmqyToKwPbBhirNSoV1Efy04i OYeQX4CvXWmxfcfvKxuPeo7r4qFsYezlhqf03PE5Raf0/tOfVPh9R5Le/hqzF2G4 Tk2Ym2/P2coHh9QR5nk09vedNbx1BVWGp+EMkqZd6lGZfWXIUpQ8J/OBgIH0atDB g2EoJlXWn004sk81tqaqorsJis5Hwi6qAb7o/OJLdHgck4dfe1g=
=TyR5
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Wed, Oct 13, 2021 at 2:50 PM Grant Edwards <grant.b.edwards@gmail.com> wrote:

Is there some reason it should default
to doing unlimited depth fetch operations?

If all you want is a repo, no reason to set the depth higher. If you
want to see the history then you'll want it all.

However, once you have an initial sync, I don't think it should go
back and fetch all the history unless you explicitly ask git to do so.
I don't see why this would cause issues after the initial sync. If
you were fetching all the history, it would be the FIRST sync that
caused all the issues. Well, unless portage is going out and trying
to pull it all in (and if so I'd think it would have done it from the
start).

Once you have the full repo then subsequent syncs should be very fast
and don't use much CPU server-side. The git client sends the remote
server its current head, and then the server walks back from its head
until it finds yours, which will only be a short distance if you've
synced recently. Then it is only the new objects in-between that have
to be sent. The whole thing is de-duplicated and copy-on-write just
due to its data structure.

I'm suspecting some sort of server-side issue - maybe an intermittent
one. Either that or portage is really trying to pull in that history
after the initial sync.

Another option is to do a pull from the github mirror. That same repo
is hosted on both gentoo's server and github, and they're identical
(the content hash tells you as much), so you should be able to do a
pull from either seamlessly. The signatures/etc are applied to both
as well. Some don't care for github not being FOSS but if you're just
using it as a mirror I'd argue it is no different than if one of the
gazillion distfile mirrors happened to run IIS or have a firmware that
wasn't coreboot. It is just another mirror.

--
Rich

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 2021-10-13, Rich Freeman <rich0@gentoo.org> wrote:

On Wed, Oct 13, 2021 at 2:50 PM Grant Edwards <grant.b.edwards@gmail.com> wrote:

Is there some reason it should default
to doing unlimited depth fetch operations?

If all you want is a repo, no reason to set the depth higher.

Then a default of 1 seems like the obvious "right" answer.
Unfortunately, it defaults to "unlimited" according to https://wiki.gentoo.org/wiki//etc/portage/repos.conf

sync-depth

Specifies sync depth to use for DVCS repositories. If set to 0,
the depth is unlimited. Defaults to 0.

If you want to see the history then you'll want it all.

Apparently, that's the default. Without any sync-depth setting the
fetch was stalling and then timing out after 60s. With sync-depth=1,
the fetch completed in about 1s.

--
Grant

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Look I mean in any other context for a git repository sync-depth=0
would be the obvious choice, why have a version control system if
you're limiting the version control. It was added to gentoo so that
developers could work on their own branches of the git repo whilst
still having it actually work with portage. And then the next logical application is making your own /usr/local/portage not suck. Using git
as the main sync instead of rsync is something that's come about way
later, and really is fairly niche, compared to what the git capability
in portage was originally designed for.

So the default being for git to act like git, instead of git to act
like an alternative to rsync, makes perfect sense in that context.

On Thu, 14 Oct 2021 at 18:22, Peter Humphrey <peter@prh.myzen.co.uk> wrote:

On Thursday, 14 October 2021 03:50:59 BST Grant Edwards wrote:

On 2021-10-13, Rich Freeman <rich0@gentoo.org> wrote:

On Wed, Oct 13, 2021 at 2:50 PM Grant Edwards

<grant.b.edwards@gmail.com> wrote:

Is there some reason it should default
to doing unlimited depth fetch operations?

If all you want is a repo, no reason to set the depth higher.

Then a default of 1 seems like the obvious "right" answer.
Unfortunately, it defaults to "unlimited" according to https://wiki.gentoo.org/wiki//etc/portage/repos.conf

sync-depth

Specifies sync depth to use for DVCS repositories. If set to 0,
the depth is unlimited. Defaults to 0.

If you want to see the history then you'll want it all.

Apparently, that's the default. Without any sync-depth setting the
fetch was stalling and then timing out after 60s. With sync-depth=1,
the fetch completed in about 1s.

I have it set to 8 here, for no reason I can remember. Is there a disadvantage
in that?

--
Regards,
Peter.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Thursday, 14 October 2021 03:50:59 BST Grant Edwards wrote:

On 2021-10-13, Rich Freeman <rich0@gentoo.org> wrote:

On Wed, Oct 13, 2021 at 2:50 PM Grant Edwards

<grant.b.edwards@gmail.com> wrote:

Is there some reason it should default
to doing unlimited depth fetch operations?

If all you want is a repo, no reason to set the depth higher.

Then a default of 1 seems like the obvious "right" answer.
Unfortunately, it defaults to "unlimited" according to https://wiki.gentoo.org/wiki//etc/portage/repos.conf

sync-depth

Specifies sync depth to use for DVCS repositories. If set to 0,
the depth is unlimited. Defaults to 0.

If you want to see the history then you'll want it all.

Apparently, that's the default. Without any sync-depth setting the
fetch was stalling and then timing out after 60s. With sync-depth=1,
the fetch completed in about 1s.

I have it set to 8 here, for no reason I can remember. Is there a disadvantage in that?

--
Regards,
Peter.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 14/10/2021 09:28, Miles Malone wrote:

So the default being for git to act like git, instead of git to act
like an alternative to rsync, makes perfect sense in that context.

And once it's sync'd the first time, I think sync depth is ignored. It
normally doesn't make much difference anyway, because git stores the
current version in full, and then just diffs going backwards.

The difference between a sync=1 and sync=0 isn't that much for linux I
gather - maybe doubles the size?

For a normal user who just wants to USE portage, a sync of 1 makes
sense. They can then always go back as far as when they first set it up, anyway.

Cheers,
Wol

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)