On 15/12/2022 21.10, Toralf Förster wrote:

On 12/15/22 20:22, Florian Schmaus wrote:

o use PORTDIR_OVERLAY and multiple repositories on their system: a
system-wide, managed by portage, and a dev repository (in your HOME),
scoped in via PORTDIR_OVERLAY.

Isn't this covered by /etc/portage/repos.conf/*

Absolutely, but this requires a manual intervention from the user. And,
of course, you can totally opt-out from portage managing (syncing) the repository, but then you have to take care of syncing yourself.

The point is that with the new portage release, portage's behavior
changes. And I would argue that portage should not, in its effort to
become more user friendly, disregard ebuild-developer friendliness.
Assuming it is achievable with a reasonable amount of additional code complexity.

- Flow

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------GrTj3XU1aDEZJHmw7gTQwfvg
Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: base64

T24gMTIvMTUvMjIgMjA6MjIsIEZsb3JpYW4gU2NobWF1cyB3cm90ZToNCj4gbyB1c2UgUE9S VERJUl9PVkVSTEFZIGFuZCBtdWx0aXBsZSByZXBvc2l0b3JpZXMgb24gdGhlaXIgc3lzdGVt OiBhIA0KPiBzeXN0ZW0td2lkZSwgbWFuYWdlZCBieSBwb3J0YWdlLCBhbmQgYSBkZXYgcmVw b3NpdG9yeSAoaW4geW91ciBIT01FKSwgDQo+IHNjb3BlZMKgaW7CoHZpYcKgUE9SVERJUl9P VkVSTEFZLg0KDQpJc24ndCB0aGlzIGNvdmVyZWQgYnkgL2V0Yy9wb3J0YWdlL3JlcG9zLmNv bmYvKg0KDQplZyBoZXJlJ3MgbXkgY29uZmlnOg0KDQpjYXQgL2V0Yy9wb3J0YWdlL3JlcG9z LmNvbmYvYWxsLmNvbmYNCltERUZBVUxUXQ0KbWFpbi1yZXBvID0gZ2VudG9vDQoNCltnZW50 b29dDQpsb2NhdGlvbiA9IC92YXIvZGIvcmVwb3MvZ2VudG9vDQphdXRvLXN5bmMgPSB5ZXMN CnN5bmMtdHlwZSA9IGdpdA0Kc3luYy11cmkgPSBodHRwczovL2dpdGh1Yi5jb20vZ2VudG9v LW1pcnJvci9nZW50b28uZ2l0DQojc3luYy1naXQtdmVyaWZ5LWNvbW1pdC1zaWduYXR1cmUg PSB0cnVlDQpwcmlvcml0eSA9IDEwDQoNCltsb2NhbF0NCmxvY2F0aW9uID0gL3Zhci9kYi9y ZXBvcy9sb2NhbA0KYXV0by1zeW5jID0gbm8NCnByaW9yaXR5ID0gOTkNCg0KW3Rncm9dDQps b2NhdGlvbiA9IC92YXIvZGIvcmVwb3MvdGdybw0KYXV0by1zeW5jID0gbm8NCg0KcHJpb3Jp dHkgPSA5MA0KDQotLSANClRvcmFsZg0KUEdQIDIzMjE3REE3IDlCODg4RjQ1DQoNCg==

--------------GrTj3XU1aDEZJHmw7gTQwfvg--

-----BEGIN PGP SIGNATURE-----

wsF5BAABCAAjFiEE4Aq096H1MGPqWQN3byNRLEwPvR0FAmObf0gFAwAAAAAACgkQbyNRLEwPvR3b rxAAnP2UxzzwmmB9QF4GT9Pm2mIYD/aaUaQXJ60ezC3XBE08YnEQhBUoytoMMQWWuapMZRjhDOyh Y9bEer7U8KbUnlhUHlj0KPy+BCrP9bI1SKeVmXGs8lEwqj+mcJoQlbB3QZZSqtJHR8dVPBLReQIg VW8OKvMoCyLzGA7gaHFO6kilBfmsepSSW2vtr6+sdQOWulLCf7SsjbV6pAabu5X9OK08G+LN1jsG ZgE2WpiQ69zQoA23hfrZYbfa8WTIB0dcKILmaQxxd8QBtZ2cVIdMlkMgTMMGgY5dnVcx/Xgf9JvC JHO9vSnluPRFWYucovrQW/WzzoqGsY1izumB+PVU+CtTP4f9KuNc+tNZ+lco9VOraRjfWUClovZj XumAvYrV/xEK2QToU+wCAGw0l6joUxgLFUvamXhnlUD3PQJ3ZikYfocyL3MX7cF1k7eKRNbiKPUk Rmo407dZJGpvyx9g5cOFSKfe53dtGfJAZ9licAnU3b1+hc5S4V9fbS/MKAK0ZMH/HZBddG+kEvRB FluTzDSLh+0pnM5emjoydUcAYBRI5bpNeukB8skJ0W7d8cAXlQ5mLhUAdVfkydt9Tohempe+A5AZ xZwH0GEPDKa6z8iNaDEDbC/x3S+fW2839R9nA5tgo4rTVSPFi/MGQkiMjiutx6zBtX/q+PNe1CiD PxY=
=3nCC
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 15 Dec 2022, at 19:22, Florian Schmaus <flow@gentoo.org> wrote:

This is a public service announcement that the recently stabilized portage version will truncate you repo's git history to 1.

I wish you'd shown us in #gentoo-portage before sending this, as it's a bit misleading / alarmist. We were actively all speaking
at the time.

Not only to reconsider the phrasing and include some important details, but also to mention the rationale, which I describe
below.

If you felt the Portage team should've written a news item for it, you were (and still are) free to say so, and we'd
do it.

While this is a good thing for the majority of Gentoo users, it affects developers that develop in a git known to portage (like me). If I understand the portage maintainers vision correctly, then future portage will assume full control over its

configured repositories and potentially perform destructive operations on them, for example "git clean" [1].

... only for repositories of sync-type=git & auto-sync=yes.

The rationale for this is that most people who use sync-type=git are doing so because they want
a quicker sync (to complete), a more reliable one (no Manifest race condition issues), and to
get changes from Gentoo faster.

Whenever Portage is syncing something, our view was that we should prioritise successful
completion of syncs, especially given it may be performed unattended.

If git is pulling from a CDN, Portage may on one sync receive state X, but
on a subsequent sync receive state X-1. This can cause an odd state
where git wants to resolve conflicts and manual intervention is required.

This situation was often worse with sync-depth=1 and would lead
to orphaned files, hence the 'git clean' PR referenced.

The motivation behind the sync depth part was because of
disk space growing unbounded otherwise.

I personally would prefer portage simply adjusting its behavior based on the owner of the repository. That is, if it's the 'portage' user then assume full control, and if it is a different user, then fall back to a preserving, conservative mode of

operation. Unfortunately, for me, this idea was received skeptically at best in a recent discussion in #gentoo-portage.

This wouldn't work for Prefix and also FEATURES="-usersync".

--

Now, looking forward in a constructive manner, I think we can
make both camps happy here with an option to indicate
If Portage can assume the repository will be touched by something
other than Portage.

I propose a configuration option called 'volatile' (thanks
Arsen for the name suggestion) which indicates that the
repository may be changed outside of Portage by any time,
and hence no destructive operations should be carried out.

If the option is on, it also prohibits some optimisations
which require assuming the integrity of the repository.

I have a draft of these changes at https://github.com/gentoo/portage/pull/961.

(https://github.com/gentoo/portage/pull/961.patch if want to view as a raw patch series.0

I am undecided on whether volatile should be default on or not. In the PR
as it is, it defaults to off, which would require a news item. I may just turn it
on given the tone of this thread, as none of it has been very encouraging
for further work on Portage.

-----BEGIN PGP SIGNATURE-----

iNUEARYKAH0WIQQlpruI3Zt2TGtVQcJzhAn1IN+RkAUCY51W018UgAAAAAAuAChp c3N1ZXItZnByQG5vdGF0aW9ucy5vcGVucGdwLmZpZnRoaG9yc2VtYW4ubmV0MjVB NkJCODhERDlCNzY0QzZCNTU0MUMyNzM4NDA5RjUyMERGOTE5MAAKCRBzhAn1IN+R kCibAP4/UftCX5BatRyrHxp0J5ed/TZx1PFt+oxyL534uUURlwD+PyD4Vto2NDly UVxcA5MTDt/2NrYlwczuzw3RKDSqtwo=
=Rs5y
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

The motivation behind the sync depth part was because of
disk space growing unbounded otherwise.

I recently did `git config --local fetch.prune true` to the repos I
don't develop (user mode).
And that seems to help with the uncontrollable growth.

# grep prune /usr/portage/.git/config -B1
[fetch]
prune = true

Michael

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 17 Dec 2022, at 05:42, Sam James <sam@gentoo.org> wrote:

... only for repositories of sync-type=git & auto-sync=yes.

Sorry, of course, the auto sync part doesn't matter.

-----BEGIN PGP SIGNATURE-----

iNUEARYKAH0WIQQlpruI3Zt2TGtVQcJzhAn1IN+RkAUCY51g5V8UgAAAAAAuAChp c3N1ZXItZnByQG5vdGF0aW9ucy5vcGVucGdwLmZpZnRoaG9yc2VtYW4ubmV0MjVB NkJCODhERDlCNzY0QzZCNTU0MUMyNzM4NDA5RjUyMERGOTE5MAAKCRBzhAn1IN+R kAfTAP9fI7ZPSMbU/B1CtQUa8YojhVW98wwyAkzNKVP6XrxmvAEAusjr8UkQdUGE I+Ph+XicyeZOI7GNwS9yAlSNUgRA5Qc=
=56oU
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Sat, Dec 17, 2022 at 05:42:43AM +0000, Sam James wrote:

On 15 Dec 2022, at 19:22, Florian Schmaus <flow@gentoo.org> wrote:

This is a public service announcement that the recently stabilized portage version will truncate you repo's git history to 1.

I wish you'd shown us in #gentoo-portage before sending this, as it's a bit misleading / alarmist. We were actively all speaking
at the time.

Not only to reconsider the phrasing and include some important details, but also to mention the rationale, which I describe
below.

If you felt the Portage team should've written a news item for it, you were (and still are) free to say so, and we'd
do it.

While this is a good thing for the majority of Gentoo users, it affects developers that develop in a git known to portage (like me). If I understand the portage maintainers vision correctly, then future portage will assume full control over its

configured repositories and potentially perform destructive operations on them, for example "git clean" [1].

... only for repositories of sync-type=git & auto-sync=yes.

The rationale for this is that most people who use sync-type=git are doing so because they want
a quicker sync (to complete), a more reliable one (no Manifest race condition issues), and to
get changes from Gentoo faster.

Whenever Portage is syncing something, our view was that we should prioritise successful
completion of syncs, especially given it may be performed unattended.

If git is pulling from a CDN, Portage may on one sync receive state X, but
on a subsequent sync receive state X-1. This can cause an odd state
where git wants to resolve conflicts and manual intervention is required.

This situation was often worse with sync-depth=1 and would lead
to orphaned files, hence the 'git clean' PR referenced.

The motivation behind the sync depth part was because of
disk space growing unbounded otherwise.

Just to make this more abundantly explicit: it repeatedly happened to
me (but others, too) that on a system using git to sync a shallow
::gentoo clone, during unattended syncs, the repository attempted a
merge with the upstream, yielding a repo in a merging state, with a
huge list of dirtied files in the working tree, plus untracked files.

I personally would prefer portage simply adjusting its behavior based on the owner of the repository. That is, if it's the 'portage' user then assume full control, and if it is a different user, then fall back to a preserving, conservative mode of

operation. Unfortunately, for me, this idea was received skeptically at best in a recent discussion in #gentoo-portage.

This wouldn't work for Prefix and also FEATURES="-usersync".

--

Now, looking forward in a constructive manner, I think we can
make both camps happy here with an option to indicate
If Portage can assume the repository will be touched by something
other than Portage.

I propose a configuration option called 'volatile' (thanks
Arsen for the name suggestion) which indicates that the
repository may be changed outside of Portage by any time,
and hence no destructive operations should be carried out.

If the option is on, it also prohibits some optimisations
which require assuming the integrity of the repository.

I have a draft of these changes at https://github.com/gentoo/portage/pull/961.

(https://github.com/gentoo/portage/pull/961.patch if want to view as a raw patch series.0

I am undecided on whether volatile should be default on or not. In the PR
as it is, it defaults to off, which would require a news item. I may just turn it
on given the tone of this thread, as none of it has been very encouraging
for further work on Portage.

-----BEGIN PGP SIGNATURE-----

iHUEABYKAB0WIQQyG9yfCrmO0LPSdG2gXq2+aa/JtQUCY552SQAKCRCgXq2+aa/J tco6AQC1glvMLl1WOJVR4QjnaHXYOEMLk2h8X1yHTqtFiD7S0wEAycT+//GOpFTa obUylEXv3QRDzBcFF4ZOT8tENeVKEwI=
=LxAd
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 17.12.22 06:42, Sam James wrote:

On 15 Dec 2022, at 19:22, Florian Schmaus <flow@gentoo.org> wrote:
I personally would prefer portage simply adjusting its behavior based on the owner of the repository. That is, if it's the 'portage' user then assume full control, and if it is a different user, then fall back to a preserving, conservative mode of

operation. Unfortunately, for me, this idea was received skeptically at best in a recent discussion in #gentoo-portage.

This wouldn't work for Prefix and also FEATURES="-usersync".

Simply do not apply the approach on Prefix. That should be fine. I am
not sure how usersync being disabled (or enabled) plays a role here, though.

I believe something like this would make everyone happy:

if volatile_explicitly_configured:
volatile = explicitly_configured_volatile_value
else if prefix
volatile = false
else if Path(repo_dir).user != (root|portage) # Or, if uid >= 1000
volatile = true
else
volatile = false

Assuming that the "if target repo is not shallow, then no shallow clone
(unless explicitly requested)" check is only applied if volatile=true,
then you even have the desired effect that users get their repo
automatically converted to shallow ones. Which makes you happy. And the
above logic would make me happy, since the "if Path(repo_dir).uid >=
1000" branch would be taken for me.

That sounds like a win/win to me.

- Flow

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 18 Dec 2022, at 10:19, Florian Schmaus <flow@gentoo.org> wrote:

On 17.12.22 06:42, Sam James wrote:

On 15 Dec 2022, at 19:22, Florian Schmaus <flow@gentoo.org> wrote:
I personally would prefer portage simply adjusting its behavior based on the owner of the repository. That is, if it's the 'portage' user then assume full control, and if it is a different user, then fall back to a preserving, conservative mode of

operation. Unfortunately, for me, this idea was received skeptically at best in a recent discussion in #gentoo-portage.

This wouldn't work for Prefix and also FEATURES="-usersync".

Simply do not apply the approach on Prefix. That should be fine. I am not sure how usersync being disabled (or enabled) plays a role here, though.

The owner of the repository and its permissions will be affected by the permissions used by Portage to sync.

I believe something like this would make everyone happy:

if volatile_explicitly_configured:
volatile = explicitly_configured_volatile_value
else if prefix
volatile = false
else if Path(repo_dir).user != (root|portage) # Or, if uid >= 1000
volatile = true
else
volatile = false

Assuming that the "if target repo is not shallow, then no shallow clone (unless explicitly requested)" check is only applied if volatile=true, then you even have the desired effect that users get their repo automatically converted to shallow ones.

Which makes you happy. And the above logic would make me happy, since the "if Path(repo_dir).uid >= 1000" branch would be taken for me.


Feel free to suggest that on the PR, it sounds like a decent compromise, if a bit automagic - but it wouldn't be the first or last case of that in Portage.

-----BEGIN PGP SIGNATURE-----

iNUEARYKAH0WIQQlpruI3Zt2TGtVQcJzhAn1IN+RkAUCY578wV8UgAAAAAAuAChp c3N1ZXItZnByQG5vdGF0aW9ucy5vcGVucGdwLmZpZnRoaG9yc2VtYW4ubmV0MjVB NkJCODhERDlCNzY0QzZCNTU0MUMyNzM4NDA5RjUyMERGOTE5MAAKCRBzhAn1IN+R kEBoAQCrQiD8B/QlECouBw1BurH1vOqfGDmMH+Vq/COqekGDNQEAmbLoRw3LCcFv rT6iDxQH1R9XNNz9AhjyFuRzAtmjUwk=
=EKaP
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)