Hi everyone. Rob emailed me about dpkg triggers and said he was happy
to redirect this. I think it's probably best to have the conversation
on a public list. (Please retain the CC to me, since I don't reliably
read debian-emacsen.)

Rob Browning writes ("dpkg trigger questions (and/or broader emacs add-on options)"):

Hey, Sean mentioned your discussion about triggers, and I ended up first thinking that it might mean triggers wouldn't work for our purposes, but
now I'm not so sure, and wondered if you might have time to discuss that
a bit. (However, if you don't have time for this right now, no
worries.)

I'll try.

Firstly, a disclaimer: it's been a very long time since I did all
this, so I am likely to have misremembered some things - it's likely
that at least something I'm going to say is false. It now seems to me
from what you quote here, and my reading of the doc, that some of the
things I said to Sean on irc were false.

Some background: the principal design goal of triggers was to make it
possible to optimise away redundant work by delaying work until the
end of processing.

Before triggers, the principle was that every package's postinst would
do all necessary processing. In the presence of multiple interacting
packages, that can mean processing more than once: typically, once for
each interacting package.

Also, some tasks are much more efficient when performed in aggregate:
for example, adding soemthing to the initramfs is done by rebuilding
the initramfs, so postponing initramfs rebuilding can help a lot with
install performance.

The model assumes that the "interested" package is more central and
lower in the dependency stack.

What led us to the current attempt to rework emacsen-common / emacs
policy is the realization that dpkg doesn't actually promise much about
the state of a package's dependencies during maintainer script
invocations, outside of "postinst configure",

I'm not sure why that's a problem for emacs addons. In this scheme
you only need to run *compilation* of some addon during that addon's
postinst, and the postinst of the emacs flavour. In each case you can
do processing only of packages that are `configured`, plus this one.

The cleanup tasks don't seem like they'd need much in the way of
working dependencies.

I mention all this because I want to make sure we all have a proper understanding.

One thing I wondered about was this triggers.txt statement:

Packages in t-awaited and t-pending demand satisfaction of their
dependencies just like packages in installed.

Does that mean that all of the dependencies of a package in one of those states should be past their postinst configure, or something else?

Yes, I think so.

However, note that dpkg does *not* bring packages out of `installed`
just because their dependencies regress to less good states, provided
the dependency isn't actually removed.

So one of the options I've been considering is to switch to triggers
(and dropping the attempt to promise that once an add-on's postinst
configure finishes it's "ready").

ISTM that you ought to aim to promise that once an add-on is in state `installed` it is ready (subject to the caveat above). That would
mean any things that invoke the addon, and Depend on it, will work
properly.

I think this means that an addon with old bytecode needs to be either
in `unpacked`, or `triggers-awaited`. Assuming we're using triggers,
that means the addon which needs rebuilds is in `triggers-awaited`.

Let's consider three packages: emacsen package E (of some flavour);
addon packages A and B where B uses features from A.

When we reinstall A we need to rebuild both it and B.

We could achive this by doing it in A's postinst but in a large run
it'll probably have to done again later, eg if emacsen are being
updated too.

Realistically, E must be interested in A, so that A gets rebuilt. (We
don't want to do this the other way around, because that would cause
an emacs to be considered not-`installed` simply because some addon
hadn't been compiled.)

If we update E, E's postinst will need to rebuild both A and B. So
E's postinst needs to be able to do a topological sort of the addons,
so that it can compile them in the right order.

Given that E's postinst can do the sort, it can reliably recompile
everything.

Ideally, though we'd not recompile unchanged packages lower in the
stack when E's postinst is running for a trigger rather than for
configure. That optimisation can be done by examining the triggering
package names.

Given (A) above, I was pondering the possibility of having some
idempotent "rebuild" trigger that would (assuming we can) just examine
what add-ons and flavors are ready (i.e. successfully past their
postinat configures), and rebuild any combinations that are stale[2].
Both flavors and add-ons would trigger "rebuild".

I think my analysis above has come to the conclusion that something
like that is indeed the best design.

Of course there are many variants of the idea, depending on what exactly
our existing semantics allow.

[2] One option might be to just do nothing in the trigger until it is
invoked when the entire add-on tree is past all of the add-on
postinst configures, and then rebuild everything that needs it --
i.e. instead of trying to rebuild stale add-ons in any *subtrees*
that are ready during intermediate trigger invocations.

dpkg tries to defer trigger processing. And you mustn't "defer work"
and return success from a trigger processing postinst invocation,
since you might never be called again. (I guess you could manually
retrigger but this seems fraught and also unnecessary.)

Ian.

--
Ian Jackson <ijackson@chiark.greenend.org.uk> These opinions are my own.

Pronouns: they/he. If I emailed you from @fyvzl.net or @evade.org.uk,
that is a private address which bypasses my fierce spamfilter.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Ian Jackson <ijackson@chiark.greenend.org.uk> writes:

I'm not sure why that's a problem for emacs addons. In this scheme
you only need to run *compilation* of some addon during that addon's postinst, and the postinst of the emacs flavour. In each case you can
do processing only of packages that are `configured`, plus this one.

The cleanup tasks don't seem like they'd need much in the way of
working dependencies.

A general issue is that the current approach was designed with incorrect assumptions, so any of the corner cases (and not so corner cases) that
we'd thought through carefully when it was established might now (and
are likely now to) be broken.

As an example related to the failures that precipitated all this, the
current emacs policy requires all add-ons to depend on emacsen-common,
and for them to call its commands at various points to handle
installs/rebuilds and removals. i.e. emacsen-common is the
library/subsystem that handles all the dependency ordering, etc.

But that's just wrong if you can't assume much about the state (or even availability) of emacsen-common from those maintainer scripts, i.e. when
we designed all this, I would never have guessed that even if you depend
on emacsen-common, it might not actually be ready (or even unpacked)
from say your prerm.

That was a surprise, and means we have to come up with an approach
that's sound given what actually *is* promised, and at least as far as dependencies go, it sounds like the only place you can (nearly
always(?)) count on your dependencies is "postinst configure" (and from
the triggers docs, I'd thought now maybe also "postinst triggered").

However, note that dpkg does *not* bring packages out of `installed`
just because their dependencies regress to less good states, provided
the dependency isn't actually removed.

Hmm, what kind of "less good state"?

ISTM that you ought to aim to promise that once an add-on is in state `installed` it is ready (subject to the caveat above). That would
mean any things that invoke the addon, and Depend on it, will work
properly.

Oh, of course -- that's how the entire current policy is designed. It
was set up to make as few assumptions as possible about what an emacs
flavor or add-on might need to do to configure or clean itself up, while
also providing an inter-add-on dependency respecting install/removal
process.

...but now that we have a lot more experience with add-ons, it could be
that we don't actually need such a general mechanism, and if so, that's
a possibility worth keeping in mind, in case it allows simpler
solutions.

It's also perhaps worth noting that we don't actually care about
triggers per se. I'd just started looking at them when among other
things, I wondered if they might be able to eliminate the need for our
own (currently a bit sketchy) dependency ordering.

Let's consider three packages: emacsen package E (of some flavour);
addon packages A and B where B uses features from A.

When we reinstall A we need to rebuild both it and B.

We could achive this by doing it in A's postinst but in a large run
it'll probably have to done again later, eg if emacsen are being
updated too.

The bigger issue I'd have imagined is that the state of B is not known
at that point?

Realistically, E must be interested in A, so that A gets rebuilt. (We
don't want to do this the other way around, because that would cause
an emacs to be considered not-`installed` simply because some addon
hadn't been compiled.)

Without having thought about it carefully -- that might be OK...
i.e. it might be fine if the whole system (add-ons and flavors) comes
out of "up to date" for a bit, as long as it re-converges eventually.

Particularly if (as may well be the case now for most flavors and
add-ons), the packages still "work fine" in the interim. Say if the install/remove process really was just recompilation, and the preinsts
could adequately clean everything up, then in that arrangement, the
worst you might get is some extraneous per-user compilation artifacts if
a user were to run an emacs flavor before all the recompilations were
finished.

If we update E, E's postinst will need to rebuild both A and B. So
E's postinst needs to be able to do a topological sort of the addons,
so that it can compile them in the right order.

Given that E's postinst can do the sort, it can reliably recompile everything.

But from E's postinst, we have no idea what state the add-ons are in?
And if so, and we had to be "completely conservative", the most we could
do is rebuild any dependency graph sub-trees we could find where the
add-ons were past their "postinst configure"s ... and then we'd have to
ensure we get to the rest later somehow.

One correct solution might involve some command we know will always "run
last" in any given apt/dpkg run, and I'd wondered if triggers might be
able to support that.

dpkg tries to defer trigger processing. And you mustn't "defer work"
and return success from a trigger processing postinst invocation,
since you might never be called again. (I guess you could manually
retrigger but this seems fraught and also unnecessary.)

I think retriggering might be necessary for the "goes last"
approach(es).


Overall, of course, I'd like to understand the facilities we currently
have available, and exactly what they promise while reasoning about
solutions.

I think I have a better handle on what dpkg promises with respect to
dependency states from maintscript invocations, i.e. generally, not
much, in the limiting case, outside "postinst configure" (and now maybe "postinst triggered"). There it's likely, and as likely as it'll ever
be, that your deps will be configured -- elsewhere, not so much.

I'd also like to understand what we might be able to count on with
respect to triggers. For example, is it plausible to put them in
roughly the same class as "postinst configure" with respect to the
dependencies will be available/configured there?


More concretely, and for any who haven't seen it, included below is one tentative idea we had been considering as a possible way to use triggers
to fix the current issues.

Don't take the specific details too seriously yet, and I would love to
find out that we/I'm just over-complicating things somehow. (If it
actually is a sensible approach, I do have some of the code sketched
out, and it also has some bits in common with the current policy/implementation.)


# Overview:

The general approach is to maintain relevant state files in /var/lib/emacsen-common/..., to manipulate those to reflect changes, and
to then trigger all the add-ons to re-evaluate the state to accommodate
changes (in some cases, an add-on will decide there's nothing to do).

We currently assume that "postinst triggered" invocations are as likely
to have the package dependencies configured as "postinst configure".
Some testing has suggested that's the case (e.g. invocations were
ordered with respect to dependencies), but if that's not generally true,
then this approach may not be suitable. This arrangement has the
advantage of letting the packaging infrastructure handle the dependency ordering as compared to the current implementation where we try to parse
dpkg deps output, tsort any add-ons found, etc.

# Approach:

Have an "installed" state file for each emacs flavor that's touched
whenever that flavor is changed and only exists if the last postinst configure/triggered for that flavor was successful, i.e. it indicates
whether add-ons should consider that flavor when building.

Have a "configured" state file for each add-on that exists whenever the
package has been successfully (postinst) configured. It may or may not
be installed for every flavor yet. It may also not exist if the add-on
is out of date with respect to its dependencies.

Have a per-flavor "installed" state file for for each add-on that's
touched whenever that add-on is successfully installed for the given
flavor. i.e. if it doesn't exist or is stale relative to the flavor's corresponding state file, then we need to reinstall for that flavor.

Whenever a flavor changes, trigger emacsen-common-add-on-assess-rebuild
All add-ons will be sensitive to that trigger, and will use the
installed state files to reinstall for the relevant flavors.

Whenever an add-on changes, remove the configured files for all reverse dependency add-ons, and trigger emacsen-common-add-on-assess-rebuild.

Each add on and flavor package must include the relevant (empty)
emacsen-common state directories.

# Background:

https://people.debian.org/~srivasta/MaintainerScripts.html#sec-6 https://www.debian.org/doc/debian-policy/ch-maintainerscripts.html#summary-of-ways-maintainer-scripts-are-called

Thanks
--
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Rob Browning writes ("Re: dpkg trigger questions (and/or broader emacs add-on options)"):

As an example related to the failures that precipitated all this, the
current emacs policy requires all add-ons to depend on emacsen-common,
and for them to call its commands at various points to handle installs/rebuilds and removals. i.e. emacsen-common is the
library/subsystem that handles all the dependency ordering, etc.

But that's just wrong if you can't assume much about the state (or even availability) of emacsen-common from those maintainer scripts, i.e. when
we designed all this, I would never have guessed that even if you depend
on emacsen-common, it might not actually be ready (or even unpacked)
from say your prerm.

According to
https://www.debian.org/doc/debian-policy/ch-maintainerscripts.html
the dependencies will have previously been configured and not
removed. I'm not sure what state you found they were in but I would
bet against there being a dpkg bug in this area.

It's also perhaps worth noting that we don't actually care about
triggers per se. I'd just started looking at them when among other
things, I wondered if they might be able to eliminate the need for our
own (currently a bit sketchy) dependency ordering.

I think triggers are probably a good thing to use here because of
their ability to defer and coalesce processing.

Realistically, E must be interested in A, so that A gets rebuilt. (We don't want to do this the other way around, because that would cause
an emacs to be considered not-`installed` simply because some addon
hadn't been compiled.)

Without having thought about it carefully -- that might be OK...
i.e. it might be fine if the whole system (add-ons and flavors) comes
out of "up to date" for a bit, as long as it re-converges eventually.

That's the basic principle behind the scheme, yes.

If we update E, E's postinst will need to rebuild both A and B. So
E's postinst needs to be able to do a topological sort of the addons,
so that it can compile them in the right order.

Given that E's postinst can do the sort, it can reliably recompile everything.

But from E's postinst, we have no idea what state the add-ons are in?

Can't you look in the filesystem? I would do this by having E's
postinst look in /usr for what things to compile. It doesn't need to
know the dpkg status.

And if so, and we had to be "completely conservative", the most we could
do is rebuild any dependency graph sub-trees we could find where the
add-ons were past their "postinst configure"s ... and then we'd have to ensure we get to the rest later somehow.

I don't think this is the right approach. Instead, recompile
everything that is physically present. This ought to work right for
anything that's not "half-installed", and in that case it can (and
should be) fixed by reinstalling the half-installed package's .deb.
In the meantime E's trigger run can fail. (But "half-installed" is
very rare.)

One correct solution might involve some command we know will always "run last" in any given apt/dpkg run, and I'd wondered if triggers might be
able to support that.

Sort of. Of course not every package can provide a hook that runs
last. You actually mean something more complicated.

The trigger system arranges that the hook you provide will run *again*
if it didn't run last. Ie, it must tolerate being run not-last, but
if anything *else* happens to trigger it after that, it will run
again.

I think this is the only reasonably plausible semantics (and it's
sufficient for your purposes).

I think I have a better handle on what dpkg promises with respect to dependency states from maintscript invocations, i.e. generally, not
much, in the limiting case, outside "postinst configure" (and now maybe "postinst triggered"). There it's likely, and as likely as it'll ever
be, that your deps will be configured -- elsewhere, not so much.

The rules are set out in some detail in the manual. (See link above.)

I'd also like to understand what we might be able to count on with
respect to triggers. For example, is it plausible to put them in
roughly the same class as "postinst configure" with respect to the dependencies will be available/configured there?

Yes, trigger processing is indeed "roughly in the same class" as
postinst configure from the pov of dependencies etc.

# Overview:

The general approach is to maintain relevant state files in /var/lib/emacsen-common/..., to manipulate those to reflect changes, and
to then trigger all the add-ons to re-evaluate the state to accommodate changes (in some cases, an add-on will decide there's nothing to do).

You want to be very careful with any separate state you maintain.
That can make things significantly more complicated. By recording
state separately, you introduce the possibility of bugs where the separately-recorded information is wrong (ie, doesn't reflect the real
state of the system).

I think you can do a simple approach, which recompiles all emacs
addons after any dpkg run, without needing any additional on-disk
state.

Ideally you'd think of any state you add as an optimisation for this
scheme. Ie, it's a cache invalidation problem. But, you might be
able to use filesystem timestamps instead.

Whatever you do, you should probably start by writing down the
invariants you are attempting to maintain, rather than diving straight
into the algorithm sketch.

# Approach:

Have an "installed" state file for each emacs flavor that's touched
whenever that flavor is changed and only exists if the last postinst configure/triggered for that flavor was successful, i.e. it indicates
whether add-ons should consider that flavor when building.

Why not do the compilation for each flavour in that flavour's trigger processing? That way you don't need to worry about the installation
state of the flavour, because you know it from context.

(This doesn't prevent you from putting the code for the compilation
framework into a common package; the flavour's postinst can call it.)

Whenever a flavor changes, trigger emacsen-common-add-on-assess-rebuild
All add-ons will be sensitive to that trigger, and will use the
installed state files to reinstall for the relevant flavors.

The trigger system works better when central packages are interested
and leaf packages do the triggering. This proposal inverts that.

I don't have time now to think through the details of this approach
but I would recommend doing it the other way.

HTH.

Ian.

--
Ian Jackson <ijackson@chiark.greenend.org.uk> These opinions are my own.

Pronouns: they/he. If I emailed you from @fyvzl.net or @evade.org.uk,
that is a private address which bypasses my fierce spamfilter.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Ian Jackson <ijackson@chiark.greenend.org.uk> writes:

According to https://www.debian.org/doc/debian-policy/ch-maintainerscripts.html the dependencies will have previously been configured and not removed.
I'm not sure what state you found they were in but I would bet against
there being a dpkg bug in this area.

I'd have to try to recall the exact details, but it didn't sound like
there was any bug. The current emacs policy had just incorrectly
assumed (among a number of related incorrect assumptions) that if an
add-on depended on emacsen-common, then emacsen-common would normally be
fully configured during the add-on's prerm, and that's just not true, as
we discovered when I read that page fairly carefully a while back (after
the related bugs were reported).

Can't you look in the filesystem? I would do this by having E's
postinst look in /usr for what things to compile. It doesn't need to
know the dpkg status.

In the the framework of the original policy it did, because we
couldn't/didn't assume what an add-on might need to do to become "ready"
in its postinst configure, i.e. the policy requires that an add-on has
to be past its successful postinst configure to be "ready" for anything
else to depend on, including compilations.

If we're willing/able to commit to the restriction that just having
"add-on files in place" is always sufficient, then that may well
simplify things. I'm not yet certain that's true (that no add-ons need
to do "more" in their configure before another add-on can "use" them),
but it may be.

I don't think this is the right approach. Instead, recompile
everything that is physically present. This ought to work right for
anything that's not "half-installed", and in that case it can (and
should be) fixed by reinstalling the half-installed package's .deb.
In the meantime E's trigger run can fail. (But "half-installed" is
very rare.)

For this to work, I suppose we'll have to be sure that (as mentioned
above) "files have all been unpacked" is sufficient for all add-ons.

The trigger system arranges that the hook you provide will run *again*
if it didn't run last. Ie, it must tolerate being run not-last, but
if anything *else* happens to trigger it after that, it will run
again.

I think this is the only reasonably plausible semantics (and it's
sufficient for your purposes).

Right, I think that was one of the expectations in the proposal at the
end of the previous message.

You want to be very careful with any separate state you maintain.
That can make things significantly more complicated. By recording
state separately, you introduce the possibility of bugs where the separately-recorded information is wrong (ie, doesn't reflect the real
state of the system).

Of course. We'd only want that complexity if we had to have it (e.g. if "merely being unpacked" isn't sufficient).

In any case, I think you may have addressed the key question -- and so
we may actually be able to use triggers, and then perhaps whether or not
we neeed custom state can be a separable question, to be determined by
whether "just being unpacked" is enough.

(This doesn't prevent you from putting the code for the compilation
framework into a common package; the flavour's postinst can call it.)

Sure, again, as long as the common package doesn't need to do anything
in its postinst configure in order to be "ready".

Otherwise, that's what we've been doing, and is the precipitant of this
whole reevaluation. Installations were failing because add-ons that
depended on emacsen-common were crashing when it wasn't ready
(configured) when they tried to use it in their maintscripts (outside
postinst configure).
e.g. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1040690#115

The trigger system works better when central packages are interested
and leaf packages do the triggering. This proposal inverts that.

OK, I'll rethink the arrangements with that in mind.

And just to double-check. It sounds like "postinst trigger" invocations
can expect their deps to be configured. Once triggered, is there any
ordering with respect to the "postinst trigger" invocations for multiple interested/awaiting packages, or is that "undefined"?

Thanks much
--
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)