• [gentoo-user] Emerge -u -k package install order - broken system

    From Alexander Puchmayr@21:1/5 to All on Mon Sep 6 16:40:02 2021
    Hi there,

    I just tried to upgrade a older installation via binary packages and this
    broke my system. After around 25 packages of almost 300 it stopped with error and failing packages.

    $ emerge
    Failed to validate a sane '/dev'.
    bash process substitution doesn't work; this may be an indication of a broken '/dev/fd'.
    $ ls -l /dev/fd/
    insgesamt 0
    lrwx------ 1 root root 64 6. Sep 14:18 0 -> /dev/pts/0
    lrwx------ 1 root root 64 6. Sep 14:18 1 -> /dev/pts/0
    lrwx------ 1 root root 64 6. Sep 14:18 2 -> /dev/pts/0
    lr-x------ 1 root root 64 6. Sep 14:18 3 -> /proc/27261/fd

    looks allright, but:

    $ bash
    bash: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by /lib64/ libreadline.so.8)

    system broken(!), cannot start any shell anymore, cannot install anything
    anymore and it's obvious that the system is bricked after reboot or even when the ssh session I'm logged in is closed.

    It seems like as if sys-libs/readline-8.1_p1-r1-1:0/8::gentoo is installed *before* installing a suitable glibc, breaking any binary that has the useflag readline (including bash).

    Two questions:
    How do I get out of this mess?
    Why does portage not work in correct package order? Portage bug?

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jack@21:1/5 to Alexander Puchmayr on Mon Sep 6 19:50:01 2021
    On 2021.09.06 10:33, Alexander Puchmayr wrote:
    Hi there,

    I just tried to upgrade a older installation via binary packages and
    this
    broke my system. After around 25 packages of almost 300 it stopped
    with error
    and failing packages.

    $ emerge
    Failed to validate a sane '/dev'.
    bash process substitution doesn't work; this may be an indication of
    a broken
    '/dev/fd'.
    $ ls -l /dev/fd/
    insgesamt 0
    lrwx------ 1 root root 64 6. Sep 14:18 0 -> /dev/pts/0
    lrwx------ 1 root root 64 6. Sep 14:18 1 -> /dev/pts/0
    lrwx------ 1 root root 64 6. Sep 14:18 2 -> /dev/pts/0
    lr-x------ 1 root root 64 6. Sep 14:18 3 -> /proc/27261/fd

    looks allright, but:

    $ bash
    bash: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by
    /lib64/
    libreadline.so.8)

    system broken(!), cannot start any shell anymore, cannot install
    anything
    anymore and it's obvious that the system is bricked after reboot or
    even when
    the ssh session I'm logged in is closed.

    It seems like as if sys-libs/readline-8.1_p1-r1-1:0/8::gentoo is
    installed
    *before* installing a suitable glibc, breaking any binary that has
    the useflag
    readline (including bash).

    Two questions:
    How do I get out of this mess?
    Why does portage not work in correct package order? Portage bug?
    It might help if you stated which version of packages you currently
    have installed - specifically glibc.

    As for recovery, you most likely need to boot to a live image (CD or
    USB) then chroot into the existing system. I'm only guessing as to
    what is the minimal list of files you will need to replace, but I'd see
    if you can find or create a binary package of the latest glibc and then install or unpack that to your system.

    As to whether this might be a bug in portage, I'd say you need to
    provide more details about exactly what you did. What emerge line,
    what you mean by installation via binary packages, and where you got
    those binary packages.

    If you do still have a running shell, does emerge still run at all?

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Alexander Puchmayr@21:1/5 to All on Tue Sep 7 09:10:01 2021
    Am Montag, 6. September 2021, 19:38:38 CEST schrieb Jack:
    On 2021.09.06 10:33, Alexander Puchmayr wrote:
    Hi there,

    I just tried to upgrade a older installation via binary packages and
    this
    broke my system. After around 25 packages of almost 300 it stopped
    with error
    and failing packages.

    $ emerge
    Failed to validate a sane '/dev'.
    bash process substitution doesn't work; this may be an indication of
    a broken
    '/dev/fd'.
    $ ls -l /dev/fd/
    insgesamt 0
    lrwx------ 1 root root 64 6. Sep 14:18 0 -> /dev/pts/0
    lrwx------ 1 root root 64 6. Sep 14:18 1 -> /dev/pts/0
    lrwx------ 1 root root 64 6. Sep 14:18 2 -> /dev/pts/0
    lr-x------ 1 root root 64 6. Sep 14:18 3 -> /proc/27261/fd

    looks allright, but:

    $ bash
    bash: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by
    /lib64/
    libreadline.so.8)

    system broken(!), cannot start any shell anymore, cannot install
    anything
    anymore and it's obvious that the system is bricked after reboot or
    even when
    the ssh session I'm logged in is closed.

    It seems like as if sys-libs/readline-8.1_p1-r1-1:0/8::gentoo is
    installed
    *before* installing a suitable glibc, breaking any binary that has
    the useflag
    readline (including bash).

    Two questions:
    How do I get out of this mess?
    Why does portage not work in correct package order? Portage bug?

    It might help if you stated which version of packages you currently
    have installed - specifically glibc.

    As for recovery, you most likely need to boot to a live image (CD or
    USB) then chroot into the existing system. I'm only guessing as to
    what is the minimal list of files you will need to replace, but I'd see
    if you can find or create a binary package of the latest glibc and then install or unpack that to your system.

    As to whether this might be a bug in portage, I'd say you need to
    provide more details about exactly what you did. What emerge line,
    what you mean by installation via binary packages, and where you got
    those binary packages.

    If you do still have a running shell, does emerge still run at all?

    What I did is:
    * On my build host, I compiled the full system and creating packages
    * On the client (i.e. The machine in question) I simply did a emerge -auvDNk world.
    * Portage did not complain about any conflicts, the amount of packages is what I expected, so I started the upgrade.
    * After about 25 packages installed successfully, emerge stopped with the errors explained before.

    For me the problem is solved now as I could recover the system, but I still would like to know how to prevent this -- installing glibc prior world may be
    a good idea.

    Thanks,
    Alex

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Alexander Puchmayr@21:1/5 to All on Tue Sep 7 09:40:02 2021
    Am Montag, 6. September 2021, 16:33:32 CEST schrieb Alexander Puchmayr:
    Hi there,

    I just tried to upgrade a older installation via binary packages and this broke my system. After around 25 packages of almost 300 it stopped with
    error and failing packages.

    $ emerge
    Failed to validate a sane '/dev'.
    bash process substitution doesn't work; this may be an indication of a
    broken '/dev/fd'.
    $ ls -l /dev/fd/
    insgesamt 0
    lrwx------ 1 root root 64 6. Sep 14:18 0 -> /dev/pts/0
    lrwx------ 1 root root 64 6. Sep 14:18 1 -> /dev/pts/0
    lrwx------ 1 root root 64 6. Sep 14:18 2 -> /dev/pts/0
    lr-x------ 1 root root 64 6. Sep 14:18 3 -> /proc/27261/fd

    looks allright, but:

    $ bash
    bash: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by /lib64/ libreadline.so.8)

    system broken(!), cannot start any shell anymore, cannot install
    anything anymore and it's obvious that the system is bricked after reboot
    or even when the ssh session I'm logged in is closed.

    It seems like as if sys-libs/readline-8.1_p1-r1-1:0/8::gentoo is installed *before* installing a suitable glibc, breaking any binary that has the useflag readline (including bash).

    Two questions:
    How do I get out of this mess?
    Why does portage not work in correct package order? Portage bug?

    The problem was caused by a newly compiled sys-libs/readline as binary
    package. The package itself was compiled on my buildhost by a emerge -auvDN world, and when readline was compiled, the latest glibc-2.33 was already installed on the buildhost, hence the readline packet had a (implicit) dependency to >=glibc-33. Portage does not seem to be aware of that (the glibc version is not a dependency of readline), so portage did not care and install it at some suitable position later. From the time on that readline was emerged from that binary package, all programs linked with it have now that implicit dependency of >=glibc-2.33, which is not yet installed (which was 2.28). Bang!

    HOW TO FIX THIS BREAKAGE:

    For the system recovery I was lucky to have a backup of the old packages, so I could extract the readline-tbz2-package file to the temporary directly and copy the so files to /usr/local/lib64 (I chose /usr/local/ to avoid overwriting the package files)

    Ls -l /usr/local/lib64/
    insgesamt 368
    lrwxrwxrwx 1 root root 15 6. Sep 14:53 libhistory.so -> libhistory.so.8 lrwxrwxrwx 1 root root 17 6. Sep 15:04 libhistory.so.8 -> libhistory.so. 8.0
    -rwxr-xr-x 1 root root 47368 6. Sep 14:53 libhistory.so.8.0
    lrwxrwxrwx 1 root root 16 6. Sep 14:53 libreadline.so -> libreadline.so.8 lrwxrwxrwx 1 root root 18 6. Sep 15:04 libreadline.so.8 -> libreadline.so.8.0
    -rwxr-xr-x 1 root root 325464 6. Sep 14:53 libreadline.so.8.0

    Then, I created a ld.so.conf config file to ensure that /usr/local/lib is at *first* place:

    echo "/usr/local/lib64" >/etc/ld.so.conf.d/10-temporary-fix.conf
    ldconfig

    Notes:
    * The config file *must* end with .conf
    * Although /usr/local/lib64 is already in the ld.so.conf file, this step is necessary to get it at first place; otherwise it is after /lib64 and /usr/ lib64, and ld.so finds the non-working version first
    * setting LD_LIBRARY_PATH does not work as ebuild does not forward the environment variables from the calling process.

    Test it with ldd /usr/bin/bash -- no errors shall be reported, and the
    readline library shall show the /usr/local/lib64 path.

    Now, I could emerge glibc-2.33 the usual way (emerge -avk1 glibc), and then afterwards remove the temporary ld.so.conf file

    rm /etc/ld.so.conf.d/10-temporary-fix.conf
    ldconfig

    Test again with ldd /usr/bin/bash -- the new readline library from /usr/lib64 and /lib64 shall now be used.
    Now, the libraries from /usr/local/lib can be removed safely.

    System recovered!

    Best regards,
    Alex

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)