• Performance degraded, Bookworm on Raspberry Pi 4B

    From Hank Barta@21:1/5 to All on Mon Mar 14 22:40:01 2022
    First of all, many many thanks for making Debian work on a Raspberry Pi 4B.
    I bounce back and forth between Bullseye and Bookworm on a Pi 4B and have noticed a performance degradation with Bookworm.

    TL;DR
    Install (current) Bookworm - performance good.
    apt update/upgrade - performance tanks.
    Revert 5.16 to 5.15 kernel - no improvement.

    It seems that one of the 103 packages that get installed during the update causes this to happen.

    More detail at https://github.com/HankB/Pi-4B-bookworm-performance

    Is there anything I can do to help track this down? (I hope the answer is
    not "install the packages one at a time to see what causes this.")

    Thanks!
    --
    Beautiful Sunny Winfield

    <div dir="ltr"><div>First of all, many many thanks for making Debian work on a Raspberry Pi 4B. I bounce back and forth between  Bullseye and Bookworm on a Pi 4B and have noticed a performance degradation with Bookworm.</div><div><br></div><div>TL;DR</
    <div>Install (current) Bookworm - performance good.</div><div>apt update/upgrade - performance tanks.</div><div>Revert 5.16 to 5.15 kernel - no improvement.</div><div><br></div><div>It seems that one of the 103 packages that get installed during the
    update causes this to happen.</div><div><br></div><div>More detail at <a href="https://github.com/HankB/Pi-4B-bookworm-performance">https://github.com/HankB/Pi-4B-bookworm-performance</a></div><div><br></div><div>Is there anything I can do to help track
    this down? (I hope the answer is not &quot;install the packages one at a time to see what causes this.&quot;)</div><div><br></div><div>Thanks!<br></div><div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">
    Beautiful Sunny Winfield</div></div></div></div>

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Hank Barta@21:1/5 to hbarta@gmail.com on Tue Mar 15 03:40:01 2022
    I started installing packages I thought were likely to cause the regression (one or two at a time) and following installation of raspi-firmware the
    problem appeared.

    On Mon, Mar 14, 2022 at 4:35 PM Hank Barta <hbarta@gmail.com> wrote:

    First of all, many many thanks for making Debian work on a Raspberry Pi
    4B. I bounce back and forth between Bullseye and Bookworm on a Pi 4B and have noticed a performance degradation with Bookworm.

    TL;DR
    Install (current) Bookworm - performance good.
    apt update/upgrade - performance tanks.
    Revert 5.16 to 5.15 kernel - no improvement.

    It seems that one of the 103 packages that get installed during the update causes this to happen.

    More detail at https://github.com/HankB/Pi-4B-bookworm-performance

    Is there anything I can do to help track this down? (I hope the answer is
    not "install the packages one at a time to see what causes this.")

    Thanks!
    --
    Beautiful Sunny Winfield



    --
    Beautiful Sunny Winfield

    <div dir="ltr">I started installing packages I thought were likely to cause the regression (one or two at a time) and following installation of raspi-firmware the problem appeared. </div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On
    Mon, Mar 14, 2022 at 4:35 PM Hank Barta &lt;<a href="mailto:hbarta@gmail.com">hbarta@gmail.com</a>&gt; wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">
    <div>First of all, many many thanks for making Debian work on a Raspberry Pi 4B. I bounce back and forth between  Bullseye and Bookworm on a Pi 4B and have noticed a performance degradation with Bookworm.</div><div><br></div><div>TL;DR</div><div>Install
    (current) Bookworm - performance good.</div><div>apt update/upgrade - performance tanks.</div><div>Revert 5.16 to 5.15 kernel - no improvement.</div><div><br></div><div>It seems that one of the 103 packages that get installed during the update causes
    this to happen.</div><div><br></div><div>More detail at <a href="https://github.com/HankB/Pi-4B-bookworm-performance" target="_blank">https://github.com/HankB/Pi-4B-bookworm-performance</a></div><div><br></div><div>Is there anything I can do to help
    track this down? (I hope the answer is not &quot;install the packages one at a time to see what causes this.&quot;)</div><div><br></div><div>Thanks!<br></div><div>-- <br><div dir="ltr"><div dir="ltr">Beautiful Sunny Winfield</div></div></div></div>
    </blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr">Beautiful Sunny Winfield</div></div>

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Gunnar Wolf@21:1/5 to All on Tue Mar 15 05:30:01 2022
    Hank Barta dijo [Mon, Mar 14, 2022 at 09:31:36PM -0500]:
    I started installing packages I thought were likely to cause the regression (one or two at a time) and following installation of raspi-firmware the problem appeared.

    As the maintainer of raspi-firmware, I have to say:

    Ouch.


    raspi-firmware, as you know, is just a blob we cannot debug or do
    anything about; it is a core component of our Raspberries, but... it
    is non-free ☹

    Given you have already pursued the situation to this detail, could you
    share... between which versions do you see the regression?

    FWIW, I cannot do much about it, other than report it as an issue in https://github.com/raspberrypi/firmware/issues (which I will do) ;
    taking a quick look at the reported issues, and found just the
    following, although I don't know if they are in any way related to
    your report:

    https://github.com/raspberrypi/firmware/issues/1619

    Greetings, and thanks for this report!

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Diederik de Haas@21:1/5 to All on Tue Mar 15 12:13:10 2022
    To: gwolf@debian.org (Gunnar Wolf)
    Copy: hbarta@gmail.com (Hank Barta)

    On Tuesday, 15 March 2022 05:20:02 CET Gunnar Wolf wrote:
    Given you have already pursued the situation to this detail, could you share... between which versions do you see the regression?

    raspi-firmware/testing 1.20220120+ds-1 arm64 [upgradable from: 1.20210805+ds-1] -----BEGIN PGP SIGNATURE-----

    iHUEABYIAB0WIQT1sUPBYsyGmi4usy/XblvOeH7bbgUCYjB0xgAKCRDXblvOeH7b bi+3AQDy7aY1chOB84H0uHI5l/guLTEWlmg88HIQYuXfMdcSegEAp52zuKJrGPtP bh0bSaw9+xUngkakNOYvHgLiOYxXJA0=
    =vwBs
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Hank Barta@21:1/5 to All on Tue Mar 15 13:40:01 2022
    Some additional information that might be useful is that the system seems unable to determine CPU frequencies. For example

    root@up:~# cpupower frequency-info
    analyzing CPU 0:
    no or unknown cpufreq driver is active on this CPU
    CPUs which run at the same hardware frequency: Not Available
    CPUs which need to have their frequency coordinated by software: Not Available
    maximum transition latency: Cannot determine or is not supported.
    Not Available
    available cpufreq governors: Not Available
    Unable to determine current policy
    current CPU frequency: Unable to call hardware
    current CPU frequency: Unable to call to kernel
    root@up:~# ls /sys/devices/system/cpu/cpu0/
    cpu_capacity crash_notes_size node0/ power/
    subsystem/ uevent
    crash_notes hotplug/ of_node/ regs/
    topology/
    root@up:~# ls -l /sys/devices/system/cpu/cpu0/
    total 0
    -r--r--r-- 1 root root 4096 Mar 15 12:22 cpu_capacity
    -r-------- 1 root root 4096 Mar 15 12:22 crash_notes
    -r-------- 1 root root 4096 Mar 15 12:22 crash_notes_size
    drwxr-xr-x 2 root root 0 Mar 15 12:22 hotplug
    lrwxrwxrwx 1 root root 0 Mar 15 12:22 node0 -> ../../node/node0
    lrwxrwxrwx 1 root root 0 Mar 15 12:22 of_node -> ../../../../firmware/devicetree/base/cpus/cpu@0
    drwxr-xr-x 2 root root 0 Mar 15 12:22 power
    drwxr-xr-x 3 root root 0 Mar 15 12:22 regs
    lrwxrwxrwx 1 root root 0 Jan 1 1970 subsystem -> ../../../../bus/cpu drwxr-xr-x 2 root root 0 Mar 15 12:16 topology
    -rw-r--r-- 1 root root 4096 Jan 1 1970 uevent
    root@up:~#

    Whereas on a Pi 4B running Bullseye without this issue

    hbarta@kweli:~$ cpupower frequency-info
    analyzing CPU 0:
    driver: cpufreq-dt
    CPUs which run at the same hardware frequency: 0 1 2 3
    CPUs which need to have their frequency coordinated by software: 0 1 2 3
    maximum transition latency: Cannot determine or is not supported.
    hardware limits: 600 MHz - 2.00 GHz
    available frequency steps: 600 MHz, 700 MHz, 800 MHz, 900 MHz, 1000 MHz, 1.10 GHz, 1.20 GHz, 1.30 GHz, 1.40 GHz, 1.50 GHz, 1.60 GHz, 1.70 GHz, 1.80
    GHz, 1.90 GHz, 2.00 GHz
    available cpufreq governors: performance schedutil
    current policy: frequency should be within 600 MHz and 2.00 GHz.
    The governor "schedutil" may decide which speed to use
    within this range.
    current CPU frequency: Unable to call hardware
    current CPU frequency: 600 MHz (asserted by call to kernel)
    hbarta@kweli:~$ cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq 2000000
    hbarta@kweli:~$

    The second example shows a Pi that has been overclocked and in fact, I discovered this issue when I tried to overclock a Pi 4B running Bookworm. I also discovered that the system did not respond to the settings in /boot/firmware/config.txt that are used to overclock. The following
    settings seem to have no effect

    arm_freq=2000
    over_voltage=6
    gpu_freq=750

    best,

    On Tue, Mar 15, 2022 at 6:13 AM Diederik de Haas <didi.debian@cknow.org>
    wrote:

    On Tuesday, 15 March 2022 05:20:02 CET Gunnar Wolf wrote:
    Given you have already pursued the situation to this detail, could you share... between which versions do you see the regression?

    raspi-firmware/testing 1.20220120+ds-1 arm64 [upgradable from: 1.20210805+ds-1]



    --
    Beautiful Sunny Winfield

    <div dir="ltr">Some additional information that might be useful is that the system seems unable to determine CPU frequencies. For example<div><br></div><div><font face="monospace">root@up:~# cpupower frequency-info<br>analyzing CPU 0:<br>  no or
    unknown cpufreq driver is active on this CPU<br>  CPUs which run at the same hardware frequency: Not Available<br>  CPUs which need to have their frequency coordinated by software: Not Available<br>  maximum transition latency:  Cannot determine or
    is not supported.<br>Not Available<br>  available cpufreq governors: Not Available<br>  Unable to determine current policy<br>  current CPU frequency: Unable to call hardware<br>  current CPU frequency:  Unable to call to kernel<br>root@up:~# ls /
    sys/devices/system/cpu/cpu0/       <br>cpu_capacity      crash_notes_size  node0/            power/            subsystem/        uevent            <br>crash_notes       hotplug/          of_node/          regs/  
              topology/         <br>root@up:~# ls -l /sys/devices/system/cpu/cpu0/<br>total 0<br>-r--r--r-- 1 root root 4096 Mar 15 12:22 cpu_capacity<br>-r-------- 1 root root 4096 Mar 15 12:22 crash_notes<br>-r-------- 1 root root 4096 Mar 15 12:
    22 crash_notes_size<br>drwxr-xr-x 2 root root    0 Mar 15 12:22 hotplug<br>lrwxrwxrwx 1 root root    0 Mar 15 12:22 node0 -&gt; ../../node/node0<br>lrwxrwxrwx 1 root root    0 Mar 15 12:22 of_node -&gt; ../../../../firmware/devicetree/base/cpus/cpu@
    0<br>drwxr-xr-x 2 root root    0 Mar 15 12:22 power<br>drwxr-xr-x 3 root root    0 Mar 15 12:22 regs<br>lrwxrwxrwx 1 root root    0 Jan  1  1970 subsystem -&gt; ../../../../bus/cpu<br>drwxr-xr-x 2 root root    0 Mar 15 12:16 topology<br>-rw-r--
    r-- 1 root root 4096 Jan  1  1970 uevent<br>root@up:~#</font></div><div><br></div><div>Whereas on a Pi 4B running Bullseye without this issue</div><div><br></div><div><font face="monospace">hbarta@kweli:~$ cpupower frequency-info<br>analyzing CPU 0:<br>
      driver: cpufreq-dt<br>  CPUs which run at the same hardware frequency: 0 1 2 3<br>  CPUs which need to have their frequency coordinated by software: 0 1 2 3<br>  maximum transition latency:  Cannot determine or is not supported.<br>  hardware
    limits: 600 MHz - 2.00 GHz<br>  available frequency steps:  600 MHz, 700 MHz, 800 MHz, 900 MHz, 1000 MHz, 1.10 GHz, 1.20 GHz, 1.30 GHz, 1.40 GHz, 1.50 GHz, 1.60 GHz, 1.70 GHz, 1.80 GHz, 1.90 GHz, 2.00 GHz<br>  available cpufreq governors: performance
    schedutil<br>  current policy: frequency should be within 600 MHz and 2.00 GHz.<br>                  The governor &quot;schedutil&quot; may decide which speed to use<br>                  within this range.<br>  current CPU frequency:
    Unable to call hardware<br>  current CPU frequency: 600 MHz (asserted by call to kernel)<br>hbarta@kweli:~$ cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq<br>2000000<br>hbarta@kweli:~$</font> <br></div><div><br></div><div>The second example
    shows a Pi that has been overclocked and in fact, I discovered this issue when I tried to overclock a Pi 4B running Bookworm. I also discovered that the system did not respond to the settings in /boot/firmware/config.txt that are used to overclock. The
    following settings seem to have no effect</div><div><br></div><div><font face="monospace">arm_freq=2000<br>over_voltage=6<br>gpu_freq=750</font><br></div><div><br></div><div> best,<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_
    attr">On Tue, Mar 15, 2022 at 6:13 AM Diederik de Haas &lt;<a href="mailto:didi.debian@cknow.org">didi.debian@cknow.org</a>&gt; wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-
    left:1ex">On Tuesday, 15 March 2022 05:20:02 CET Gunnar Wolf wrote:<br>
    &gt; Given you have already pursued the situation to this detail, could you<br> &gt; share... between which versions do you see the regression?<br>

    raspi-firmware/testing 1.20220120+ds-1 arm64 [upgradable from: 1.20210805+ds-1]</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr">Beautiful Sunny Winfield</div></div>

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Diederik de Haas@21:1/5 to All on Tue Mar 15 15:01:59 2022
    Copy: gwolf@debian.org (Gunnar Wolf)
    Copy: hbarta@gmail.com (Hank Barta)

    On Tuesday, 15 March 2022 13:32:37 CET Hank Barta wrote:
    Some additional information that might be useful is that the system seems unable to determine CPU frequencies. For example

    root@up:~# cpupower frequency-info
    analyzing CPU 0:
    no or unknown cpufreq driver is active on this CPU
    CPUs which run at the same hardware frequency: Not Available
    CPUs which need to have their frequency coordinated by software: Not Available
    maximum transition latency: Cannot determine or is not supported.
    Not Available
    available cpufreq governors: Not Available
    Unable to determine current policy
    current CPU frequency: Unable to call hardware
    current CPU frequency: Unable to call to kernel

    I lack the knowledge to properly/fully understand the issue Gunnar linked to, but it looks applicable ... still (despite the last msg indicating the issue was fixed).
    I don't know exactly the role that clocks play, but they seem quite important. And when the value of a/multiple clock(s) return 0, it doesn't surprise me
    that it would cause other things to fail, like the ones you reported above, which in turn cause the CPU (f.e.) to be way slower then it can be.

    Personally I think it's more useful if Hank reported the issue himself to https://github.com/raspberrypi/firmware/ then let Gunnar do it on his behalf.

    Having a bug in Debian's BTS (against raspi-firmware) regarding this issue seems useful as well and with the 'forwarded' keyword it can be linked to the upstream issue.

    Cheers,
    Diederik
    -----BEGIN PGP SIGNATURE-----

    iHUEABYIAB0WIQT1sUPBYsyGmi4usy/XblvOeH7bbgUCYjCcVwAKCRDXblvOeH7b brNyAQCNHS90El4uFZ8tx6tO8Sp1FLWIO3Pdd+Vow5MitTobOwD6Anm7olH4Y1P0 OgFzzqz/c9+sY1UXZPExVsGVWVv87A8=
    =8SyV
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Gunnar Wolf@21:1/5 to All on Tue Mar 15 17:10:01 2022
    Diederik de Haas dijo [Tue, Mar 15, 2022 at 03:01:59PM +0100]:
    Personally I think it's more useful if Hank reported the issue himself to https://github.com/raspberrypi/firmware/ then let Gunnar do it on his behalf.

    I agree with you, Diederik. I think Hank has a better grasp of the
    problems he is reporting than me, and it would be better if he were
    able to send this information to the upstream authors; of course, it
    makes sense to link to this thread.

    Hank, are you OK with this suggestion?

    -----BEGIN PGP SIGNATURE-----

    iHUEABYIAB0WIQRgswk9lhCOXLlxQu/i9jtDU/RZiQUCYjC5eAAKCRDi9jtDU/RZ iaUEAP9Z8R7X+/Z32FhaTw3kHE31PVBfiQFUuIL5ghWCbvc1ygD9GIrsePTZubFu UuH6XNrUau21lMIkJdXIDrI2DKq1AQo=
    =gPRA
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Hank Barta@21:1/5 to All on Tue Mar 15 16:40:01 2022
    I tested this morning with an install on an SD card and the disk benchmarks were not significantly different. The updated `raspi-firmware` did seem to result in slightly slower performance (7188 events before and 6525 after)
    but I attribute this to a low CPU clock rather than a problem with the SD
    card driver. Details at https://github.com/HankB/Pi-4B-bookworm-performance#2022-03-15-testing-with-sd-card
    .

    I'll be happy to file the bug reports. https://github.com/raspberrypi/firmware/issues/1705

    Debian BTS next.

    On Tue, Mar 15, 2022 at 9:02 AM Diederik de Haas <didi.debian@cknow.org>
    wrote:

    On Tuesday, 15 March 2022 13:32:37 CET Hank Barta wrote:
    Some additional information that might be useful is that the system seems unable to determine CPU frequencies. For example

    root@up:~# cpupower frequency-info
    analyzing CPU 0:
    no or unknown cpufreq driver is active on this CPU
    CPUs which run at the same hardware frequency: Not Available
    CPUs which need to have their frequency coordinated by software: Not Available
    maximum transition latency: Cannot determine or is not supported.
    Not Available
    available cpufreq governors: Not Available
    Unable to determine current policy
    current CPU frequency: Unable to call hardware
    current CPU frequency: Unable to call to kernel

    I lack the knowledge to properly/fully understand the issue Gunnar linked
    to,
    but it looks applicable ... still (despite the last msg indicating the
    issue
    was fixed).
    I don't know exactly the role that clocks play, but they seem quite important.
    And when the value of a/multiple clock(s) return 0, it doesn't surprise me that it would cause other things to fail, like the ones you reported
    above,
    which in turn cause the CPU (f.e.) to be way slower then it can be.

    Personally I think it's more useful if Hank reported the issue himself to https://github.com/raspberrypi/firmware/ then let Gunnar do it on his
    behalf.

    Having a bug in Debian's BTS (against raspi-firmware) regarding this issue seems useful as well and with the 'forwarded' keyword it can be linked to
    the
    upstream issue.

    Cheers,
    Diederik



    --
    Beautiful Sunny Winfield

    <div dir="ltr"><div>I tested this morning with an install on an SD card and the disk benchmarks were not significantly different. The updated `raspi-firmware` did seem to result in slightly slower performance (7188 events before and 6525 after) but I
    attribute this to a low CPU clock rather than a problem with the SD card driver. Details at <a href="https://github.com/HankB/Pi-4B-bookworm-performance#2022-03-15-testing-with-sd-card">https://github.com/HankB/Pi-4B-bookworm-performance#2022-03-15-
    testing-with-sd-card</a>.</div><div><br></div><div>I&#39;ll be happy to file the  bug reports. <a href="https://github.com/raspberrypi/firmware/issues/1705">https://github.com/raspberrypi/firmware/issues/1705</a></div><div><br></div><div>Debian BTS next.
    <br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Mar 15, 2022 at 9:02 AM Diederik de Haas &lt;<a href="mailto:didi.debian@cknow.org">didi.debian@cknow.org</a>&gt; wrote:<br></div><blockquote class="gmail_quote" style=
    "margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Tuesday, 15 March 2022 13:32:37 CET Hank Barta wrote:<br>
    &gt; Some additional information that might be useful is that the system seems<br>
    &gt; unable to determine CPU frequencies. For example<br>
    &gt; <br>
    &gt; root@up:~# cpupower frequency-info<br>
    &gt; analyzing CPU 0:<br>
    &gt;   no or unknown cpufreq driver is active on this CPU<br>
    &gt;   CPUs which run at the same hardware frequency: Not Available<br> &gt;   CPUs which need to have their frequency coordinated by software: Not<br>
    &gt; Available<br>
    &gt;   maximum transition latency:  Cannot determine or is not supported.<br>
    &gt; Not Available<br>
    &gt;   available cpufreq governors: Not Available<br>
    &gt;   Unable to determine current policy<br>
    &gt;   current CPU frequency: Unable to call hardware<br>
    &gt;   current CPU frequency:  Unable to call to kernel<br>

    I lack the knowledge to properly/fully understand the issue Gunnar linked to, <br>
    but it looks applicable ... still (despite the last msg indicating the issue <br>
    was fixed).<br>
    I don&#39;t know exactly the role that clocks play, but they seem quite important. <br>
    And when the value of a/multiple clock(s) return 0, it doesn&#39;t surprise me <br>
    that it would cause other things to fail, like the ones you reported above, <br>
    which in turn cause the CPU (f.e.) to be way slower then it can be.<br>

    Personally I think it&#39;s more useful if Hank reported the issue himself to<br>
    <a href="https://github.com/raspberrypi/firmware/" rel="noreferrer" target="_blank">https://github.com/raspberrypi/firmware/</a> then let Gunnar do it on his behalf.<br>

    Having a bug in Debian&#39;s BTS (against raspi-firmware) regarding this issue <br>
    seems useful as well and with the &#39;forwarded&#39; keyword it can be linked to the <br>
    upstream issue.<br>

    Cheers,<br>
      Diederik</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr">Beautiful Sunny Winfield</div></div>

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)