Dear all,
there is a boot regression in effect in Linux v6.4-rc3 that affects at
least:
* rx2620 (w/2 x Montecito and zx1)
* rx2800-i2 (w/1 x Tukwila)
...(see second part of [1] and following posts for more details, [2] and
[3] for the respective logs), example here:
```
ELILO v3.16 for EFI/IA-64
..
Uncompressing Linux... done
Loading file AC100221.initrd.img...done
[ 0.000000] Linux version 6.4.0-rc3 (root@x4270) (ia64-linux-gcc
(GCC) 12.2.0, GNU ld (GNU Binutils) 2.39) #1 SMP Thu May 25 15:52:20
CEST 2023
[ 0.000000] efi: EFI v1.1 by HP
[ 0.000000] efi: SALsystab=0x3ee7a000 ACPI 2.0=0x3fe2a000
ESI=0x3ee7b000 SMBIOS=0x3ee7c000 HCDP=0x3fe28000
[ 0.000000] PCDP: v3 at 0x3fe28000
[ 0.000000] earlycon: uart8250 at MMIO 0x00000000f4050000 (options '9600n8')
[ 0.000000] printk: bootconsole [uart8250] enabled
[ 0.000000] ACPI: Early table checksum verification disabled
[ 0.000000] ACPI: RSDP 0x000000003FE2A000 000028 (v02 HP )
[ 0.000000] ACPI: XSDT 0x000000003FE2A02C 0000CC (v01 HP rx2620 00000000 HP 00000000)
[...]
[ 3.793350] Run /init as init process
Loading, please wait...
Starting systemd-udevd version 252.6-1
[ 3.951100] ------------[ cut here ]------------
[ 3.951100] WARNING: CPU: 6 PID: 140 at kernel/module/main.c:1547 __layout_sections+0x370/0x3c0
[ 3.949512] Unable to handle kernel paging request at virtual address 1000000000000000
[ 3.951100] Modules linked in:
[ 3.951100] CPU: 6 PID: 140 Comm: (udev-worker) Not tainted 6.4.0-rc3 #1
[ 3.956161] (udev-worker)[142]: Oops 11003706212352 [1]
[ 3.951774] Hardware name: hp server rx2620 , BIOS
04.29
11/30/2007
[ 3.951774]
[ 3.951774] Call Trace:
[ 3.958339] Unable to handle kernel paging request at virtual address 1000000000000000
[ 3.956161] Modules linked in:
[ 3.951774] [<a0000001000156d0>] show_stack.part.0+0x30/0x60
[ 3.951774] sp=e000000183a67b20 bsp=e000000183a61628
[ 3.956161]
[ 3.956161]
```
[1]: https://lists.debian.org/debian-ia64/2023/05/msg00010.html
[2]: https://pastebin.com/SAUKbG7Z
[3]: https://pastebin.com/v1TTB2x3
Hi Frank,
Thanks for the report.
It seems the error happened during the WARN_ON_ONCE. Could you
please try whether something like the following fixes it?
diff --git i/kernel/module/main.c w/kernel/module/main.c
index 0f9183f1ca9f..ae42dfc1a815 100644
--- i/kernel/module/main.c
+++ w/kernel/module/main.c
@@ -1537,7 +1537,7 @@ static void __layout_sections(struct module
*mod, struct load_info *info, bool i
|| is_init != module_init_layout_section(sname))
continue;
- if (WARN_ON_ONCE(type == MOD_INVALID))
+ if (type == MOD_INVALID)
continue;
s->sh_entsize =
module_get_offset_and_type(mod, type, s, i);
If that doesn't work, maybe we need something like this:
diff --git i/arch/ia64/kernel/module.c w/arch/ia64/kernel/module.c
index 3661135da9d9..4e9a7f0498e2 100644
--- i/arch/ia64/kernel/module.c
+++ w/arch/ia64/kernel/module.c
@@ -815,7 +815,7 @@ apply_relocate_add (Elf64_Shdr *sechdrs, const
char *strtab, unsigned int symind
uint64_t gp;
struct module_memory *mod_mem;
- mod_mem = &mod->mem[MOD_DATA];
+ mod_mem = &mod->mem[MOD_TEXT];
if (mod_mem->size > MAX_LTOFF)
/*
* This takes advantage of fact that
SHF_ARCH_SMALL gets allocated
Hi Song,
On 26.05.23 18:49, Song Liu wrote:
Hi Frank,
Thanks for the report.
Sure, thanks for your help in this.
It seems the error happened during the WARN_ON_ONCE. Could you
please try whether something like the following fixes it?
diff --git i/kernel/module/main.c w/kernel/module/main.c
index 0f9183f1ca9f..ae42dfc1a815 100644
--- i/kernel/module/main.c
+++ w/kernel/module/main.c
@@ -1537,7 +1537,7 @@ static void __layout_sections(struct module
*mod, struct load_info *info, bool i
|| is_init != module_init_layout_section(sname))
continue;
- if (WARN_ON_ONCE(type == MOD_INVALID))
+ if (type == MOD_INVALID)
continue;
s->sh_entsize =
module_get_offset_and_type(mod, type, s, i);
Ok, tried that as -patch1 on top of v6.4-rc3, but didn't help, see [1].
[1]: https://pastebin.com/UK9v30Ae
If that doesn't work, maybe we need something like this:
diff --git i/arch/ia64/kernel/module.c w/arch/ia64/kernel/module.c
index 3661135da9d9..4e9a7f0498e2 100644
--- i/arch/ia64/kernel/module.c
+++ w/arch/ia64/kernel/module.c
@@ -815,7 +815,7 @@ apply_relocate_add (Elf64_Shdr *sechdrs, const
char *strtab, unsigned int symind
uint64_t gp;
struct module_memory *mod_mem;
- mod_mem = &mod->mem[MOD_DATA];
+ mod_mem = &mod->mem[MOD_TEXT];
if (mod_mem->size > MAX_LTOFF)
/*
* This takes advantage of fact that SHF_ARCH_SMALL gets allocated
Tried that one as -patch2 on top of v6.4-rc3, but didn't help, see [2].
[2]: https://pastebin.com/gLupBndU
I also tried both patches as -patch1plus2 on top of v6.4-rc3 with a
similar result, see [3].
[3]: https://pastebin.com/7pXBG5vx
Not saying that debugging commit ac3b4328392344 ("module: replace module_layout with module_memory") is going to be impossible, quite
the contrary I think it would be good to root cause it, if possible,
as perhaps it may also be similar to some other future oddball arch
bug later that may come up.
On Fri, May 26, 2023 at 2:59 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
Not saying that debugging commit ac3b4328392344 ("module: replace module_layout with module_memory") is going to be impossible, quite
the contrary I think it would be good to root cause it, if possible,
as perhaps it may also be similar to some other future oddball arch
bug later that may come up.
I don't have any context - the mailing lists in question that
apparently this came in on aren't in lore.
That said, that commit looks odd for the ia64 part.
In particular, this part:
- if (mod->core_layout.size > MAX_LTOFF)
+ struct module_memory *mod_mem;
+
+ mod_mem = &mod->mem[MOD_DATA];
in apply_relocate_add() (file: arch/ia64/kernel/module.c) seems suspect.
The previous place that used to look at "mod->core_layout.base"
converted that to "mod->mem[MOD_TEXT].base". As do other changes in
other architectures.
So that "MOD_DATA" looks *very* wrong. Shouldn't core_layout. be
translated to use "MOD_TEXT" instead?
Dear all,
there is a boot regression in effect in Linux v6.4-rc3 that affects at
least:
* rx2620 (w/2 x Montecito and zx1)
* rx2800-i2 (w/1 x Tukwila)
Thanks for running the test.
I am not very familiar with the code, but I think we shouldn't hit that WARN_ON_ONCE. Could you please try with the follow patch to see
which section caused this issue?
Thanks,
Song
diff --git i/kernel/module/main.c w/kernel/module/main.c
index 0f9183f1ca9f..caf3d30cd133 100644
--- i/kernel/module/main.c
+++ w/kernel/module/main.c
@@ -1537,8 +1537,11 @@ static void __layout_sections(struct module
*mod, struct load_info *info, bool i
|| is_init != module_init_layout_section(sname))
continue;
- if (WARN_ON_ONCE(type == MOD_INVALID))
+ if (WARN_ON_ONCE(type == MOD_INVALID)) {
+ pr_warn("%s: section %s (sh_flags
%llx) matched to MOD_INVALID\n", __func__,
+ sname, s->sh_flags);
continue;
+ }
s->sh_entsize =
module_get_offset_and_type(mod, type, s, i);
pr_debug("\t%s\n", sname);
[...]
But this is my "monkey see, monkey do" pattern matching reaction, not
from any deeper understanding of the problem (I can't even see the
report) or really even the code.
If it is of any help, my initial report is available for example via:
https://marc.info/?l=linux-ia64&m=168509859125505&w=2
...the whole thread is currently at:
https://marc.info/?t=168509868200003&r=1&w=2
Anyway, the WARN_ON() is likely related, but the bug is clearly an
unexpected page fault in __copy_user() when called by load_module().
The ia64 oops output is nasty, presumably because ia64 aggressively
inlines things. It would help a lot if you enabled debug info (maybe
you already do?)
and then run the oops through
./scripts/decode_stacktrace.sh which will figure out line numbers,
inlining etc.
Because I don't even see why it would call __copy_user() in the first
place. This is 'finit_module()' that loads the module data from a
file, not user space.
So I guess it must be the strndup_user() in
mod->args = strndup_user(uargs, ~0UL >> 1);
but that doesn't look like it should even care about any module
layout. Plus I would have expected to see strndup_user() in the call
trace, but whatever.
End result: that ia64 trace is very hard to read, and _maybe_ running
it through the decode script might give more information about what it
is that triggers...
Ok, I put the decoded console messages on [2].
[2]: https://pastebin.com/dLYMijfS
On Sat, May 27, 2023 at 11:41 AM Frank Scheiner <frank.scheiner@web.de> wrote:
Ok, I put the decoded console messages on [2].
[2]: https://pastebin.com/dLYMijfS
Ugh. Apparently ia64 decoding isn't great. But at least it gives
multiple line numbers:
load_module (kernel/module/main.c:2291 kernel/module/main.c:2412 kernel/module/main.c:2868)
except your kernel obviously has those test-patches, so I still don't
know exactly where they are.
But it looks like it is in move_module(). Strange. I don't know how it
gets to "__copy_user" from there...
[ Looks at the ia64 code ]
Oh.
It turns out that it *says* __copy_user(), but the code is actually
shared with the regular memcpy() function, which does
GLOBAL_ENTRY(memcpy)
and r28=0x7,in0
and r29=0x7,in1
mov f6=f0
mov retval=in0
br.cond.sptk .common_code
;;
where that ".common_code" label is - surprise surprise - the common
copy code, and so when the oops reports that the problem happened in __copy_user(), it actually is in this case just a normal memcpy.
Ok, so it's probably the
memcpy(dest, (void *)shdr->sh_addr, shdr->sh_size);
in move_module() that takes a fault. And looking at the registers,
the destination is in r17/r18, and your dump has
unable to handle kernel paging request at virtual address 1000000000000000
...
r17 : 0fffffffffffffff r18 : 1000000000000000
so it's almost certainly that 'dest' that is bad.
From the log, the following sections are assigned to MOD_INVALID:
Frank, could you please give it a try?
Thanks,
Song
diff --git i/kernel/module/main.c w/kernel/module/main.c
index 0f9183f1ca9f..e4e723e1eb21 100644
--- i/kernel/module/main.c
+++ w/kernel/module/main.c
@@ -1514,14 +1514,14 @@ static void __layout_sections(struct module
*mod, struct load_info *info, bool i
MOD_RODATA,
MOD_RO_AFTER_INIT,
MOD_DATA,
- MOD_INVALID, /* This is needed to match the masks array */
+ MOD_DATA,
};
static const int init_m_to_mem_type[] = {
MOD_INIT_TEXT,
MOD_INIT_RODATA,
MOD_INVALID,
MOD_INIT_DATA,
- MOD_INVALID, /* This is needed to match the masks array */
+ MOD_INIT_DATA,
};
for (m = 0; m < ARRAY_SIZE(masks); ++m) {
Thanks, that patch (as -patch4 on top of v6.4-rc3) fixes the boot
regression for me on the rx2620:
Hi again,
On 28.05.23 09:30, Frank Scheiner wrote:
[...]
Thanks, that patch (as -patch4 on top of v6.4-rc3) fixes the boot regression for me on the rx2620:
[...]
Great! I'll give it a try on my rx2800-i2, too, but assume it wil work there, too.
Indeed, -patch4 also makes it work on the rx2800-i2:
```
ELILO v3.16 for EFI/IA-64
..
Uncompressing Linux... done
Loading file AC10027B.initrd.img...done
[ 0.000000] Linux version 6.4.0-rc3-44c026a73be8038f03dbdeef028b642880cf1511-patch4 (root@x4270) (ia64-linux-gcc (GCC) 12.2.0, GNU ld (GNU Binutils) 2.39) #1 SMP Sun May
28 09:08:44 CEST 2023
[ 0.000000] efi: EFI v2.1 by HP
[ 0.000000] efi: SALsystab=0xdfdd63a18 ESI=0xdfdd63f18 ACPI
2.0=0x3d3c4014 HCDP=0xdffff8798 SMBIOS=0x3d368000
[ 0.000000] PCDP: v3 at 0xdffff8798
[ 0.000000] earlycon: uart8250 at I/O port 0x4000 (options '115200n8')
[ 0.000000] printk: bootconsole [uart8250] enabled
[ 0.000000] ACPI: Early table checksum verification disabled
[ 0.000000] ACPI: RSDP 0x000000003D3C4014 000024 (v02 HP )
[ 0.000000] ACPI: XSDT 0x000000003D3C4580 000124 (v01 HP RX2800-2 00000001 01000013)
[...]
[ 36.649531] Run /init as init process
Loading, please wait...
Starting systemd-udevd version 252.6-1
[ 36.865635] pps_core: LinuxPPS API ver. 1 registered
[ 36.869321] pps_core: Software ver. 5.3.6 - Copyright 2005-2007
Rodolfo Giometti <giometti@linux.it>
[ 36.885025] PTP clock support registered
[ 36.910852] ACPI: bus type USB registered
[ 36.918198] usbcore: registered new interface driver usbfs
[ 36.924762] usbcore: registered new interface driver hub
[ 36.922133] igb: Intel(R) Gigabit Ethernet Network Driver
[ 36.931386] usbcore: registered new device driver usb
[ 36.938149] igb: Copyright (c) 2007-2014 Intel Corporation.
[...]
[ OK ] Finished apt-daily-upgrade… apt upgrade and clean activities.
Debian GNU/Linux 12 rx2800-i2 -
rx2800-i2 login:
```
I'll try to test this on other machines (rx4640 w/Madison, rx2660 w/Montecito/Montvale, rx6600 w/Montvale) as well, starting on Tuesday if possible.
****
There is an issue - only for the rx2800-i2 - seemingly related to it's
PCIe NIC(s) and MSIs, but this is already there in 6.3.x (see first part
of [1]) and **not related to the problem of this thread (AFAICT)** and -
more important - doesn't affect operation: The machine is working
diskless using one of those interfaces so it can't be that bad. I'll
look into bisecting that issue as well.
[1]: https://lists.debian.org/debian-ia64/2023/05/msg00010.html
Cheers,
Frank
This does make it clear just how great a mailing list archive lore is. Konstantin, is there any particular reason why
linux-ia64@vger.kernel.org isn't in lore? Is it just a rational hatred
of all things itanium?
We only add things to lore when someone asks, and nobody's asked. :) I guess I'll consider this an ask and put it on the radar.
On Tue, May 30, 2023 at 4:21 PM Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
We only add things to lore when someone asks, and nobody's asked. :) I guess
I'll consider this an ask and put it on the radar.
Thanks. It would probably be good to see if there are any other vger.kernel.org lists with any appreciable traffic that aren't on
lore.
[...]
Thanks for running the test!
I will send the official patch.
Thanks,
Song
Looking forward to the next occasion - for your sake maybe on another architecture, but can't promise... ;-)
Sysop: | Keyop |
---|---|
Location: | Huddersfield, West Yorkshire, UK |
Users: | 349 |
Nodes: | 16 (0 / 16) |
Uptime: | 144:53:37 |
Calls: | 7,614 |
Calls today: | 2 |
Files: | 12,790 |
Messages: | 5,684,750 |
Posted today: | 2 |