Dear Debian user list members,
I am trying to run network related stuff (Samba, Zabbix) on a Raspberry
Pi 4B in a virtualized environment using Debian Bookworm and Xen. I am running into reproducible complete system crashes/reboots due to a Xen watchdog triggering under certain, seemingly strange conditions (the
number of VLANs involved seems to play a role, running tcpdump on
certain interfaces prevents this issue, ...). If you are interested in
the long version, you can find it here [1].
Some people on xen-devel pointed out to me two unhandled SMC calls in
the boot logs which could be the root of the problem. I am now trying to
find out where these calls come from to get closer to the root cause.
The suspected calls are the following ones:
(XEN) d0v0 Unhandled SMC/HVC: 0x84000050
(XEN) d0v0 Unhandled SMC/HVC: 0x8600ff01
These calls happen during the Dom0 boot process, so it's something from inside Linux and nothing Xen related, I've been told. The current
working hypothesis is that the calls are trying to find some module not emulated by Xen and are therefore failing, leading to Linux waiting for
the reply, and subsequently to the Xen watchdog triggering and rebooting.
From what I could find out in ARM documentation, the unhandled SMC
calls probably have the following purpose:
0x84000050 = TRNG_VERSION, returns the implemented TRNG (True Random
Number Generator) ABI version [2]
0x8600ff01 = Call UID Query for Vendor Specific Hypervisor Service,
Returns a unique identifier of the service provider [3]
The more likely cause is the second call to the address 0x8600ff01.
Now I simply have no idea how to find out where in the Linux boot
process these calls are made. I tried poking into the Linux sources a
bit, and I couldn't find an exact match for these call addresses, so I
assume these addresses are assembled from different parts. There are
some matches for "0x8600" and for "ff01", but I couldn't identify if
these matches are relevant.
I tried to find out if strace could help, but from what I understand,
this is related to commands coming from userspace, so I am not sure that strace helps during the boot process.
I'd appreciate it if somebody more knowledgeable would point me in the
right direction. If more information is needed, I can provide it.
Thanks,
Paul
[1] https://lists.xenproject.org/archives/html/xen-devel/2023-10/msg00796.html [2] https://developer.arm.com/documentation/den0098/latest/
[3] https://documentation-service.arm.com/static/628b755ce3c4322a76af56de?token=
Am 25.01.2024 um 22:28 schrieb Paul Leiber:
Paul
[1] https://lists.xenproject.org/archives/html/xen-devel/2023-10/msg00796.html [2] https://developer.arm.com/documentation/den0098/latest/
[3] https://documentation-service.arm.com/static/628b755ce3c4322a76af56de?token=
Hm, no reply so far. Is this maybe the wrong list? Should I post this rather somewhere else?
Paul
Am 25.01.2024 um 22:28 schrieb Paul Leiber:
Dear Debian user list members,
I am trying to run network related stuff (Samba, Zabbix) on a Raspberry
Pi 4B in a virtualized environment using Debian Bookworm and Xen. I am running into reproducible complete system crashes/reboots due to a Xen watchdog triggering under certain, seemingly strange conditions (the
number of VLANs involved seems to play a role, running tcpdump on
certain interfaces prevents this issue, ...). If you are interested in
the long version, you can find it here [1].
Some people on xen-devel pointed out to me two unhandled SMC calls in
the boot logs which could be the root of the problem. I am now trying to find out where these calls come from to get closer to the root cause.
The suspected calls are the following ones:
(XEN) d0v0 Unhandled SMC/HVC: 0x84000050
(XEN) d0v0 Unhandled SMC/HVC: 0x8600ff01
These calls happen during the Dom0 boot process, so it's something from inside Linux and nothing Xen related, I've been told. The current
working hypothesis is that the calls are trying to find some module not emulated by Xen and are therefore failing, leading to Linux waiting for the reply, and subsequently to the Xen watchdog triggering and rebooting.
From what I could find out in ARM documentation, the unhandled SMC
calls probably have the following purpose:
0x84000050 = TRNG_VERSION, returns the implemented TRNG (True Random Number Generator) ABI version [2]
0x8600ff01 = Call UID Query for Vendor Specific Hypervisor Service, Returns a unique identifier of the service provider [3]
The more likely cause is the second call to the address 0x8600ff01.
Now I simply have no idea how to find out where in the Linux boot
process these calls are made. I tried poking into the Linux sources a
bit, and I couldn't find an exact match for these call addresses, so I assume these addresses are assembled from different parts. There are
some matches for "0x8600" and for "ff01", but I couldn't identify if
these matches are relevant.
I tried to find out if strace could help, but from what I understand,
this is related to commands coming from userspace, so I am not sure that strace helps during the boot process.
I'd appreciate it if somebody more knowledgeable would point me in the right direction. If more information is needed, I can provide it.
On Wed, Jan 31, 2024 at 08:02:47AM +0100, Paul Leiber wrote:
Am 25.01.2024 um 22:28 schrieb Paul Leiber:
Paul
[1]
https://lists.xenproject.org/archives/html/xen-devel/2023-10/msg00796.html >>> [2] https://developer.arm.com/documentation/den0098/latest/
[3] https://documentation-service.arm.com/static/628b755ce3c4322a76af56de?token=
Hm, no reply so far. Is this maybe the wrong list? Should I post this rather >> somewhere else?
Paul
debian-arm / OFTC IRC #debian-arm or #debian-raspberrypi
Xen in Debian is team maintained, I think, but many people have moved
away from Xen in favour of other virtualisation/paravirtualisation
solutions and containers.
All the very best, as ever,
Andy
(amacater@debian.org)
On Wed, 2024-01-31 at 08:02 +0100, Paul Leiber wrote:
Am 25.01.2024 um 22:28 schrieb Paul Leiber:
[...]
Some people on xen-devel pointed out to me two unhandled SMC calls in the boot logs which could be the root of the problem. I am now trying to find out where these calls come from to get closer to the root cause. The suspected calls are the following ones:
(XEN) d0v0 Unhandled SMC/HVC: 0x84000050
(XEN) d0v0 Unhandled SMC/HVC: 0x8600ff01
These calls happen during the Dom0 boot process, so it's something from inside Linux and nothing Xen related, I've been told. The current working hypothesis is that the calls are trying to find some module not emulated by Xen and are therefore failing, leading to Linux waiting for the reply, and subsequently to the Xen watchdog triggering and rebooting.
From what I could find out in ARM documentation, the unhandled SMC calls probably have the following purpose:
0x84000050 = TRNG_VERSION, returns the implemented TRNG (True Random Number Generator) ABI version [2]
0x8600ff01 = Call UID Query for Vendor Specific Hypervisor Service, Returns a unique identifier of the service provider [3]
The more likely cause is the second call to the address 0x8600ff01.
Now I simply have no idea how to find out where in the Linux boot process these calls are made. I tried poking into the Linux sources a bit, and I couldn't find an exact match for these call addresses, so I assume these addresses are assembled from different parts. There are some matches for "0x8600" and for "ff01", but I couldn't identify if these matches are relevant.
I tried to find out if strace could help, but from what I understand, this is related to commands coming from userspace, so I am not sure that strace helps during the boot process.
I'd appreciate it if somebody more knowledgeable would point me in the right direction. If more information is needed, I can provide it.
I would search for the message 'Unhandled SMC/HVC' itself, or even for 'Unhandled', not for the address. The address is probably determined
at runtime and not hardcoded.
On Wed, 2024-01-31 at 21:59 +0100, hw wrote:
On Wed, 2024-01-31 at 08:02 +0100, Paul Leiber wrote:
Am 25.01.2024 um 22:28 schrieb Paul Leiber:
[...]
Some people on xen-devel pointed out to me two unhandled SMC calls in
the boot logs which could be the root of the problem. I am now trying to >>>> find out where these calls come from to get closer to the root cause.
The suspected calls are the following ones:
(XEN) d0v0 Unhandled SMC/HVC: 0x84000050
(XEN) d0v0 Unhandled SMC/HVC: 0x8600ff01
These calls happen during the Dom0 boot process, so it's something from >>>> inside Linux and nothing Xen related, I've been told. The current
working hypothesis is that the calls are trying to find some module not >>>> emulated by Xen and are therefore failing, leading to Linux waiting for >>>> the reply, and subsequently to the Xen watchdog triggering and rebooting. >>>>
From what I could find out in ARM documentation, the unhandled SMC
calls probably have the following purpose:
0x84000050 = TRNG_VERSION, returns the implemented TRNG (True Random
Number Generator) ABI version [2]
0x8600ff01 = Call UID Query for Vendor Specific Hypervisor Service,
Returns a unique identifier of the service provider [3]
The more likely cause is the second call to the address 0x8600ff01.
Now I simply have no idea how to find out where in the Linux boot
process these calls are made. I tried poking into the Linux sources a
bit, and I couldn't find an exact match for these call addresses, so I >>>> assume these addresses are assembled from different parts. There are
some matches for "0x8600" and for "ff01", but I couldn't identify if
these matches are relevant.
I tried to find out if strace could help, but from what I understand,
this is related to commands coming from userspace, so I am not sure that >>>> strace helps during the boot process.
I'd appreciate it if somebody more knowledgeable would point me in the >>>> right direction. If more information is needed, I can provide it.
I would search for the message 'Unhandled SMC/HVC' itself, or even for
'Unhandled', not for the address. The address is probably determined
at runtime and not hardcoded.
I sure those hex values aren't 'addresses' but the ID's for the secure monitor calls Paul already identified.
Looking at the Linux sources I found the header for constructing these monitor calls: include/linux/arm-smccc.h
So it might be worth looking at the files that include that. There are various drivers for firmware, and a watchdog driver amongst other
things... drivers/watchdog/arm_smc_wdt.c
| Sysop: | Keyop |
|---|---|
| Location: | Huddersfield, West Yorkshire, UK |
| Users: | 297 |
| Nodes: | 16 (2 / 14) |
| Uptime: | 34:38:17 |
| Calls: | 6,669 |
| Calls today: | 1 |
| Files: | 12,216 |
| Messages: | 5,338,372 |