• debian bootup hangs on qemu-system-mips64el

    From Jun Sun@21:1/5 to jsun@junsun.net on Sun Jan 5 05:40:02 2020
    Cross-post to the *right* linux-mips list and hopefully we can have more
    expert eyes looking at this ...

    Thanks.

    Jun

    On Wed, Jan 1, 2020 at 6:50 PM Jun Sun <jsun@junsun.net> wrote:

    Hi, all,

    Debian/mips64el runs well on qemu-system-mips64el until I turned on smp option with a number of 2 or higher. See the error message below. It usually hangs pretty early on.

    I have experimented with different CPU types (e.g., 5KEc) and different memory amont with no avail. Does anybody have an idea?

    I attached the kernel config for reference. It appears SMP kernel configs are enabled properly.

    Thanks a lot!

    Jun

    qemu-system-mips64el -M malta -smp 2 -cpu MIPS64R2-generic -m 1314 -hda hda.qcow2 -kernel vmlinux -initrd initrd.img -append 'root=/dev/sda1 console=ttyS0 nokaslr' -nographic -net user,hostfwd=tcp::2222-:22,hostfwd=tcp::8080-:80,hostfwd=tcp::8443-:443
    -net nic -serial mon:stdio
    qemu-system-mips64el: warning: SPD cannot represent final 2MB of SDRAM
    [ 0.000000] Linux version 4.19.0-6-5kc-malta ( debian-kernel@lists.debian.org) (gcc version 8.3.0 (Debian 8.3.0-6)) #1
    SMP Debian 4.19.67-2+deb10u1 (2019-09-20)
    [ 0.000000] earlycon: uart8250 at I/O port 0x3f8 (options '38400n8')
    [ 0.000000] bootconsole [uart8250] enabled
    [ 0.000000] CPU0 revision is: 00010000 (MIPS GENERIC QEMU)
    [ 0.000000] FPU revision is: 007f0000
    [ 0.000000] Checking for the multiply/shift bug... no.
    [ 0.000000] Checking for the daddiu bug... no.
    [ 0.000000] MIPS: machine is mti,malta
    [ 0.000000] Software DMA cache coherency enabled
    [ 0.000000] Determined physical RAM map:
    [ 0.000000] memory: 0000000010000000 @ 0000000000000000 (usable)
    [ 0.000000] memory: 0000000042200000 @ 0000000090000000 (usable)

    [HANG FOREVER AT HERE ...]



    <div dir="ltr">Cross-post to the *right* linux-mips list and hopefully we can have more expert eyes looking at this ...  <br><div><br></div><div>Thanks.</div><div><br></div><div>Jun</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_
    attr">On Wed, Jan 1, 2020 at 6:50 PM Jun Sun &lt;<a href="mailto:jsun@junsun.net">jsun@junsun.net</a>&gt; wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="
    ltr">Hi, all,<div><br></div><div>Debian/mips64el runs well on qemu-system-mips64el until I turned on smp option with a number of 2 or higher.  See the error message below.  It usually hangs pretty early on.</div><div><br></div><div>I have experimented
    with different CPU types (e.g., 5KEc) and different memory amont with no avail. Does anybody have an idea?</div><div><br></div><div>I attached the kernel config for reference.  It appears SMP kernel configs are enabled properly.</div><div><br></div><
    Thanks a lot!</div><div><br></div><div>Jun</div><div><br></div><div>qemu-system-mips64el -M malta -smp 2 -cpu MIPS64R2-generic -m 1314 -hda hda.qcow2 -kernel vmlinux -initrd initrd.img -append &#39;root=/dev/sda1 console=ttyS0 nokaslr&#39; -nographic
    -net user,hostfwd=tcp::2222-:22,hostfwd=tcp::8080-:80,hostfwd=tcp::8443-:443 -net nic -serial mon:stdio<br>qemu-system-mips64el: warning: SPD cannot represent final 2MB of SDRAM<br>[    0.000000] Linux version 4.19.0-6-5kc-malta (<a href="mailto:debian-
    kernel@lists.debian.org" target="_blank">debian-kernel@lists.debian.org</a>) (gcc version 8.3.0 (Debian 8.3.0-6)) #1 SMP Debian 4.19.67-2+deb10u1 (2019-09-20)<br>[    0.000000] earlycon: uart8250 at I/O port 0x3f8 (options &#39;38400n8&#39;)<br>[    
    0.000000] bootconsole [uart8250] enabled<br>[    0.000000] CPU0 revision is: 00010000 (MIPS GENERIC QEMU)<br>[    0.000000] FPU revision is: 007f0000<br>[    0.000000] Checking for the multiply/shift bug... no.<br>[    0.000000] Checking for the
    daddiu bug... no.<br>[    0.000000] MIPS: machine is mti,malta<br>[    0.000000] Software DMA cache coherency enabled<br>[    0.000000] Determined physical RAM map:<br>[    0.000000]  memory: 0000000010000000 @ 0000000000000000 (usable)<br>[  
    0.000000]  memory: 0000000042200000 @ 0000000090000000 (usable)<br><br></div><div>[HANG FOREVER AT HERE ...]</div><div><br></div></div>
    </blockquote></div>

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jun Sun@21:1/5 to jsun@junsun.net on Sun Jan 5 05:50:02 2020
    Cross-post to the *right* linux-mips list with the *right*
    "plain/text" format (sigh)... and hopefully we can have more expert
    eyes looking at this ...

    (Really sorry to qemu-discuss and debian-mips list ...)

    Thanks.

    Jun


    On Wed, Jan 1, 2020 at 6:50 PM Jun Sun <jsun@junsun.net> wrote:

    Hi, all,

    Debian/mips64el runs well on qemu-system-mips64el until I turned on smp option with a number of 2 or higher. See the error message below. It usually hangs pretty early on.

    I have experimented with different CPU types (e.g., 5KEc) and different memory amont with no avail. Does anybody have an idea?

    I attached the kernel config for reference. It appears SMP kernel configs are enabled properly.

    Thanks a lot!

    Jun

    qemu-system-mips64el -M malta -smp 2 -cpu MIPS64R2-generic -m 1314 -hda hda.qcow2 -kernel vmlinux -initrd initrd.img -append 'root=/dev/sda1 console=ttyS0 nokaslr' -nographic -net user,hostfwd=tcp::2222-:22,hostfwd=tcp::8080-:80,hostfwd=tcp::8443-:443 -
    net nic -serial mon:stdio
    qemu-system-mips64el: warning: SPD cannot represent final 2MB of SDRAM
    [ 0.000000] Linux version 4.19.0-6-5kc-malta (debian-kernel@lists.debian.org) (gcc version 8.3.0 (Debian 8.3.0-6)) #1 SMP Debian 4.19.67-2+deb10u1 (2019-09-20)
    [ 0.000000] earlycon: uart8250 at I/O port 0x3f8 (options '38400n8')
    [ 0.000000] bootconsole [uart8250] enabled
    [ 0.000000] CPU0 revision is: 00010000 (MIPS GENERIC QEMU)
    [ 0.000000] FPU revision is: 007f0000
    [ 0.000000] Checking for the multiply/shift bug... no.
    [ 0.000000] Checking for the daddiu bug... no.
    [ 0.000000] MIPS: machine is mti,malta
    [ 0.000000] Software DMA cache coherency enabled
    [ 0.000000] Determined physical RAM map:
    [ 0.000000] memory: 0000000010000000 @ 0000000000000000 (usable)
    [ 0.000000] memory: 0000000042200000 @ 0000000090000000 (usable)

    [HANG FOREVER AT HERE ...]


    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Aurelien Jarno@21:1/5 to Jun Sun on Thu Jan 9 22:30:02 2020
    On 2020-01-04 20:46, Jun Sun wrote:
    On Wed, Jan 1, 2020 at 6:50 PM Jun Sun <jsun@junsun.net> wrote:

    Hi, all,

    Debian/mips64el runs well on qemu-system-mips64el until I turned on smp option with a number of 2 or higher. See the error message below. It usually hangs pretty early on.


    QEMU/mips doesn't really support SMP. The -smp option used to be for
    SMT, but I am not even sure it really worked.

    All that said, it's still a bug in QEMU to accept this option.

    --
    Aurelien Jarno GPG: 4096R/1DDD8C9B aurelien@aurel32.net http://www.aurel32.net

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Aurelien Jarno@21:1/5 to Jun Sun on Sun Jan 12 21:00:01 2020
    Hi,

    On 2020-01-12 11:34, Jun Sun wrote:
    Are you sure QEMU/MIPS does not support SMP? Below link written in
    2016 clearly shows something was working then.

    https://www.mips.com/blog/how-to-run-smp-linux-in-qemu-on-a-mips64-release-6-cpu/

    It's not supported for the malta/mips64r2 combination you are trying to
    use. It's clearly possible that it is supported for mips64r6, but it's a different instruction set, so you can't run a standard Debian
    distribution on it.

    Also, as Lixing mentioned in our private exchange, there is SMP
    support for loongson. Lixing, I assume loongson SMP support is
    already merged in QEMU repo, right? Can you point out the relevant
    source code directories?

    Lixing mentioned virtualization. In that case it is indeed supported
    like on other mips64r5 CPUs. For that you need to use add the -enable-kvm option to the qemu command.

    Aurelien

    --
    Aurelien Jarno GPG: 4096R/1DDD8C9B aurelien@aurel32.net http://www.aurel32.net

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jun Sun@21:1/5 to aurelien@aurel32.net on Sun Jan 12 20:40:02 2020
    Are you sure QEMU/MIPS does not support SMP? Below link written in
    2016 clearly shows something was working then.

    https://www.mips.com/blog/how-to-run-smp-linux-in-qemu-on-a-mips64-release-6-cpu/

    Also, as Lixing mentioned in our private exchange, there is SMP
    support for loongson. Lixing, I assume loongson SMP support is
    already merged in QEMU repo, right? Can you point out the relevant
    source code directories?

    Would love to get more pointers here! Thanks!

    Jun

    On Thu, Jan 9, 2020 at 1:24 PM Aurelien Jarno <aurelien@aurel32.net> wrote:

    On 2020-01-04 20:46, Jun Sun wrote:
    On Wed, Jan 1, 2020 at 6:50 PM Jun Sun <jsun@junsun.net> wrote:

    Hi, all,

    Debian/mips64el runs well on qemu-system-mips64el until I turned on smp option with a number of 2 or higher. See the error message below. It usually hangs pretty early on.


    QEMU/mips doesn't really support SMP. The -smp option used to be for
    SMT, but I am not even sure it really worked.

    All that said, it's still a bug in QEMU to accept this option.

    --
    Aurelien Jarno GPG: 4096R/1DDD8C9B aurelien@aurel32.net http://www.aurel32.net

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jiaxun Yang@21:1/5 to All on Mon Jan 13 02:50:02 2020
    于 2020年1月13日 GMT+08:00 上午3:34:18, Jun Sun <jsun@junsun.net> 写到: >Are you sure QEMU/MIPS does not support SMP? Below link written in
    2016 clearly shows something was working then.

    https://www.mips.com/blog/how-to-run-smp-linux-in-qemu-on-a-mips64-release-6-cpu/

    Also, as Lixing mentioned in our private exchange, there is SMP
    support for loongson. Lixing, I assume loongson SMP support is
    already merged in QEMU repo, right? Can you point out the relevant
    source code directories?
    Hi Jun,

    As far as I known, Loongson is using there own SMP IPI implementation instead of MIPS MT One.
    There QEMU implementation could be found here [1].
    QEMU have implemented MIPS MT ASE for SMP.

    Thanks

    Would love to get more pointers here! Thanks!

    Jun

    On Thu, Jan 9, 2020 at 1:24 PM Aurelien Jarno <aurelien@aurel32.net>
    wrote:

    On 2020-01-04 20:46, Jun Sun wrote:
    On Wed, Jan 1, 2020 at 6:50 PM Jun Sun <jsun@junsun.net> wrote:

    Hi, all,

    Debian/mips64el runs well on qemu-system-mips64el until I turned
    on smp option with a number of 2 or higher. See the error message
    below. It usually hangs pretty early on.


    QEMU/mips doesn't really support SMP. The -smp option used to be for
    SMT, but I am not even sure it really worked.

    All that said, it's still a bug in QEMU to accept this option.

    --
    Aurelien Jarno GPG: 4096R/1DDD8C9B
    aurelien@aurel32.net http://www.aurel32.net

    [1]: https://gitee.com/loongsonlab/qemu

    --
    Jiaxun Yang

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jun Sun@21:1/5 to jiaxun.yang@flygoat.com on Mon Jan 13 03:40:01 2020
    Thanks, Jiaxun. See more questions below.

    Jun

    On Sun, Jan 12, 2020 at 5:17 PM Jiaxun Yang <jiaxun.yang@flygoat.com> wrote:



    于 2020年1月13日 GMT+08:00 上午3:34:18, Jun Sun <jsun@junsun.net> 写到:
    Are you sure QEMU/MIPS does not support SMP? Below link written in
    2016 clearly shows something was working then.

    https://www.mips.com/blog/how-to-run-smp-linux-in-qemu-on-a-mips64-release-6-cpu/

    Also, as Lixing mentioned in our private exchange, there is SMP
    support for loongson. Lixing, I assume loongson SMP support is
    already merged in QEMU repo, right? Can you point out the relevant
    source code directories?
    Hi Jun,

    As far as I known, Loongson is using there own SMP IPI implementation instead of MIPS MT One.
    There QEMU implementation could be found here [1].
    QEMU have implemented MIPS MT ASE for SMP.


    Do you know how to enable and run MIPS MT ASE in qemu? For with CPU options?

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jun Sun@21:1/5 to aurelien@aurel32.net on Mon Jan 13 03:20:01 2020
    See below.

    On Sun, Jan 12, 2020 at 11:58 AM Aurelien Jarno <aurelien@aurel32.net> wrote:

    Hi,

    On 2020-01-12 11:34, Jun Sun wrote:
    Are you sure QEMU/MIPS does not support SMP? Below link written in
    2016 clearly shows something was working then.

    https://www.mips.com/blog/how-to-run-smp-linux-in-qemu-on-a-mips64-release-6-cpu/

    It's not supported for the malta/mips64r2 combination you are trying to
    use. It's clearly possible that it is supported for mips64r6, but it's a different instruction set, so you can't run a standard Debian
    distribution on it.


    How do I specify mips64r6 cpu to start qemu? A simple cpu listing
    below shows no such CPU. If mips64r6 is indeed supported with SMP, I
    would think mips64r2 userland should run fine.

    $ qemu-system-mips64el -cpu ?
    MIPS '4Kc'
    MIPS '4Km'
    MIPS '4KEcR1'
    MIPS '4KEmR1'
    MIPS '4KEc'
    MIPS '4KEm'
    MIPS '24Kc'
    MIPS '24KEc'
    MIPS '24Kf'
    MIPS '34Kf'
    MIPS '74Kf'
    MIPS 'M14K'
    MIPS 'M14Kc'
    MIPS 'P5600'
    MIPS 'mips32r6-generic'
    MIPS 'R4000'
    MIPS 'VR5432'
    MIPS '5Kc'
    MIPS '5Kf'
    MIPS '20Kc'
    MIPS 'MIPS64R2-generic'
    MIPS '5KEc'
    MIPS '5KEf'
    MIPS 'I6400'
    MIPS 'Loongson-2E'
    MIPS 'Loongson-2F'
    MIPS 'mips64dspr2'

    Also, as Lixing mentioned in our private exchange, there is SMP
    support for loongson. Lixing, I assume loongson SMP support is
    already merged in QEMU repo, right? Can you point out the relevant
    source code directories?

    Lixing mentioned virtualization. In that case it is indeed supported
    like on other mips64r5 CPUs. For that you need to use add the -enable-kvm option to the qemu command.

    I'm not sure if loongson support has to be kvm. Maybe Lixing can clarify.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Huacai Chen@21:1/5 to jsun@junsun.net on Mon Jan 13 04:40:02 2020
    Hi, all

    I'm working on Loongson-3's SMP support in QEMU, and I'll send my patches soon.

    Huacai

    On Mon, Jan 13, 2020 at 10:35 AM Jun Sun <jsun@junsun.net> wrote:

    Thanks, Jiaxun. See more questions below.

    Jun

    On Sun, Jan 12, 2020 at 5:17 PM Jiaxun Yang <jiaxun.yang@flygoat.com> wrote:



    于 2020年1月13日 GMT+08:00 上午3:34:18, Jun Sun <jsun@junsun.net> 写到:
    Are you sure QEMU/MIPS does not support SMP? Below link written in
    2016 clearly shows something was working then.

    https://www.mips.com/blog/how-to-run-smp-linux-in-qemu-on-a-mips64-release-6-cpu/

    Also, as Lixing mentioned in our private exchange, there is SMP
    support for loongson. Lixing, I assume loongson SMP support is
    already merged in QEMU repo, right? Can you point out the relevant >source code directories?
    Hi Jun,

    As far as I known, Loongson is using there own SMP IPI implementation instead of MIPS MT One.
    There QEMU implementation could be found here [1].
    QEMU have implemented MIPS MT ASE for SMP.


    Do you know how to enable and run MIPS MT ASE in qemu? For with CPU options?

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jiaxun Yang@21:1/5 to All on Mon Jan 13 04:50:01 2020
    于 2020年1月13日 GMT+08:00 上午10:35:18, Jun Sun <jsun@junsun.net> 写到:
    Thanks, Jiaxun. See more questions below.

    Jun

    On Sun, Jan 12, 2020 at 5:17 PM Jiaxun Yang <jiaxun.yang@flygoat.com>
    wrote:



    于 2020年1月13日 GMT+08:00 上午3:34:18, Jun Sun <jsun@junsun.net> 写到:
    Are you sure QEMU/MIPS does not support SMP? Below link written in
    2016 clearly shows something was working then.

    https://www.mips.com/blog/how-to-run-smp-linux-in-qemu-on-a-mips64-release-6-cpu/

    Also, as Lixing mentioned in our private exchange, there is SMP
    support for loongson. Lixing, I assume loongson SMP support is
    already merged in QEMU repo, right? Can you point out the relevant
    source code directories?
    Hi Jun,

    As far as I known, Loongson is using there own SMP IPI implementation >instead of MIPS MT One.
    There QEMU implementation could be found here [1].
    QEMU have implemented MIPS MT ASE for SMP.


    Do you know how to enable and run MIPS MT ASE in qemu? For with CPU
    options?

    I tried maxcpus=2 cpu=34kf for malta long ago and it was working.
    But I'm not sure whether it still works now.


    --
    Jiaxun Yang

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Paul Burton@21:1/5 to Jun Sun on Mon Jan 13 20:30:02 2020
    Hi Jun Sun,

    On Sun, Jan 12, 2020 at 06:14:12PM -0800, Jun Sun wrote:
    On Sun, Jan 12, 2020 at 11:58 AM Aurelien Jarno <aurelien@aurel32.net> wrote:
    On 2020-01-12 11:34, Jun Sun wrote:
    Are you sure QEMU/MIPS does not support SMP? Below link written in
    2016 clearly shows something was working then.

    https://www.mips.com/blog/how-to-run-smp-linux-in-qemu-on-a-mips64-release-6-cpu/

    It's not supported for the malta/mips64r2 combination you are trying to use. It's clearly possible that it is supported for mips64r6, but it's a different instruction set, so you can't run a standard Debian
    distribution on it.

    How do I specify mips64r6 cpu to start qemu? A simple cpu listing
    below shows no such CPU. If mips64r6 is indeed supported with SMP, I
    would think mips64r2 userland should run fine.

    I know that using QEMU's I6400 or I6500 CPU emulation works fine with
    SMP as far as -smp 8. It emulates a single core system with up to 8 VPs (hardware threads), and Linux works fine with it. I most often use the
    Boston board emulation with a 64r6el_defconfig kernel for this, but it
    ought to work for Malta too.

    Both I6400 & I6500 are MIPS64r6 CPUs, so as Aurelien said you'd need to
    ensure you're using a suitable MIPSr6 root filesystem.

    If you wanted to run as I do it'd go something like this:

    $ qemu-system-mips64el \
    -M boston \
    -cpu I6500 \
    -smp 8 \
    -kernel arch/mips/boot/vmlinux.gz.itb

    Where that's run from the root of a kernel source tree within which
    you've built a 64r6el_defconfig kernel.

    Not sure what the behavior is with older CPUs; QEMU's CM & CPC emulation
    is somewhat incomplete, and whilst I6400 & I6500 have a relatively straightforward VP model older CPUs have the more complex MT ASE with
    VPEs & TCs to worry about.

    Thanks,
    Paul

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)