On Sun, 3 Jan 2021, Eero Tamminen wrote:

...
* While kernel runs init a minute after being
booted [1], installer is *much* slower.

E.g. after pressing Enter to select another
country, it takes 5-10 mins until installer
presents me with a list from which I can select
Europe

Is that a kernel regression or an installer regression?

...
Testing things this far took at least hour for
each run, due to installer slowness. Does anybody
happen to have an already installed minimal and up
to date Debian m68k disk image?

You may want to try to debootstrap one. ISTR there are instructions on wiki.debian.org.

[1] Hatari profiler shows kernel CPU usage to go
in boot, until first sbrk() system call, to: ------------------------------------------------
Executed instructions:
19.16% 25366973 memset
7.97% 10548067 memcpy
3.31% 4379652 _parse_integer
1.88% 2489653 bit_putcs
1.85% 2452344 link_path_walk
1.82% 2406272 atafb_mfb_linefill
1.26% 1672580 memmap_init_zone
1.26% 1664687 timekeeping_advance
1.10% 1452290 strlen
1.07% 1422344 get_page_from_freelist
1.05% 1388376 psi_group_change.constprop.0
0.94% 1240372 kmem_cache_alloc
0.71% 936593 add_interrupt_randomness
0.69% 909479 __d_lookup_rcu
0.68% 896967 __ashldi3
0.64% 849576 atafb_imageblit
0.62% 822723 kernfs_name_hash
0.62% 820971 __list_add_valid
0.54% 708561 notify_change
0.51% 679840 __free_one_page ------------------------------------------------

In cycles, memcpy() uses a bit more compared to
memset(), but memset() still takes most CPU.

Attached are partial callgraphs showing where
they're most called from. Percentages in the
arrow lines indicate each function node's share
of those calls.

If this is a new phenomenon, git bisect may reveal what changed.

Could the memcpy calls be related to 64-bit timekeeping?

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi,

On 1/3/21 1:38 AM, Finn Thain wrote:

On Sun, 3 Jan 2021, Eero Tamminen wrote:

...
* While kernel runs init a minute after being
booted [1], installer is *much* slower.

E.g. after pressing Enter to select another
country, it takes 5-10 mins until installer
presents me with a list from which I can select
Europe

Is that a kernel regression or an installer regression?

This is the first time I've run Debian kernel +
installer in Hatari, so I have no idea whether
it's a regression. Sorry, I wasn't clear on that!

Based on some profiling I did, slowness is
completely on the user-space side. [1]

In cycles, memcpy() uses a bit more compared to
memset(), but memset() still takes most CPU.

Attached are partial callgraphs showing where
they're most called from. Percentages in the
arrow lines indicate each function node's share
of those calls.

If this is a new phenomenon, git bisect may reveal what changed.

I've been testing upstream kernel boot times
since v5.2, and I don't think there's been any
significant regression...

Could the memcpy calls be related to 64-bit timekeeping?

...but if you're interested, I could profile and
provide similar callgraph for another kernel
you're interested about. Just point me to kernel
+ initrd + symbols.map packages (or files) you'd
like me to check.


[1] There isn't any way to profile Linux user-
space in Hatari, except for having single number
of how much time whole user-space consumes
compared to kernel.

However, I can get very accurate profiling data
for all of the kernel side (depending on symbols
file content), and profiling whole Linux bootup is
trivial in Hatari.

Profiling kernel usage during specific user-
space activity would require that user-space
activity to start & end with call to some kernel
symbol/syscall that's not otherwise called
(so that emulator breakpoints can be set to
start and of the profiling).


- Eero

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Sun, 3 Jan 2021, Eero Tamminen wrote:

On 1/3/21 1:38 AM, Finn Thain wrote:

On Sun, 3 Jan 2021, Eero Tamminen wrote:

...
* While kernel runs init a minute after being
booted [1], installer is *much* slower.

E.g. after pressing Enter to select another
country, it takes 5-10 mins until installer
presents me with a list from which I can select
Europe

Is that a kernel regression or an installer regression?

This is the first time I've run Debian kernel +
installer in Hatari, so I have no idea whether
it's a regression. Sorry, I wasn't clear on that!

Based on some profiling I did, slowness is
completely on the user-space side. [1]

If you want to check for a regression in the installer, you can find
various ISO images for m68k going back to 2018 at https://cdimage.debian.org/cdimage/ports/

In cycles, memcpy() uses a bit more compared to
memset(), but memset() still takes most CPU.

Attached are partial callgraphs showing where
they're most called from. Percentages in the
arrow lines indicate each function node's share
of those calls.

If this is a new phenomenon, git bisect may reveal what changed.

I've been testing upstream kernel boot times
since v5.2, and I don't think there's been any
significant regression...

Could the memcpy calls be related to 64-bit timekeeping?

...but if you're interested, I could profile and
provide similar callgraph for another kernel
you're interested about. Just point me to kernel
+ initrd + symbols.map packages (or files) you'd
like me to check.

It's not hard to build that yourself, thanks to Debian's cross-compiler packages. But since I do a lot of these builds, I've built one for you to
try.

https://www.telegraphics.com.au/~fthain/build/linux-m68k-image-4.14.0-multi.tar.gz
https://www.telegraphics.com.au/~fthain/build/vmlinux-4.14.0-multi

MD5 (linux-m68k-image-4.14.0-multi.tar.gz) = fdd15dafc843a86c3e50afd24c864159 MD5 (vmlinux-4.14.0-multi) = 0129bce08219f49a6cdb7b3d21c7afd1

It boots to busybox in aranym but YMMV. It may not work with the debian installer in hatari. Note that v4.14.213 has fewer bugs but it too may
have performance regressions.

[1] There isn't any way to profile Linux user-
space in Hatari, except for having single number
of how much time whole user-space consumes
compared to kernel.

That's quite a useful ratio, I think. If you kept the userspace workload constant and swapped the kernel release for an older one, you could tell whether the kernel had regressed. Similarly, you could keep the kernel
build constant and vary the installer release, to look for changes there.

However, I can get very accurate profiling data
for all of the kernel side (depending on symbols
file content), and profiling whole Linux bootup is
trivial in Hatari.

Profiling kernel usage during specific user-
space activity would require that user-space
activity to start & end with call to some kernel
symbol/syscall that's not otherwise called
(so that emulator breakpoints can be set to
start and of the profiling).

I guess profiling should start at kernel_execve("/sbin/init", ...)

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)