1. Forum
  2. Usenet
  3. LINUX.KERNEL

  • [PATCH RFC V4 5/6] perf top: switch to backward overwrite mode

    From acme@kernel.org@21:1/5 to All on Mon Oct 2 22:50:01 2017
    Em Mon, Oct 02, 2017 at 05:19:41PM +0000, Liang, Kan escreveu:


    On Fri, Sep 29, 2017 at 07:47:56AM -0700, kan.liang@intel.com wrote:
    From: Kan Liang <kan.liang@intel.com>

    perf_top__mmap_read has severe performance issue in Knights
    Landing/Mill, when monitoring in heavy load system. It costs several minutes to finish, which is unacceptable.

    perf top was overwrite mode. But it is changed to non overwrite mode since commit 93fc64f14472 ("perf top: Switch to non overwrite mode").
    For non overwrite mode, it tries to read everything in the ring buffer and does not check the messup. Once there are lots of samples
    delivered shortly, the processing time could be very long.
    Knights Landing/Mill as a manycore processor contains a large number
    of small cores. Because of the huge core number, it will generated
    lots of samples in a heavy load system. Also, since the huge sample#,
    the mmap writer probably bite the tail and mess up the samples.

    Also, to avoid the problems which is described in commit 9ecda41acb97 ("perf/core: Add ::write_backward attribute to perf event"), switch to backward overwrite mode.
    Pausing the ring-buffer during perf_top__mmap_read to ensure the ring-buffer is stable.
    There would be some records lost in backward overwrite mode. Removing
    the lost events checking.

    I'm getting perf top hogging the cpu completely with this change


    I think I find the root cause of the cpu hogging.
    perf_mmap__read_catchup discards the md->prev from previous mmap_read. Current mmap_read doesn't know which data has already been processed by previous mmap_read. So it has to go through all the valid data in the ring buffer,
    even most of the data has been processed by previous mmap_read.

    Also, it looks perf record has the similar issue.
    The previous location will be discarded as well in backward overwrite mode. That will be an issue when --overwrite and --switch-output are enabled.
    The new output will always include the old data in the previous output, which should be wrong.

    I think I will rewrite the perf_mmap__read_backward and perf_mmap__read_catchup
    to fix this issue in a separate thread. Those functions should be common backward
    mmap_read functions for all tools and tests.

    BTW, are you OK with patch 1-4?
    Those patches multithreading the machine__synthesize_threads, which is irrelevant with the overwrite mode.
    I think they can be merged separately.

    I'll try and process 1-4 while you work on the read_backward case.

    - Arnaldo

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Liang, Kan@21:1/5 to All on Mon Oct 2 22:50:06 2017
    On Fri, Sep 29, 2017 at 07:47:56AM -0700, kan.liang@intel.com wrote:
    From: Kan Liang <kan.liang@intel.com>

    perf_top__mmap_read has severe performance issue in Knights
    Landing/Mill, when monitoring in heavy load system. It costs several minutes to finish, which is unacceptable.

    perf top was overwrite mode. But it is changed to non overwrite mode
    since commit 93fc64f14472 ("perf top: Switch to non overwrite mode").
    For non overwrite mode, it tries to read everything in the ring buffer
    and does not check the messup. Once there are lots of samples
    delivered shortly, the processing time could be very long.
    Knights Landing/Mill as a manycore processor contains a large number
    of small cores. Because of the huge core number, it will generated
    lots of samples in a heavy load system. Also, since the huge sample#,
    the mmap writer probably bite the tail and mess up the samples.

    Also, to avoid the problems which is described in commit 9ecda41acb97 ("perf/core: Add ::write_backward attribute to perf event"), switch to backward overwrite mode.
    Pausing the ring-buffer during perf_top__mmap_read to ensure the ring-buffer is stable.
    There would be some records lost in backward overwrite mode. Removing
    the lost events checking.

    I'm getting perf top hogging the cpu completely with this change


    I think I find the root cause of the cpu hogging.
    perf_mmap__read_catchup discards the md->prev from previous mmap_read.
    Current mmap_read doesn't know which data has already been processed by previous mmap_read. So it has to go through all the valid data in the ring buffer,
    even most of the data has been processed by previous mmap_read.

    Also, it looks perf record has the similar issue.
    The previous location will be discarded as well in backward overwrite mode. That will be an issue when --overwrite and --switch-output are enabled.
    The new output will always include the old data in the previous output, which should be wrong.

    I think I will rewrite the perf_mmap__read_backward and perf_mmap__read_catchup to fix this issue in a separate thread. Those functions should be common backward
    mmap_read functions for all tools and tests.

    BTW, are you OK with patch 1-4?
    Those patches multithreading the machine__synthesize_threads, which is irrelevant with the overwrite mode.
    I think they can be merged separately.

    Thanks,
    Kan

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)