Hello,
I am using mmap() to map device file "mtdblock0" into virtual address space of one process1. Some data whose value changes continuously for some time are stored in memory mapped region as well. I want to prevent these data changes to be flushed
from memory mapped region to the device file for some time until the calculations are over(so that process2 which is mapping to the same file does not see these changes). Note: The same device file is mapped by another process also to read the data.

Assuming that mmap() syncing works for the device files in the same way as for regular files, I tried setting "dirty_writeback_centisecs" to 0 to disable the kernel threads that perform the flushing. Then I'm setting "dirty_writeback_centisecs" to
default value after some time to again enable the flushing. This is working partially. But the data is still getting flushed to the device file occassionally even when "dirty_writeback_centisecs" is 0. Please help me to know why this is not working.

Does the mmap() syncing works for the device files in the same way as for regular files?

Are the same set of threads that are responsible for flushing the page cache are also responsible for flushing the memory mapped region to the underlying file?

Files mapped using mmap() are directly mapped from the disk to the virtual address space of process. They don't use intermediate page cache. So any flushing done through msync() happens directly between memory map and disk. Is my understanding right?

If this method is not possible, are there any other ways of achieving this?(I have tried mysnc() with INVALIDATE and it does not work)

Thanks in Advance

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Pavankumar S V <pavankumarsv96@gmail.com> writes:

I am using mmap() to map device file "mtdblock0" into
virtual address space of one process1. Some data whose value changes continuously for some time are stored in memory mapped region as
well. I want to prevent these data changes to be flushed from memory
mapped region to the device file for some time until the calculations
are over(so that process2 which is mapping to the same file does not
see these changes). Note: The same device file is mapped by another
process also to read the data.

Assuming that mmap() syncing works for the device files in the same
way as for regular files, I tried setting "dirty_writeback_centisecs"
to 0 to disable the kernel threads that perform the flushing. Then I'm setting "dirty_writeback_centisecs" to default value after some time
to again enable the flushing. This is working partially. But the data
is still getting flushed to the device file occassionally even when "dirty_writeback_centisecs" is 0. Please help me to know why this is
not working.

The kernel will also flush if there is more than a certain number of
modified pages.

Does the mmap() syncing works for the device files in the same way as
for regular files?

Are the same set of threads that are responsible for flushing the page
cache are also responsible for flushing the memory mapped region to
the underlying file?

Files mapped using mmap() are directly mapped from the disk to the
virtual address space of process. They don't use intermediate page
cache. So any flushing done through msync() happens directly between
memory map and disk. Is my understanding right?

If this method is not possible, are there any other ways of achieving
this?(I have tried mysnc() with INVALIDATE and it does not work)

I don’t think this is a viable strategy in its current form; you need
some explicit synchronization between the two processes.

--
https://www.greenend.org.uk/rjk/

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Monday, 6 March 2023 at 22:50:28 UTC+5:30, Richard Kettlewell wrote:

Pavankumar S V <pavanku...@gmail.com> writes:

Thanks for your inputs.

The kernel will also flush if there is more than a certain number of modified pages.

Yes. In combination with "dirty_writeback_centisecs", I had also tried increasing the values of kernel parameters "dirty_background_ratio", "dirty_ratio" to large values to increase the number of dirty pages that can be kept in memory before flushing to
disk.
But this also did not work.

I don’t think this is a viable strategy in its current form; you need
some explicit synchronization between the two processes.

I am trying now with explicit synchronization mechanism like this:
1) Created a shared memory object of size 1 byte in /dev/shm(let's consider its name as "prevent_read"). Memory mapped this to the virtual address space of both the processes.
2) When I want to prevent the process2 from reading "mtdblock0" region, then I'm writing "prevent_read" with '1' from process1.
3) process2 checks "prevent_read"(value is set as 1 by process1) before reading "mtdblock0" region and it should not read.
4) After some time, process1 writes '0' to "prevent_read".
5) process2 checks "prevent_read"(value is set as 0 by process1) before reading "mtdblock0" region and it should be able to read.

But above mechanism is also not working. Note: Both process1 and process2 are independent processes running on different cores.

As per my understanding, since shared memory object "prevent_read" uses the mmap() concept, again kernel has to do the flushing of data from the memory mapped region to the shared memory object "prevent_read".
So whenever process1 writes '1' to ""prevent_read", it actually writes to the memory mapped region of process1. So kernel has to do the flushing from memory mapped region to shared memory object "prevent_read". If this flushing is delayed, then process2
can read
'0' from "prevent_read" and can access "mtdblock0" until the flushing is completed. So this mechanism is failing sometimes.
Is there any way to do flushing immediately that will help in this case? (I tried msync, but msync is not syncing because msync's behavior is undefined if used on a shared memory object as per the man page.)

Thanks in Advance.

--
https://www.greenend.org.uk/rjk/

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Pavankumar S V <pavankumarsv96@gmail.com> writes:

I am trying now with explicit synchronization mechanism like this:
1) Created a shared memory object of size 1 byte in /dev/shm(let's
consider its name as "prevent_read"). Memory mapped this to the
virtual address space of both the processes.
2) When I want to prevent the process2 from reading "mtdblock0"
region, then I'm writing "prevent_read" with '1' from process1.
3) process2 checks "prevent_read"(value is set as 1 by process1)
before reading "mtdblock0" region and it should not read.
4) After some time, process1 writes '0' to "prevent_read".
5) process2 checks "prevent_read"(value is set as 0 by process1)
before reading "mtdblock0" region and it should be able to read.

That has a time-of-check/time-of-use bug even on a single core, and
there’s no chance it’ll work across multiple cores due to the lack of memory synchronization.

But above mechanism is also not working. Note: Both process1 and
process2 are independent processes running on different cores.

By synchronization mechanism I mean something like fctnl with F_SETLK,
or pthread_mutex_lock (with a mutex initialized with
PTHREAD_PROCESS_SHARED).

--
https://www.greenend.org.uk/rjk/

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Wednesday, 8 March 2023 at 14:23:03 UTC+5:30, Richard Kettlewell wrote:

Pavankumar S V <pavanku...@gmail.com> writes:

That has a time-of-check/time-of-use bug even on a single core, and there’s no chance it’ll work across multiple cores due to the lack of memory synchronization.

Ok understood. I was trying to use shared memory concept to achieve explicit synchronization (as Shared memory is the fastest way of IPC). I think intention of shared memory concept is just to transfer some data from one process to another quickly. We
cannot use Shared memory concept for synchronization purpose. Because, even though shared memory is very fast, data written by one process will not be visible to another immediately. Is my understanding right?

By synchronization mechanism I mean something like fctnl with F_SETLK,
or pthread_mutex_lock (with a mutex initialized with PTHREAD_PROCESS_SHARED).

ok. I'll try pthread_mutex_lock or semaphores to achieve the synchronization.

Thank you very much.

--
https://www.greenend.org.uk/rjk/

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)