For some time now when I run synth builds lang/rust
never completes,running out of swap space and making
over a dozen other ports that wait for it to get
built be skipped.
In the last few days,however,the swap space (I have 19G)
running out has not only caused a failure of rust or
a termination of synth,but made other applications terminate
("unable to recover memory") and logged me out,and when
I log in I discover that programs such as httpd and sendmail
are no longer running...
and furthermore,when I try a shutdown to reboot,it doesn't
complete the process,just sits after syncer is finished
and never powers down from shutdown -p
(and despite all the controlled shutdowns that it has
gone through,when I manually power down and up I have
to fsck everything).
-=-=-
The World Trade Center towers MUST rise again,
at least as tall as before...or terror has triumphed.
Louis Epstein <le@main.lekno.ws> wrote:
For some time now when I run synth builds lang/rust
never completes,running out of swap space and making
over a dozen other ports that wait for it to get
built be skipped.
In the last few days,however,the swap space (I have 19G)
running out has not only caused a failure of rust or
a termination of synth,but made other applications terminate
("unable to recover memory") and logged me out,and when
I log in I discover that programs such as httpd and sendmail
are no longer running...
and furthermore,when I try a shutdown to reboot,it doesn't
complete the process,just sits after syncer is finished
and never powers down from shutdown -p
(and despite all the controlled shutdowns that it has
gone through,when I manually power down and up I have
to fsck everything).
shutdown -r also stops cold after sync-ing.
-=-=-
The World Trade Center towers MUST rise again,
at least as tall as before...or terror has triumphed.
In the last few days,however,the swap space (I have 19G)
running out has not only caused a failure of rust or
a termination of synth,but made other applications terminate
("unable to recover memory") and logged me out,and when
I log in I discover that programs such as httpd and sendmail
are no longer running...
And killing the rust build when 7% of swap space is in use
still leads to an unable to get swap space message just
as killing it when 99.9% of swap space is in use before
the shutdown.
Louis Epstein <le@main.lekno.ws> wrote:
In the last few days,however,the swap space (I have 19G)
running out has not only caused a failure of rust or
a termination of synth,but made other applications terminate
("unable to recover memory") and logged me out,and when
I log in I discover that programs such as httpd and sendmail
are no longer running...
and later added:
And killing the rust build when 7% of swap space is in use
still leads to an unable to get swap space message just
as killing it when 99.9% of swap space is in use before
the shutdown.
Wild stab in the dark ...
If you're using tmpfs and it's an unknown tmpfs bug,
perhaps disable tmpfs and see if that makes a difference?
-WBE
Winston <wbe@ubeblock.psr.com.invalid> wrote:
Louis Epstein <le@main.lekno.ws> wrote:
In the last few days,however,the swap space (I have 19G)
running out has not only caused a failure of rust or
a termination of synth,but made other applications terminate
("unable to recover memory") and logged me out,and when
I log in I discover that programs such as httpd and sendmail
are no longer running...
and later added:
And killing the rust build when 7% of swap space is in use
still leads to an unable to get swap space message just
as killing it when 99.9% of swap space is in use before
the shutdown.
Wild stab in the dark ...
If you're using tmpfs and it's an unknown tmpfs bug,
perhaps disable tmpfs and see if that makes a difference?
-WBE
Will have to consider that,tmpfs is certainly
shown on a pre-shutdown df with lots of synth material.
The problem had gone away for a while but returned recently.
-=-=-
The World Trade Center towers MUST rise again,
at least as tall as before...or terror has triumphed.
Louis Epstein <le@main.lekno.ws> wrote:
Winston <wbe@ubeblock.psr.com.invalid> wrote:
Louis Epstein <le@main.lekno.ws> wrote:
In the last few days,however,the swap space (I have 19G)
running out has not only caused a failure of rust or
a termination of synth,but made other applications terminate
("unable to recover memory") and logged me out,and when
I log in I discover that programs such as httpd and sendmail
are no longer running...
and later added:
And killing the rust build when 7% of swap space is in use
still leads to an unable to get swap space message just
as killing it when 99.9% of swap space is in use before
the shutdown.
Wild stab in the dark ...
If you're using tmpfs and it's an unknown tmpfs bug,
perhaps disable tmpfs and see if that makes a difference?
-WBE
Will have to consider that,tmpfs is certainly
shown on a pre-shutdown df with lots of synth material.
The problem had gone away for a while but returned recently.
Now it may indeed by gone...after a rust config-option
deletion.
-=-=-
The World Trade Center towers MUST rise again,
at least as tall as before...or terror has triumphed.
Now it seems to have returned...
Louis Epstein <le@main.lekno.ws> wrote:
Now it seems to have returned...
Is there a description of your setup somewhere? I didn't see it.
bob prohaska <bp@www.zefox.net> wrote:
Louis Epstein <le@main.lekno.ws> wrote:
Now it seems to have returned...
Is there a description of your setup somewhere? I didn't see it.
What details do you need?
I'm running 13.2-RELEASE-p4 on an Intel i7-3770K.
8G DDR3 RAM and 19G swap that gets swallowed by these crashes.
In the last few days,however,the swap space (I have 19G)
running out has not only caused a failure of rust or
a termination of synth,but made other applications terminate
("unable to recover memory") and logged me out,and when
I log in I discover that programs such as httpd and sendmail
are no longer running...
And killing the rust build when 7% of swap space is in use
still leads to an unable to get swap space message just
as killing it when 99.9% of swap space is in use before
the shutdown.
Wild stab in the dark ...
If you're using tmpfs and it's an unknown tmpfs bug,
perhaps disable tmpfs and see if that makes a difference?
Will have to consider that,tmpfs is certainly
shown on a pre-shutdown df with lots of synth material.
The problem had gone away for a while but returned recently.
Louis Epstein <le@main.lekno.ws> wrote:
In the last few days,however,the swap space (I have 19G)
running out has not only caused a failure of rust or
a termination of synth,but made other applications terminate
("unable to recover memory") and logged me out,and when
I log in I discover that programs such as httpd and sendmail
are no longer running...
and later added:
And killing the rust build when 7% of swap space is in use
still leads to an unable to get swap space message just
as killing it when 99.9% of swap space is in use before
the shutdown.
to which I replied:
Wild stab in the dark ...
If you're using tmpfs and it's an unknown tmpfs bug,
perhaps disable tmpfs and see if that makes a difference?
Louis Epstein <le@main.lekno.ws> then replied:
Will have to consider that,tmpfs is certainly
shown on a pre-shutdown df with lots of synth material.
The problem had gone away for a while but returned recently.
Did you ever try the experiment of disabling tmpfs to see if
that helps?
FWIW, your issue sounds similar to the title of bug 219399
from 2017 which is still Open:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=219399
In that bug, though, the problem was seen with an early AMD Ryzen
processor, and (skimming through the discussion), it seems like
the problem was a hardware problem where temperature caused
various RAM and CPU errors.
-WBE
Did you ever try the experiment of disabling tmpfs to see if
that helps?
Given that using tmpfs is supposed to speed things up and
the synth builds already take hours and hours when the
batch is large and the swap-swallows prevent using the
computer for anything else,I was cautious about this.
Is there a way to dismount the /tmpfs entries of df
that are there when I kill synth before the swap-swallow
has crashed the system, so I know if they play a part in
the failure of the shutdown program to execute completely?
shutdown -h and shutdown -p both end up in a circle.
As part of the thread, I asked:
Did you ever try the experiment of disabling tmpfs to see if
that helps?
to which Louis Epstein <le@main.lekno.ws> responded:
Given that using tmpfs is supposed to speed things up and
the synth builds already take hours and hours when the
batch is large and the swap-swallows prevent using the
computer for anything else,I was cautious about this.
Is there a way to dismount the /tmpfs entries of df
that are there when I kill synth before the swap-swallow
has crashed the system, so I know if they play a part in
the failure of the shutdown program to execute completely?
shutdown -h and shutdown -p both end up in a circle.
I'm not confident I understand what you're asking, but
perhaps one or more of the following address your question.
While I would think that disabling tmpfs is the only
definitive way to test whether it's a tmpfs bug, if you
don't wish to disable tmpfs:
* You could run pstat or swapinfo from time to see how much
swap space is in use.
* If you have >1 swap area, you could consider suspending
the build and then using swapoff to remove one of them.
It'll take a while to move the active content to the
remaining swap space. Then use swapon to re-add it.
Hopefully, tmpfs handles swap space addition and removal
well.
See if the reported swap space used changes significantly.
If so, maybe it's a tmpfs bug (not freeing no-longer-used
swap space).
* protect(1) can be use to keep processes from being killed
when swap space runs out, but, as the man page warns,
that can also lead to the system hanging instead of
crashing.
* Maybe check temperatures from time to time while the
build is running, too.
* Periodic logging: maybe write a trivial shell loop to run
every few seconds or so to write the output of swapinfo,
df -h /tmp, temperatures, etc. to a file which you could
examine after rebooting.
If swap space doesn't grow gradually, it starts to look
more like a bit error or really-hard-to-find bug.
In the case of bug 219399, small increases in voltages
reduced, but didn't eliminate, bit errors caused by heat.
HTH,
-WBE
Winston <wbe@ubeblock.psr.com.invalid> wrote:
As part of the thread, I asked:
Did you ever try the experiment of disabling tmpfs to see if
that helps?
to which Louis Epstein <le@main.lekno.ws> responded:
Given that using tmpfs is supposed to speed things up and
the synth builds already take hours and hours when the
batch is large and the swap-swallows prevent using the
computer for anything else,I was cautious about this.
Is there a way to dismount the /tmpfs entries of df
that are there when I kill synth before the swap-swallow
has crashed the system, so I know if they play a part in
the failure of the shutdown program to execute completely?
shutdown -h and shutdown -p both end up in a circle.
I'm not confident I understand what you're asking, but
perhaps one or more of the following address your question.
While I would think that disabling tmpfs is the only
definitive way to test whether it's a tmpfs bug, if you
don't wish to disable tmpfs:
* You could run pstat or swapinfo from time to see how much
swap space is in use.
The "dashboard" on synth includes a steadily updating
percentage of how much swap space is in use...I also
run top on another terminal session at times to see
the figure there.
* If you have >1 swap area, you could consider suspending
the build and then using swapoff to remove one of them.
I don't think synth takes an interrupt.
It'll take a while to move the active content to the
remaining swap space. Then use swapon to re-add it.
Hopefully, tmpfs handles swap space addition and removal
well.
See if the reported swap space used changes significantly.
If so, maybe it's a tmpfs bug (not freeing no-longer-used
swap space).
* protect(1) can be use to keep processes from being killed
when swap space runs out, but, as the man page warns,
that can also lead to the system hanging instead of
crashing.
But would this solve what makes shutdown unable to complete
after I have killed synth between errors but before the
whole system has gone down?
* Maybe check temperatures from time to time while the
build is running, too.
* Periodic logging: maybe write a trivial shell loop to run
every few seconds or so to write the output of swapinfo,
df -h /tmp, temperatures, etc. to a file which you could
examine after rebooting.
If swap space doesn't grow gradually, it starts to look
more like a bit error or really-hard-to-find bug.
In the case of bug 219399, small increases in voltages
reduced, but didn't eliminate, bit errors caused by heat.
HTH,
-WBE
-=-=-
The World Trade Center towers MUST rise again,
at least as tall as before...or terror has triumphed.
* You could run pstat or swapinfo from time to see how much
swap space is in use.
The "dashboard" on synth includes a steadily updating
percentage of how much swap space is in use...I also
run top on another terminal session at times to see
the figure there.
Further experimentation has revealed that while synth is
still running before the system crashes, doing a kill -9
rather than plain kill saves the shutdown program from
becoming unable to complete, and doing umount -a before a
shutdown after it HAS crashed doesn't save the shutdown
program or remove the /tmpfs entries from df,but it DOES
avoid all the fsck'ing of the drives after the power-cycling.
And killing the rust build when 7% of swap space is in
use still leads to an unable to get swap space message
just as killing it when 99.9% of swap space is in use
before the shutdown.
I'm still mostly just taking stabs in the dark, but here goes ...
As part of this thread, I previously suggested:
* You could run pstat or swapinfo from time to see how much
swap space is in use.
to which Louis Epstein <le@main.lekno.ws> replied:
The "dashboard" on synth includes a steadily updating
percentage of how much swap space is in use...I also
run top on another terminal session at times to see
the figure there.
I misread that for a long time. I read "steadily decreasing"
rather than what you actually wrote: steadily updating.
So, is swap space decreasing as the build progresses? If so,
does df /tmp steadily increase to match? If it does, check
what's filling up /tmp(fs).
Louis followed up what I quoted above by adding:
Further experimentation has revealed that while synth is
still running before the system crashes, doing a kill -9
rather than plain kill saves the shutdown program from
becoming unable to complete, and doing umount -a before a
shutdown after it HAS crashed doesn't save the shutdown
program or remove the /tmpfs entries from df,but it DOES
avoid all the fsck'ing of the drives after the power-cycling.
Since part of shutting down is syncing the vnodes, if it
doesn't get that far, it's not surprising that a fsck could be
needed. That's probably why the umount helps. I'm a little
surprised that umount /tmp(fs) wouldn't remove the df entry,
though.
After rebooting, can you resume the build from the point of the
crash (and does it complete?), or do you start over?
I note that /sbin/shutdown runs /usr/bin/wall and execs
/sbin/reboot, /sbin/halt, etc. In those cases where your
system is out of swap space, that may contribute to shutdown
not working, ... Even a subroutine call that needs heap space
and can't get it could cause trouble.
But, several articles ago, you wrote:
And killing the rust build when 7% of swap space is in
use still leads to an unable to get swap space message
just as killing it when 99.9% of swap space is in use
before the shutdown.
So then it's not strictly an out-of-swap-space issue.
This from the "man tmpfs" might be relevant:
Metadata, including the directory content, is never
swapped out by the current implementation. Keep this
in mind when planning the mount limits, especially when
expecting to place many small files on a tmpfs mount.
I can imagine a case where a smaller RAM is full from the
combination of locked memory, active processes, and lots of
tmpfs file metadata, even when there's unused swap space. As
such a situation develops, I would think there'd be a *lot* of
swapping of whichever processes are still swappable, making
things really slow.
Trying to help, but not confident I'm succeeding, :)
-WBE
I just got rust to build overnight after increasing the
swap space on the machine.
So far the increased swap space has done the immediate need,
get lang/rust to complete the build process.
Louis Epstein <le@main.lekno.ws> wrote:
I just got rust to build overnight after increasing the
swap space on the machine.
So far the increased swap space has done the immediate need,
get lang/rust to complete the build process.
So, just to wrap things up ... Once the build was finished,
the swap space all reappeared or was accounted for by tmpfs
files?
-WBE
I just got rust to build overnight after increasing the
swap space on the machine.
So far the increased swap space has done the immediate need,
get lang/rust to complete the build process.
So, just to wrap things up ... Once the build was finished,
the swap space all reappeared or was accounted for by tmpfs
files?
Synth runs now continue past the building of rust,
and complete the rebuilding of the local repository.
Louis Epstein <le@main.lekno.ws> wrote:
I just got rust to build overnight after increasing the
swap space on the machine.
So far the increased swap space has done the immediate need,
get lang/rust to complete the build process.
prompting me to ask:
So, just to wrap things up ... Once the build was finished,
the swap space all reappeared or was accounted for by tmpfs
files?
Louis Epstein <le@main.lekno.ws> replied:
Synth runs now continue past the building of rust,
and complete the rebuilding of the local repository.
OK, but my question (above) was more about whether you saw
swap + tmpfs space lost along the way (bug), or was
swap + tmpfs space all back to normal afterward (no bug)?
-WBE
| Sysop: | Keyop |
|---|---|
| Location: | Huddersfield, West Yorkshire, UK |
| Users: | 355 |
| Nodes: | 16 (2 / 14) |
| Uptime: | 05:42:57 |
| Calls: | 7,656 |
| Files: | 12,812 |
| Messages: | 5,700,890 |