Hi all, a bit of a conundrum; maybe I'm missing something.

RPI4 (4Gb) running 13.1, no humungous tasks running on it, just a small
mail, web and file- server for the family.

But the swap is all eaten up, and there seem to be no swapped processes
to account:

root@kirk:~ # uptime
9:02AM up 20 days, 21:21, 2 users, load averages: 1.17, 0.87, 0.51

root@kirk:~ # swapinfo -h
Device Size Used Avail Capacity
/dev/da0s2b 2.0G 1.7G 267M 87%

root@kirk:~ # ps axwju | grep W
root 12 0 0 0 0 WL - 196:45.89 [intr]
23.9 0.0 0 368 14Jul22
root 19009 18964 19008 53800 2 S+ 2 0:00.01 grep W
0.0 0.0 12672 1900 09:01


It completely ran out of swap last night, so I'll reboot. But what's
chewing it all up? I've had much the same system running on i386 for
years without seeing this problem.

TIA.


--
Mike Scott
Harlow, England

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Mike Scott <usenet.16@scottsonline.org.uk.invalid> wrote:

RPI4 (4Gb) running 13.1, no humungous tasks running on it, just a small
mail, web and file- server for the family.

But the swap is all eaten up, and there seem to be no swapped processes
to account:

root@kirk:~ # uptime
9:02AM up 20 days, 21:21, 2 users, load averages: 1.17, 0.87, 0.51

root@kirk:~ # swapinfo -h
Device Size Used Avail Capacity
/dev/da0s2b 2.0G 1.7G 267M 87%

root@kirk:~ # ps axwju | grep W
root 12 0 0 0 0 WL - 196:45.89 [intr]
23.9 0.0 0 368 14Jul22
root 19009 18964 19008 53800 2 S+ 2 0:00.01 grep W
0.0 0.0 12672 1900 09:01

It completely ran out of swap last night, so I'll reboot. But what's
chewing it all up? I've had much the same system running on i386 for
years without seeing this problem.

Does top have anything interesting to say?

This question might be worth putting on the freebsd-arm mailing list: https://lists.freebsd.org/subscription/freebsd-arm

I read the list and don't recall seeing anything similar, nor have
I experienced anything similar on a 13.1 Pi3 or -current Pi4.

HTH,

bob prohaska

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 07/08/2022 21:17, bob prohaska wrote:

Mike Scott <usenet.16@scottsonline.org.uk.invalid> wrote:

RPI4 (4Gb) running 13.1, no humungous tasks running on it, just a small
mail, web and file- server for the family.

But the swap is all eaten up, and there seem to be no swapped processes
to account:

root@kirk:~ # uptime
9:02AM up 20 days, 21:21, 2 users, load averages: 1.17, 0.87, 0.51

root@kirk:~ # swapinfo -h
Device Size Used Avail Capacity
/dev/da0s2b 2.0G 1.7G 267M 87%

root@kirk:~ # ps axwju | grep W
root 12 0 0 0 0 WL - 196:45.89 [intr]
23.9 0.0 0 368 14Jul22
root 19009 18964 19008 53800 2 S+ 2 0:00.01 grep W
0.0 0.0 12672 1900 09:01


It completely ran out of swap last night, so I'll reboot. But what's
chewing it all up? I've had much the same system running on i386 for
years without seeing this problem.

Does top have anything interesting to say?

This question might be worth putting on the freebsd-arm mailing list: https://lists.freebsd.org/subscription/freebsd-arm

I read the list and don't recall seeing anything similar, nor have
I experienced anything similar on a 13.1 Pi3 or -current Pi4.

HTH,

bob prohaska

Thanks for the comment. I'm wary about the discussion lists: too much is
over my head, but I'll take a look, thanks.

I've just this moment experimented a bit. After posting the above, the
system's been running for a few days.

top says:

last pid: 74530; load averages: 1.21, 1.14, 0.80 up 3+23:34:36
08:41:25
108 processes: 2 running, 106 sleeping
CPU: 0.4% user, 0.0% nice, 0.3% system, 0.1% interrupt, 99.3% idle
Mem: 138M Active, 1052M Inact, 1278M Laundry, 580M Wired, 260M Buf, 809M
Free
Swap: 2048M Total, 179M Used, 1869M Free, 8% Inuse


I just did swapoff -a (took a few seconds) and top said:

last pid: 74538; load averages: 0.71, 1.03, 0.78 up 3+23:35:25
08:42:14
97 processes: 1 running, 96 sleeping
CPU: 0.4% user, 0.0% nice, 0.3% system, 0.1% interrupt, 99.3% idle
Mem: 128M Active, 679M Inact, 1447M Laundry, 490M Wired, 171M Buf, 1114M
Free


I've never known what "Laundry" might be :-{, but I would observe that
swap usage went from 179M to none, while Laundry increased by /roughly/
the same amount. Also, the number of "sleeping" process dropped a lot.

(Although as the system's been running a dump script while doing this, I
can't be sure about the figures)

The swap usage just seems to increase monotonically with time. If
there's enough memory, a swapoff/swapon fixes it for a while. If the
swap gets too full, it's can't all be brought into main memory, so the
swapoff won't work. It's not a truly busy machine - 4Gb memory should be
way more than it needs for operation (used to all run with just 1Gb
(i386) a couple of years ago without issue).

I'm tempted to run swapoff/swapon overnight, but then I'd never find
what was wrong :-}

Thanks.





--
Mike Scott
Harlow, England

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Mike Scott <usenet.16@scottsonline.org.uk.invalid> wrote:

On 07/08/2022 21:17, bob prohaska wrote:

Mike Scott <usenet.16@scottsonline.org.uk.invalid> wrote:

RPI4 (4Gb) running 13.1, no humungous tasks running on it, just a small
mail, web and file- server for the family.

But the swap is all eaten up, and there seem to be no swapped processes
to account:

root@kirk:~ # uptime
9:02AM up 20 days, 21:21, 2 users, load averages: 1.17, 0.87, 0.51

root@kirk:~ # swapinfo -h
Device Size Used Avail Capacity
/dev/da0s2b 2.0G 1.7G 267M 87%

root@kirk:~ # ps axwju | grep W
root 12 0 0 0 0 WL - 196:45.89 [intr]
23.9 0.0 0 368 14Jul22
root 19009 18964 19008 53800 2 S+ 2 0:00.01 grep W
0.0 0.0 12672 1900 09:01


It completely ran out of swap last night, so I'll reboot. But what's
chewing it all up? I've had much the same system running on i386 for
years without seeing this problem.

Does top have anything interesting to say?

This question might be worth putting on the freebsd-arm mailing list:
https://lists.freebsd.org/subscription/freebsd-arm

I read the list and don't recall seeing anything similar, nor have
I experienced anything similar on a 13.1 Pi3 or -current Pi4.

HTH,

bob prohaska

Thanks for the comment. I'm wary about the discussion lists: too much is
over my head, but I'll take a look, thanks.

If you've found a memory leak it'll be of interest to the developers.
Almost certainly you'll be asked for the output of uname -a and a
description of the storage hardware and the software running.

I've just this moment experimented a bit. After posting the above, the system's been running for a few days.

top says:

last pid: 74530; load averages: 1.21, 1.14, 0.80 up 3+23:34:36
08:41:25
108 processes: 2 running, 106 sleeping
CPU: 0.4% user, 0.0% nice, 0.3% system, 0.1% interrupt, 99.3% idle
Mem: 138M Active, 1052M Inact, 1278M Laundry, 580M Wired, 260M Buf, 809M
Free
Swap: 2048M Total, 179M Used, 1869M Free, 8% Inuse

I just did swapoff -a (took a few seconds) and top said:

last pid: 74538; load averages: 0.71, 1.03, 0.78 up 3+23:35:25
08:42:14
97 processes: 1 running, 96 sleeping
CPU: 0.4% user, 0.0% nice, 0.3% system, 0.1% interrupt, 99.3% idle
Mem: 128M Active, 679M Inact, 1447M Laundry, 490M Wired, 171M Buf, 1114M
Free

I've never known what "Laundry" might be :-{, but I would observe that
swap usage went from 179M to none, while Laundry increased by /roughly/
the same amount. Also, the number of "sleeping" process dropped a lot.

AIUI, laundry is memory (virtual or physical) that holds data which will
never be used because the originating process is gone. There some amount
of work in putting it back in the free pool, so the system doesn't bother
so long as it does not need to.

(Although as the system's been running a dump script while doing this, I can't be sure about the figures)

The swap usage just seems to increase monotonically with time. If
there's enough memory, a swapoff/swapon fixes it for a while. If the
swap gets too full, it's can't all be brought into main memory, so the swapoff won't work. It's not a truly busy machine - 4Gb memory should be
way more than it needs for operation (used to all run with just 1Gb
(i386) a couple of years ago without issue).

Times have changed considerably 8-) 64-bit does not help.

I'm tempted to run swapoff/swapon overnight, but then I'd never find
what was wrong :-}

Top reports (at least on my machines) SIZE and RES. I think size is the
total memory used by a process and RES is how much is actually resident
in phyiscal RAM, with the difference being (I think) in swap. Those values might bear watching.

hth,

bob prohaska

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

In article <tcrvmj$11jmd$1@dont-email.me>,
bob prohaska <bp@www.zefox.net> wrote:

If you've found a memory leak it'll be of interest to the developers.
Almost certainly you'll be asked for the output of uname -a and a
description of the storage hardware and the software running.

Is the OP using tmpfs?
# mount -t tmpfs

Top reports (at least on my machines) SIZE and RES. I think size is the
total memory used by a process and RES is how much is actually resident
in phyiscal RAM, with the difference being (I think) in swap. Those values >might bear watching.

<URL:https://docs.freebsd.org/en/books/handbook/basics/#basics-processes>
| Below the header is a series of columns containing similar
| information to the output from ps(1), such as the PID, username,
| amount of CPU time, and the command that started the process. By
| default, top(1) also displays the amount of memory space taken by
| the process. This is split into two columns: one for total size and
| one for resident size. Total size is how much memory the application
| has needed and the resident size is how much it is actually using
| now.

I think top(1) output would be useful when swapinfo(8) is reporting
high usage.
# top -d 1 -o res -w

John
groenveld@acm.org

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 09/08/2022 01:02, John D Groenveld wrote:

(Thanks to you and to Bob for replying)

In article <tcrvmj$11jmd$1@dont-email.me>,
bob prohaska <bp@www.zefox.net> wrote:

If you've found a memory leak it'll be of interest to the developers.
Almost certainly you'll be asked for the output of uname -a and a
description of the storage hardware and the software running.

Is the OP using tmpfs?
# mount -t tmpfs

Yes - and I'd forgotten that so thanks for the reminder. But it's not
the cause, as it's limited to 50Mb, and currently using only 4.5k

Top reports (at least on my machines) SIZE and RES. I think size is the

total memory used by a process and RES is how much is actually resident
in phyiscal RAM, with the difference being (I think) in swap. Those

values

might bear watching.

....>
I think top(1) output would be useful when swapinfo(8) is reporting
high usage.
# top -d 1 -o res -w

That just returns nothing for me. However,
# top -w -o swap 10000 | cat

returns a list of 106 processes, all with 0B swap usage listed. Yet
swapinfo says:

Device Size Used Avail Capacity
/dev/da0s2b 2.0G 246M 1.8G 12%


(wrote that yesterday, did a swapoff/swapon to clear things out, and
left a monitor script running swapinfo overnight. Then today.....)

At ~3.05 am, something started gobbling up swap space: it cranks up from
0 at 03:05 to 129M by 03:20. Then at 04:03 it started increasing again
reaching 372Mb by 04:16.

That's what it was this morning when I did a 'top -w 10000 |cat', which
tells me every process has 0B swap usage.

Yet, AFAICS crontab is free of anything running at those times apart
from periodic.daily (03:01); maybe I need to shift the time of that to
check.

I do have 2 jails running, for web and sendmail servers. stopping the
mail jail reduces usage to 143Mb, stopping web jail takes it down to
94Mb (restarting them keeps the reduced level). Yet clamd (which takes,
I've found, a horrendous amount of memory) I stopped yesterday without affecting the swap usage one jot.


Presumably there's something big (not necessarily jailed) that runs that
forces other processes into swap before it exits, but if that's the
case, everything left ought to fit into main memory, which it clearly
doesn't as swap usage just goes monotonically upwards to the point where swapoff can't work.

I guess I'll have to change the monitor script to do a ps when swapinfo
says swap is on the up.

Tedious.



--
Mike Scott
Harlow, England

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Mike Scott <usenet.16@scottsonline.org.uk.invalid> wrote:

Presumably there's something big (not necessarily jailed) that runs that forces other processes into swap before it exits, but if that's the
case, everything left ought to fit into main memory, which it clearly
doesn't as swap usage just goes monotonically upwards to the point where swapoff can't work.

Are you using a custom kernel, perchance?

[risking carrying coals to Newcastle]
Here's a script prepared with the help of Mark Millard which was used to investigate problems during self-hosting on Pi2 and Pi3 systems. It'll
have to be hacked to look at what you want, but might be a starting point.

I found it not-too-hard to grep the swapinfo lines, sort them and then
search buildworld.log to find what was happening when swap use soared.
You'd probably want to replace the sysctl with ps -aux or something like
it (apparently I was looking for power supply problems at last use).

hth,

bob prohaska

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 11/08/2022 02:31, bob prohaska wrote:

Mike Scott <usenet.16@scottsonline.org.uk.invalid> wrote:

Presumably there's something big (not necessarily jailed) that runs that
forces other processes into swap before it exits, but if that's the
case, everything left ought to fit into main memory, which it clearly
doesn't as swap usage just goes monotonically upwards to the point where
swapoff can't work.

Are you using a custom kernel, perchance?

Thanks for comments. No. Can't face the thought of building one on the
Pi :-| Works fine as is though.

[risking carrying coals to Newcastle]
Here's a script prepared with the help of Mark Millard which was used to investigate problems during self-hosting on Pi2 and Pi3 systems. It'll
have to be hacked to look at what you want, but might be a starting point.

Seems to have been left off :(

However, here's my offering give or take line wrap:

#!/bin/sh

LOG="/var/tmp/swap.topinfo"
DELAY=60

echo >>$LOG
echo "starting...." >>$LOG

lastswap="XXXX"

while true ; do
swap=`swapinfo -h | grep /dev | awk '{print $3}'`
if [ "$swap" != "$lastswap" ] ; then
echo "==================" >>$LOG

echo -n `date +"%Y-%m-%dT%H:%M:%S "` >>$LOG
swapinfo -h | grep /dev >>$LOG

top -wS 10000 >>$LOG # make terminal wide enough!!!!!!!! truncates if not
echo >>$LOG

ps axwwud >>$LOG
echo >>$LOG

lastswap="$swap"
fi

sleep $DELAY
done


I had it running last night - or rather managed to accidentally run 2
copies using the same log file -- and top had truncated the last column
anyway. Very muddly - but it's nevertheless clear that /everything/ is
listed as 0B in the swap used column, even when swapinfo shows the space rocketed to >400Mb used (later dropped to 190Mb). Nothing obvious amiss otherwise.

I've restarted (one copy this time!) and swap used forced back to 0.
Have to await what happens tonight.


--
Mike Scott
Harlow, England

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Mike Scott <usenet.16@scottsonline.org.uk.invalid> wrote:

On 11/08/2022 02:31, bob prohaska wrote:

Are you using a custom kernel, perchance?

Thanks for comments. No. Can't face the thought of building one on the
Pi :-| Works fine as is though.

Kernels aren't bad. Clang can be difficult.

If you've got a standard kernel using all its swap the folks
at freebsd.org would probably want to know it.

[risking carrying coals to Newcastle]

Seems to have been left off :(

Here's another try:
bob@www:~/fbsd % more swapscript
#!/bin/sh
while true
sysctl hw.regulator.5v0.min_uvolt ; do vmstat ; gstat -abd -I 10s ; date ; swapinfo ; tail \
-n 2 /var/log/messages ; netstat -m | grep "mbuf clusters" ; tail -n 1 \
/usr/src/buildworld.log
done

The idea is to log things of interest (swap use) with things like swap use numbers and (in this case) make output lines that can be searched for in
the log file. Grep and sort find the interesting swap numbers, and a search
in more on the log files finds what the system was doing at the time.

It's tedious.....

However, here's my offering give or take line wrap:

#!/bin/sh

LOG="/var/tmp/swap.topinfo"
DELAY=60

echo >>$LOG
echo "starting...." >>$LOG

lastswap="XXXX"

while true ; do
swap=`swapinfo -h | grep /dev | awk '{print $3}'`
if [ "$swap" != "$lastswap" ] ; then
echo "==================" >>$LOG

Are you looking for changes in swap use here?

echo -n `date +"%Y-%m-%dT%H:%M:%S "` >>$LOG
swapinfo -h | grep /dev >>$LOG

top -wS 10000 >>$LOG # make terminal wide enough!!!!!!!! truncates if not
echo >>$LOG

ps axwwud >>$LOG
echo >>$LOG

lastswap="$swap"
fi

sleep $DELAY
done

I had it running last night - or rather managed to accidentally run 2
copies using the same log file -- and top had truncated the last column anyway. Very muddly - but it's nevertheless clear that /everything/ is
listed as 0B in the swap used column, even when swapinfo shows the space rocketed to >400Mb used (later dropped to 190Mb). Nothing obvious amiss otherwise.

I don't claim to understand the intent of your script, but I'd suggest
not worrying if the output is readable, as in trying to find the problem. Rather make it a series of snapshots of the machine state over time.
Let the resulting file be big, with long lines. Then tease out the
times when swap use is high and and use the swap numbers as indexes
to the log file to see what's going on.

I can't help but wonder if the swap use is somehow coming from the kernel.

Apologies in advance if I'm missing your point! This does seem worth
a post or two on the freebsd mailing list(s)

bob prohaska

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Den 2022-08-04 skrev Mike Scott <usenet.16@scottsonline.org.uk.invalid>:

Hi all, a bit of a conundrum; maybe I'm missing something.

RPI4 (4Gb) running 13.1, no humungous tasks running on it, just a small
mail, web and file- server for the family.

But the swap is all eaten up, and there seem to be no swapped processes
to account:

root@kirk:~ # uptime
9:02AM up 20 days, 21:21, 2 users, load averages: 1.17, 0.87, 0.51

root@kirk:~ # swapinfo -h
Device Size Used Avail Capacity
/dev/da0s2b 2.0G 1.7G 267M 87%

root@kirk:~ # ps axwju | grep W
root 12 0 0 0 0 WL - 196:45.89 [intr]
23.9 0.0 0 368 14Jul22
root 19009 18964 19008 53800 2 S+ 2 0:00.01 grep W
0.0 0.0 12672 1900 09:01

It completely ran out of swap last night, so I'll reboot. But what's
chewing it all up? I've had much the same system running on i386 for
years without seeing this problem.

If you aren't seeing processes in top eating more and more memory, you
could try having a look at vmstat -m which will show how the kernel
has allocated heap memory. It could be something in the kernel being
the culprit.

TIA.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

In article <td2g5g$24nfl$1@dont-email.me>,
Mike Scott <usenet.16@scottsonline.org.uk.invalid> wrote:

Thanks for comments. No. Can't face the thought of building one on the
Pi :-| Works fine as is though.

Ok, have you updated to p1 as advised? <URL:https://www.freebsd.org/security/advisories/FreeBSD-SA-22:11.vm.asc>

Seems to have been left off :(

You have left off the full output of ps and top when your swapinfo
usage exceeds your preferred maximum.

John
groenveld@acm.org

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 12/08/2022 18:04, John D Groenveld wrote:

In article <td2g5g$24nfl$1@dont-email.me>,
Mike Scott <usenet.16@scottsonline.org.uk.invalid> wrote:

Thanks for comments. No. Can't face the thought of building one on the
Pi :-| Works fine as is though.

Ok, have you updated to p1 as advised? <URL:https://www.freebsd.org/security/advisories/FreeBSD-SA-22:11.vm.asc>

Seems to have been left off :(

You have left off the full output of ps and top when your swapinfo
usage exceeds your preferred maximum.

Yes, they seemed long, and I wasn't sure about the security of
publishing the entire contents, which would divulge exactly what
software my server runs.

But, meanwhile, I've been checking, changing and testing. It seems (and
IMBW) that clamd may be part of the problem. It's the biggest job on the
system at around 1.3G, and it turns out (which I didn't know before)
that it replicates itself when updating its virus tables, to ensure
continuity of service. That would chew up most of the available main memory.

So I've changed clamd's config to not do that, and the monotonic
increase in swap seems to have gone away. At least, it's been stable at
up to only ~280M after running for a few days.

FWIW I've now done a top and ps listing, then swapoff/swapon to clear
swap usage, and repeated the top/ps listing. Swap dropped from 284M to
0, but a diff on the two sets of listings shows nothing (that I can
spot) untoward.

I'll run for some days more and see what happens, then put clamd back to
being 'memory hungry' and see if the problem recurs. Can't think of
anything better ATM.

Thanks all.

John
groenveld@acm.org

--
Mike Scott
Harlow, England

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 12/08/2022 16:17, Andreas Kempe wrote:

Den 2022-08-04 skrev Mike Scott <usenet.16@scottsonline.org.uk.invalid>:

Hi all, a bit of a conundrum; maybe I'm missing something.

RPI4 (4Gb) running 13.1, no humungous tasks running on it, just a small
mail, web and file- server for the family.

But the swap is all eaten up, and there seem to be no swapped processes
to account:

.....

If you aren't seeing processes in top eating more and more memory, you
could try having a look at vmstat -m which will show how the kernel
has allocated heap memory. It could be something in the kernel being
the culprit.

TIA.

Thanks to all for comments.

I tinkered with the setup for clamav's update, first removing the
memory-greedy update stuff - which seemed to fix the problem - then
putting it back, which didn't bring the problem back.

So I have a system now the same as when the problem reared its head, but behaving reasonably. Swap use goes up to around 500Mb, then drops to
250Mb or thereabouts. There have been system updates in the interim, I
believe, so maybe something changed there.

I can't troubleshoot further as it's "gone away" and I remain bemused by
the original symptoms of programs that should fit main memory, but
don't, and nothing being marked as swapped out.

It's irritating that there seems no easy way of finding a list of stuff
using the swap file.

But at least things are now manageable, and I've learned a bit.

Thanks again.


--
Mike Scott
Harlow, England

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)