I have tried with NFS, and it is not great.
I have tried with NFS, and it is not great.
I have a setup in which my system has a limited amount of disk space,
but I am in a network with access to a server with all but infinite
amounts of disk space available. I need to use my system to carry out
lengthy software compilations, the kind that takes lots of disk space. I therefore have to access disk space from the server above over the 100
Mb LAN.
My question would be, What filesystem is likely to deliver the
best performance under the circumstances described?
I have tried with NFS, and it is not great. I will try next SSHFS
and WebDAV, but if anybody has experience with these, and other
networked filesystems, your feedback would be much appreciated.
On 25/11/2020 19:00, Harold Johanssen wrote:
I have a setup in which my system has a limited amount of disk
space, but I am in a network with access to a server with all but
infinite amounts of disk space available. I need to use my system to
carry out lengthy software compilations, the kind that takes lots of disk
space. I therefore have to access disk space from the server above over
the 100 Mb LAN.
That's going to limit your bandwidth to about 10 MB per second. Is that
your bottleneck?
My question would be, What filesystem is likely to deliver the
best performance under the circumstances described?
I have tried with NFS, and it is not great. I will try next SSHFS
and WebDAV, but if anybody has experience with these, and other networked
filesystems, your feedback would be much appreciated.
sshfs will encrypt your traffic (usually). If you have slow processors
to go with your slow LAN, that might be a bottleneck. But if your
processors are fast enough for that not to be a problem, then you can
use the "-C" flag to sshfs to get compression in the traffic - for compressible files, that will let you get higher bandwidth than the LAN supports directly.
But the obvious solution to increased speed is to join the 21st century
and upgrade to 1Gb Ethernet, if that is practical for you.
On Wed, 25 Nov 2020 18:00:36 +0000, Harold Johanssen wrote:
I have a setup in which my system has a limited amount of disk space,
but I am in a network with access to a server with all but infinite
amounts of disk space available. I need to use my system to carry out
lengthy software compilations, the kind that takes lots of disk space. I
therefore have to access disk space from the server above over the 100
Mb LAN.
My question would be, What filesystem is likely to deliver the
best performance under the circumstances described?
I have tried with NFS, and it is not great. I will try next SSHFS
and WebDAV, but if anybody has experience with these, and other
networked filesystems, your feedback would be much appreciated.
Thanks to all who replied. I have tried to do some fine tuning in
my NSF settings, as described in a number of places in the net. As for my network - well, I'd love to be able to have a 1 Gb network, but that is
not going to happen in the short term, unfortunately.
My intent was, when posing this question, to get feedback on the performance of different network filesystems with the networking
limitations that I described, from anybody who has had experience with comparable setups. This is an all-Linux network.
On Wed, 25 Nov 2020 18:00:36 +0000, Harold Johanssen wrote:
I have a setup in which my system has a limited amount of disk space,
but I am in a network with access to a server with all but infinite
amounts of disk space available. I need to use my system to carry out
lengthy software compilations, the kind that takes lots of disk space. I
therefore have to access disk space from the server above over the 100
Mb LAN.
My question would be, What filesystem is likely to deliver the
best performance under the circumstances described?
I have tried with NFS, and it is not great. I will try next SSHFS
and WebDAV, but if anybody has experience with these, and other
networked filesystems, your feedback would be much appreciated.
Thanks to all who replied. I have tried to do some fine tuning in
my NSF settings, as described in a number of places in the net. As for my network - well, I'd love to be able to have a 1 Gb network, but that is
not going to happen in the short term, unfortunately.
My intent was, when posing this question, to get feedback on the performance of different network filesystems with the networking
limitations that I described, from anybody who has had experience with comparable setups. This is an all-Linux network.
On 25/11/2020 21.26, Harold Johanssen wrote:...
My intent was, when posing this question, to get feedback on the
performance of different network filesystems with the networking
limitations that I described, from anybody who has had experience with
comparable setups. This is an all-Linux network.
Depends on what the actual load your software exerts. Large files? Small files? Random r/w access? That's the worst.
On 25/11/2020 21.26, Harold Johanssen wrote:
On Wed, 25 Nov 2020 18:00:36 +0000, Harold Johanssen wrote:
I have a setup in which my system has a limited amount of disk space,
but I am in a network with access to a server with all but infinite
amounts of disk space available. I need to use my system to carry out
lengthy software compilations, the kind that takes lots of disk space. I >>> therefore have to access disk space from the server above over the 100
Mb LAN.
My question would be, What filesystem is likely to deliver the
best performance under the circumstances described?
I have tried with NFS, and it is not great. I will try next SSHFS >>> and WebDAV, but if anybody has experience with these, and other
networked filesystems, your feedback would be much appreciated.
Thanks to all who replied. I have tried to do some fine tuning in
my NSF settings, as described in a number of places in the net. As for my
network - well, I'd love to be able to have a 1 Gb network, but that is
not going to happen in the short term, unfortunately.
My intent was, when posing this question, to get feedback on the
performance of different network filesystems with the networking
limitations that I described, from anybody who has had experience with
comparable setups. This is an all-Linux network.
Depends on what the actual load your software exerts. Large files? Small files? Random r/w access? That's the worst. Just write a file and send
it whole? The later could be cached locally and then sent slowly while
the process is doing calculations.
I would simply test it.
sshfs should be slower, as it uses encryption (more CPU load). But it
can also use compression. So, test it.
Is a double network interface an option? Just a wild idea. No, I have
never done that.
On Wed, 2020-11-25, Carlos E.R. wrote:
On 25/11/2020 21.26, Harold Johanssen wrote:...
My intent was, when posing this question, to get feedback on the
performance of different network filesystems with the networking
limitations that I described, from anybody who has had experience with
comparable setups. This is an all-Linux network.
Depends on what the actual load your software exerts. Large files? Small
files? Random r/w access? That's the worst.
But (I think) it could also be the scenario where you gain the most by offloading the actual disk I/O to another machine, which perhaps has
disks with better random access, and more cache.
(Not that I know anything about this -- I've always managed to avoid networked file systems. I use rsync or Git to make the data show up
in a local file system.)
/Jorgen
On 2020-11-26, Jorgen Grahn <grahn+nntp@snipabacken.se> wrote:
On Wed, 2020-11-25, Carlos E.R. wrote:
On 25/11/2020 21.26, Harold Johanssen wrote:...
My intent was, when posing this question, to get feedback on the
performance of different network filesystems with the networking
limitations that I described, from anybody who has had experience with >>>> comparable setups. This is an all-Linux network.
Depends on what the actual load your software exerts. Large files? Small >>> files? Random r/w access? That's the worst.
But (I think) it could also be the scenario where you gain the most by
offloading the actual disk I/O to another machine, which perhaps has
disks with better random access, and more cache.
network speeds are (always?) slower than disk bus access speeds. Thus i
wold expect [ local<-> network<->remote computer <-> disk] to be slower than [local computer <-> disk.]
Channel bundling can help on a good network setup, but it's unlikely to
help if he has cheapo or ancient switches, as he must have, since he is
on 100 Mb - they won't get enough parallel port-to-port transfers to
give significant improvements in throughput. (And if he has hubs,
rather than switches, it certainly won't help.)
On 11/26/20 12:22 AM, David Brown wrote:
Channel bundling can help on a good network setup, but it's unlikely to
help if he has cheapo or ancient switches, as he must have, since he is
on 100 Mb - they won't get enough parallel port-to-port transfers to
give significant improvements in throughput. (And if he has hubs,
rather than switches, it certainly won't help.)
It's a lot easier to upgrade a switch than to upgrade the wires in the
walls. One can have a modern switch and old wiring. I'd not be so sure
the person has "ancient" switches.
The odds of encountering someone with a hub is almost nil. I remember
when the first consumer switches hit the market and that was more than
20 years ago.
I suspect that in older networks, you might still find a few hubs -
equipment like this rarely gets replaced unless there is a problem, and
just like old wiring, there is often old equipment still in use.
On 11/28/20 7:22 AM, David Brown wrote:
I suspect that in older networks, you might still find a few hubs -
equipment like this rarely gets replaced unless there is a problem, and
just like old wiring, there is often old equipment still in use.
Nah... A switch can be had for $10 and have been in production for far
longer than hubs.
When was the last time you could even find a hub for
sale in a store? 2005?
On 28/11/2020 21:37, Johann Beretta wrote:
On 11/28/20 7:22 AM, David Brown wrote:
I suspect that in older networks, you might still find a few hubs -
equipment like this rarely gets replaced unless there is a problem, and
just like old wiring, there is often old equipment still in use.
Nah... A switch can be had for $10 and have been in production for far
longer than hubs.
The switches that existed when hubs were popular were massively
expensive. And no switch has ever cost $10 - excluding thrown-out second-hand devices that would not be used in a serious network.
This is a multi-part message in MIME format. --------------F2A5B81AEF67C13D33627186^^^^^^^^^
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
On 11/29/20 4:26 AM, David Brown wrote:
On 28/11/2020 21:37, Johann Beretta wrote:nd
On 11/28/20 7:22 AM, David Brown wrote:
I suspect that in older networks, you might still find a few hubs -
equipment like this rarely gets replaced unless there is a problem, a=
just like old wiring, there is often old equipment still in use.
Nah... A switch can be had for $10 and have been in production for far=
longer than hubs.=20
The switches that existed when hubs were popular were massively
expensive. And no switch has ever cost $10 - excluding thrown-out
^^^^^^^^^second-hand devices that would not be used in a serious network.
=20
Be careful when making absolute claims. You said no switch, including
thrown-out second hand devices....
https://www.ebay.com/itm/HP-Procurve-2610-48-J9088A-48-Port-10-100-1000-E= thernet-Switch/274592476654?hash=3Ditem3feefcadee:g:3sUAAOSw~kFfwUyq
$10 opening bid (which means it MIGHT sell for $10.)
On 11/29/20 4:26 AM, David Brown wrote:
On 28/11/2020 21:37, Johann Beretta wrote:
On 11/28/20 7:22 AM, David Brown wrote:
I suspect that in older networks, you might still find a few hubs -
equipment like this rarely gets replaced unless there is a problem, and >>>> just like old wiring, there is often old equipment still in use.
Nah... A switch can be had for $10 and have been in production for far
longer than hubs.
The switches that existed when hubs were popular were massively
expensive. And no switch has ever cost $10 - excluding thrown-out
second-hand devices that would not be used in a serious network.
Be careful when making absolute claims. You said no switch, including thrown-out second hand devices....
https://www.ebay.com/itm/HP-Procurve-2610-48-J9088A-48-Port-10-100-1000-Ethernet-Switch/274592476654?hash=item3feefcadee:g:3sUAAOSw~kFfwUyq
$10 opening bid (which means it MIGHT sell for $10.)
Yes - it's an old used, second-hand device, as I said.
But I was perhaps a bit quick in suggesting that no one would buy such a device for a serious network. When a new 48 port switch can be bought
for as little as about $100, I don't think companies would want to get critical infrastructure as left-overs found cheaply on Ebay. However,
it is entirely possible that someone would want this particular type of switch to replace an existing one, and buy it for the model rather than
for the price.
^^^^^^^^^=20
The switches that existed when hubs were popular were massively
expensive. And no switch has ever cost $10 - excluding thrown-out
^^^^^^^^^second-hand devices that would not be used in a serious network.
=20
Be careful when making absolute claims. You said no switch, including
thrown-out second hand devices....
https://www.ebay.com/itm/HP-Procurve-2610-48-J9088A-48-Port-10-100-1000-E= >> thernet-Switch/274592476654?hash=3Ditem3feefcadee:g:3sUAAOSw~kFfwUyq
$10 opening bid (which means it MIGHT sell for $10.)
On 11/29/20 1:02 PM, David Brown wrote:
Yes - it's an old used, second-hand device, as I said.
I know. I misread "excluding" as "including". Mea culpa
But I was perhaps a bit quick in suggesting that no one would buy such a
device for a serious network. When a new 48 port switch can be bought
for as little as about $100, I don't think companies would want to get
critical infrastructure as left-overs found cheaply on Ebay. However,
it is entirely possible that someone would want this particular type of
switch to replace an existing one, and buy it for the model rather than
for the price.
Agreed.
I have a setup in which my system has a limited amount of disk
space, but I am in a network with access to a server with all but
infinite amounts of disk space available. I need to use my system to
carry out lengthy software compilations, the kind that takes lots of disk space. I therefore have to access disk space from the server above over
the 100 Mb LAN.
My question would be, What filesystem is likely to deliver the
best performance under the circumstances described?
I have tried with NFS, and it is not great. I will try next SSHFS
and WebDAV, but if anybody has experience with these, and other networked filesystems, your feedback would be much appreciated.
Harold Johanssen wrote:
I have a setup in which my system has a limited amount of diskYou might hate me for this, sshfs
space, but I am in a network with access to a server with all but
infinite amounts of disk space available. I need to use my system to
carry out lengthy software compilations, the kind that takes lots of disk
space. I therefore have to access disk space from the server above over
the 100 Mb LAN.
My question would be, What filesystem is likely to deliver the
best performance under the circumstances described?
I have tried with NFS, and it is not great. I will try next SSHFS
and WebDAV, but if anybody has experience with these, and other networked
filesystems, your feedback would be much appreciated.
I have a setup in which my system has a limited amount of disk
space, but I am in a network with access to a server with all but
infinite amounts of disk space available. I need to use my system to
carry out lengthy software compilations, the kind that takes lots of disk space. I therefore have to access disk space from the server above over
the 100 Mb LAN.
My question would be, What filesystem is likely to deliver the
best performance under the circumstances described?
I have tried with NFS, and it is not great. I will try next SSHFS
and WebDAV, but if anybody has experience with these, and other networked filesystems, your feedback would be much appreciated.
| Sysop: | Keyop |
|---|---|
| Location: | Huddersfield, West Yorkshire, UK |
| Users: | 296 |
| Nodes: | 16 (2 / 14) |
| Uptime: | 58:03:58 |
| Calls: | 6,652 |
| Calls today: | 4 |
| Files: | 12,200 |
| Messages: | 5,331,127 |