• [gentoo-user] NAS suggestions for home user

    From Mark Knecht@21:1/5 to All on Fri Oct 1 00:00:01 2021
    I'm in the study phase on some sort of NAS backup system for my home. I'll
    be building (or buying) a new desktop/server machine in the next few months
    - my i980 machine doesn't have the right instruction set for running
    Tensorflow anymore - so I want to figure out backups before I put together
    the new machine.

    I have about $400 credit at NewEgg and would like to keep my additional
    costs on the NAS down to about $100-$200. I expect the new machine to
    probably be 4TB RAID but it would be quite a while before that gets filled
    up.

    What do I need to be thinking about? Do I need 8TB in the NAS box? Are 2 or
    4 bay NAS boxes generally RAID? I do backups today about once a week. I do
    not currently keep any snapshots. I just back up files so over time the
    backup carries a lot of stuff that I don't need anymore and I have to go
    clean it up if I run out of space.

    The NAS would be turned off most of the time. If I need to use it I'll just power it up.

    I've been looking at a few software solutions based on another thread here
    but so far nothing has excited me so recommendations for what makes sense
    for high reliability home backup is of great interest, especially if it
    helps me somehow in cleaning up the backups after deleting stuff on my main machine on purpose and therefore not needing it on the backup.

    Thanks in advance for any ideas.

    Cheers,
    Mark

    <div dir="ltr">I&#39;m in the study phase on some sort of NAS backup system for my home. I&#39;ll be building (or buying) a new desktop/server machine in the next few months - my i980 machine doesn&#39;t have the right instruction set for running
    Tensorflow anymore - so I want to figure out backups before I put together the new machine.<div><br></div><div>I have about $400 credit at NewEgg and would like to keep my additional costs on the NAS down to about $100-$200. I expect the new machine to
    probably be 4TB RAID but it would be quite a while before that gets filled up.</div><div><br></div><div>What do I need to be thinking about? Do I need 8TB in the NAS box? Are 2 or 4 bay NAS boxes generally RAID? I do backups today about once a week. I do
    not currently keep any snapshots. I just back up files so over time the backup carries a lot of stuff that I don&#39;t need anymore and I have to go clean it up if I run out of space.</div><div><br></div><div>The NAS would be turned off most of the time.
    If I need to use it I&#39;ll just power it up.</div><div><br></div><div>I&#39;ve been looking at a few software solutions based on another thread here but so far nothing has excited me so recommendations for what makes sense for high reliability home
    backup is of great interest, especially if it helps me somehow in cleaning up the backups after deleting stuff on my main machine on purpose and therefore not needing it on the backup.</div><div><br></div><div>Thanks in advance for any ideas.</div><div><
    </div><div>Cheers,</div><div>Mark</div></div>

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Rich Freeman@21:1/5 to markknecht@gmail.com on Fri Oct 1 00:40:04 2021
    On Thu, Sep 30, 2021 at 5:50 PM Mark Knecht <markknecht@gmail.com> wrote:

    I've been looking at a few software solutions based on another thread here but so far nothing has excited me so recommendations for what makes sense for high reliability home backup is of great interest, especially if it helps me somehow in cleaning up
    the backups after deleting stuff on my main machine on purpose and therefore not needing it on the backup.

    It is probably going to stretch your budget, but you should look into distributed filesystems like CephFS/MooseFS/LizardFS. The latter two
    at least should run fine on hardware like a Pi4 if you aren't doing
    too much IOPS. You could actually run them on as little as a single
    host, which would probably be cheaper than a commercial NAS though
    really no better. The big advantages is that you aren't limited by
    the drive capacity of a single host, and you have redundancy at the
    host level. That is, you can pull a plug on any host and the whole
    thing just keeps running.

    Again, I realize this isn't exactly what you asked for. IMO this is
    the long-term direction storage is trending towards though. I can't
    vouch for the hardware requirements for Ceph, but that can scale
    incredibly well and is pretty-much the future. I've heard it isn't so
    great on just a few hosts though.

    --
    Rich

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Wols Lists@21:1/5 to Mark Knecht on Fri Oct 1 13:10:01 2021
    On 30/09/21 22:50, Mark Knecht wrote:
    I'm in the study phase on some sort of NAS backup system for my home.
    I'll be building (or buying) a new desktop/server machine in the next
    few months - my i980 machine doesn't have the right instruction set for running Tensorflow anymore - so I want to figure out backups before I
    put together the new machine.

    Okay, your NAS box is going to be shut down most of the time? Why not re-purpose your old box as the NAS? If it was running a lot, I'd be
    concerned about the power consumption,but here it sounds okay.

    I have about $400 credit at NewEgg and would like to keep my additional
    costs on the NAS down to about $100-$200. I expect the new machine to probably be 4TB RAID but it would be quite a while before that gets
    filled up.

    If that includes buying new drives, you're pushing your luck here ... a
    new "suitable for raid" 4TB drive will probably blow that budget in one hit!

    What do I need to be thinking about? Do I need 8TB in the NAS box?

    Ummm... I'd be inclined to buy a single 8TB drive. It's not much more
    than the 4TB - I'd suggest suggest Seagate Ironwolf. The Tosh N300s
    aren't bad afaik, but I've no personal experience. You could get a WD
    Red Pro, but do NOT get a plain Red, and given the way they've mucked
    people about I personally wouldn't get a WD.

    Are 2 or 4 bay NAS boxes generally RAID? I do backups today about once a week.
    I do not currently keep any snapshots.

    2-bay will be mirrored, 4-bay is normally raid-5

    I just back up files so over time
    the backup carries a lot of stuff that I don't need anymore and I have
    to go clean it up if I run out of space.

    Get that 8TB Ironwolf/N300, format it btrfs (it pains me to say that
    :-), and set up your backup to do an "in place" rsync then snapshot the
    volume.

    That way, each backup will be an incremental, but the snapshot will give
    you a full backup. Think of it like git, it will store the current
    state, with diffs so you can checkout any previous state if you want.

    Just MAKE SURE the drive never actually fills up - btrfs has an
    appalling rep for surviving a disk full with snapshots.

    If you need more space you can then think about getting a second (and
    third, and fourth) 8TB drive and going to raid or just adding them to
    the btrfs. Again, just remember that, at the moment, btrfs parity raid
    also has a pretty appalling rap. Mirroring is fine.

    The NAS would be turned off most of the time. If I need to use it I'll
    just power it up.

    I've been looking at a few software solutions based on another thread
    here but so far nothing has excited me so recommendations for what makes sense for high reliability home backup is of great interest, especially
    if it helps me somehow in cleaning up the backups after deleting stuff
    on my main machine on purpose and therefore not needing it on the backup.

    Thanks in advance for any ideas.

    Take a read of the raid wiki - https://raid.wiki.kernel.org/index.php/Linux_Raid

    In particular the overview section. My current (new) setup is ext4, over
    lvm, over mirror raid, over dm-integrity, over two IronWolves. I'm
    planning to add a Barracuda and go raid-5, but not without a solid
    backup first !!!

    Make sure you read the warnings, and the timeout mismatch section! It
    talks about Barracudas, and WD Reds!

    On a different topic, I don't know anything about it but some file
    systems do block level de-duplication - zfs I believe for one. That way,
    you can create your backups in dated directories, and the filesystem
    will store each duplicated file only once, without you having to worry
    apart from maybe having to trigger a manual de-duplicate once in a
    while. But with 8TB that won't be often.

    Cheers,
    Wol

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Mark Knecht@21:1/5 to rich0@gentoo.org on Fri Oct 1 18:00:01 2021
    On Thu, Sep 30, 2021 at 3:35 PM Rich Freeman <rich0@gentoo.org> wrote:

    On Thu, Sep 30, 2021 at 5:50 PM Mark Knecht <markknecht@gmail.com> wrote:

    I've been looking at a few software solutions based on another thread
    here but so far nothing has excited me so recommendations for what makes
    sense for high reliability home backup is of great interest, especially if
    it helps me somehow in cleaning up the backups after deleting stuff on my
    main machine on purpose and therefore not needing it on the backup.

    It is probably going to stretch your budget, but you should look into distributed filesystems like CephFS/MooseFS/LizardFS. The latter two
    at least should run fine on hardware like a Pi4 if you aren't doing
    too much IOPS. You could actually run them on as little as a single
    host, which would probably be cheaper than a commercial NAS though
    really no better. The big advantages is that you aren't limited by
    the drive capacity of a single host, and you have redundancy at the
    host level. That is, you can pull a plug on any host and the whole
    thing just keeps running.

    Again, I realize this isn't exactly what you asked for. IMO this is
    the long-term direction storage is trending towards though. I can't
    vouch for the hardware requirements for Ceph, but that can scale
    incredibly well and is pretty-much the future. I've heard it isn't so
    great on just a few hosts though.

    --
    Rich

    Rich,
    The idea of using a Raspberry Pi hadn't entered my mind. I took
    a very quick look at CepfFS and immediately felt a bit swamped
    but I will look into all the file systems and get back to you if that's ok.

    If the new machine just needs a standard backup plan then IOPS
    should be pretty low. If the backup system is low power then I could
    just run it Friday afternoons and incremental backups wouldn't be
    a problem at all.

    Physically I am sort of looking for a stand along chassis, or that's
    the picture in my head anyway. If I can find a Pi4 in a chassis with
    power supply and 1-2 drive bays at the right cost that could possibly
    make sense for home. The other issue going that way is how much
    time do I spend managing the OS on this backup box? Best answer
    is zero.

    Thanks for the ideas.

    Cheers,
    Mark

    <div dir="ltr"><br><br>On Thu, Sep 30, 2021 at 3:35 PM Rich Freeman &lt;<a href="mailto:rich0@gentoo.org">rich0@gentoo.org</a>&gt; wrote:<br>&gt;<br>&gt; On Thu, Sep 30, 2021 at 5:50 PM Mark Knecht &lt;<a href="mailto:markknecht@gmail.com">markknecht@
    gmail.com</a>&gt; wrote:<br>&gt; &gt;<br>&gt; &gt; I&#39;ve been looking at a few software solutions based on another thread here but so far nothing has excited me so recommendations for what makes sense for high reliability home backup is of great
    interest, especially if it helps me somehow in cleaning up the backups after deleting stuff on my main machine on purpose and therefore not needing it on the backup.<br>&gt;<br>&gt; It is probably going to stretch your budget, but you should look into<br>
    &gt; distributed filesystems like CephFS/MooseFS/LizardFS.  The latter two<br>&gt; at least should run fine on hardware like a Pi4 if you aren&#39;t doing<br>&gt; too much IOPS.  You could actually run them on as little as a single<br>&gt; host, which
    would probably be cheaper than a commercial NAS though<br>&gt; really no better.  The big advantages is that you aren&#39;t limited by<br>&gt; the drive capacity of a single host, and you have redundancy at the<br>&gt; host level.  That is, you can
    pull a plug on any host and the whole<br>&gt; thing just keeps running.<br>&gt;<br>&gt; Again, I realize this isn&#39;t exactly what you asked for.  IMO this is<br>&gt; the long-term direction storage is trending towards though.  I can&#39;t<br>&gt;
    vouch for the hardware requirements for Ceph, but that can scale<br>&gt; incredibly well and is pretty-much the future.  I&#39;ve heard it isn&#39;t so<br>&gt; great on just a few hosts though.<br>&gt;<br>&gt; --<br>&gt; Rich<div><br></div><div>Rich,</
    <div>   The idea of using a Raspberry Pi hadn&#39;t entered my mind. I took </div><div>a very quick look at CepfFS and immediately felt a bit swamped </div><div>but I will look into all the file systems and get back to you if that&#39;s ok.</div><
    <br></div><div>   If the new machine just needs a standard backup plan then IOPS</div><div>should be pretty low. If the backup system is low power then I could</div><div>just run it Friday afternoons and incremental backups wouldn&#39;t be</div><
    a problem at all.</div><div><br></div><div>   Physically I am sort of looking for a stand along chassis, or that&#39;s </div><div>the picture in my head anyway. If I can find a Pi4 in a chassis with</div><div>power supply and 1-2 drive bays at the
    right cost that could possibly</div><div>make sense for home. The other issue going that way is how much</div><div>time do I spend managing the OS on this backup box? Best answer</div><div>is zero.</div><div><br></div><div>   Thanks for the ideas.</div>
    <div><br></div><div>Cheers,</div><div>Mark</div></div>

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Wols Lists@21:1/5 to Mark Knecht on Fri Oct 1 18:40:02 2021
    On 01/10/2021 17:08, Mark Knecht wrote:
    This old machine is now about 10 years old. It's a big Cooler Master
    case, 6 or 8 removable drive bays, heavy. It collects dust and
    sometimes the fans are quite noisy. If I was going this direction
    I think I'd have to tear the whole thing down, redo the case fans at
    least. If I did all that then I think I'd use it for the new machine,
    but you do have a point.

    Okay, get your new disk drive, stick it in your old server, put btrfs on
    it, learn to play with the backups etc. You can hoover out the inside at
    the same time, and possibly replace the fans - they might be noisy
    because the bearings are shot.

    There's no reason why your backup drive has to be in a different machine
    (other than the physical safety of it being separate), so play with it
    as part of your current machine. Learn btrfs, learn rsync, learn all
    that stuff.

    (Your case sounds a bit like the N300 I've just bought. I want to put a
    whole load of 1TB drives in it as a raid testbed - you might have
    noticed my name on the raid wiki :-)

    The other thing, if you are interested and happy with just one disk not
    raid, look at getting one of these HOST MANAGED shingled drives, and use
    a log-structured file system. Again, I don't know anything about these
    other than what they are, but for backups it should be a good and
    reasonably cheap solution.

    If you want to go down the pi route, I think you can get little cases,
    and I've got a USB thingy into which you can plug two drives. But at
    about £30-40 each, that's $100 for hardware over and above your drive.
    I'd recycle the old machine :-)

    Cheers,
    Wol

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Mark Knecht@21:1/5 to antlists@youngman.org.uk on Fri Oct 1 18:20:02 2021
    On Fri, Oct 1, 2021 at 4:00 AM Wols Lists <antlists@youngman.org.uk> wrote:

    On 30/09/21 22:50, Mark Knecht wrote:
    I'm in the study phase on some sort of NAS backup system for my home.
    I'll be building (or buying) a new desktop/server machine in the next
    few months - my i980 machine doesn't have the right instruction set for running Tensorflow anymore - so I want to figure out backups before I
    put together the new machine.

    Okay, your NAS box is going to be shut down most of the time? Why not re-purpose your old box as the NAS? If it was running a lot, I'd be
    concerned about the power consumption,but here it sounds okay.

    The number 1 reason would be I want the backup system running
    _before_ I decommission this old box. I'd like to spend some time
    using the old box, running backups, learning the software, etc, before
    the new machine comes online.

    Still it's a possibility. I could live on my old laptop for a few months,
    take the current server machine, decommission, rebuild it, bring up
    the NAS/backup solution/whatever it is. It's possible.

    This old machine is now about 10 years old. It's a big Cooler Master
    case, 6 or 8 removable drive bays, heavy. It collects dust and
    sometimes the fans are quite noisy. If I was going this direction
    I think I'd have to tear the whole thing down, redo the case fans at
    least. If I did all that then I think I'd use it for the new machine,
    but you do have a point.



    I have about $400 credit at NewEgg and would like to keep my additional costs on the NAS down to about $100-$200. I expect the new machine to probably be 4TB RAID but it would be quite a while before that gets
    filled up.

    If that includes buying new drives, you're pushing your luck here ... a
    new "suitable for raid" 4TB drive will probably blow that budget in one
    hit!

    Currently my thinking about hardware was pretty modest:

    https://www.newegg.com/synology-ds220/p/N82E16822108743

    and a couple of 4TB drives at $100+ would be in the $500-$500 range.

    I know NOTHING about boxes like that in terms of OS or maintaining
    them.

    Will respond to the rest of the post when I get back from lunch in
    Phoenix. 2 hour drive to meet up with the bass player in one of
    my college progressive rock bands. (It's tough being in your
    60's...)

    Thanks much,
    Mark

    <div dir="ltr"><br><br>On Fri, Oct 1, 2021 at 4:00 AM Wols Lists &lt;<a href="mailto:antlists@youngman.org.uk">antlists@youngman.org.uk</a>&gt; wrote:<br>&gt;<br>&gt; On 30/09/21 22:50, Mark Knecht wrote:<br>&gt; &gt; I&#39;m in the study phase on some
    sort of NAS backup system for my home.<br>&gt; &gt; I&#39;ll be building (or buying) a new desktop/server machine in the next<br>&gt; &gt; few months - my i980 machine doesn&#39;t have the right instruction set for<br>&gt; &gt; running Tensorflow anymore
    - so I want to figure out backups before I<br>&gt; &gt; put together the new machine.<br>&gt;<br>&gt; Okay, your NAS box is going to be shut down most of the time? Why not<br>&gt; re-purpose your old box as the NAS? If it was running a lot, I&#39;d be<br>
    &gt; concerned about the power consumption,but here it sounds okay.<div><br></div><div>The number 1 reason would be I want the backup system running</div><div>_before_ I decommission this old box. I&#39;d like to spend some time</div><div>using the old
    box, running backups, learning the software, etc, before</div><div>the new machine comes online.</div><div><br></div><div>Still it&#39;s a possibility. I could live on my old laptop for a few months,</div><div>take the current server machine,
    decommission, rebuild it, bring up</div><div>the NAS/backup solution/whatever it is. It&#39;s possible.</div><div><br></div><div>This old machine is now about 10 years old. It&#39;s a big Cooler Master</div><div>case, 6 or 8 removable drive bays, heavy.
    It collects dust and </div><div>sometimes the fans are quite noisy. If I was going this direction</div><div>I think I&#39;d have to tear the whole thing down, redo the case fans at</div><div>least. If I did all that then I think I&#39;d use it for the
    new machine, </div><div>but you do have a point.</div><div><br></div><div><br>&gt; &gt;<br>&gt; &gt; I have about $400 credit at NewEgg and would like to keep my additional<br>&gt; &gt; costs on the NAS down to about $100-$200. I expect the new machine
    to<br>&gt; &gt; probably be 4TB RAID but it would be quite a while before that gets<br>&gt; &gt; filled up.<br>&gt;<br>&gt; If that includes buying new drives, you&#39;re pushing your luck here ... a<br>&gt; new &quot;suitable for raid&quot; 4TB drive
    will probably blow that budget in one hit!</div><div><br></div><div>Currently my thinking about hardware was pretty modest:</div><div><br></div><div><a href="https://www.newegg.com/synology-ds220/p/N82E16822108743">https://www.newegg.com/synology-ds220/p/
    N82E16822108743</a><br></div><div><br></div><div>and a couple of 4TB drives at $100+ would be in the $500-$500 range.</div><div><br></div><div>I know NOTHING about boxes like that in terms of OS or maintaining</div><div>them.</div><div><br>Will respond
    to the rest of the post when I get back from lunch in </div><div>Phoenix. 2 hour drive to meet up with the bass player in one of</div><div>my college progressive rock bands. (It&#39;s tough being in your </div><div>60&#39;s...)</div><div><br></div><div>
    Thanks much,</div><div>Mark</div></div>

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Mark Knecht@21:1/5 to antlists@youngman.org.uk on Tue Dec 7 22:00:01 2021
    On Fri, Oct 1, 2021 at 9:39 AM Wols Lists <antlists@youngman.org.uk> wrote:

    On 01/10/2021 17:08, Mark Knecht wrote:
    This old machine is now about 10 years old. It's a big Cooler Master
    case, 6 or 8 removable drive bays, heavy. It collects dust and
    sometimes the fans are quite noisy. If I was going this direction
    I think I'd have to tear the whole thing down, redo the case fans at
    least. If I did all that then I think I'd use it for the new machine,
    but you do have a point.

    Okay, get your new disk drive, stick it in your old server, put btrfs on
    it, learn to play with the backups etc. You can hoover out the inside at
    the same time, and possibly replace the fans - they might be noisy
    because the bearings are shot.

    There's no reason why your backup drive has to be in a different machine (other than the physical safety of it being separate), so play with it
    as part of your current machine. Learn btrfs, learn rsync, learn all
    that stuff.

    (Your case sounds a bit like the N300 I've just bought. I want to put a
    whole load of 1TB drives in it as a raid testbed - you might have
    noticed my name on the raid wiki :-)

    The other thing, if you are interested and happy with just one disk not
    raid, look at getting one of these HOST MANAGED shingled drives, and use
    a log-structured file system. Again, I don't know anything about these
    other than what they are, but for backups it should be a good and
    reasonably cheap solution.

    If you want to go down the pi route, I think you can get little cases,
    and I've got a USB thingy into which you can plug two drives. But at
    about £30-40 each, that's $100 for hardware over and above your drive.
    I'd recycle the old machine :-)

    Cheers,
    Wol


    So here I am reporting back after a couple of months of not working on
    this task.

    I dug around in the garage and found an old i5 Clarksdale machine that literally hadn't been turned on since we sold a house back in about
    2013. Unboxed it, cleaned it up a bit, took out all the old hard
    drives, the CD and the floppy and put in 2 500GB WD Enterprise drives
    I had sitting here from a previous upgrade. Darned if the machine
    didn't boot right up from a FreeNAS (now TrueNAS Core) flash drive. I
    installed the OS to a second USB thumb drive, booted the machine,
    created a 500GB ZFS mirrored pool, created a user directory and 3
    hours later I'm doing backups. So far it's done 30GB of about 450GB
    and just seems to be humming along nicely. CPU usage is only about 5%
    most of the time. The processor is only 2 cores, 4 threads, but most
    of the time it's only using about 5% CPU.

    No idea how stable it will be but the computer itself was always a
    good machine 10 years ago so I'll keep my fingers crossed and see how
    it goes.

    I'll be adding a SSD front end cache to the storage pool later this
    week and will likely move the OS to something internal (SSD or maybe
    an old HDD) as I don't like the idea of depending on a USB boot.

    Cheers,
    Mark

    Anyone looking for some similar solution so far I really couldn't be
    happier with how easy it was to get this up and running.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)