I plan to buy another hard drive pretty soon. Next month is possible.
If there is nothing available that does what I want, is there a way to
use rsync and have it set to backup files starting with "a" through "k"
to one spot and then backup "l" through "z" to another? I could then
split the files into two parts.
Thoughts? Ideas?
Dale
:-) :-)
<div>Do you happen to have an old computer laying around? If so check </div><div>out TrueNAS Core.</div><div><br></div><div>You'd need one small OS drive and 2 backup drives - your current </div><div>7TB and one more to build the recommendedRAID. It compresses, </div><div>saves older revs of files if possible with snapshots. Is supports</div><div>NFS mounts for media/etc, chroot jails and lots of other stuff.<br></div><div><br></div><div>The default version has been FreeBSD based so I had
<div>number of things you are asking for.</div><div><br></div><div>HTH,</div><div>Mark</div></div>
On Sun, Aug 14, 2022 at 3:44 PM Dale <rdalek1967@gmail.com <mailto:rdalek1967@gmail.com>> wrote:
<SNIP>
Thoughts? Ideas?
Dale
:-) Â :-)
Do you happen to have an old computer laying around? If so checkÂ
out TrueNAS Core.
You'd need one small OS drive and 2 backup drives - your currentÂ
7TB and one more to build the recommended RAID. It compresses,Â
saves older revs of files if possible with snapshots. Is supports
NFS mounts for media/etc, chroot jails and lots of other stuff.
The default version has been FreeBSD based so I had someÂ
learning to do but I think there's now a Linux version.
It appears that possibly you have your backup drive in your
computer so it moves backups to a separate machine which
you can locate remotely so you're probably safer in terms of
a fire or some other catastrophic event.
If this appeals and you have the hardware you can buildÂ
a box and mess with it but it certainly does the minimalÂ
number of things you are asking for.
HTH,
Mark
Right now, my backups are external hard drives. I have a 3TB, a 6TB and
a 8TB that sadly is SMR. They are encrypted and after I do my backup updates, they go in a fire safe. I tend to do updates once a week,
usually while I'm doing OS updates.
Howdy,
With my new fiber internet, my poor disks are getting a work out, and
also filling up. First casualty, my backup disk. I have one directory that is . . . well . . . huge. It's about 7TBs or so. This is where it
is right now and it's still trying to pack in files.
/dev/mapper/8tb           7.3T 7.1T 201G 98% /mnt/8tb
Right now, I'm using rsync which doesn't compress files but does just
update things that have changed. I'd like to find some way, software
but maybe there is already a tool I'm unaware of, to compress data and
work a lot like rsync otherwise. I looked in app-backup and there is a
lot of options but not sure which fits best for what I want to do.
Again, backup a directory, compress and only update with changed or new files. Generally, it only adds files but sometimes a file gets replaced
as well. Same name but different size.
I was trying to go through the list in app-backup one by one but to be honest, most links included only go to github or something and usually doesn't tell anything about how it works or anything. Basically, as far
as seeing if it does what I want, it's useless. It sort of reminds me of quite a few USE flag descriptions.
I plan to buy another hard drive pretty soon. Next month is possible.
If there is nothing available that does what I want, is there a way to
use rsync and have it set to backup files starting with "a" through "k"
to one spot and then backup "l" through "z" to another? I could then
split the files into two parts. I use a script to do this now, if one
could call my little things scripts, so even a complicated command could work, just may need help figuring out the command.
Thoughts? Ideas?
Dale
:-)Â :-)
On Sun, Aug 14, 2022 at 6:44 PM Dale <rdalek1967@gmail.com> wrote:
Right now, I'm using rsync which doesn't compress files but does just update things that have changed. I'd like to find some way, software
but maybe there is already a tool I'm unaware of, to compress data and
work a lot like rsync otherwise.
So, how important is it that it work exactly like rsync?
I use duplicity, in part because I've been using it forever. Restic
seems to be a similar program most are using these days which I
haven't looked at super-closely but I'd look at that first if starting
out.
Duplicity uses librsync, so it backs up exactly the same data as rsync
would, except instead of replicating entire files, it creates streams
of data more like something like tar. So if you back up a million
small files you might get out 1-3 big files. It can compress and
encrypt the data as you wish. The downside is that you don't end up
with something that looks like your original files - you have to run
the restore process to extract them all back out. It is extremely space-efficient though - if 1 byte changes in the middle of a 10GB
file you'll end up just backing up maybe a kilobyte or so (whatever
the block size is), which is just like rsync.
Typically you rely on metadata to find files that change which is
fast, but I'm guessing you can tell these programs to do a deep scan
which of course requires reading the entire contents, and that will
discover anything that was modified without changing ctime/mtime.
The output files can be split to any size, and the index info (the
metadata) is separate from the raw data. If you're storing to offline/remote/cloud/whatever storage typically you keep the metadata
cached locally to speed retrieval and to figure out what files have
changed for incrementals. However, if the local cache isn't there
then it will fetch just the indexes from wherever it is stored
(they're small).
It has support for many cloud services - I store mine to AWS S3.
There are also some options that are a little closer to rsync like
rsnapshot and burp. Those don't store compressed (unless there is an
option for that or something), but they do let you rotate through
multiple backups and they'll set up hard links/etc so that they are de-duplicated. Of course hard links are at the file level so if 1
byte inside a file changes you'll end up with two full copies. It
will still only transfer a single block so the bandwidth requirements
are similar to rsync.
Howdy,
With my new fiber internet, my poor disks are getting a work out, and
also filling up. First casualty, my backup disk. I have one directory
that is . . . well . . . huge. It's about 7TBs or so. This is where it
is right now and it's still trying to pack in files.
/dev/mapper/8tb 7.3T 7.1T 201G 98% /mnt/8tb
Thoughts? Ideas?
On Sun, Aug 14, 2022 at 6:44 PM Dale <rdalek1967@gmail.com> wrote:
Right now, I'm using rsync which doesn't compress files but does justSo, how important is it that it work exactly like rsync?
update things that have changed. I'd like to find some way, software
but maybe there is already a tool I'm unaware of, to compress data and
work a lot like rsync otherwise.
I use duplicity, in part because I've been using it forever. Restic
seems to be a similar program most are using these days which I
haven't looked at super-closely but I'd look at that first if starting
out.
Duplicity uses librsync, so it backs up exactly the same data as rsync
would, except instead of replicating entire files, it creates streams
of data more like something like tar. So if you back up a million
small files you might get out 1-3 big files. It can compress and
encrypt the data as you wish. The downside is that you don't end up
with something that looks like your original files - you have to run
the restore process to extract them all back out. It is extremely space-efficient though - if 1 byte changes in the middle of a 10GB
file you'll end up just backing up maybe a kilobyte or so (whatever
the block size is), which is just like rsync.
Typically you rely on metadata to find files that change which is
fast, but I'm guessing you can tell these programs to do a deep scan
which of course requires reading the entire contents, and that will
discover anything that was modified without changing ctime/mtime.
The output files can be split to any size, and the index info (the
metadata) is separate from the raw data. If you're storing to offline/remote/cloud/whatever storage typically you keep the metadata
cached locally to speed retrieval and to figure out what files have
changed for incrementals. However, if the local cache isn't there
then it will fetch just the indexes from wherever it is stored
(they're small).
It has support for many cloud services - I store mine to AWS S3.
There are also some options that are a little closer to rsync like
rsnapshot and burp. Those don't store compressed (unless there is an
option for that or something), but they do let you rotate through
multiple backups and they'll set up hard links/etc so that they are de-duplicated. Of course hard links are at the file level so if 1
byte inside a file changes you'll end up with two full copies. It
will still only transfer a single block so the bandwidth requirements
are similar to rsync.
Hello,
On 8/14/22 18:44, Dale wrote:
Thoughts? Ideas?
You might be interested in borgbackup [1]
It takes delta backups and has de-duplication and compression to save
some space. It supports encryption too.
It's packaged in ::gentoo and you run it on whatever machine you want
to backup and give it its destination, it can be local or on a remote machine.
I've been using it for a while and it works well. I have it configured
on a crontab and it backups my files every night
[1] https://www.borgbackup.org/
On 15/8/22 06:44, Dale wrote:
Howdy,The questions you need to ask is how compressible is the data and how
With my new fiber internet, my poor disks are getting a work out, and
also filling up. First casualty, my backup disk. I have one directory
that is . . . well . . . huge. It's about 7TBs or so. This is where it >> is right now and it's still trying to pack in files.
/dev/mapper/8tb           7.3T 7.1T 201G 98% /mnt/8tb
Right now, I'm using rsync which doesn't compress files but does just
update things that have changed. I'd like to find some way, software
but maybe there is already a tool I'm unaware of, to compress data and
work a lot like rsync otherwise. I looked in app-backup and there is a
lot of options but not sure which fits best for what I want to do.
Again, backup a directory, compress and only update with changed or new
files. Generally, it only adds files but sometimes a file gets replaced
as well. Same name but different size.
I was trying to go through the list in app-backup one by one but to be
honest, most links included only go to github or something and usually
doesn't tell anything about how it works or anything. Basically, as far
as seeing if it does what I want, it's useless. It sort of reminds me of
quite a few USE flag descriptions.
I plan to buy another hard drive pretty soon. Next month is possible.
If there is nothing available that does what I want, is there a way to
use rsync and have it set to backup files starting with "a" through "k"
to one spot and then backup "l" through "z" to another? I could then
split the files into two parts. I use a script to do this now, if one
could call my little things scripts, so even a complicated command could
work, just may need help figuring out the command.
Thoughts? Ideas?
Dale
:-)Â :-)
much duplication is in there. Rsync's biggest disadvantage is it
doesn't keep history, so if you need to restore something from last
week you are SOL. Honestly, rsync is not a backup program and should
only be used the way you do for data that don't value as an rsync
archive is a disaster waiting to happen from a backup point of view.
Look into dirvish - uses hard links to keep files current but safe, is
easy to restore (looks like a exact copy so you cp the files back if needed. Downside is it hammers the hard disk and has no compression
so its only deduplication via history (my backups stabilised about 2x original size for ~2yrs of history - though you can use something like
btrfs which has filesystem level compression.
My current program is borgbackup which is very sophisticated in how it
stores data - its probably your best bet in fact. I am storing
literally tens of Tb of raw data on a 4Tb usb3 disk (going back years
and yes, I do restore regularly, and not just for disasters but for
space efficient long term storage I access only rarely.
e.g.:
A single host:
------------------------------------------------------------------------------
                      Original size     Compressed size Deduplicated
size
All archives:Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â 3.07 TBÂ Â Â Â Â Â Â Â Â Â Â Â Â 1.96 TBÂ Â Â Â Â Â Â Â Â Â Â
151.80 GB
                      Unique chunks        Total chunks
Chunk index:Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â 1026085Â Â Â Â Â Â Â Â Â Â Â Â 22285913
Then there is my offline storage - it backs up ~15 hosts (in repos
like the above) + data storage like 22 years of email etc. Each host
backs up to its own repo then the offline storage backs that up. The deduplicated size is the actual on disk size ... compression varies as
its whatever I used at the time the backup was taken ... currently I
have it set to "auto,zstd,11" but it can be mixed in the same repo (a
repo is a single backup set - you can nest repos which is what I do -
so ~45Tb stored on a 4Tb offline disk). One advantage of a system
like this is chunked data rarely changes, so its only the differences
that are backed up (read the borgbackup docs - interesting)
------------------------------------------------------------------------------
                      Original size     Compressed size Deduplicated
size
All archives:Â Â Â Â Â Â Â Â Â Â Â Â Â Â 28.69 TBÂ Â Â Â Â Â Â Â Â Â Â Â 28.69 TBÂ Â Â Â Â Â Â Â Â Â Â Â Â
3.81 TB
                      Unique chunks        Total chunks
Chunk index:
William Kenworthy wrote:
On 15/8/22 06:44, Dale wrote:
Howdy,The questions you need to ask is how compressible is the data and how
With my new fiber internet, my poor disks are getting a work out, and
also filling up. First casualty, my backup disk. I have one directory
that is . . . well . . . huge. It's about 7TBs or so. This is where it >> is right now and it's still trying to pack in files.
/dev/mapper/8tb 7.3T 7.1T 201G 98% /mnt/8tb
Right now, I'm using rsync which doesn't compress files but does just
update things that have changed. I'd like to find some way, software
but maybe there is already a tool I'm unaware of, to compress data and
work a lot like rsync otherwise. I looked in app-backup and there is a
lot of options but not sure which fits best for what I want to do.
Again, backup a directory, compress and only update with changed or new
files. Generally, it only adds files but sometimes a file gets replaced >> as well. Same name but different size.
I was trying to go through the list in app-backup one by one but to be
honest, most links included only go to github or something and usually
doesn't tell anything about how it works or anything. Basically, as far >> as seeing if it does what I want, it's useless. It sort of reminds me of >> quite a few USE flag descriptions.
I plan to buy another hard drive pretty soon. Next month is possible.
If there is nothing available that does what I want, is there a way to
use rsync and have it set to backup files starting with "a" through "k"
to one spot and then backup "l" through "z" to another? I could then
split the files into two parts. I use a script to do this now, if one
could call my little things scripts, so even a complicated command could >> work, just may need help figuring out the command.
Thoughts? Ideas?
Dale
:-) :-)
much duplication is in there. Rsync's biggest disadvantage is it
doesn't keep history, so if you need to restore something from last
week you are SOL. Honestly, rsync is not a backup program and should
only be used the way you do for data that don't value as an rsync
archive is a disaster waiting to happen from a backup point of view.
Look into dirvish - uses hard links to keep files current but safe, is
easy to restore (looks like a exact copy so you cp the files back if needed. Downside is it hammers the hard disk and has no compression
so its only deduplication via history (my backups stabilised about 2x original size for ~2yrs of history - though you can use something like btrfs which has filesystem level compression.
My current program is borgbackup which is very sophisticated in how it stores data - its probably your best bet in fact. I am storing
literally tens of Tb of raw data on a 4Tb usb3 disk (going back years
and yes, I do restore regularly, and not just for disasters but for
space efficient long term storage I access only rarely.
e.g.:
A single host:
------------------------------------------------------------------------------
Original size Compressed size Deduplicated
size
All archives: 3.07 TB 1.96 TB
151.80 GB
Unique chunks Total chunks
Chunk index: 1026085 22285913
Then there is my offline storage - it backs up ~15 hosts (in repos
like the above) + data storage like 22 years of email etc. Each host
backs up to its own repo then the offline storage backs that up. The deduplicated size is the actual on disk size ... compression varies as
its whatever I used at the time the backup was taken ... currently I
have it set to "auto,zstd,11" but it can be mixed in the same repo (a
repo is a single backup set - you can nest repos which is what I do -
so ~45Tb stored on a 4Tb offline disk). One advantage of a system
like this is chunked data rarely changes, so its only the differences
that are backed up (read the borgbackup docs - interesting)
------------------------------------------------------------------------------
Original size Compressed size Deduplicated
size
All archives: 28.69 TB 28.69 TB
3.81 TB
Unique chunks Total chunks
Chunk index:
For the particular drive in question, it is 99.99% videos. I don't want
to lose any quality but I'm not sure how much they can be compressed to
be honest. It could be they are already as compressed as they can be
without losing resolution etc. I've been lucky so far. I don't think
I've ever needed anything and did a backup losing what I lost on working copy. Example. I update a video only to find the newer copy is corrupt
and wanting the old one back. I've done it a time or two but I tend to
find that before I do backups. Still, it is a downside and something
I've thought about before. I figure when it does happen, it will be something hard to replace. Just letting the devil have his day. :-(
For that reason, I find the version type backups interesting. It is a
safer method. You can have a new file but also have a older file as
well just in case new file takes a bad turn. It is a interesting
thought. It's one not only I should consider but anyone really.
As I posted in another reply, I found a 10TB drive that should be here
by the time I do a fresh set of backups. This will give me more time to consider things. Have I said this before a while back??? :/
Rich Freeman wrote:
Duplicity uses librsync, so it backs up exactly the same data as rsync would, except instead of replicating entire files, it creates streams
of data more like something like tar. So if you back up a million
small files you might get out 1-3 big files. It can compress and
encrypt the data as you wish.
Duplicity sounds interesting except that I already have the drive
encrypted.
The reason I mentioned being like rsync, I don't want to rebuild a backup from scratch each time as that would be time consuming.
I have been using restic for a while, and although it does not do compression, there are a couple of nice things it does
Am Mon, 15 Aug 2022 12:50:37 +0200
schrieb Gerrit Kühn <gerrit.kuehn@aei.mpg.de>:
Being a happy restic user myself, I'd like to mention that compression is available meanwhile (https://restic.readthedocs.io/en/latest/047_tuning_backup_parameters.html #compression). However, the feature is rather new, I did not use it so
far.
https://forum.restic.net/t/compression-support-has-landed-in-master/4997
Just adding another link to the official announcement from earlier this
year.
cu
Gerrit
Hello,
On 8/14/22 18:44, Dale wrote:
Thoughts? Ideas?
You might be interested in borgbackup [1]
It takes delta backups and has de-duplication and compression to save
some space. It supports encryption too.
It's packaged in ::gentoo and you run it on whatever machine you want to backup and give it its destination, it can be local or on a remote machine.
Mark Knecht wrote:
On Sun, Aug 14, 2022 at 3:44 PM Dale <rdalek1967@gmail.com> wrote:<SNIP>
<SNIP>
Thoughts? Ideas?
Dale
:-) :-)
Do you happen to have an old computer laying around? If so check
out TrueNAS Core.
That may be a option later. I'm actually considering build a NAS butright now, costs are preventing that. I almost have enough that I could
This new fiber thing is going to take some getting used too. ;-)
I see lots of talk of NAS and zfs/btrfs and snapshots. IMO these are
NOT really great solutions for backup. NAS can work of course but it
is overkill for backup storage.
zfs would solve your problem of corruption, even without versioning.
You do a scrub at short intervals and at least you would know if the
file is corrupted. Of course, redundancy is better, such as mirroring
and backups take a very short time because sending from one zfs to
another it knows exactly what bytes to send.
The main issue I think you're going to have is having support for multi-volume backups if you need to be able to split a backup across
drives. The only thing I've found on Linux that does this is bacula,
and it is a royal pain that I'm embarrassed to even mention. If
somebody knows of another backup solution that can write the output to
disk (a filesystem, not /dev/rmt) and then pause to mount a new disk
when one fills up, I'm all ears.
I just hope this 10TB drive isn't a SMR. I googled around and the best
I could find is anything above 8TB is CMR. It's a WD101EDBZ-11B1DA0. I hope that is right. I'm not totally opposed to SMR even as a backup but
I'd rather not. The deal I found was for a pull and costs about $110 including shipping. I looked at a 14TB but my jaw dropped. $$$$$$$$
On 15/08/2022 11:11, Rich Freeman wrote:
I see lots of talk of NAS and zfs/btrfs and snapshots. IMO these are
NOT really great solutions for backup. NAS can work of course but it
is overkill for backup storage.
Do you want multiple *independent* backups, or do you want *incremental* backups so you can go back in time. It's nice to have both, but
snapshotting gives you full backups for the price of incremental.
Rich Freeman wrote:
On Sun, Aug 14, 2022 at 6:44 PM Dale <rdalek1967@gmail.com> wrote:
Right now, I'm using rsync which doesn't compress files but does just
update things that have changed. I'd like to find some way, software
but maybe there is already a tool I'm unaware of, to compress data and
work a lot like rsync otherwise.
So, how important is it that it work exactly like rsync?
I use duplicity, in part because I've been using it forever. Restic
seems to be a similar program most are using these days which I
haven't looked at super-closely but I'd look at that first if starting
out.
Duplicity uses librsync, so it backs up exactly the same data as rsync would, except instead of replicating entire files, it creates streams
of data more like something like tar. So if you back up a million
small files you might get out 1-3 big files. It can compress and
encrypt the data as you wish. The downside is that you don't end up
with something that looks like your original files - you have to run
the restore process to extract them all back out. It is extremely space-efficient though - if 1 byte changes in the middle of a 10GB
file you'll end up just backing up maybe a kilobyte or so (whatever
the block size is), which is just like rsync.
Typically you rely on metadata to find files that change which is
fast, but I'm guessing you can tell these programs to do a deep scan
which of course requires reading the entire contents, and that will discover anything that was modified without changing ctime/mtime.
The output files can be split to any size, and the index info (the metadata) is separate from the raw data. If you're storing to offline/remote/cloud/whatever storage typically you keep the metadata cached locally to speed retrieval and to figure out what files have
changed for incrementals. However, if the local cache isn't there
then it will fetch just the indexes from wherever it is stored
(they're small).
It has support for many cloud services - I store mine to AWS S3.
There are also some options that are a little closer to rsync like rsnapshot and burp. Those don't store compressed (unless there is an option for that or something), but they do let you rotate through
multiple backups and they'll set up hard links/etc so that they are de-duplicated. Of course hard links are at the file level so if 1
byte inside a file changes you'll end up with two full copies. It
will still only transfer a single block so the bandwidth requirements
are similar to rsync.
Duplicity sounds interesting except that I already have the drive
encrypted. Keep in mind, these are external drives that I hook up long enough to complete the backups then back in a fire safe they go. The
reason I mentioned being like rsync, I don't want to rebuild a backup
from scratch each time as that would be time consuming. I thought of
using Kbackup ages ago and it rebuilds from scratch each time but it
does have the option of compressing. That might work for small stuff
but not many TBs of it. Back in the early 90's, I remember using a
backup software that was incremental. It would only update files that changed and would do it over several floppy disks and compressed it as
well. Something like that nowadays is likely rare if it exists at all
since floppies are long dead. I either need to split my backup into two pieces or compress my data. That is why I mentioned if there is a way
to backup first part of alphabet in one command, switch disks and then
do second part of alphabet to another disk.
On Monday, August 15, 2022 12:44:11 AM CEST Dale wrote:
Howdy,<snipped>
With my new fiber internet, my poor disks are getting a work out, and
also filling up. First casualty, my backup disk. I have one directory
that is . . . well . . . huge. It's about 7TBs or so. This is where it
is right now and it's still trying to pack in files.
/dev/mapper/8tb 7.3T 7.1T 201G 98% /mnt/8tb
Thoughts? Ideas?Plenty, see below:
For backups to external disks, I would recommend having a look at "dar" :
$ eix -e dar
* app-backup/dar
Available versions: 2.7.6^t ~2.7.7^t {argon2 curl dar32 dar64 doc gcrypt
gpg lz4 lzo nls rsync threads xattr}
Homepage: http://dar.linux.free.fr/
Description: A full featured backup tool, aimed for disks
It's been around for a while and the developer is active and responds quite well to questions.
It supports compression (different compression methods), incremental backups (only need a catalogue of the previous backup for the incremental) and encryption.
The NAS options others mentioned would also work as they can compress data on disk and you'd only notice a delay in writing/reading (depending on the compression method used). I would recommend using one that uses ZFS on-disk as
it's more reliable and robust then BTRFS.
One option that comes available for you now that you are no longer limited to slow ADSL: Cloud backups.
I use Backblaze (B2) to store compressed backups that haven't been stored on tape to off-site locations.
But, you can also encrypt the backups locally and store the encrypted+compressed backupfiles on other cloud storage.
--
Joost
Actually, there still is a piece of software that does this:
" app-backup/dar "
You can tell it to split the backups into slices of a specific size.
J. Roeleveld wrote:
On Monday, August 15, 2022 12:44:11 AM CEST Dale wrote:
Howdy,
With my new fiber internet, my poor disks are getting a work out, and
also filling up. First casualty, my backup disk. I have one directory
that is . . . well . . . huge. It's about 7TBs or so. This is where it >> is right now and it's still trying to pack in files.
/dev/mapper/8tb 7.3T 7.1T 201G 98% /mnt/8tb
<snipped>
Thoughts? Ideas?
Plenty, see below:
For backups to external disks, I would recommend having a look at "dar" :
$ eix -e dar
* app-backup/dar
Available versions: 2.7.6^t ~2.7.7^t {argon2 curl dar32 dar64 doc
gcrypt
gpg lz4 lzo nls rsync threads xattr}
Homepage: http://dar.linux.free.fr/
Description: A full featured backup tool, aimed for disks
It's been around for a while and the developer is active and responds
quite
well to questions.
It supports compression (different compression methods), incremental backups (only need a catalogue of the previous backup for the
incremental) and encryption.
The NAS options others mentioned would also work as they can compress data on disk and you'd only notice a delay in writing/reading (depending on
the compression method used). I would recommend using one that uses ZFS on-disk as it's more reliable and robust then BTRFS.
One option that comes available for you now that you are no longer limited to slow ADSL: Cloud backups.
I use Backblaze (B2) to store compressed backups that haven't been stored on tape to off-site locations.
But, you can also encrypt the backups locally and store the encrypted+compressed backupfiles on other cloud storage.
--
Joost
Dar does sound interesting. It sounds a lot like what I used way back
in the 90's. I'm sure it is different software but could work on
floppies then like it does on USB sticks etc today. Same principal.
I looked into ZFS as well. Google helped me find a interesting page. I notice it is also used on some NAS setups as well. It seems to be
advanced and maintained well. It sounds a little like LVM but may have
more features, such as compression maybe? I haven't read that far yet.
I notice it mentions snapshots which LVM also uses.
Getting plenty of ideas. I just wish I had a separate building to put a
NAS in that would be safe and climate controlled. I got a out building
but it gets plenty hot in the summer. No A/C or anything. I only heat
it enough to prevent freezing but computers would likely like that anyway.
On Monday, 15 August 2022 11:58:14 BST Gerrit Kühn wrote:
Am Mon, 15 Aug 2022 12:50:37 +0200I think In Dale's use case compression is a solution seeking to address the problem of not enough storage space for backups, but it only makes sense if the data can be effectively and efficiently compressed. He mentioned 99.99% of his backup data is video. Video files are not particularly compressible, although small space savings can be achieved. For example using basic enough zst parameters '-19 --rsyncable -z' I got just a 1.6% file reduction:
schrieb Gerrit Kühn <gerrit.kuehn@aei.mpg.de>:
Being a happy restic user myself, I'd like to mention that compression is >>> available meanwhilehttps://forum.restic.net/t/compression-support-has-landed-in-master/4997
(https://restic.readthedocs.io/en/latest/047_tuning_backup_parameters.html >>> #compression). However, the feature is rather new, I did not use it so
far.
Just adding another link to the official announcement from earlier this
year.
cu
Gerrit
Frames Skips Compressed Uncompressed Ratio Check
1 0 88.9 MiB 90.3 MiB 1.016 XXH64
Even if compression delivers some small space saving, given Dale's new faster Internet link and local video storage tendencies, compression will only kick the can down the road. If these are not private or rare videos and remain available on public streaming platforms, perhaps local storage is no longer necessary?
Given my weird way of doing backups, rsync may be the best option.
Plus, easier to restore from as well since it just requires a copy
command, any of them will do.
On 15/08/2022 08:52, Dale wrote:
I just hope this 10TB drive isn't a SMR. I googled around and the best
I could find is anything above 8TB is CMR. It's a WD101EDBZ-11B1DA0. I >> hope that is right. I'm not totally opposed to SMR even as a backup but
I'd rather not. The deal I found was for a pull and costs about $110
including shipping. I looked at a 14TB but my jaw dropped. $$$$$$$$
Just done a search for you (it helps I know what I'm looking for), but
CMR it is ...
https://nascompares.com/answer/list-of-wd-cmr-and-smr-hard-drives-hdd/
I tend to go for OEM stuff, and know the names of the range I'm
looking for - Seagate Ironwolf, Toshiba N300 - about £170 for 8TB ...
Cheers,
Wol
On Sun, Aug 14, 2022 at 4:21 PM Dale <rdalek1967@gmail.com <mailto:rdalek1967@gmail.com>> wrote:
Mark Knecht wrote:
Do you happen to have an old computer laying around? If so check<SNIP>
out TrueNAS Core.
That may be a option later. I'm actually considering build a NASbut right now, costs are preventing that. I almost have enough that I
could build another computer. I have a mobo, memory, CPU and such. I think I only need a power supply and maybe a video card. Could use a
case for it to but could mount it on a wall somewhere. Good air flow.
 lol
<SNIP>
This new fiber thing is going to take some getting used too. Â ;-)
I experienced much of the same thing (more data) when my connection
got faster.Â
Expense of a separate system to build a NAS is always an issue and
you've received excellent guidance from other folks here about how to
do it locally so I think you're set.
A couple of things:
1) I didn't see mentioned so I will - the NAS, being on the network,
is connected over gigabit Ethernet in my case so backups are
significantly faster than using USB drives, or at least much faster
than my older USB. I get about 800mbit/Sec sustained transfers. Once
you get the main backup done the incremental ones are very fast. (Go
to the kitchen fast)Â
2) The NAS, when attached, is mounted over NFS as a directory and I
use rsync to do the transfers so it's all very familiar on the client
side. I think that's important to you today but likely won't be as
much of an issue if you get used to some new backup application.
3) Compression is done on the NAS and is transparent from the client
side. I can browse directories and retrieve individual files. As I
think you mentioned you won't get much compression - close to zero -
for movies but for my general data and VMs overall I'm getting about
40% so there's a big disk saving. Compute requirements are pretty low.
I bought a used MB with a 6th gen i5 Core processor with 4 cores and
it hardly works to do the compression.Â
Good luck with whatever you do.
Mark
Actually, you can with the "-p / --pause" option.
Also, as per the man-page, if you forget this, the process will simply inform you the target location is full and you can move slices away to a different location:
"
If the destination filesystem is too small to contain all the slices of the backup, the -p option (pausing before starting new slices) might be of interest. Else, in the case the filesystem is full, dar will suspend the operation, asking for the user to make free space, then continue its operation. To make free space, the only thing you cannot do is to touch the slice being written.
"
The pause-option will actually stop between slices and you can umount the target location and mount a different disk there.
This option has been around for a while.
As it is, I have several options. In a way, I wish I could tell rsync todo 1st half of alphabet to one drive and then with next command tell it to
</div>
Glad to know what I found was good info. I just wonder how long it will
be before even 10TB drives will be SMR. I also dread having to search
out a 14TB drive later. :/
As it is, I have several options. In a way, I wish I could tellrsync to do 1st half of alphabet to one drive and then with next
command tell it to do the 2nd half of alphabet. That would likely
split the data in half for each one. Â
You can do that, at least with a small kludge I'm sure. rsync supports excluding directories and file names. As an example:
rsync -avx --port=873
--exclude={.cache,.nv,'google-chrome*',DiskImages} /home/mark mark@truenas1:/mn
t/MyPool/mark/Backups/science/.
There's a test open (?? -n maybe ??) that allows you to see what it
would do.
I'm sure you can figure that part out. The above line is just in a
script file for me
HTH,
Mark
Mark Knecht wrote:
As it is, I have several options. In a way, I wish I could tellrsync to do 1st half of alphabet to one drive and then with next
command tell it to do the 2nd half of alphabet. That would likely
split the data in half for each one.
You can do that, at least with a small kludge I'm sure. rsync supports excluding directories and file names. As an example:
rsync -avx --port=873
--exclude={.cache,.nv,'google-chrome*',DiskImages} /home/mark mark@truenas1:/mn
t/MyPool/mark/Backups/science/.
There's a test open (?? -n maybe ??) that allows you to see what it
would do.
I'm sure you can figure that part out. The above line is just in a
script file for me
HTH,
Mark
In the directory I'm backing up, there is over 400 directories. That
would be a LOT of --exclude options. Also, I would have to adjust the exclude options each time I added a new directory, which can be several
a day sometimes. The word nightmare comes to mind. Loss of hair is
also a thought. :-D
I'm just glad I got a bigger hard drive coming. That's the easiest fix
at the moment.
Dale
:-) :-)
On Mon, Aug 15, 2022 at 3:41 PM Dale <rdalek1967@gmail.com> wrote:
Glad to know what I found was good info. I just wonder how long it willI think it will be a long time if ever, and here is why.
be before even 10TB drives will be SMR. I also dread having to search
out a 14TB drive later. :/
There are good reasons and bad reasons to use SMR. The reason you
would WANT to use SMR is that you have a task that is well-suited to
their limitations like backup or applications that can use log-style
storage. Ideally you'd want host-managed SMR for this. The benefit
is higher density for the cost, so you'd be doing it to get a drive
that is cheaper than it otherwise would be. However, these are all
things that would appeal to experts who really know what they're
doing.
The bad reason to use SMR is that you're a manufacturer trying to
squeeze out a bit more profit margin, not passing on the savings. In
this case you want to sell the drive to somebody who DOESN'T know what they're doing, and make it drive-managed.
This is why we've seen SMR in medium-sized drives and not big ones as
would be expected if you assumed it would be employed for the good
reasons. The only people buying 14TB hard drives are people who tend
to know what they're doing, which makes them less of a target for unscrupulous manufacturers. You wouldn't see them as much in small
drives as the return in capacity isn't as much. The medium sized
drives are big enough to get a return out of using SMR, but small
enough that suckers will be willing to buy them.
At least, that's my theory...
On Monday, August 15, 2022 9:07:41 PM CEST Dale wrote:
J. Roeleveld wrote:If it was during the 90's, then it wasn't. First version was released in 2002.
On Monday, August 15, 2022 12:44:11 AM CEST Dale wrote:Dar does sound interesting. It sounds a lot like what I used way back
Howdy,<snipped>
With my new fiber internet, my poor disks are getting a work out, and
also filling up. First casualty, my backup disk. I have one directory >>>> that is . . . well . . . huge. It's about 7TBs or so. This is where it >>>> is right now and it's still trying to pack in files.
/dev/mapper/8tb 7.3T 7.1T 201G 98% /mnt/8tb
Thoughts? Ideas?Plenty, see below:
For backups to external disks, I would recommend having a look at "dar" : >>> $ eix -e dar
* app-backup/dar
Available versions: 2.7.6^t ~2.7.7^t {argon2 curl dar32 dar64 doc
gcrypt
gpg lz4 lzo nls rsync threads xattr}
Homepage: http://dar.linux.free.fr/
Description: A full featured backup tool, aimed for disks
It's been around for a while and the developer is active and responds
quite
well to questions.
It supports compression (different compression methods), incremental
backups (only need a catalogue of the previous backup for the
incremental) and encryption.
The NAS options others mentioned would also work as they can compress data >>> on disk and you'd only notice a delay in writing/reading (depending on
the compression method used). I would recommend using one that uses ZFS
on-disk as it's more reliable and robust then BTRFS.
One option that comes available for you now that you are no longer limited >>> to slow ADSL: Cloud backups.
I use Backblaze (B2) to store compressed backups that haven't been stored >>> on tape to off-site locations.
But, you can also encrypt the backups locally and store the
encrypted+compressed backupfiles on other cloud storage.
--
Joost
in the 90's. I'm sure it is different software but could work on
floppies then like it does on USB sticks etc today. Same principal.
I looked into ZFS as well. Google helped me find a interesting page. IZFS does a lot more then just LVM+Ext4. But it really needs multiple disks for
notice it is also used on some NAS setups as well. It seems to be
advanced and maintained well. It sounds a little like LVM but may have
more features, such as compression maybe? I haven't read that far yet.
I notice it mentions snapshots which LVM also uses.
all the anti-corruption features as well.
Getting plenty of ideas. I just wish I had a separate building to put aIf you can keep it between optimal temperatures (and stable) the NAS should manage. There is NO need to keep it at 18C (like some places do).
NAS in that would be safe and climate controlled. I got a out building
but it gets plenty hot in the summer. No A/C or anything. I only heat
it enough to prevent freezing but computers would likely like that anyway.
Also, consider a small AC unit that only cools a small box big enough for the NAS. No need to cool an entire room.
--
Joost
Even if compression delivers some small space saving, given Dale's new
faster Internet link and local video storage tendencies, compression
will only kick the can down the road. If these are not private or rare videos and remain available on public streaming platforms, perhaps local storage is no longer necessary?
On Mon, Aug 15, 2022 at 1:17 PM Dale <rdalek1967@gmail.com <mailto:rdalek1967@gmail.com>> wrote:
Mark Knecht wrote:
As it is, I have several options. In a way, I wish I could tellrsync to do 1st half of alphabet to one drive and then with next
command tell it to do the 2nd half of alphabet. That would likely
split the data in half for each one. Â
You can do that, at least with a small kludge I'm sure. rsync supports excluding directories and file names. As an example:
rsync -avx --port=873
--exclude={.cache,.nv,'google-chrome*',DiskImages} /home/mark mark@truenas1:/mn
t/MyPool/mark/Backups/science/.
There's a test open (?? -n maybe ??) that allows you to see what it
would do.
I'm sure you can figure that part out. The above line is just in a
script file for me
HTH,
Mark
In the directory I'm backing up, there is over 400 directories. That would be a LOT of --exclude options. Also, I would have to adjust the exclude options each time I added a new directory, which can be several
a day sometimes. The word nightmare comes to mind. Loss of hair is
also a thought. Â :-D
I'm just glad I got a bigger hard drive coming. That's the easiest fix
at the moment.
Dale
:-) Â :-)
I have my movies in a directory called VideoLib and then
subdirectories A, B, C, etc. If you rearranged a little bit you could subdivide your movies and use symbolic links to make it look more
flat. That sort of arrangement would mean 2 files with 13 excludes in
each for the movies. You could leave the rest alone, create a 3rd file
and exclude the top level movie directories.
Anyway, I just wanted to provide an option. Only you know what works
for your work flow.
Good luck,
Mark
| Sysop: | Keyop |
|---|---|
| Location: | Huddersfield, West Yorkshire, UK |
| Users: | 303 |
| Nodes: | 16 (2 / 14) |
| Uptime: | 72:59:00 |
| Calls: | 6,805 |
| Calls today: | 1 |
| Files: | 12,325 |
| Messages: | 5,399,878 |