Main back-up medium for me is writable DVDs. My back-ups are not as
regular as they ought to be , it could happen that days happen
between successive back-ups. I have a BASH function called crucial
which creates a tar file of all the files (including directories) I
want on back-up. crucial accepts a single argument which is number
of days since last time the files were modified. If I want on the
tar file everything "crucial" then I will run the function with a
large enough argument that all the "crucial" files are covered.
Generally a back-up cycle is that I run crucial with a large argument
and on following days with smaller ones until I have enough tar files
so that not much of the DVD goes to waste , so say at least
4,300,000,000 bytes total. As an intermediate measure before burning
a DVD , I may copy some of the tar files on a memory stick.
RAID would be ideal but it's not a possibility with the hardware I'm
using at present.
I figured it may be instructive to have a thread about this kind of
thing.
By policies/practices I mean not just the back-ups themselves but also whether you have any such policies on using your back-ups for recovering
from various disaster scenarios and whether you ever got to test those policies for real.
On Sun, 04 Jun 2023 12:14:30 +0000, Spiros Bousbouras wrote:
I figured it may be instructive to have a thread about this kind of
thing.
By policies/practices I mean not just the back-ups themselves but also
whether you have any such policies on using your back-ups for
recovering from various disaster scenarios and whether you ever got to
test those policies for real.
I run a backup script every night. I use UFS on BSD, so I use the dump utility since it is arguably the best for this filesystem.
I back up complete filesystems, including the system files, for a quick restore.
On Saturdays it does a complete backup; on other days it does a
differential backup *with the previous Saturday as a base*. In other
wprds, a restore only needs the last differential and also the last
Saturday one. The backups go to the file server, with the file server
itself being backed up elsewhere.
Every Saturday the full backup is also copied to USB storage. That gets cycled; there are seven USBs per machine. 0, 1,2, 3, 0, 1, 2, 4, 0, 1,
2, 5, 0, 1, 2, 6. So I always have last last four weeks and beyone that
the three previous months.
Every 16 weeks (when the cycle ends) I copy the latest full backup to
DVD.
I may reconsider the medium, as the file server now takes 7 DVDs, and
two others use 3 or more. I generate 2 copies of those DVDs, one being
kept on a different floor of the building, and the other being taken to
a storage facility 10 miles away.
In addition, files that I can't easily recreate (system files are not in
this category) are sent to a rolling backup on tarsnap. The last month
(every day), the last year (every month), and forever (every year, going
back 10 years so far).
I also regularly dump more static files (big ones) into Amazon S3 Deep Archive.
I figured it may be instructive to have a thread about this kind
of thing.
By policies/practices I mean not just the back-ups themselves but
also whether you have any such policies on using your back-ups
for recovering from various disaster scenarios and whether you
ever got to test those policies for real.
Spiros Bousbouras <spibou@gmail.com> wrote:
Main back-up medium for me is writable DVDs. My back-ups are not as regular as they ought to be , it could happen that days happen
between successive back-ups. I have a BASH function called crucial
which creates a tar file of all the files (including directories) I
want on back-up. crucial accepts a single argument which is number
of days since last time the files were modified. If I want on the
tar file everything "crucial" then I will run the function with a
large enough argument that all the "crucial" files are covered.
Generally a back-up cycle is that I run crucial with a large argument
and on following days with smaller ones until I have enough tar files
so that not much of the DVD goes to waste , so say at least
4,300,000,000 bytes total. As an intermediate measure before burning
a DVD , I may copy some of the tar files on a memory stick.
Do you also compress the tar files? (gzip, bzip2, xz, or lrzip?).
RAID would be ideal but it's not a possibility with the hardware I'm
using at present.
Can you explain why not?
I've had to recover three times so far since I set this up. Once was
failure of boot/home disk in main desktop. Second was when one of the
disks in the RAID dropped out. That event led to the "third disk" I
mention above, as I ordered it expecting to replace the disk that
dropped. When "third disk" arrived, it turned out that the dropped
disk was just a SATA cable had worked itself loose. So reseating, and
RAID rebuild onto that disk, brought the RAID back to full redundancy.
And as I now had a "spare" I turned it into the "third" backup copy.
In both of the RAID events, the RAID continued operation, albiet with slightly degraded performance, while the single disks were dropped out. Downtime was only the time to shutdown, swap disks (or reseat SATA
cable), blow out accumulated dust, and boot back up.
Stuff like /home gets backed up more frequently than stuff like /boot,
since the latter changes so rarely.
I’ve restored after disk failure a number of times over the years and it went fine.
I'm surprised that people are still using DVDs.
By policies/practices I mean not just the back-ups themselves but
also whether you have any such policies on using your back-ups
for recovering from various disaster scenarios and whether you
ever got to test those policies for real.
Main back-up medium for me is writable DVDs.
Crucial files include all the settings for software I use often
like BASH functions and settings , vim functions and settings ,
etc. So in theory I could restore a working system exactly as I
want it from a single tar file. But I've never had to test it
because I've never had a failure of a hard disk ! Lucky I guess.
I forgot to say that the backup files are currently compressed with gzip (moving to xz though). And the DVDs are not filled, as I add ECC for
recovery if a spot is unreadable.
On 4 Jun 2023 13:31:37 GMT Bob Eager <news0009@eager.cx> wrote:
I forgot to say that the backup files are currently compressed with
gzip (moving to xz though). And the DVDs are not filled, as I add ECC
for recovery if a spot is unreadable.
Is there preexisting software which does this and can also use the ECC
for recovery or have you devised your own scheme ?
On Sun, 4 Jun 2023 12:50:34 -0000 (UTC)
Rich <rich@example.invalid> wrote:
Spiros Bousbouras <spibou@gmail.com> wrote:
RAID would be ideal but it's not a possibility with the hardware
I'm using at present.
Can you explain why not?
Because I don't think my hardware supports it but then again I haven't really searched. I'm not even sure which part of the hardware is responsible ; motherboard ?
Your comments about RAID prompted me to have a look in the wikipedia
article and it turned out that I had a very simplistic view of what a
RAID does. I was under the impression that with a RAID everything
simply gets copied automatically to several disks and that made me
wonder why with 1 less disk functioning , the performance would drop
as opposed to becoming higher (with the tradeoff of lower
redundancy). But it is (or can be) a lot more complicated than this.
I've had to recover three times so far since I set this up. Once was
failure of boot/home disk in main desktop. Second was when one of the
disks in the RAID dropped out. That event led to the "third disk" I
mention above, as I ordered it expecting to replace the disk that
dropped. When "third disk" arrived, it turned out that the dropped
disk was just a SATA cable had worked itself loose. So reseating, and
RAID rebuild onto that disk, brought the RAID back to full redundancy.
And as I now had a "spare" I turned it into the "third" backup copy.
I assume there was no software based way to realise that it was the cable rather than the disk which caused the problem ; the only way was to open
a computer case and see the loose cable.
On 5 Jun 2023 09:17:37 +1000
not@telling.you.invalid (Computer Nerd Kev) wrote:
I'm surprised that people are still using DVDs.
As long as everything fits into one DVD then it's a cheap and space
efficient way to do back-ups.
Whether it's more reliable than alternative methods of storage I
don't know. Does anyone know of any graphs which show as a function
of time the probability of a storage medium going "bad" ?
Of course , there are different levels of badness ranging from
completely unrecoverable to <everything recoverable if you jump
through enough hoops>.
Spiros Bousbouras <spibou@gmail.com> wrote:
On Sun, 4 Jun 2023 12:50:34 -0000 (UTC)
Rich <rich@example.invalid> wrote:
Spiros Bousbouras <spibou@gmail.com> wrote:
RAID would be ideal but it's not a possibility with the hardware I'm
using at present.
Can you explain why not?
Because I don't think my hardware supports it but then again I haven't
really searched. I'm not even sure which part of the hardware is
responsible ; motherboard ?
No special hardware is required for using Linux's 'md' driver. It is
pure software RAID. Any motherboard with sufficient disk ports or to
which an expansion card can be installed to provide sufficient disk
ports can be used with the md driver to provide RAID. In fact, one
/could/ simply attach a set of disks via USB port and use md to RAID
them. Performance over USB, unless one had USB3 or better, would not be
that great, but the md driver would combine them into a RAID array. The motherboard I'm using for my raid box has no hardware support for raid,
and only two PATA disk ports on board, yet it has been performing RAID5
with three drives on a PCI 4-port SATA board just fine for somewhere
going on about ten years or so now.
I figured it may be instructive to have a thread about this kind
of thing.
By policies/practices I mean not just the back-ups themselves but
also whether you have any such policies on using your back-ups
for recovering from various disaster scenarios and whether you
ever got to test those policies for real.
On 5 Jun 2023 09:17:37 +1000
not@telling.you.invalid (Computer Nerd Kev) wrote:
I'm surprised that people are still using DVDs.
As long as everything fits into one DVD then it's a cheap and space
efficient way to do back-ups.
Whether it's more reliable than
alternative methods of storage I don't know. Does anyone know of
any graphs which show as a function of time the probability of a
storage medium going "bad" ?
Spiros Bousbouras <spibou@gmail.com> wrote:
On Sun, 4 Jun 2023 12:50:34 -0000 (UTC)
Rich <rich@example.invalid> wrote:
Spiros Bousbouras <spibou@gmail.com> wrote:
RAID would be ideal but it's not a possibility with the hardware
I'm using at present.
Can you explain why not?
Because I don't think my hardware supports it but then again I haven't really
searched. I'm not even sure which part of the hardware is responsible ; motherboard ?
No special hardware is required for using Linux's 'md' driver. It is
pure software RAID. Any motherboard with sufficient disk ports or to
which an expansion card can be installed to provide sufficient disk
ports can be used with the md driver to provide RAID.
Spiros Bousbouras <spibou@gmail.com> wrote:
On 5 Jun 2023 09:17:37 +1000
not@telling.you.invalid (Computer Nerd Kev) wrote:
Whether it's more reliable than alternative methods of storage I
don't know. Does anyone know of any graphs which show as a function
of time the probability of a storage medium going "bad" ?
For DVD's, that highly depends upon the quality of the disks when they
were manufactured. At one point some 'el cheapo' DVD-R's would fail
only a few years after being burned.
Secondary effects result from the
choice of dye on the DVD, and of course manner of storage adds a huge variable. Storing them in a hot car, in full sunlight, will degrade
them faster than in a light tight box in a cool basement. Of course
the 'cool basement' adds the possibility of a dampness factor, which
can again change the variables.
I buy Verbatim or TDK , both respectable brands as far as I know.
On 04/06/2023 13:14, Spiros Bousbouras wrote:
I figured it may be instructive to have a thread about this kind
of thing.
By policies/practices I mean not just the back-ups themselves but
also whether you have any such policies on using your back-ups
for recovering from various disaster scenarios and whether you
ever got to test those policies for real.
herewith my contribution :-)
I have several computers. Mostly windows boxes but with a few Linux ones
too.
The Win boxes all have a 2nd hard drive to back up to amd also an
automatic backup to back up to a networked NAS.
The Linux boxes can be set up with a cron job and rum rsync to the 2nd internal drive and to the NAS.
I have a directory for each networked device on the NAS.
The NAS use either RAID 6 or RAID 10 ( 4 drives in total and can
tolerate failure of up to two drives.)
You can use Linux MD or FreeBSD's ZFS or use a hardware based RAID
controller (cheap as chips such as the adaptec 7805 whcih will support 8
SATA drives) to do a RAID array.
The adaptec 7805's have onboard 512 MB RAM with a battery backup unit so
that data in transit is not lost during a power cut due to the BBU
(battery backup unit). The NAS is aslo supported by a UPS.
you will find backing up to SATA HDDs or SSDs will be faster than DVDs
and by automating as much of it as possible makes life much easier.
HDDs and SSDs are also much bigger than DVDs, heck a 500 GB SSD will
hold 100 full DVD's worth.
I figured it may be instructive to have a thread about this kind
of thing.
By policies/practices I mean not just the back-ups themselves but
also whether you have any such policies on using your back-ups
for recovering from various disaster scenarios and whether you
ever got to test those policies for real.
Spiros Bousbouras <spibou@gmail.com> wrote:
Whether it's more reliable than alternative methods of storage I
don't know. Does anyone know of any graphs which show as a function
of time the probability of a storage medium going "bad" ?
For DVD's, that highly depends upon the quality of the disks when they
were manufactured. At one point some 'el cheapo' DVD-R's would fail
only a few years after being burned. Secondary effects result from the choice of dye on the DVD, and of course manner of storage adds a huge variable. Storing them in a hot car, in full sunlight, will degrade
them faster than in a light tight box in a cool basement. Of course
the 'cool basement' adds the possibility of a dampness factor, which
can again change the variables.
command.you will find backing up to SATA HDDs or SSDs will be faster than DVDs
and by automating as much of it as possible makes life much easier.
Speed is not really an issue because it's not as if I'm sitting around waiting for the burning to finish , I do my various activities as normal. So > it's just a matter of entering the DVD in the drive and typing the
HDDs and SSDs are also much bigger than DVDs, heck a 500 GB SSD will
hold 100 full DVD's worth.
But present a single point of failure.
command.you will find backing up to SATA HDDs or SSDs will be faster than DVDs
and by automating as much of it as possible makes life much easier.
Speed is not really an issue because it's not as if I'm sitting around
waiting for the burning to finish , I do my various activities as
normal. So > it's just a matter of entering the DVD in the drive and
typing the
fair enough :-)
HDDs and SSDs are also much bigger than DVDs, heck a 500 GB SSD will
hold 100 full DVD's worth.
But present a single point of failure.
Agreed with, and I found this out the hard way when I had 4 IDE drives
on an adaptec 2400A controller..... the controller failed.... so could not access any of the 4 drives as they had been configured as a single
RAID 5 array.
I now use an Adaptec 7805 which allows me to connect up to 8 SATA drives.
I can configure these to be as 8 single drives and formatted to ext4 or
ext5.
So back up to SDA, then rsync SDA to SDB, ditto SDB to SDC, ditto SDC to
SDD, ditto SDD to SDE, ditto SDE to SDF, ditto SDF to SDG and finally
SDG to SDH.
If all 8 drives are 500 GB say, you;s be able to store 100 DVDs on each drive, so you'd have 8 copies of the 100 DVDs. So you;d have to be
really unlucky to have all 8 drives fail simultaneously
I keep a few USB to SATA caddies spare, so if the single controller
fails, I can choose any one of those 8 HDDs and stick in a caddy and
then use my fave linux Live distro or spare Linux box to get at my
backups if I am not able to source a replacement controller.
I have had up to 3 NASes, all syncing to each other so with a 2TB drive
in each NAS, you'd be able to back up 400 DVD's thrice over :-)
the OneDrive thing is a good off-site back up as my last line of defence.
The Win boxes all have a 2nd hard drive to back up to amd also an
automatic backup to back up to a networked NAS.
On Tue, 06 Jun 2023 10:55:47 +0000, Spiros Bousbouras wrote:
I buy Verbatim or TDK , both respectable brands as far as I know.
I look for any brand that uses Taiyo Yuden dye, etc. Manufacturers change what they use over time.
I always check read what I have written.
It sounds like you keep an archive of all your old backups
(as opposed to using DVD-R/W discs).
P.S I have seen some Icydock hard disc caddies that have both a USB
socket and a SATA & Power socket in the back.
The PC has a backplane that the caddies plugs into and works over SATA.
You can pull any caddy out, use a USB lead to connect the caddy to a PC
and then mount and browse the partitions.
(this only works if you have NOT set the caddies up as part of a
Hardware based RAID array or if Software RAID based, you'd have to trust
that all the caddies will work over USB with Linux MDAM / MD ...)
On 6 Jun 2023 11:09:30 GMT
Bob Eager <news0009@eager.cx> wrote:
On Tue, 06 Jun 2023 10:55:47 +0000, Spiros Bousbouras wrote:
I buy Verbatim or TDK , both respectable brands as far as I know.
I look for any brand that uses Taiyo Yuden dye, etc. Manufacturers change
what they use over time.
Is there any reason to think that dye manufacturers keep their products
more constant (or improving) over time than DVD manufacturers (or manufacturers of any other writable medium for that matter) ?
SH <i.love@spam.com> writes:
The Win boxes all have a 2nd hard drive to back up to amd also an
automatic backup to back up to a networked NAS.
What do you use to back up in Windows? I have a SpiderOak subscription
which allows convenient choosing of folders and files to back up but
it's for remote backups and I'd like local backups too. Not that I have
that much to back up in Windows.
As I understand it, Windows 10 ships a broken backup tool (to promote OneDrive, I suppose) and also a working one which is called "Backup and Restore Windows 7" which works but it's pretty clunky. I think I tried
Aomei too and some others but didn't find a program I like.
Media get backed up to BD-R. I knocked together some scripts to manage[...]
this:
https://gitlab.com/salfter/bdarchiver
A database that tracks what files are on what disc is maintained by the scripts and written to each disc. I write 20 GB to single-layer BD-Rs,
I've also
occasionally needed to recover individual files that had somehow
disappeared. The database makes locating the right disc trivial.
On 07/06/2023 10:43, Spiros Bousbouras wrote:[...]
On 6 Jun 2023 11:09:30 GMT
Bob Eager <news0009@eager.cx> wrote:
On Tue, 06 Jun 2023 10:55:47 +0000, Spiros Bousbouras wrote:
I buy Verbatim or TDK , both respectable brands as far as I know.
I look for any brand that uses Taiyo Yuden dye, etc. Manufacturers change >> what they use over time.
Is there any reason to think that dye manufacturers keep their products more constant (or improving) over time than DVD manufacturers (or manufacturers of any other writable medium for that matter) ?
Developing a new DVD dye costs millions and it is now much harder to
recoup that investment in R&D.
So what will happen is the current commercial formulations will continue
to be used until eventually the DVD market becomes too small and
commercially unviable abd then all production of dvd discs, DVD players/recorders and DVD drives will cease.
I have a number of packs of Verbatim DVD blank media I have for sale on
eBay for several months and no one has bought them yet.... :-(
If anyone is interested in blank DVD +R and DVD-R and DVD-RAM media and
also in either SATA or EIDE DVD burner drives, reply to this post :-)
[...]
Developing a new DVD dye costs millions and it is now much harder to
recoup that investment in R&D.
So what will happen is the current commercial formulations will continue
to be used until eventually the DVD market becomes too small and
commercially unviable abd then all production of dvd discs, DVD
players/recorders and DVD drives will cease.
Do you have any timescale in mind ? Because I can easily find floppy drives and disks (an even older technology) on amazon.
I have a number of packs of Verbatim DVD blank media I have for sale on
eBay for several months and no one has bought them yet.... :-(
Why would anyone search on ebay for this kind of thing ? I mean I can find writable DVDs on amazon (among many other websites , I'm sure) , several pound
shops (the usual well known brands of DVDs) and also big chains selling stationery.
[...]
Developing a new DVD dye costs millions and it is now much harder to
recoup that investment in R&D.
So what will happen is the current commercial formulations will continue >>> to be used until eventually the DVD market becomes too small and
commercially unviable abd then all production of dvd discs, DVD
players/recorders and DVD drives will cease.
Do you have any timescale in mind ? Because I can easily find floppy
drives
and disks (an even older technology) on amazon.
Good question. much of the floppy drives and flopp discs for sale is 2nd hand. The last floppy disc manufacturer closed soem time ago and I think there is a Japanese or US compnay trading in 2nd hand floppies & drives
and they occasionally buy up job lots of discovered still sealed
floppies and drives.
Coming back to DVDs, I'd say 10 ish years, then it will be legacy unsold stock thereafter
I have a number of packs of Verbatim DVD blank media I have for sale on
eBay for several months and no one has bought them yet.... :-(
Why would anyone search on ebay for this kind of thing ? I mean I can
find
writable DVDs on amazon (among many other websites , I'm sure) ,
several pound
shops (the usual well known brands of DVDs) and also big chains selling
stationery.
I did wonder about selling on Amazon but I don't know if Amazon will
accept private sellers in the same way as eBay does?
Synctoy is your friend!
https://updov.com/download-synctoy/
You will also need to download and install .NET frameworks too.
By policies/practices I mean not just the back-ups themselves but
also whether you have any such policies on using your back-ups
for recovering from various disaster scenarios and whether you
ever got to test those policies for real.
I'm surprised that people are still using DVDs.
On Tue, 06 Jun 2023 17:19:21 GMT
scott@alfter.diespammersdie.us wrote:
Media get backed up to BD-R. I knocked together some scripts to manage[...]
this:
https://gitlab.com/salfter/bdarchiver
A database that tracks what files are on what disc is maintained by the
scripts and written to each disc. I write 20 GB to single-layer BD-Rs,
I've also
occasionally needed to recover individual files that had somehow
disappeared. The database makes locating the right disc trivial.
Do you also have a copy of the database on your hard disk (or equivalent) ?
And how do you identify the BD-R ? Do you write anything on their surface with a marker ?
I have a database (just a text file with an appropriate format) of what's
on each DVD I burn and I write on each DVD with a marker some identifying number.
On 5 Jun 2023 09:17:37 +1000, Computer Nerd Kev wrote:
I'm surprised that people are still using DVDs.
I have quite a lot of customers still using all sorts of archaic
phyiscal media. Tape is quite popular as well.
Cheap, easily stored, easily destroyed, certifiable.
Matthew Ernisse <matt@going-flying.com> wrote:
On 5 Jun 2023 09:17:37 +1000, Computer Nerd Kev wrote:
I'm surprised that people are still using DVDs.
I have quite a lot of customers still using all sorts of archaic
phyiscal media. Tape is quite popular as well.
I would imagine only the newer tape formats that store vastly more
data than a DVD though.
Cheap, easily stored, easily destroyed, certifiable.
Fair enough, but these days I'd say it only makes sense to use
DVD for backups if you can fit everything onto one disc.
| Sysop: | Keyop |
|---|---|
| Location: | Huddersfield, West Yorkshire, UK |
| Users: | 299 |
| Nodes: | 16 (2 / 14) |
| Uptime: | 77:09:13 |
| Calls: | 6,695 |
| Files: | 12,228 |
| Messages: | 5,347,353 |