debian-user:

I have a SOHO file server with ~1 TB of data. I would like archive the
data by burning it to a series of optical discs organized by time (e.g.
mtime). I expect to periodically burn additional discs in the future,
each covering a span of time from the previous last disc to the
then-current time.


I am looking for FOSS software for Unix platforms that goes beyond a
disc burner with multi-volume spanning. The term "archive management
system" comes to mind.


Comments or suggestions?


David

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 1/22/24 21:28, David Christensen wrote:

debian-user:

I have a SOHO file server with ~1 TB of data.  I would like archive the
data by burning it to a series of optical discs organized by time (e.g. mtime).  I expect to periodically burn additional discs in the future,
each covering a span of time from the previous last disc to the
then-current time.

I am looking for FOSS software for Unix platforms that goes beyond a
disc burner with multi-volume spanning.  The term "archive management system" comes to mind.

Comments or suggestions?

Take a look at amanda, although the optical is not on my radar because
of the low capacity of a dvd at 4.7gigs. Amanda can use an lvm for
v-disks, which if you want say 60 days back as bare metal recover, would actually be the contents of 60 directories in that lvm, allowing any
file up to 60 days old to be recovered.

Here, with all my machine on my local network, and before I started with
3d printers which need maybe 30 gigabyte of gcode to drive the printers
to make one complex part, 5 machines backed up for 60 days, was filling
a 1T drive to around 87%. So I bought 2 2T seagates, got buster
installed and everything running smoothly. Installed buster on oe drive, configure amanda to use the other. 3 or 4 weeks later, both of those
seagates dropped off the sata controller in the night, and I rebuilt
with SSD's and bookworm which has been the disaster you all have been
trying to help me with since, but its not running well enough to be
worth reinventing my amanda setup which just grew since about 1999.

Amanda was developed for use with qic or thereabouts tapes so its prime directive is to juggle the backup levels to fully fill the tape(s) while
doing a level0 on everything withing the time limit in days between
level0's. It can use a tape library of however many tapes the library
contains. Bring big red wagonloads of cash for those. But I gave up on
tapes 15 years ago, they simply weren't dependable enough, while
spinning rust, never shut down can sit there and spin for 50,000+ hours, they've done it for me.

The 1T I took out, a cheap 5400 rev barracuda, had just under 70,000
spinning hours on it with maybe 40 power downs on it, when I retired it
cuz it was getting too small, for the 2T that lasted less than a month.

In spinning rust, power downs are the drive killers, so spin them up and
leave them spinning, the heads aren't wearing while they are flying on 3 microns of air between the head and the platter.
Set it up right, and amanda will have your back till the place is a few
inches of ashes. And you can set to cycle the storage drive(s) offsite, exchanging them weekly or monthly, what ever you're comfortable with.
That adds to the powerdown count however so I never did that.

David

Take care, stay warm, dry and well, David.

.

Cheers, Gene Heskett.
--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
- Louis D. Brandeis

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Mon, 22 Jan 2024 18:27:51 -0800
David Christensen <dpchrist@holgerdanske.com> wrote:

debian-user:

I have a SOHO file server with ~1 TB of data. I would like archive
the data by burning it to a series of optical discs organized by time
(e.g. mtime). I expect to periodically burn additional discs in the
future, each covering a span of time from the previous last disc to
the then-current time.

I am looking for FOSS software for Unix platforms that goes beyond a
disc burner with multi-volume spanning. The term "archive management
system" comes to mind.

Comments or suggestions?

gene heskett <gheskett@shentel.net>'s suggestion of Amanda is a good
one. It has its kinks, but is solid and reliable. Amanda also handles compression and encryption for you. I currently use Amanda to back up
to a RAID array. I then use rsnapshot to back portions of that
(including the Amanda virtual tapes) to one of three rotating off-site
USB external drives. I suspect the latter could be adapted to your requirements.

If you don't find anything readily available, I'd look at using find
and the mtimes to copy to a holding disk, which you can then burn to
archive media.

I suggest you look at Blu-Ray for archiving.


--
Does anybody read signatures any more?

https://charlescurley.com
https://charlescurley.com/blog/

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 1/22/24 19:44, gene heskett wrote:

On 1/22/24 21:28, David Christensen wrote:

debian-user:

I have a SOHO file server with ~1 TB of data.  I would like archive
the data by burning it to a series of optical discs organized by time
(e.g. mtime).  I expect to periodically burn additional discs in the
future, each covering a span of time from the previous last disc to
the then-current time.


I am looking for FOSS software for Unix platforms that goes beyond a
disc burner with multi-volume spanning.  The term "archive management
system" comes to mind.


Comments or suggestions?

Take a look at amanda, although the optical is not on my radar because
of the low capacity of a dvd at 4.7gigs.  Amanda can use an lvm for
v-disks, which if you want say 60 days back as bare metal recover, would actually be the contents of 60 directories in that lvm, allowing any
file up to 60 days old to be recovered.

Here, with all my machine on my local network, and before I started with
3d printers which need maybe 30 gigabyte of gcode to drive the printers
to make one complex part, 5 machines backed up for 60 days, was filling
a 1T drive to around 87%. So I bought 2 2T seagates, got buster
installed and everything running smoothly. Installed buster on oe drive, configure amanda to use the other.  3 or 4 weeks later, both of those seagates dropped off the sata controller in the night, and I rebuilt
with SSD's and bookworm which has been the disaster you all have been
trying to help me with since, but its not running well enough to be
worth reinventing my amanda setup which just grew since about 1999.

Amanda was developed for use with qic or thereabouts tapes so its prime directive is to juggle the backup levels to fully fill the tape(s) while doing a level0 on everything withing the time limit in days between
level0's. It can use a tape library of however many tapes the library contains. Bring big red wagonloads of cash for those.  But I gave up on tapes 15 years ago, they simply weren't dependable enough, while
spinning rust, never shut down can sit there and spin for 50,000+ hours, they've done it for me.

The 1T I took out, a cheap 5400 rev barracuda, had just under 70,000
spinning hours on it with maybe 40 power downs on it, when I retired it
cuz it was getting too small, for the 2T that lasted less than a month.

In spinning rust, power downs are the drive killers, so spin them up and leave them spinning, the heads aren't wearing while they are flying on 3 microns of air between the head and the platter.
Set it up right, and amanda will have your back till the place is a few inches of ashes.  And you can set to cycle the storage drive(s) offsite, exchanging them weekly or monthly, what ever you're comfortable with.
That adds to the powerdown count however so I never did that.

Thank you for the reply. :-)


David

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 1/22/24 20:30, Charles Curley wrote:

On Mon, 22 Jan 2024 18:27:51 -0800
David Christensen <dpchrist@holgerdanske.com> wrote:

debian-user:

I have a SOHO file server with ~1 TB of data. I would like archive
the data by burning it to a series of optical discs organized by time
(e.g. mtime). I expect to periodically burn additional discs in the
future, each covering a span of time from the previous last disc to
the then-current time.


I am looking for FOSS software for Unix platforms that goes beyond a
disc burner with multi-volume spanning. The term "archive management
system" comes to mind.


Comments or suggestions?

gene heskett <gheskett@shentel.net>'s suggestion of Amanda is a good
one. It has its kinks, but is solid and reliable. Amanda also handles compression and encryption for you. I currently use Amanda to back up
to a RAID array. I then use rsnapshot to back portions of that
(including the Amanda virtual tapes) to one of three rotating off-site
USB external drives. I suspect the latter could be adapted to your requirements.

If you don't find anything readily available, I'd look at using find
and the mtimes to copy to a holding disk, which you can then burn to
archive media.

I suggest you look at Blu-Ray for archiving.

Thank you for the reply. :-)


David

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi,

David Christensen wrote:

I have a SOHO file server with ~1 TB of data. I would like archive the data by burning it to a series of optical discs organized by time (e.g. mtime).
I expect to periodically burn additional discs in the future, each covering
a span of time from the previous last disc to the then-current time.

I use my own software for making incremental multi-volume backups, based
on file timestamps (m and c), inode numbers, and content checksums.
http://scdbackup.webframe.org/main_eng.html
http://scdbackup.webframe.org/examples.html#incremental

The software and the texts are quite old. The proposed backup scheme
is not in use here any more.
Instead i have four independent backup families, each comprised of
level 0 to N with no repetitions below the current level N.
Further i have backups of the configuration and memorized file lists
on 4 CDs.

Level 0 fills dozens of BD-RE discs. The other levels fill at most one
BD-RE. Level N of each family exists in three copies which get larger
with each backup run of that level. Whenever this level BD threatens to overflow, i archive the latest BD of that level and start level N+1.
That step is a bit bumpy, because i have to restore the file lists of
level N from CD after a backup has been planned but was not performed.
When overflow is foreseeable, i make a copy of the file lists on disk
before i start the planning run, or i simply start level N+1 without
waiting for the overflow.

I use scdbackup for the slowly growing bulk of my file collection.
The agile parts of my hard disk are only about 5 GB and get covered by incremental multi-session backups of xorriso (which learned a lot about incrementality from scdbackup). With zisofs compression i can put about
30 incremental backups of one DVD+RW or 250 backups on one BD-RE.
Day by day.

The term "archive management system" comes to mind.

I would not attribute this title to scdbackup. It was created to scratch
my itch when hard disks grew much faster in capacity than the backup
media. (Also it was the motivation to start programming on ISO 9660
producers and burn programs.)

So it might be that you are better off with a more professional backup
system. :))

(Else we will probably have to read together
http://scdbackup.webframe.org/cd_backup_planer_help
and my backup configurations to compose configurations for you.)

----------------------------------------------------------------------
About timestamps and incremental backup:

If you only go for mtime, them you miss changes of file attributes
which are indicated by ctime.
Even more, timestamps alone are not a reliable way to determine which
files are new at their current location in the directory tree.
If you move a file from one directory to the other, then the timestamps
of the file do _not_ get updated. Only the two involved directories get
new timestamps.
So when the backup tool encounters directories with young timestamps
it has to use other means to determine whether their data files were
moved. scdbackup uses recorded device and inode numbers, and as last
resort recorded MD5 sums for that purpose.

(Of course, content MD5 comparison is slow and causes high disk load,
compared to simple directory tree traversal with timestamps and inode
numbers. So scdbackup tries to avoid this when possible and allowed
by the -changetest_options in the backup configuration file.)


Have a nice day :)

Thomas

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)