Howdy,
I sort of started this on another thread but wanted to nail a few things
down first. I'm wanting to encrypt some parts of my data on /home.
This is what I got hard drive wise.
root@fireball / # pvs
PV VG Fmt Attr PSize PFree
/dev/sda7 OS lvm2 a-- <124.46g 21.39g
/dev/sdb1 Home2 lvm2 a-- <5.46t 0
/dev/sdc1 Home2 lvm2 a-- <7.28t 0
/dev/sdd1 Home2 lvm2 a-- <7.28t 0
/dev/sde1 backup lvm2 a-- 698.63g 0
root@fireball / #
I've done some checking on sizes of things I want to encrypt and am
weighing options. I use LVM which should help make things easier. I've downloaded and printed some howtos regarding shrinking the file system
and LVM thingys. It seems I need to shrink the file system while my
/home partition is unmounted. Then move the data off whichever drive I
want to remove and then remove the drive itself. After that I can
encrypt the just removed drive and start moving files over, using rsync
is my plan. I think that is the basic steps.
My question now comes to this. When I encrypt one of the drives, can I
then expand that drive with it being encrypted or is that not a option?
I plan to encrypt two of the drives as one volume group and leave one
other volume group as normal. I just want to be sure whether or not I
can expand a encrypted LVM drive the same as a normal LVM since both
uses LVM. I use cryptsetup commands to accomplish the encryption if
that matters. So as a example, I start with one 7TB drive encrypted,
move some data to it, then want to add either the 5TB or 7TB drive. Can
I just expand it like a normal LVM or does it being encrypted change
things?
Thoughts? My remove steps look sensible? Expanding encrypted LVM possible?
Dale
Wol wrote:
On 27/03/2022 20:17, Dale wrote:
Howdy,One big piece of missing information. What does fdisk say about
I sort of started this on another thread but wanted to nail a few things >>> down first. I'm wanting to encrypt some parts of my data on /home.
This is what I got hard drive wise.
root@fireball / # pvs
PV VG Fmt Attr PSize PFree
/dev/sda7 OS lvm2 a-- <124.46g 21.39g
/dev/sdb1 Home2 lvm2 a-- <5.46t 0
/dev/sdc1 Home2 lvm2 a-- <7.28t 0
/dev/sdd1 Home2 lvm2 a-- <7.28t 0
/dev/sde1 backup lvm2 a-- 698.63g 0
root@fireball / #
sd[b,c,d]1? And can you add sdf1?
I have the entire drive as one large partition for each drive. I could
have done it as a whole device but I wanted partitions to give a hint
that the drive is in use, if booted from other medium for example.
I have enough extra space that I can remove either a 6TB or a 8TB
drive. Once that is done, I can start to encrypt and move data around.
This is some additional info from df for /home:
/dev/mapper/Home2-Home2 20T 8.7T 12T 45% /home
If I remove a 8TB drive, I'd still have enough room for my data. I
could then rebuild /home starting with the 8TB drive just freed up.
Then as I move data, I could expand them one at a time encrypting as I
go. I'd rather not have to buy a hard drive right now. Tight budget
given other things I got going on. I do have backups, more than one in
a couple important data spots.
I'm guessing you've got three 8TB drives? Or is it two 8s and a 6? Can
you get a third 8TB? And if you're encrypting *parts* of /home ...
what parts?
I've done some checking on sizes of things I want to encrypt and am
weighing options. I use LVM which should help make things easier. I've >>> downloaded and printed some howtos regarding shrinking the file system
and LVM thingys. It seems I need to shrink the file system while my
/home partition is unmounted. Then move the data off whichever drive I >>> want to remove and then remove the drive itself. After that I can
encrypt the just removed drive and start moving files over, using rsync
is my plan. I think that is the basic steps.
Not necessarily.
My question now comes to this. When I encrypt one of the drives, can I >>> then expand that drive with it being encrypted or is that not a option?
I plan to encrypt two of the drives as one volume group and leave one
other volume group as normal. I just want to be sure whether or not I
can expand a encrypted LVM drive the same as a normal LVM since both
uses LVM. I use cryptsetup commands to accomplish the encryption if
that matters. So as a example, I start with one 7TB drive encrypted,
move some data to it, then want to add either the 5TB or 7TB drive. Can >>> I just expand it like a normal LVM or does it being encrypted change
things?
Thoughts? My remove steps look sensible? Expanding encrypted LVM
possible?
If you are using LVM to do the encryption, then I can't see any
problems adding a new PV to an encrypted VG.
Personally, I'd use dm-crypt to encrypt the drive, and then the whole
Dale
lot is encrypted, and put plain LVM over that. I've got dedicated
layers for everything.
It looks like your home2 is 6TB+8TB+8TB. I'd get a new 8TB, put
dm-crypt on it, and add it. Now I can remove the first 8TB, dm-crypt
it and re-add it. Same with the second 8TB. Now remove the 6TB and
there you are ...
My layout's rather different from yours, so I don't think I ought to
say too much :-)
Cheers,
Wol
What is the advantage of dm-crypt over cryptsetup? I've learned how to
use cryptsetup with my external drive so was hoping to stick with what I already know. Unless there is a advantage to dm-crypt.
On 27/03/2022 21:13, Dale wrote:
Wol wrote:Do you need to shrink your fs first though?
On 27/03/2022 20:17, Dale wrote:
Howdy,One big piece of missing information. What does fdisk say about
I sort of started this on another thread but wanted to nail a few
things
down first. I'm wanting to encrypt some parts of my data on /home.
This is what I got hard drive wise.
root@fireball / # pvs
PV VG Fmt Attr PSize PFree
/dev/sda7 OS lvm2 a-- <124.46g 21.39g
/dev/sdb1 Home2 lvm2 a-- <5.46t 0
/dev/sdc1 Home2 lvm2 a-- <7.28t 0
/dev/sdd1 Home2 lvm2 a-- <7.28t 0
/dev/sde1 backup lvm2 a-- 698.63g 0
root@fireball / #
sd[b,c,d]1? And can you add sdf1?
I have the entire drive as one large partition for each drive. I could
have done it as a whole device but I wanted partitions to give a hint
that the drive is in use, if booted from other medium for example.
I have enough extra space that I can remove either a 6TB or a 8TB
drive. Once that is done, I can start to encrypt and move data around.
This is some additional info from df for /home:
/dev/mapper/Home2-Home2 20T 8.7T 12T 45% /home
If I remove a 8TB drive, I'd still have enough room for my data. I
could then rebuild /home starting with the 8TB drive just freed up.
Then as I move data, I could expand them one at a time encrypting as I
go. I'd rather not have to buy a hard drive right now. Tight budget
given other things I got going on. I do have backups, more than one in
a couple important data spots.
My three 3TB partitions are raided, and /dev/md/home is my PV. I've
only allocated the space to LVs that they need, so I could probably
shrink the PV and remove a drive without needing to mess about with my
LVs at all. I get the impression you may have allocated all your
space, not a good idea.
My attitude is my data is backed up, expanding an LV/FS is low risk,
I'll just grow stuff as I need to ... my /home partition contains
proper home drives, things like videos may be in /home/videos, but
they're actually a separate partition, etc etc.
I don't know either. I'm just far more familiar with the dm/md layer
I'm guessing you've got three 8TB drives? Or is it two 8s and a 6? Can
you get a third 8TB? And if you're encrypting *parts* of /home ...
what parts?
I've done some checking on sizes of things I want to encrypt and am
weighing options. I use LVM which should help make things easier.
I've
downloaded and printed some howtos regarding shrinking the file system >>>> and LVM thingys. It seems I need to shrink the file system while my
/home partition is unmounted. Then move the data off whichever
drive I
want to remove and then remove the drive itself. After that I can
encrypt the just removed drive and start moving files over, using
rsync
is my plan. I think that is the basic steps.
Not necessarily.
My question now comes to this. When I encrypt one of the drives,
can I
then expand that drive with it being encrypted or is that not a
option?
I plan to encrypt two of the drives as one volume group and leave one
other volume group as normal. I just want to be sure whether or not I >>>> can expand a encrypted LVM drive the same as a normal LVM since both
uses LVM. I use cryptsetup commands to accomplish the encryption if
that matters. So as a example, I start with one 7TB drive encrypted, >>>> move some data to it, then want to add either the 5TB or 7TB
drive. Can
I just expand it like a normal LVM or does it being encrypted change
things?
Thoughts? My remove steps look sensible? Expanding encrypted LVM
possible?
If you are using LVM to do the encryption, then I can't see any
problems adding a new PV to an encrypted VG.
Personally, I'd use dm-crypt to encrypt the drive, and then the whole
Dale
lot is encrypted, and put plain LVM over that. I've got dedicated
layers for everything.
It looks like your home2 is 6TB+8TB+8TB. I'd get a new 8TB, put
dm-crypt on it, and add it. Now I can remove the first 8TB, dm-crypt
it and re-add it. Same with the second 8TB. Now remove the 6TB and
there you are ...
My layout's rather different from yours, so I don't think I ought to
say too much :-)
Cheers,
Wol
What is the advantage of dm-crypt over cryptsetup? I've learned how to
use cryptsetup with my external drive so was hoping to stick with what I
already know. Unless there is a advantage to dm-crypt.
because I run md-raid over dm-integrity. Hence dm-crypt.
Is cryptsetup a layer in its own right, or part of lvm? I prefer the
Unix "use several tools each of which does one thing well", other
people prefer a swiss army knife like ZFS or btrfs. I don't know where cryptsetup lies on that spectrum, and I don't know your preferences on
that spectrum.
Cheers,
Wol
On 27/03/2022 20:17, Dale wrote:
Howdy,One big piece of missing information. What does fdisk say about
I sort of started this on another thread but wanted to nail a few things
down first. I'm wanting to encrypt some parts of my data on /home.
This is what I got hard drive wise.
root@fireball / # pvs
PV VG Fmt Attr PSize PFree
/dev/sda7 OS lvm2 a-- <124.46g 21.39g
/dev/sdb1 Home2 lvm2 a-- <5.46t 0
/dev/sdc1 Home2 lvm2 a-- <7.28t 0
/dev/sdd1 Home2 lvm2 a-- <7.28t 0
/dev/sde1 backup lvm2 a-- 698.63g 0
root@fireball / #
sd[b,c,d]1? And can you add sdf1?
I'm guessing you've got three 8TB drives? Or is it two 8s and a 6? Can
you get a third 8TB? And if you're encrypting *parts* of /home ...
what parts?
I've done some checking on sizes of things I want to encrypt and am
weighing options. I use LVM which should help make things easier. I've >> downloaded and printed some howtos regarding shrinking the file system
and LVM thingys. It seems I need to shrink the file system while my
/home partition is unmounted. Then move the data off whichever drive I
want to remove and then remove the drive itself. After that I can
encrypt the just removed drive and start moving files over, using rsync
is my plan. I think that is the basic steps.
Not necessarily.
My question now comes to this. When I encrypt one of the drives, can I
then expand that drive with it being encrypted or is that not a option?
I plan to encrypt two of the drives as one volume group and leave one
other volume group as normal. I just want to be sure whether or not I
can expand a encrypted LVM drive the same as a normal LVM since both
uses LVM. I use cryptsetup commands to accomplish the encryption if
that matters. So as a example, I start with one 7TB drive encrypted,
move some data to it, then want to add either the 5TB or 7TB drive. Can
I just expand it like a normal LVM or does it being encrypted change
things?
Thoughts? My remove steps look sensible? Expanding encrypted LVM
possible?
If you are using LVM to do the encryption, then I can't see any
problems adding a new PV to an encrypted VG.
Personally, I'd use dm-crypt to encrypt the drive, and then the whole
Dale
lot is encrypted, and put plain LVM over that. I've got dedicated
layers for everything.
It looks like your home2 is 6TB+8TB+8TB. I'd get a new 8TB, put
dm-crypt on it, and add it. Now I can remove the first 8TB, dm-crypt
it and re-add it. Same with the second 8TB. Now remove the 6TB and
there you are ...
My layout's rather different from yours, so I don't think I ought to
say too much :-)
Cheers,
Wol
What is the advantage of dm-crypt over cryptsetup? I've learned how to
use cryptsetup with my external drive so was hoping to stick with what I already know. Unless there is a advantage to dm-crypt.
I don't know either. I'm just far more familiar with the dm/md layer
because I run md-raid over dm-integrity. Hence dm-crypt.
Is cryptsetup a layer in its own right, or part of lvm? I prefer the
Unix "use several tools each of which does one thing well", other people prefer a swiss army knife like ZFS or btrfs. I don't know where
cryptsetup lies on that spectrum, and I don't know your preferences on
that spectrum.
On 27/03/2022 21:36, Wol wrote:
I don't know either. I'm just far more familiar with the dm/md layer
because I run md-raid over dm-integrity. Hence dm-crypt.
Is cryptsetup a layer in its own right, or part of lvm? I prefer the
Unix "use several tools each of which does one thing well", other
people prefer a swiss army knife like ZFS or btrfs. I don't know
where cryptsetup lies on that spectrum, and I don't know your
preferences on that spectrum.
Just seen Rich's message, so now I know :-)
But it's just hit me - you have three PV's joined into one LV? Is that effectively raid-0? If so, you know you have just TREBLED your risk of
losing your home drive? (Although I do know the risk is low to start
with.)
Don't know what really to suggest though, other than getting a new 8TB
drive and converting it to a 3x8TB 16TB raid-5 ... and you said you
didn't want to splash out on a new drive ...
Cheers,
Wol
Wol wrote:
My three 3TB partitions are raided, and /dev/md/home is my PV. I've
only allocated the space to LVs that they need, so I could probably
shrink the PV and remove a drive without needing to mess about with my
LVs at all. I get the impression you may have allocated all your
space, not a good idea.
I did allocate all the space because at the time, I wasn't considering encrypting any of that data or dividing it up. Things have changed and
I want to move things around. This is one of the good things about ext4
and LVM. They can shrink in size fairly easy. Of course, backups are
always a good idea.
My attitude is my data is backed up, expanding an LV/FS is low risk,
I'll just grow stuff as I need to ... my /home partition contains
proper home drives, things like videos may be in /home/videos, but
they're actually a separate partition, etc etc.
That's sort of what I'm going to do. I'm going to divide things into sections with some encrypted and some not.
Based on the reply from Rich, thanks for the info, cryptsetup is just a
upper level of dm-crypt. Basically, cryptsetup just has some user
friendly bits added on top of it. Security wise, should be secure
either way.
The biggest thing, can I encrypt a LVM group and then expand it. It
seems I can. I've found where google results say the same but some
results are dated. Things change. Sometimes for the good, sometimes not.
On Sunday, 27 March 2022 22:04:45 BST Dale wrote:
I wonder if all you want to do is to encrypt some directories on your /home, then a different level of encryption would be more appropriate? Instead of encrypting a whole block device, you could just encrypt a directory tree or two, using ext4 encryption. e4crypt has been kicking around for a few years now and it is meant to be an improvement on eCryptfs.
That's sort of what I'm going to do. I'm going to divide things into
sections with some encrypted and some not.
https://lwn.net/Articles/639427/
https://wiki.gentoo.org/wiki/Ext4_encryption
WARNING: I'm not qualified to speak about this topic because my experience is
limited, but I'm interested all the same in reading your approach and other contributors advice.
I don't have RAID at all. Just three drives being used as /home on
LVM. I should use RAID but I have a backup that gets done each week. I wouldn't lose much even if it crashed and burned badly. The biggest
loss might would be emails. I think I have gmail set up to save them so
I think it would download whatever was missing from the last backup restoration. I need to check that.
On 27/03/2022 22:34, Dale wrote:
I don't have RAID at all. Just three drives being used as /home on
LVM. I should use RAID but I have a backup that gets done each week. I >> wouldn't lose much even if it crashed and burned badly. The biggest
loss might would be emails. I think I have gmail set up to save them so
I think it would download whatever was missing from the last backup
restoration. I need to check that.
You might have copies of email everywhere without realising it. If
gmail is set up as imap, then it will keep a copy of all your mail. I
have dovecot set up which keeps a copy of all mine.
Suck it and see, but if you have /home on one drive, and dovecot cache
on another, then you shouldn't lose anything. Mail clients should err
on the side of caution, and if they disagree with the server they
should assume the worst and sync keeping not sync deleting.
Cheers,
Wol
I looked into setting up a local mail server, it is just way over my
head. When I have time to deal with it, I'll look into it again. I'm
sure I can do it but at the moment, got to much else going on. It's the season for catching catfish anyway.;-)
On Monday, 28 March 2022 04:59:04 BST Dale wrote:
Michael wrote:OK, that's a different plan to what I'm talking about. I am talking about filesystem based encryption.
On Sunday, 27 March 2022 22:04:45 BST Dale wrote:That is the basic plan. I'll have /home as a normal open mount point.
That's sort of what I'm going to do. I'm going to divide things intoI wonder if all you want to do is to encrypt some directories on your
sections with some encrypted and some not.
/home, then a different level of encryption would be more appropriate?
Instead of encrypting a whole block device, you could just encrypt a
directory tree or two, using ext4 encryption. e4crypt has been kicking
around for a few years now and it is meant to be an improvement on
eCryptfs.
https://lwn.net/Articles/639427/
https://wiki.gentoo.org/wiki/Ext4_encryption
WARNING: I'm not qualified to speak about this topic because my
experience is limited, but I'm interested all the same in reading your
approach and other contributors advice.
That way I can login without a encryption password being needed. After
that, I plan to have other mount point(s) that are encrypted.
You are talking about block device level encryption. Whether these other mountpoints you want to encrypt are for normal partitions, LVM, what-ever, you're dealing with whole block devices and mechanisms for encrypting these block devices *before* a filesystem and data is even placed on them.
With a filesystem level encryption you will be using the kernel's native filesystem encryption and the fscrypt API to manage it. The sys-fs/fscrypt userspace package can be used to manage the kernel based fscrypt API to encrypt/decrypt some directories selectively within a filesystem. This can be
set up to happen transparently via PAM, using the user's login, if desired.
You can read more about the kernel fscrypt mechanism here:
https://www.kernel.org/doc/html/latest/filesystems/fscrypt.html
For how to set it up and run it from userspace details are here:
https://wiki.archlinux.org/title/Fscrypt
It may beIf you have 8TB of data you want encrypted, then a block level encryption would make sense. However, if you only have a few MB or GB of data to encrypt
/home/dale/Data or something to that effect. I'm still doing some
checking but the normal non-encrypted stuff should easily fit on a 6TB
drive without encryption. I can then rebuild the two 8TB drives as
encrypted mount points with a different volume group thingy. When I
boot up, I can login in as usual then decrypt the other mount point and
access it as needed or close it and it be encrypted until needed.
in a couple of directories, then it may be more appropriate to consider native
filesystem encryption with fscrypt.
I've considered just encrypting /home completely but I don't have theWith fscrypt you can run manually 'fscrypt lock Vault-for-Dale' and only the directory 'Vault-for-Dale' with any files in it will be encrypted. All the remaining fs will be left as it was.
option of closing it while I'm logged into KDE. KDE wouldn't be able to
access /home/dale/.kde or .config plus if I leave Seamonkey open, it
will want to store new emails to .mozilla as well. So, some things need
to be available and I'm not to worried about them being encrypted
anyway. So encrypting all of /home would be overkill plus would be a
problem for some things too, such as Seamonkey and KDE.
Anyway, I'm only mentioning this as an alternative to consider, in case it fits
your use case better than buying more disks and whole block device encryption methods.
I googled and was trying to find info on how long it will take to shrink
from 20TB to around 12TBs or so. It didn't lead to much except for
someone with some seriously large drives and it taking about a week to shrink. That's a problem. If it will even take a few hours, I need a
new plan. Anyone have ideas on how long that will take?
Michael wrote:
On Sunday, 27 March 2022 22:04:45 BST Dale wrote:
That's sort of what I'm going to do. I'm going to divide things into
sections with some encrypted and some not.
I wonder if all you want to do is to encrypt some directories on your /home, then a different level of encryption would be more appropriate? Instead of encrypting a whole block device, you could just encrypt a directory tree or two, using ext4 encryption. e4crypt has been kicking around for a few years now and it is meant to be an improvement on eCryptfs.
https://lwn.net/Articles/639427/
https://wiki.gentoo.org/wiki/Ext4_encryption
WARNING: I'm not qualified to speak about this topic because my
experience is limited, but I'm interested all the same in reading your approach and other contributors advice.
That is the basic plan. I'll have /home as a normal open mount point.
That way I can login without a encryption password being needed. After
that, I plan to have other mount point(s) that are encrypted.
It may be
/home/dale/Data or something to that effect. I'm still doing some
checking but the normal non-encrypted stuff should easily fit on a 6TB
drive without encryption. I can then rebuild the two 8TB drives as
encrypted mount points with a different volume group thingy. When I
boot up, I can login in as usual then decrypt the other mount point and access it as needed or close it and it be encrypted until needed.
I've considered just encrypting /home completely but I don't have the
option of closing it while I'm logged into KDE. KDE wouldn't be able to access /home/dale/.kde or .config plus if I leave Seamonkey open, it
will want to store new emails to .mozilla as well. So, some things need
to be available and I'm not to worried about them being encrypted
anyway. So encrypting all of /home would be overkill plus would be a
problem for some things too, such as Seamonkey and KDE.
On 28/03/2022 13:52, Dale wrote:
I googled and was trying to find info on how long it will take to shrink
from 20TB to around 12TBs or so. It didn't lead to much except for
someone with some seriously large drives and it taking about a week to
shrink. That's a problem. If it will even take a few hours, I need a
new plan. Anyone have ideas on how long that will take?
Well, formatting my 3TB and 4TB drives I was looking at hours.
dm-integrity I think it was a day or so.
If you're shrinking 8TB off it, depends how much data it has to move
around, but I'd allow at least a day, maybe more ...
Cheers,
Wol
Time for plan B. I expect a drive purchase soon. $$$ Heck, it would
be faster to do backups, redo the whole thing and copy it all back. I
could copy it in chunks. First chunk gets me running and then copy
remaining stuff. Hmmmmm.
To be clear, cryptsetup is just a userspace command line tool for manipulating dm-crypt/LUKS stuff. dm-crypt is the kernel part. Lots of tools are structured this way. LVM is a thing in the kernel, and the lvcreate/pvcreate/etc. command line tools are just the userspace user interface. I assume they probably use a bunch of complicated ioctl()
calls on the LVM block devices to do their magic.
Howdy,
I sort of started this on another thread but wanted to nail a few things
down first. I'm wanting to encrypt some parts of my data on /home.
<<< SNIP >>>
On Mon, Mar 28, 2022 at 9:32 AM Dale <rdalek1967@gmail.com> wrote:
Time for plan B. I expect a drive purchase soon. $$$ Heck, it wouldIf you start having large volumes of data it probably makes sense to
be faster to do backups, redo the whole thing and copy it all back. I
could copy it in chunks. First chunk gets me running and then copy
remaining stuff. Hmmmmm.
split that off and handle it differently. I am storing my large stuff
on lizardfs for this reason (though if starting today I'd take another
look at moosefs or ceph). When you don't care so much about IOPS, or efficiency of small files, there are a lot of constraints you can
avoid with ext4. Distributed filesystems also have scaling benefits
because you don't have to try to cram your 10 hard drives into a
single host.
Ext4 can be grown online, but it can't be shrunk online. When you
start getting to large filesystems you need to consider
backup/restoration time and if you want availability you really want solutions that feature RAID and which can do all the operations you
need online. Simply having a backup might not be satisfactory if your
backup requires dozens of hours to restore, except as a last resort.
Dale wrote:
Howdy,
I sort of started this on another thread but wanted to nail a few things
down first. I'm wanting to encrypt some parts of my data on /home.
<<< SNIP >>>
OK. I looked into another hard drive but budget right now says no. So,
I went back to plan A. I managed to remove the 6TB drive and it is now
on it's own LVM thingy. I've moved enough over to mount it and use it
as my /home directory. I'm in the process of copying other non-critical >files over so I can move things around some more. Anyway, I'm using
rsync to copy things over. It works great, can restart if I need to
stop it etc etc but it has one thing that annoys me. While it is
copying things over, it makes my system slow to respond. Once the cache
in memory gets pretty full, it takes a while to switch desktops or for >programs to show up when I do get to a desktop. Seamonkey seems to be
hit hardest with this. I tried putting ionice in front of the command
but it is still slow. My CPU cores are a bit busy but nowhere near
100%. Most cores are switching from almost idle to around 40% at their >peak. If I added them all up, I'd say the total would average around
10%, 20% at the very most. I've got swapiness set pretty low and it
isn't using swap according to gkrellm.
Anyone have a idea how to make rsync not cause this problem? Is there >something besides ionice I need to use?
Thanks.
Dale
:-) :-)
down first. I'm wanting to encrypt some parts of my data on /home. <br> <<&l
Rsync has a bwlimit argument which helps here. Note that rsync copies
the whole file on what it considers local storage (which can be mounted network shares) ... this can cause a real slowdown.
Am Sun, Apr 03, 2022 at 09:59:22AM +0100 schrieb Wols Lists:
On 03/04/2022 02:15, Bill Kenworthy wrote:
Rsync has a bwlimit argument which helps here. Note that rsync copiesIt won't help on the initial copy, but look at the - I think it is -
the whole file on what it considers local storage (which can be mounted
network shares) ... this can cause a real slowdown.
--in-place option.
This one is mostly useful if space on the destination is tight or the data link (for FS commands) is slooow, because normally rsync creates a new temp file and moves it into place once the transfer is complete. This to ensure you never lose data due to a broken connection. If space is tight you could also consider --delete-before instead, to first do all deletions before copying the new stuff.
On 03/04/2022 02:15, Bill Kenworthy wrote:
Rsync has a bwlimit argument which helps here. Note that rsync copies
the whole file on what it considers local storage (which can be mounted network shares) ... this can cause a real slowdown.
It won't help on the initial copy, but look at the - I think it is - --in-place option.
It won't help with the "read and compare", but it only writes what has changed, so if a big file has changed slightly, it'll stop it re-copying
the whole file.
On 03/04/2022 02:15, Bill Kenworthy wrote:
Rsync has a bwlimit argument which helps here. Note that rsync copies
the whole file on what it considers local storage (which can be mounted network shares) ... this can cause a real slowdown.
It won't help on the initial copy, but look at the - I think it is - --in-place option.
It won't help with the "read and compare", but it only writes what has changed, so if a big file has changed slightly, it'll stop it re-copying the whole file.
Rsync has a bwlimit argument which helps here. Note that rsync copies the whole file on what it considers local storage (which can be mounted network shares) ... this can cause a real slowdown.It won't help on the initial copy, but look at the - I think it is - --in-place option.
This one is mostly useful if space on the destination is tight or the data link (for FS commands) is slooow, because normally rsync creates a new temp file and moves it into place once the transfer is complete.
And making a temporary file may be exactly what you DON'T want. I make heavy use of hard links,
and "make a temp file" absolutely buggers file system integrity ...
And my use case with LVM and backups, "make a temp file" does both
absolutely nothing to protect file system integrity
and makes every backup waste far more space than is necessary ...
So --in-place actually has a lot of uses outside your two examples. I have oodles of space, and both my source and target are on fast sata links in the same computer, but not using --in-place would be *very* costly for me.
Am Sun, Apr 03, 2022 at 05:05:07PM +0100 schrieb Wol:
Rsync has a bwlimit argument which helps here. Note that rsync copies >>>>> the whole file on what it considers local storage (which can be mounted >>>>> network shares) ... this can cause a real slowdown.It won't help on the initial copy, but look at the - I think it is -
--in-place option.
This one is mostly useful if space on the destination is tight or the data >>> link (for FS commands) is slooow, because normally rsync creates a new temp >>> file and moves it into place once the transfer is complete.
And making a temporary file may be exactly what you DON'T want. I make heavy >> use of hard links,
… which the -H option is for.
and "make a temp file" absolutely buggers file system integrity ...
Can you elaborate? A temp file is just like any other file. If you do in-place syncing, then it “buggers up” the original file instead of a temp.
And my use case with LVM and backups, "make a temp file" does both
absolutely nothing to protect file system integrity
I was not actually talking about file *system* integrity, but that if you overwrite in-place and the connection is lost during transfer, you are left with an incomplete file. Hence you just lost data on the destination. The FS itself is not adversely affected by that, only the file.
and makes every backup waste far more space than is necessary ...
How so? If it’s a COW filesystem, I can understand it (insofar as if you do in-place, then the existing file is deleted first, which gives the FS more space for the new copy).
So --in-place actually has a lot of uses outside your two examples. I have >> oodles of space, and both my source and target are on fast sata links in the >> same computer, but not using --in-place would be *very* costly for me.
Well I admit I don’t know much outside of my realm of local disks with ext4,
zfs and ssh. While I do run LVM on my laptop, it’s just so that I can have “partitions” on an otherwise fully encrypted block device. I never used any
of its more fancy features.
Rsync has a bwlimit argument which helps here. Note that rsync copies
the whole file on what it considers local storage (which can be
mounted network shares) ... this can cause a real slowdown.
BillK
that google didn't help with. I went and helped my sis-n-law with some garden stuff. That helped. ;-)-----Original Message-----
From: Dale <rdalek1967@gmail.com>
Sent: Monday, April 4, 2022 5:42 AM
To: gentoo-user@lists.gentoo.org
Subject: Re: [gentoo-user] LVM and moving things around
Bill Kenworthy wrote:
Rsync has a bwlimit argument which helps here. Note that rsync copiesI ended up just letting it do its thing. I didn't want to slow it down by much, just make my desktop able to respond better. I used nice and ionice to do this with emerge and it works great. I just thought I was missing some option for that command
the whole file on what it considers local storage (which can be
mounted network shares) ... this can cause a real slowdown.
BillK
added or not. I may not at the moment. Of course, once fiber internet gets here, that may change pretty soon.
As it stands now, I've copied enough over to get a free 8TB drive. I set up LUKS, which includes LVM, on the drive and am copying some more files onto the newly encrypted drive. Once everything is transferred, I'll then see if I need the other drive
until I figured out how to get the two together. Even tho I got it working, I'm still not real clear on how one part of it works. I'm just not clear enough on it to write one myself. A Gentoo wiki would be nice. There's one for the two separately but
If someone is really knowledgeable about LVM and LUKS and how to set up a encrypted hard drive, not a whole install but just a data drive, a howto for this would be really nice. I had to use a LUKS howto and a LVM howto and sort of merge commands
A little late to the party, but the other setting to look at when you're doing this kind of thing is "sysctl vm.dirty_ratio".
Now to find something to do while rsync copies over some 6TBs of files. O_O >>
Dale
:-) :-)
Basically it's what percentage of the disk cache can be dirty before the system forces all IO operations to be synchronous.
Setting it higher lets the system keep more data up-in-the-air while you're shuffling lots of stuff around. Of course, it also risks losing more if the system crashes in the middle of it all, so use it judiciously.
Setting dirty_ratio dirty_background_ratio, and dirty_writeback_centisecs appropriately when doing things with large amounts of data can significantly improve system responsiveness and, with rotational drives, throughput.
LMP
rsync, it creeps some in. When it does, that results in some slowness. I have a little script thing that clears all that but still, I may set it to 3 or maybe 2 for a bit. Me ponders the thought.-----Original Message-----
From: Dale <rdalek1967@gmail.com>
Sent: Monday, April 4, 2022 4:37 PM
To: gentoo-user@lists.gentoo.org
Subject: Re: [gentoo-user] LVM and moving things around
One thing that annoys me, it trying to use swap. I don't want to disable it because on occasion Firefox goes nuts and starting hogging memory really bad. I have swappiness set to like 5 or something which means it shouldn't use it but when using
I'm making progress. Feel sorry for those hard drives tho. ;-)
Dale
:-) :-)
I'm told that you can use cgroups for dealing with that kind of thing such that, for example, only Firefox is allowed to be swapped. I haven't had time to dig into it, but it seems like a useful tool.
Also, the compressed swap and zram swap devices with backing stores offer a fairly significant boost to the speed of swap so long as the data being swapped is compressible.
I don't know how you take advantage of it, but linux by default caches
disk i/o. You can tell it to "don't cache" and apparently it makes a
major difference. Given that rsync reads once and then never uses it
again, you don't want it cached.
| Sysop: | Keyop |
|---|---|
| Location: | Huddersfield, West Yorkshire, UK |
| Users: | 302 |
| Nodes: | 16 (2 / 14) |
| Uptime: | 100:53:57 |
| Calls: | 6,767 |
| Calls today: | 5 |
| Files: | 12,295 |
| Messages: | 5,376,432 |
| Posted today: | 1 |