• NTFS -v- exFAT

    From Jeff Gaines@21:1/5 to All on Wed Aug 11 09:37:20 2021
    Here's a question for the techies :-)

    In my sort out of what data I keep where, and how, I formatted 4 x 2TB
    SSDs as exFAT so I could use them with both Windows Machines and Macs.

    As a check that I am backing things up properly I am doing directory size checks and there is an enormous difference between the NTFS and exFAT
    total directory size. One directory contains nearly 13,000 small files of Flight Simulator Scenery and TreeSize Free is measuring the NTFS directory
    at 673 MB and the exFAT at 7.6 GB. The overall impact is that FS2002 is
    3.6 GB on NTFS and 32.9 GB on exFAT.

    It makes it look like I've screwed up my backup but the number of files is
    the same so it seems to be just the different file systems. Does that make sense?

    --
    Jeff Gaines Wiltshire UK
    Indecision is the key to flexibility

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jaimie Vandenbergh@21:1/5 to jgaines_newsid@yahoo.co.uk on Wed Aug 11 14:58:07 2021
    On 11 Aug 2021 at 10:37:20 BST, ""Jeff Gaines""
    <jgaines_newsid@yahoo.co.uk> wrote:


    Here's a question for the techies :-)

    In my sort out of what data I keep where, and how, I formatted 4 x 2TB
    SSDs as exFAT so I could use them with both Windows Machines and Macs.

    As a check that I am backing things up properly I am doing directory size checks and there is an enormous difference between the NTFS and exFAT
    total directory size. One directory contains nearly 13,000 small files of Flight Simulator Scenery and TreeSize Free is measuring the NTFS directory
    at 673 MB and the exFAT at 7.6 GB. The overall impact is that FS2002 is
    3.6 GB on NTFS and 32.9 GB on exFAT.

    It makes it look like I've screwed up my backup but the number of files is the same so it seems to be just the different file systems. Does that make sense?

    It's just the difference in filesystems. They don't store files by the
    byte, they store in blocks. Some filesystems will store the leftover bit
    at the end of a file, and tiny files, cleverly - exFAT does not.

    You can choose what size blocks your exFAT is formatted at formatting
    time, but they've probably defauled to 64kb on a 2TB filesystem. Which
    means on average 32kb of waste empty space per file, if the file sizes
    are evenly distributed.

    Even so, that does seem a little excessive.... are lots of the files
    really really tiny?

    (You may like to know that macOS can read NTFS disks, btw - so you could
    have one read/write exFAT and the rest NTFS, if workloads suited that.
    Or leave them plugged into one machine and access over the network,
    where it doesn't matter what the underlying filesystem is as the host
    computer handles it. Like a NAS. Weren't you going to put these in the
    NAS anyway?)

    Cheers - Jaimie
    --
    To every complex problem there is a solution which
    is simple, neat and wrong.
    -- HL Mencken

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jeff Gaines@21:1/5 to Vandenbergh on Wed Aug 11 16:05:58 2021
    On 11/08/2021 in message <ini6rvFmpjhU1@mid.individual.net> Jaimie
    Vandenbergh wrote:

    Even so, that does seem a little excessive.... are lots of the files
    really really tiny?

    (You may like to know that macOS can read NTFS disks, btw - so you could
    have one read/write exFAT and the rest NTFS, if workloads suited that.
    Or leave them plugged into one machine and access over the network,
    where it doesn't matter what the underlying filesystem is as the host >computer handles it. Like a NAS. Weren't you going to put these in the
    NAS anyway?)

    Hi Jaimie.

    I wrote a quick app in C# to do some measuring, it tells me that FS2002
    has 66,032 files in 497 directories amounting to 3.571 MB. The scenery
    texture files are 43 KB each and there are 44,155 of them in 12
    directories amounting to 1.845 MB for all the scenery files (it's an add
    on mapping program).

    Interestingly when I run the program over the exFAT drive it barfs and I
    get a Windows message saying th message pump has stalled, if I tell it to
    carry on it does and comes up with exactly the same figure as the NTFS
    drive which produced the result in seconds.

    I've just run AS SSD benchmark, the read speed for the WD Blue OS drive is
    492 MB/s (NTFS) the exFAT drive 377 MB/s and my brand new WD Blue NVMe (in
    a PCIe card) 375 MB/s, oh deary me why is that so low? I think I need to
    go off and see if WD do a driver for their NVMe.

    This is fascinating and I keep getting distracted in polishing the app,
    now where's the swamp pump...

    --
    Jeff Gaines Wiltshire UK
    I've been through the desert on a horse with no name.
    It was a right bugger to get him back when he ran off.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jeff Gaines@21:1/5 to Gaines on Wed Aug 11 16:32:45 2021
    On 11/08/2021 in message <xn0n1ihyz1f4415004@news.individual.net> Jeff
    Gaines wrote:

    I think I need to go off and see if WD do a driver for their NVMe.

    WD dashboard says I am using an unsupported driver but doesn't tell me how
    to get an updated one :-(

    I have put a question in to WD support.

    --
    Jeff Gaines Wiltshire UK
    If you ever find something you like buy a lifetime supply because they
    will stop making it

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jeff Gaines@21:1/5 to Gaines on Thu Aug 12 08:36:10 2021
    On 11/08/2021 in message <xn0n1iimt1g2koc005@news.individual.net> Jeff
    Gaines wrote:

    On 11/08/2021 in message <xn0n1ihyz1f4415004@news.individual.net> Jeff
    Gaines wrote:

    I think I need to go off and see if WD do a driver for their NVMe.

    WD dashboard says I am using an unsupported driver but doesn't tell me how
    to get an updated one :-(

    I have put a question in to WD support.

    I have dusted off my Asus Z170-K machine. The WD NVMe is reading at just
    under 2,000 MB/s but WD dashboard still complains of an unsupported
    driver. The Samsung NVMe (boot device) reads at 2448 MB/s.

    Definite gremlin problems though. Pinging the desktop machine from the
    laptop frequently times out which may be the cause ofe the hesistancy and stuttering I often see. To cap it off my Gigabit landline 'phone started
    to make crackiling noises last night and I can smell magic smoke!

    Whoops, spell checker not installed, E&OE.

    --
    Jeff Gaines Wiltshire UK
    George Washington was a British subject until well after his 40th birthday. (Margaret Thatcher, speech at the White House 17 December 1979)

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Chris Ridd@21:1/5 to Jaimie Vandenbergh on Fri Aug 13 19:51:30 2021
    On 11/08/2021 15:58, Jaimie Vandenbergh wrote:
    On 11 Aug 2021 at 10:37:20 BST, ""Jeff Gaines""
    <jgaines_newsid@yahoo.co.uk> wrote:


    Here's a question for the techies :-)

    In my sort out of what data I keep where, and how, I formatted 4 x 2TB
    SSDs as exFAT so I could use them with both Windows Machines and Macs.

    As a check that I am backing things up properly I am doing directory size
    checks and there is an enormous difference between the NTFS and exFAT
    total directory size. One directory contains nearly 13,000 small files of
    Flight Simulator Scenery and TreeSize Free is measuring the NTFS directory >> at 673 MB and the exFAT at 7.6 GB. The overall impact is that FS2002 is
    3.6 GB on NTFS and 32.9 GB on exFAT.

    It makes it look like I've screwed up my backup but the number of files is >> the same so it seems to be just the different file systems. Does that make >> sense?

    It's just the difference in filesystems. They don't store files by the
    byte, they store in blocks. Some filesystems will store the leftover bit
    at the end of a file, and tiny files, cleverly - exFAT does not.

    You can choose what size blocks your exFAT is formatted at formatting
    time, but they've probably defauled to 64kb on a 2TB filesystem. Which
    means on average 32kb of waste empty space per file, if the file sizes
    are evenly distributed.

    Even so, that does seem a little excessive.... are lots of the files
    really really tiny?

    (You may like to know that macOS can read NTFS disks, btw - so you could
    have one read/write exFAT and the rest NTFS, if workloads suited that.

    If you decide on NTFS as the common filesystem between your 2 worlds,
    Paragon sell a read/write NTFS driver for Macs: https://www.paragon-software.com/home/ntfs-mac/

    I'd be more inclined to use ZFS instead, but don't know how stable it is
    on Macs and Windows. At least it was designed this century ;-)

    Option C would be to avoid the problem and move your SSDs into a NAS, as
    Jaimie also says.

    --
    Chris

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)