• Re: mkzip leads to 1 sec mtime difference

    From Helmut Giese@21:1/5 to Alexandru on Wed Dec 22 23:14:43 2021
    Alexandru <alexandru.dadalau@meshparts.de> schrieb:

    I think (not 100% sure) I discovered an issue with the mkzip package.
    I'm packing files into an 7z archive using mkzip.
    Unpacking the files is done in Windows using the 7z file manager.
    The result is that many files have 1 sec mtime difference compared to the original.

    After analysing the code, I think the issue could be related to this function below, which gets the "file mtime" for each source file and sets the result as the new mtime property to the compressed file:

    proc ::zipfile::mkzip::timet_to_dos {time_t} {
    set s [clock format $time_t -format {%Y %m %e %k %M %S}]
    scan $s {%d %d %d %d %d %d} year month day hour min sec
    expr {(($year-1980) << 25) | ($month << 21) | ($day << 16)
    | ($hour << 11) | ($min << 5) | ($sec >> 1)}
    }

    I don't quite understand how the code works but could if be that it leads to 1 sec difference to the input variable?

    Many thanks
    Alexandru
    Hi,
    ISTR from long ago that the mkzip file format under DOS could only
    implement 2 second intervalls - there were probably only 5 bits left
    for encoding the seconds - and this carried over until today.
    That's from memory so take it with a grain of salt.
    HTH
    Helmut

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Alexandru@21:1/5 to All on Wed Dec 22 13:44:04 2021
    I think (not 100% sure) I discovered an issue with the mkzip package.
    I'm packing files into an 7z archive using mkzip.
    Unpacking the files is done in Windows using the 7z file manager.
    The result is that many files have 1 sec mtime difference compared to the original.

    After analysing the code, I think the issue could be related to this function below, which gets the "file mtime" for each source file and sets the result as the new mtime property to the compressed file:

    proc ::zipfile::mkzip::timet_to_dos {time_t} {
    set s [clock format $time_t -format {%Y %m %e %k %M %S}]
    scan $s {%d %d %d %d %d %d} year month day hour min sec
    expr {(($year-1980) << 25) | ($month << 21) | ($day << 16)
    | ($hour << 11) | ($min << 5) | ($sec >> 1)}
    }

    I don't quite understand how the code works but could if be that it leads to 1 sec difference to the input variable?

    Many thanks
    Alexandru

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Alexandru@21:1/5 to Helmut Giese on Wed Dec 22 14:40:09 2021
    Helmut Giese schrieb am Mittwoch, 22. Dezember 2021 um 23:37:12 UTC+1:
    Hi,
    if you look at the last term
    proc ::zipfile::mkzip::timet_to_dos {time_t} {
    set s [clock format $time_t -format {%Y %m %e %k %M %S}]
    scan $s {%d %d %d %d %d %d} year month day hour min sec
    expr {(($year-1980) << 25) | ($month << 21) | ($day << 16)
    | ($hour << 11) | ($min << 5) | ($sec >> 1)}
    ^^^^^^^^^
    }
    you see that the seconds are shifted right one position, that is they
    are divided by 2 - I'd say it is intentional.
    Helmut

    Okay, so why would this be intentional?
    Can this function be modified, so that the mtime is preserved?

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Helmut Giese@21:1/5 to All on Wed Dec 22 23:37:08 2021
    Hi,
    if you look at the last term
    proc ::zipfile::mkzip::timet_to_dos {time_t} {
    set s [clock format $time_t -format {%Y %m %e %k %M %S}]
    scan $s {%d %d %d %d %d %d} year month day hour min sec
    expr {(($year-1980) << 25) | ($month << 21) | ($day << 16)
    | ($hour << 11) | ($min << 5) | ($sec >> 1)}

    ^^^^^^^^^
    }
    you see that the seconds are shifted right one position, that is they
    are divided by 2 - I'd say it is intentional.
    Helmut

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Rich@21:1/5 to Alexandru on Thu Dec 23 03:39:02 2021
    Alexandru <alexandru.dadalau@meshparts.de> wrote:
    Helmut Giese schrieb am Mittwoch, 22. Dezember 2021 um 23:37:12 UTC+1:
    Hi,
    if you look at the last term
    proc ::zipfile::mkzip::timet_to_dos {time_t} {
    set s [clock format $time_t -format {%Y %m %e %k %M %S}]
    scan $s {%d %d %d %d %d %d} year month day hour min sec
    expr {(($year-1980) << 25) | ($month << 21) | ($day << 16)
    | ($hour << 11) | ($min << 5) | ($sec >> 1)}
    ^^^^^^^^^
    }
    you see that the seconds are shifted right one position, that is they
    are divided by 2 - I'd say it is intentional.
    Helmut

    Okay, so why would this be intentional?

    Because the zip format uses the MS-DOS time storage format, and for
    MS-DOS time storage format the resolution of the time is two seconds.

    Can this function be modified, so that the mtime is preserved?

    No, because it is inherent in the zip format, the mkzip package is
    simply adhering to the zip format requirements.

    If you want perfectly accurate mtime packaging and extraction you will
    need to either use a different archive format that supports storing the
    full time value, or you'll have to store the full times externally and
    apply them manually after extracting from a zip.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Alexandru@21:1/5 to Rich on Thu Dec 23 03:32:08 2021
    Rich schrieb am Donnerstag, 23. Dezember 2021 um 04:39:08 UTC+1:
    Alexandru <alexandr...@meshparts.de> wrote:
    Helmut Giese schrieb am Mittwoch, 22. Dezember 2021 um 23:37:12 UTC+1:
    Hi,
    if you look at the last term
    proc ::zipfile::mkzip::timet_to_dos {time_t} {
    set s [clock format $time_t -format {%Y %m %e %k %M %S}]
    scan $s {%d %d %d %d %d %d} year month day hour min sec
    expr {(($year-1980) << 25) | ($month << 21) | ($day << 16)
    | ($hour << 11) | ($min << 5) | ($sec >> 1)}
    ^^^^^^^^^
    }
    you see that the seconds are shifted right one position, that is they
    are divided by 2 - I'd say it is intentional.
    Helmut

    Okay, so why would this be intentional?
    Because the zip format uses the MS-DOS time storage format, and for
    MS-DOS time storage format the resolution of the time is two seconds.
    Can this function be modified, so that the mtime is preserved?
    No, because it is inherent in the zip format, the mkzip package is
    simply adhering to the zip format requirements.

    If you want perfectly accurate mtime packaging and extraction you will
    need to either use a different archive format that supports storing the
    full time value, or you'll have to store the full times externally and
    apply them manually after extracting from a zip.

    When I use the 7z file manager (under Windows) the time stamps are preserved, so the 7z format supports this.
    The questions is: Is there a way in Tcl to create a 7z file with correct time stamps?
    Maybe not using the zip package, maybe by modifying the package.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Rich@21:1/5 to Alexandru on Thu Dec 23 17:27:15 2021
    Alexandru <alexandru.dadalau@meshparts.de> wrote:
    Rich schrieb am Donnerstag, 23. Dezember 2021 um 04:39:08 UTC+1:
    Alexandru <alexandr...@meshparts.de> wrote:
    Helmut Giese schrieb am Mittwoch, 22. Dezember 2021 um 23:37:12 UTC+1:
    Hi,
    if you look at the last term
    proc ::zipfile::mkzip::timet_to_dos {time_t} {
    set s [clock format $time_t -format {%Y %m %e %k %M %S}]
    scan $s {%d %d %d %d %d %d} year month day hour min sec
    expr {(($year-1980) << 25) | ($month << 21) | ($day << 16)
    | ($hour << 11) | ($min << 5) | ($sec >> 1)}
    ^^^^^^^^^
    }
    you see that the seconds are shifted right one position, that is they
    are divided by 2 - I'd say it is intentional.
    Helmut

    Okay, so why would this be intentional?
    Because the zip format uses the MS-DOS time storage format, and for
    MS-DOS time storage format the resolution of the time is two seconds.
    Can this function be modified, so that the mtime is preserved?
    No, because it is inherent in the zip format, the mkzip package is
    simply adhering to the zip format requirements.

    If you want perfectly accurate mtime packaging and extraction you will
    need to either use a different archive format that supports storing the
    full time value, or you'll have to store the full times externally and
    apply them manually after extracting from a zip.

    When I use the 7z file manager (under Windows) the time stamps are
    preserved, so the 7z format supports this.

    7z is not zip. 7z is a newer archive format, and presumably it
    supports higher resoultion timestamps than the MS-DOS timestamps used
    in zip. [1]

    The questions is: Is there a way in Tcl to create a 7z file with
    correct time stamps? Maybe not using the zip package, maybe by
    modifying the package.

    Yes, when you (or someone else) writes the appropriate Tcl code module
    to create a proper 7z file.


    [1] The zip format was created in the MS-DOS days - so inheriting
    MS-DOS's timestamp storage format made sense at that time. The
    timestamp storage format has not been changed since (most likely in the
    name of backwards compatibility.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Alexandru@21:1/5 to Rich on Fri Dec 24 13:21:00 2021
    Rich schrieb am Donnerstag, 23. Dezember 2021 um 18:27:20 UTC+1:
    Alexandru <alexandr...@meshparts.de> wrote:
    Rich schrieb am Donnerstag, 23. Dezember 2021 um 04:39:08 UTC+1:
    Alexandru <alexandr...@meshparts.de> wrote:
    Helmut Giese schrieb am Mittwoch, 22. Dezember 2021 um 23:37:12 UTC+1: >> >> Hi,
    if you look at the last term
    proc ::zipfile::mkzip::timet_to_dos {time_t} {
    set s [clock format $time_t -format {%Y %m %e %k %M %S}]
    scan $s {%d %d %d %d %d %d} year month day hour min sec
    expr {(($year-1980) << 25) | ($month << 21) | ($day << 16)
    | ($hour << 11) | ($min << 5) | ($sec >> 1)}
    ^^^^^^^^^
    }
    you see that the seconds are shifted right one position, that is they >> >> are divided by 2 - I'd say it is intentional.
    Helmut

    Okay, so why would this be intentional?
    Because the zip format uses the MS-DOS time storage format, and for
    MS-DOS time storage format the resolution of the time is two seconds.
    Can this function be modified, so that the mtime is preserved?
    No, because it is inherent in the zip format, the mkzip package is
    simply adhering to the zip format requirements.

    If you want perfectly accurate mtime packaging and extraction you will
    need to either use a different archive format that supports storing the
    full time value, or you'll have to store the full times externally and
    apply them manually after extracting from a zip.

    When I use the 7z file manager (under Windows) the time stamps are preserved, so the 7z format supports this.
    7z is not zip. 7z is a newer archive format, and presumably it
    supports higher resoultion timestamps than the MS-DOS timestamps used
    in zip. [1]
    The questions is: Is there a way in Tcl to create a 7z file with
    correct time stamps? Maybe not using the zip package, maybe by
    modifying the package.
    Yes, when you (or someone else) writes the appropriate Tcl code module
    to create a proper 7z file.


    [1] The zip format was created in the MS-DOS days - so inheriting
    MS-DOS's timestamp storage format made sense at that time. The
    timestamp storage format has not been changed since (most likely in the
    name of backwards compatibility.

    I thought I'm already creating an archive in 7z format.
    So the actual format is zip not the native 7z format?

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Christian Gollwitzer@21:1/5 to All on Fri Dec 24 22:48:48 2021
    Am 24.12.21 um 22:21 schrieb Alexandru:
    Rich schrieb am Donnerstag, 23. Dezember 2021 um 18:27:20 UTC+1:
    [1] The zip format was created in the MS-DOS days - so inheriting
    MS-DOS's timestamp storage format made sense at that time. The
    timestamp storage format has not been changed since (most likely in the
    name of backwards compatibility.

    I thought I'm already creating an archive in 7z format.
    So the actual format is zip not the native 7z format?

    Most probably yes. You can use the "file" command on Linux or
    Cygwin/MSYS to test the actual file format:

    (base) Apfelkiste:Sources chris$ file HZB-MLBG.7z
    HZB-MLBG.7z: 7-zip archive data, version 0.4
    (base) Apfelkiste:Sources chris$ file RawTherapee_OSX_10.9_64_5.4.zip RawTherapee_OSX_10.9_64_5.4.zip: Zip archive data, at least v2.0 to extract


    "file" looks into the file, it does not check the ending.

    Christian

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Rich@21:1/5 to Alexandru on Sat Dec 25 03:05:49 2021
    Alexandru <alexandru.dadalau@meshparts.de> wrote:
    Rich schrieb am Donnerstag, 23. Dezember 2021 um 18:27:20 UTC+1:
    Alexandru <alexandr...@meshparts.de> wrote:
    Rich schrieb am Donnerstag, 23. Dezember 2021 um 04:39:08 UTC+1:
    Alexandru <alexandr...@meshparts.de> wrote:
    Okay, so why would this be intentional?
    Because the zip format uses the MS-DOS time storage format, and for
    MS-DOS time storage format the resolution of the time is two seconds.
    Can this function be modified, so that the mtime is preserved?
    No, because it is inherent in the zip format, the mkzip package is
    simply adhering to the zip format requirements.

    If you want perfectly accurate mtime packaging and extraction you will
    need to either use a different archive format that supports storing the >> >> full time value, or you'll have to store the full times externally and
    apply them manually after extracting from a zip.

    When I use the 7z file manager (under Windows) the time stamps are
    preserved, so the 7z format supports this.
    7z is not zip. 7z is a newer archive format, and presumably it
    supports higher resoultion timestamps than the MS-DOS timestamps used
    in zip. [1]
    The questions is: Is there a way in Tcl to create a 7z file with
    correct time stamps? Maybe not using the zip package, maybe by
    modifying the package.
    Yes, when you (or someone else) writes the appropriate Tcl code module
    to create a proper 7z file.

    [1] The zip format was created in the MS-DOS days - so inheriting
    MS-DOS's timestamp storage format made sense at that time. The
    timestamp storage format has not been changed since (most likely in the
    name of backwards compatibility.

    I thought I'm already creating an archive in 7z format.
    So the actual format is zip not the native 7z format?

    What are you creating the archive with? Your original post implied you
    were using Tcl's mkzip - which produces "zip" format archives.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Alexandru@21:1/5 to Rich on Sat Dec 25 10:16:51 2021
    Rich schrieb am Samstag, 25. Dezember 2021 um 04:05:54 UTC+1:
    Alexandru <alexandr...@meshparts.de> wrote:
    Rich schrieb am Donnerstag, 23. Dezember 2021 um 18:27:20 UTC+1:
    Alexandru <alexandr...@meshparts.de> wrote:
    Rich schrieb am Donnerstag, 23. Dezember 2021 um 04:39:08 UTC+1:
    Alexandru <alexandr...@meshparts.de> wrote:
    Okay, so why would this be intentional?
    Because the zip format uses the MS-DOS time storage format, and for
    MS-DOS time storage format the resolution of the time is two seconds. >> >> > Can this function be modified, so that the mtime is preserved?
    No, because it is inherent in the zip format, the mkzip package is
    simply adhering to the zip format requirements.

    If you want perfectly accurate mtime packaging and extraction you will >> >> need to either use a different archive format that supports storing the >> >> full time value, or you'll have to store the full times externally and >> >> apply them manually after extracting from a zip.

    When I use the 7z file manager (under Windows) the time stamps are
    preserved, so the 7z format supports this.
    7z is not zip. 7z is a newer archive format, and presumably it
    supports higher resoultion timestamps than the MS-DOS timestamps used
    in zip. [1]
    The questions is: Is there a way in Tcl to create a 7z file with
    correct time stamps? Maybe not using the zip package, maybe by
    modifying the package.
    Yes, when you (or someone else) writes the appropriate Tcl code module
    to create a proper 7z file.

    [1] The zip format was created in the MS-DOS days - so inheriting
    MS-DOS's timestamp storage format made sense at that time. The
    timestamp storage format has not been changed since (most likely in the
    name of backwards compatibility.

    I thought I'm already creating an archive in 7z format.
    So the actual format is zip not the native 7z format?
    What are you creating the archive with? Your original post implied you
    were using Tcl's mkzip - which produces "zip" format archives.

    I don't remember exactly how I got this impression. It was obviously wrong.
    I thought the package cann also produce 7z archives. The ending of the created archives is 7z so that was the missleading point. Now I'm thinking: Maybe to just use the precompiles 7-zip executable to create the archives instead mkzip.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Rich@21:1/5 to Alexandru on Sun Dec 26 15:04:50 2021
    Alexandru <alexandru.dadalau@meshparts.de> wrote:
    Rich schrieb am Samstag, 25. Dezember 2021 um 04:05:54 UTC+1:
    Alexandru <alexandr...@meshparts.de> wrote:
    I thought I'm already creating an archive in 7z format.
    So the actual format is zip not the native 7z format?

    What are you creating the archive with? Your original post implied
    you were using Tcl's mkzip - which produces "zip" format archives.

    I don't remember exactly how I got this impression. It was obviously
    wrong.

    I thought the package cann also produce 7z archives. The ending of
    the created archives is 7z so that was the missleading point.

    Ah, the ms-windows stupidity of "file extension defines file format".
    The name of a file can be anything, that has no bearing on what format
    makes up the contents. Naming a zip format file *.7z no more makes it
    a 7z format file than renaming a .png file to .txt makes the .png into
    a text file.

    Now I'm thinking: Maybe to just use the precompiles 7-zip executable
    to create the archives instead mkzip.

    If having perfect timestamps is important, then this is one way to go
    (although test to verify that actual 7z format archives do in fact
    store full resolution timestamps).

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Schelte@21:1/5 to Rich on Sun Dec 26 22:05:20 2021
    On 26/12/2021 16:04, Rich wrote:
    Alexandru <alexandru.dadalau@meshparts.de> wrote:
    Now I'm thinking: Maybe to just use the precompiles 7-zip executable
    to create the archives instead mkzip.

    If having perfect timestamps is important, then this is one way to go (although test to verify that actual 7z format archives do in fact
    store full resolution timestamps).

    In addition to zip files, the 7z program can also process tgz files. Tar archives contain timestamps with at least 1 second resolution. So,
    depending on your requirements, you may be able to use the tar package
    from tcllib together with zlib to create compressed files that can be
    unpacked by 7z with correct time stamps.

    Something along the lines of (untested):

    package require tar
    set fd [open output.tgz wb]
    zlib push gzip $fd -level 9
    tar::create $fd $files -chan
    close $fd


    Schelte.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Alexandru@21:1/5 to Schelte on Tue Dec 28 11:01:23 2021
    Schelte schrieb am Sonntag, 26. Dezember 2021 um 22:05:26 UTC+1:
    On 26/12/2021 16:04, Rich wrote:
    Alexandru <alexandr...@meshparts.de> wrote:
    Now I'm thinking: Maybe to just use the precompiles 7-zip executable
    to create the archives instead mkzip.

    If having perfect timestamps is important, then this is one way to go (although test to verify that actual 7z format archives do in fact
    store full resolution timestamps).

    In addition to zip files, the 7z program can also process tgz files. Tar archives contain timestamps with at least 1 second resolution. So,
    depending on your requirements, you may be able to use the tar package
    from tcllib together with zlib to create compressed files that can be unpacked by 7z with correct time stamps.

    Something along the lines of (untested):

    package require tar
    set fd [open output.tgz wb]
    zlib push gzip $fd -level 9
    tar::create $fd $files -chan
    close $fd


    Schelte.

    Thanks Schelte, your code seems to solve the issue and it's even easier that with the mkzip, because it automatically handles files from different directories or different drives. It automatically creates the nested folder structure inside the archives.
    Many thanks!
    Alexandru

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)