1. Forum
  2. Usenet
  3. COMP.COMPRESSION

  • Additions to MTF?

    From Harry Potter@21:1/5 to All on Sat Oct 21 09:33:42 2023
    Hi! Recently, I found some bugs in my rendition of Adaptive Huffman codes, and the bug fixes cost me big time. :( After reimplementing and tweaking MTF, I found it to be better than Adaptive Huffman, but I'm not totally sure yet, as I need to debug it.
    Right now, the 16-bit version seems to be doing about 4.6% better than 7Zip, but I have two problem test files--and one of them is uncompressible using my technique but gives me about 7.8% using Deflate--and am still .4% off my long-term goal there. I
    had to kill some methods in the process. :( Is there anything I can add to MTF to make it better? Thank you.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Harry Potter@21:1/5 to All on Sat Oct 21 14:58:05 2023
    Good news: I'm doing 5.4% better than 7Zip now. :) Bad news: the problem files are still there. :( This particular technique is to be close-source for a while after publication, but I'm willing to reveal the following in the hopes that somebody here
    can help me fix the problem files:

    * I use Last16, where a recent LZ77 block is shortened to how many blocks ago was a match.
    * I use my own version of BPE.
    * I am using an MTF variant where the output is separated into ranges such that small numbers take fewer bits
    * I modified Deflate's lazy mechanism such that, if compressing a two-byte LZ77 reference, the next block has to do at least two bytes better before skipping the current byte.
    * I have a way to optimize values that are not in the range of a power of 2.

    When I look at the problem test file that is uncompressible with VBinDiff, I find at the beginning of the file a lot of 0's, so it should be more compressible. I don't know why it's not compressing.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)