Skip to content
This repository was archived by the owner on Feb 8, 2023. It is now read-only.
This repository was archived by the owner on Feb 8, 2023. It is now read-only.

Creating zstd dictionaries for compressible mime types #429

Open

Description

@RubenKelevra

I did some testing with the zstd dictionary feature, which speeds up compression/decompression speeds significantly and increases the compression ratio too.

I like to create a set of dictionaries for the future using them in ipfs for compression.

Currently, I only got a dictionary for the Linux-Kernel source code which can achieve a compression ratio of 5.931x. It should work on other c-projects quite well, too. (maybe some testing is needed)

There's no need for those files to have an open license (IMHO) since those dictionaries will just analyze the whole dataset for common patterns.

If you like to help: I search for large datasets of files (which are considered compressible), to make the most general dictionary for those file types as possible. Feel free to share either links here, via mail or via matrix ( ruben_kelevra ) with me.

More details on my research: ipld/specs#76 (comment)

(Please assign this ticket to me)

Activity

  1. hsanjuan commented on Apr 2, 2020

    @hsanjuan
    Contributor

    This does not seem to be describing a go-ipfs feature or how it should work. Right now this looks like a message for https://discuss.ipfs.io. Would you mind moving it or fixing the feature request? Thanks!

  2. RubenKelevra commented on Apr 3, 2020

    @RubenKelevra
    Author

    This does not seem to be describing a go-ipfs feature or how it should work. Right now this looks like a message for https://discuss.ipfs.io. Would you mind moving it or fixing the feature request? Thanks!

    This was actually just a ticket which I like to complete. It will take some time to create set precompiled dictionaries. This ticket is meant to track the progress.

    Maybe move it to ipfs/notes? :)

  3. ribasushi commented on Apr 4, 2020

    @ribasushi

    Yeah, let's move it to ipfs/notes. Having the go-ipfs binary carry predefined dictionaries ( of any kind ) is something I would be very opposed to. The research is extremely valuable on its own, but its results would fit in a separate standalone tool, not go-ipfs.

  4. transferred this issue fromipfs/kuboon Apr 4, 2020
  5. wallabra commented on Jan 24, 2022

    @wallabra

    How's progress in this? This entire feature is quite interesting and I can't wait to see where it goes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions