You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Feb 8, 2023. It is now read-only.
Repository navigation
This repository was archived by the owner on Feb 8, 2023. It is now read-only.
Creating zstd dictionaries for compressible mime types #429
I did some testing with the zstd dictionary feature, which speeds up compression/decompression speeds significantly and increases the compression ratio too.
I like to create a set of dictionaries for the future using them in ipfs for compression.
Currently, I only got a dictionary for the Linux-Kernel source code which can achieve a compression ratio of 5.931x. It should work on other c-projects quite well, too. (maybe some testing is needed)
There's no need for those files to have an open license (IMHO) since those dictionaries will just analyze the whole dataset for common patterns.
If you like to help: I search for large datasets of files (which are considered compressible), to make the most general dictionary for those file types as possible. Feel free to share either links here, via mail or via matrix ( ruben_kelevra ) with me.
This does not seem to be describing a go-ipfs feature or how it should work. Right now this looks like a message for https://discuss.ipfs.io. Would you mind moving it or fixing the feature request? Thanks!
This does not seem to be describing a go-ipfs feature or how it should work. Right now this looks like a message for https://discuss.ipfs.io. Would you mind moving it or fixing the feature request? Thanks!
This was actually just a ticket which I like to complete. It will take some time to create set precompiled dictionaries. This ticket is meant to track the progress.
Yeah, let's move it to ipfs/notes. Having the go-ipfs binary carry predefined dictionaries ( of any kind ) is something I would be very opposed to. The research is extremely valuable on its own, but its results would fit in a separate standalone tool, not go-ipfs.
Activity
hsanjuan commentedon Apr 2, 2020
This does not seem to be describing a go-ipfs feature or how it should work. Right now this looks like a message for https://discuss.ipfs.io. Would you mind moving it or fixing the feature request? Thanks!
RubenKelevra commentedon Apr 3, 2020
This was actually just a ticket which I like to complete. It will take some time to create set precompiled dictionaries. This ticket is meant to track the progress.
Maybe move it to ipfs/notes? :)
ribasushi commentedon Apr 4, 2020
Yeah, let's move it to
ipfs/notes. Having thego-ipfsbinary carry predefined dictionaries ( of any kind ) is something I would be very opposed to. The research is extremely valuable on its own, but its results would fit in a separate standalone tool, not go-ipfs.wallabra commentedon Jan 24, 2022
How's progress in this? This entire feature is quite interesting and I can't wait to see where it goes.