You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Feb 25, 2026. It is now read-only.
Repository navigation
This repository was archived by the owner on Feb 25, 2026. It is now read-only.
Next step is to write some more tests to make sure it's compatible with the existing Node.js implementation, and then integrate it into peermaps/ingest which will be the commandline tool for converting pbf files into an on-disk format. Then I'll look at the ipfs and filecoin apis to see how to best push the files onto the network and record usability notes along the way.
Maybe get into contact with the osm ops team. They might be able to help as well to make this happen https://twitter.com/OSM_Tech/status/1308465290263629825
They've been running the official osm tiles server and are seeking help for hosting. IPFS hosting would be a great solution.
(ps: If it'd be possible to do ipfs clustering or whatever the final solution will be. Please make it something that it's possible to configure the ipfs node to only pin a section of it. So that people who want to contribute some bandwidth and storage but only want to dedicate x amounts of GB's have that option)
I am the person who put the @OSM_Tech tweet out. The tile CDN renders on demand, for tile layer 0 to maybe zoom 14 the cache hit ratio is high, but decreases rapidly to zoom 19 where the majority of requests are rendered on demand (~1% hit ratio). This tool is handy for working out the number of map tiles there are: https://tools.geofabrik.de/calc/#type=geofabrik_standard&bbox=-144.227563,-75.111889,233.16404,81.381733 and the likely 3TB storage required over 16.5 billion tiles.
Reacted by Thibault Molleman, swedneck and Dietrich Ayala
Hi all! Thanks for chiming in with all this interesting information and links to existing conversation surrounding maptiles. Just want to give a status update --
We're doing preliminary testing on small metro areas and working on improving the performance for parsing and packing the full planet osm into the peermaps format. We're a bit behind schedule and it's hard to say if we'll have a full working demo of the full planet parsed by the end of the year; the goal is to get it from OSM dump to updated on the peer network daily, but that takes a significant lift of algorithms work.
We have made good progress on the ingest cli tool, and the code can be found here. We are blocked currently on a full demo with planet.osm, as the eyros database needs some more bug fixes and tweaks from @substack, including:
NaN bug @ 50m records
updating the wasm code and making sure that still works
double-checking read profile on queries with different input distributions
However, I have enough to work with the current version of the database to compile a report about feasibility and usability when building on both IPFS and Filecoin, just on a smaller metro area. I'll be working on that over the next two weeks (less than part-time) and then report back
Reacted by Dietrich Ayala and Marcin Rataj
Reacted by bertrandfalguiere, Dietrich Ayala and Marcin Rataj
Hi @autonome , @substack is still working diligently on the database but it's almost ready for primetime. The peermaps ingest pipeline has processed 6.04 billion records in 84h39m, and counting.
Once this is complete the team will create some demos, put it on the network, and finish up the grant. Thanks so much for your patience during this as the team worked out the kinks with the underlying database.
@substack @okdistribute hi! the updates last quarter are exciting and inspiring. would be great to chat about how to wrap up this phase, and talk about follow-on work.
Might be interesting to mention here: https://phabricator.wikimedia.org/T187601#7642399
Basically this guy has maintained a very popular osm tile server, but didn't really get much compensation for it. So he's shutting it down unless someone takes it over. Resulting in a lot of sites looking like this atm:
We're really happy to announce that we've completed the deliverables of this grant and have a working demo of peer-to-peer maps!
See peermaps/ingest which is a library for ingesting and diffing OpenStreetMap data, handling periodic changes. This tool converts OpenStreetMap pbf into the Peermaps format.
We also released a more 'all-in-one' commandline tool to diff .pbf files, convert to the Peermaps format, and pin both data formats on IPFS.
Dropping some comments here that were in gdoc, so we don't lose them.
IPFS peermaps experience
The peermaps dataset is generated from planet-osm.pbf, a binary mult-GB file of OpenStreetMap data with inline compression. The OSM features are denormalized and converted into a format that is easier to render, and each record is written into a spatial database.
The spatial database is stored as a collection of small files that can link to each other. These
files are very often less than 1 megabyte in size. The peermaps database is 100 GB split across 2.6 million files.
Using ipfs add on the directory structure finished in 4.8 hours which is not very much time
considering it took ~4 days to process the data from planet-osm.pbf. The resulting storage overhead was also not very much and a single machine appears entirely capable of seeding millions of hashes.
The biggest limitation has been the retrieval time for content. Using a local ipfs node and ipfs
companion browser extension with an empty cache, loading a new area of the map with this took could take several minutes:
But once the cache is primed and the data is available locally, the results are very fast. The
results are similar but slightly better using an ipfs http gateway: several minutes to fetch a new
area. Each request has a huge amount of latency, and multiple requests are required to walk the tree structure used by the spatial index.
Running an http server on the vps that is seeding to ipfs, the map loads much more quickly, at a
similar speed to other web maps.
Given these performance findings, it seems best for now to use ipfs for non-interactive bulk-loading purposes and not realtime rendering purposes. A web client could use a list of user-configurable http gateways that are known to have the entire peermaps dataset cached and those gateways could use tipfs to fetch the initial content and seed the dataset.
It may be possible to use libp2p to drive ipfs manually to fetch data more directly from peers which are known or highly likely to have files in the peermaps dataset. This manual strategy might have similar latency as fetching from a known list of http servers.
Installing lotus was not easy to try ingesting data into the Filecoin API. Would be great if there was a prepackaged executable. We had to use the Filecoin CLI, because of the size of the dataset. The commandline tool is called 'Lotus' and it needs to be built from source.
Activity
okdistribute commentedon Aug 17, 2020
Thanks @parkan and @substack for the update! We're currently working on Deliverable 1 "v1 of the peermaps/ingest library in Rust"
okdistribute commentedon Aug 25, 2020
I just pushed up a first pass of
georender-packin Rust, which encodespbffiles into a smaller format, currently called peermaps/bufferschema.Next step is to write some more tests to make sure it's compatible with the existing Node.js implementation, and then integrate it into peermaps/ingest which will be the commandline tool for converting pbf files into an on-disk format. Then I'll look at the ipfs and filecoin apis to see how to best push the files onto the network and record usability notes along the way.
thibaultmol commentedon Sep 22, 2020
Maybe get into contact with the osm ops team. They might be able to help as well to make this happen https://twitter.com/OSM_Tech/status/1308465290263629825
They've been running the official osm tiles server and are seeking help for hosting. IPFS hosting would be a great solution.
(ps: If it'd be possible to do ipfs clustering or whatever the final solution will be. Please make it something that it's possible to configure the ipfs node to only pin a section of it. So that people who want to contribute some bandwidth and storage but only want to dedicate x amounts of GB's have that option)
Firefishy commentedon Sep 22, 2020
I am the person who put the @OSM_Tech tweet out. The tile CDN renders on demand, for tile layer 0 to maybe zoom 14 the cache hit ratio is high, but decreases rapidly to zoom 19 where the majority of requests are rendered on demand (~1% hit ratio). This tool is handy for working out the number of map tiles there are: https://tools.geofabrik.de/calc/#type=geofabrik_standard&bbox=-144.227563,-75.111889,233.16404,81.381733 and the likely 3TB storage required over 16.5 billion tiles.
okdistribute commentedon Nov 22, 2020
Hi all! Thanks for chiming in with all this interesting information and links to existing conversation surrounding maptiles. Just want to give a status update --
We're doing preliminary testing on small metro areas and working on improving the performance for parsing and packing the full planet osm into the peermaps format. We're a bit behind schedule and it's hard to say if we'll have a full working demo of the full planet parsed by the end of the year; the goal is to get it from OSM dump to updated on the peer network daily, but that takes a significant lift of algorithms work.
autonome commentedon Jan 20, 2021
@okdistribute happy new year! Can you share where things are at?
okdistribute commentedon Jan 28, 2021
Hi @autonome !
We have made good progress on the ingest cli tool, and the code can be found here. We are blocked currently on a full demo with planet.osm, as the eyros database needs some more bug fixes and tweaks from @substack, including:
However, I have enough to work with the current version of the database to compile a report about feasibility and usability when building on both IPFS and Filecoin, just on a smaller metro area. I'll be working on that over the next two weeks (less than part-time) and then report back
okdistribute commentedon Jul 31, 2021
Hi @autonome , @substack is still working diligently on the database but it's almost ready for primetime. The peermaps ingest pipeline has processed 6.04 billion records in 84h39m, and counting.
Once this is complete the team will create some demos, put it on the network, and finish up the grant. Thanks so much for your patience during this as the team worked out the kinks with the underlying database.
autonome commentedon Aug 10, 2021
6.04 billion 😱
whyrusleeping commentedon Aug 10, 2021
@okdistribute is there a way I can pull a copy of the dataset? I have sufficient disk space and bandwidth.
whyrusleeping commentedon Aug 10, 2021
@substack oh cool, how are you doing the ingestion? Im super curious
whyrusleeping commentedon Aug 10, 2021
(reading through https://github.com/peermaps/ingest now)
autonome commentedon Feb 9, 2022
@substack @okdistribute hi! the updates last quarter are exciting and inspiring. would be great to chat about how to wrap up this phase, and talk about follow-on work.
thibaultmol commentedon Feb 25, 2022
Might be interesting to mention here:

https://phabricator.wikimedia.org/T187601#7642399
Basically this guy has maintained a very popular osm tile server, but didn't really get much compensation for it. So he's shutting it down unless someone takes it over. Resulting in a lot of sites looking like this atm:
okdistribute commentedon Apr 1, 2022
We're really happy to announce that we've completed the deliverables of this grant and have a working demo of peer-to-peer maps!
See peermaps/ingest which is a library for ingesting and diffing OpenStreetMap data, handling periodic changes. This tool converts OpenStreetMap pbf into the Peermaps format.
We also released a more 'all-in-one' commandline tool to diff .pbf files, convert to the Peermaps format, and pin both data formats on IPFS.
Here's the demo: https://ipfs.io/ipfs/Qmb4eNFHMoDpd5qxZsLuHxfTSwvzYnR6ReUtXqESEeD2aW/#bbox=-149.91,61.213,-149.89,61.223
And we have a blog post up and coming!
autonome commentedon Aug 16, 2023
For folks interested in decentralized spatial...
A WG here, who holds ~monthly talks: https://easierdata.org/dgwwg
Can sign up for dGWWG updates and talk announcements here: https://zc.vg/Dck0z
autonome commentedon Dec 1, 2023
Dropping some comments here that were in gdoc, so we don't lose them.