Skip to content
This repository was archived by the owner on Feb 25, 2026. It is now read-only.
This repository was archived by the owner on Feb 25, 2026. It is now read-only.

[TRACKING] Open Street Map #59

Open

Description

@parkan

Activity

  1. added
    type:open-grantOpen Grant Tracking (https://github.com/protocol/ipfs-grants/blob/master/open-grants/README.md)
    tracking issueTracking issue for a grant proposal in review
    on Aug 15, 2020
  2. self-assigned this
    on Aug 15, 2020
  3. okdistribute commented on Aug 17, 2020

    @okdistribute
    Contributor

    Thanks @parkan and @substack for the update! We're currently working on Deliverable 1 "v1 of the peermaps/ingest library in Rust"

  4. okdistribute commented on Aug 25, 2020

    @okdistribute
    Contributor

    I just pushed up a first pass of georender-pack in Rust, which encodes pbf files into a smaller format, currently called peermaps/bufferschema.

    Next step is to write some more tests to make sure it's compatible with the existing Node.js implementation, and then integrate it into peermaps/ingest which will be the commandline tool for converting pbf files into an on-disk format. Then I'll look at the ipfs and filecoin apis to see how to best push the files onto the network and record usability notes along the way.

  5. thibaultmol commented on Sep 22, 2020

    @thibaultmol

    Maybe get into contact with the osm ops team. They might be able to help as well to make this happen https://twitter.com/OSM_Tech/status/1308465290263629825
    They've been running the official osm tiles server and are seeking help for hosting. IPFS hosting would be a great solution.

    (ps: If it'd be possible to do ipfs clustering or whatever the final solution will be. Please make it something that it's possible to configure the ipfs node to only pin a section of it. So that people who want to contribute some bandwidth and storage but only want to dedicate x amounts of GB's have that option)

  6. Firefishy commented on Sep 22, 2020

    @Firefishy

    I am the person who put the @OSM_Tech tweet out. The tile CDN renders on demand, for tile layer 0 to maybe zoom 14 the cache hit ratio is high, but decreases rapidly to zoom 19 where the majority of requests are rendered on demand (~1% hit ratio). This tool is handy for working out the number of map tiles there are: https://tools.geofabrik.de/calc/#type=geofabrik_standard&bbox=-144.227563,-75.111889,233.16404,81.381733 and the likely 3TB storage required over 16.5 billion tiles.

  7. okdistribute commented on Nov 22, 2020

    @okdistribute
    Contributor

    Hi all! Thanks for chiming in with all this interesting information and links to existing conversation surrounding maptiles. Just want to give a status update --

    We're doing preliminary testing on small metro areas and working on improving the performance for parsing and packing the full planet osm into the peermaps format. We're a bit behind schedule and it's hard to say if we'll have a full working demo of the full planet parsed by the end of the year; the goal is to get it from OSM dump to updated on the peer network daily, but that takes a significant lift of algorithms work.

  8. autonome commented on Jan 20, 2021

    @autonome
    Contributor

    @okdistribute happy new year! Can you share where things are at?

  9. okdistribute commented on Jan 28, 2021

    @okdistribute
    Contributor

    Hi @autonome !

    We have made good progress on the ingest cli tool, and the code can be found here. We are blocked currently on a full demo with planet.osm, as the eyros database needs some more bug fixes and tweaks from @substack, including:

    • NaN bug @ 50m records
    • updating the wasm code and making sure that still works
    • double-checking read profile on queries with different input distributions

    However, I have enough to work with the current version of the database to compile a report about feasibility and usability when building on both IPFS and Filecoin, just on a smaller metro area. I'll be working on that over the next two weeks (less than part-time) and then report back

  10. okdistribute commented on Jul 31, 2021

    @okdistribute
    Contributor

    Hi @autonome , @substack is still working diligently on the database but it's almost ready for primetime. The peermaps ingest pipeline has processed 6.04 billion records in 84h39m, and counting.

    Once this is complete the team will create some demos, put it on the network, and finish up the grant. Thanks so much for your patience during this as the team worked out the kinks with the underlying database.

  11. autonome commented on Aug 10, 2021

    @autonome
    Contributor

    6.04 billion 😱

  12. whyrusleeping commented on Aug 10, 2021

    @whyrusleeping
    Member

    @okdistribute is there a way I can pull a copy of the dataset? I have sufficient disk space and bandwidth.

  13. whyrusleeping commented on Aug 10, 2021

    @whyrusleeping
    Member

    @substack oh cool, how are you doing the ingestion? Im super curious

  14. whyrusleeping commented on Aug 10, 2021

    @whyrusleeping
    Member
  15. autonome commented on Feb 9, 2022

    @autonome
    Contributor

    @substack @okdistribute hi! the updates last quarter are exciting and inspiring. would be great to chat about how to wrap up this phase, and talk about follow-on work.

  16. thibaultmol commented on Feb 25, 2022

    @thibaultmol

    Might be interesting to mention here:
    https://phabricator.wikimedia.org/T187601#7642399
    Basically this guy has maintained a very popular osm tile server, but didn't really get much compensation for it. So he's shutting it down unless someone takes it over. Resulting in a lot of sites looking like this atm:
    image

  17. okdistribute commented on Apr 1, 2022

    @okdistribute
    Contributor

    We're really happy to announce that we've completed the deliverables of this grant and have a working demo of peer-to-peer maps!

    See peermaps/ingest which is a library for ingesting and diffing OpenStreetMap data, handling periodic changes. This tool converts OpenStreetMap pbf into the Peermaps format.

    We also released a more 'all-in-one' commandline tool to diff .pbf files, convert to the Peermaps format, and pin both data formats on IPFS.

    Here's the demo: https://ipfs.io/ipfs/Qmb4eNFHMoDpd5qxZsLuHxfTSwvzYnR6ReUtXqESEeD2aW/#bbox=-149.91,61.213,-149.89,61.223

    And we have a blog post up and coming!

  18. autonome commented on Aug 16, 2023

    @autonome
    Contributor

    For folks interested in decentralized spatial...

    A WG here, who holds ~monthly talks: https://easierdata.org/dgwwg

    Can sign up for dGWWG updates and talk announcements here: https://zc.vg/Dck0z

  19. autonome commented on Dec 1, 2023

    @autonome
    Contributor

    Dropping some comments here that were in gdoc, so we don't lose them.

    IPFS peermaps experience

    The peermaps dataset is generated from planet-osm.pbf, a binary mult-GB file of OpenStreetMap data with inline compression. The OSM features are denormalized and converted into a format that is easier to render, and each record is written into a spatial database.

    The spatial database is stored as a collection of small files that can link to each other. These
    files are very often less than 1 megabyte in size. The peermaps database is 100 GB split across 2.6 million files.

    Using ipfs add on the directory structure finished in 4.8 hours which is not very much time
    considering it took ~4 days to process the data from planet-osm.pbf. The resulting storage overhead was also not very much and a single machine appears entirely capable of seeding millions of hashes.

    The biggest limitation has been the retrieval time for content. Using a local ipfs node and ipfs
    companion browser extension with an empty cache, loading a new area of the map with this took could take several minutes:

    https://ipfs.io/ipfs/QmS24zmgDz2jFdakvd6aT6sRXSGRXWJaB62aPTbvmpguBB/

    But once the cache is primed and the data is available locally, the results are very fast. The
    results are similar but slightly better using an ipfs http gateway: several minutes to fetch a new
    area. Each request has a huge amount of latency, and multiple requests are required to walk the tree structure used by the spatial index.

    Running an http server on the vps that is seeding to ipfs, the map loads much more quickly, at a
    similar speed to other web maps.

    Given these performance findings, it seems best for now to use ipfs for non-interactive bulk-loading purposes and not realtime rendering purposes. A web client could use a list of user-configurable http gateways that are known to have the entire peermaps dataset cached and those gateways could use tipfs to fetch the initial content and seed the dataset.

    It may be possible to use libp2p to drive ipfs manually to fetch data more directly from peers which are known or highly likely to have files in the peermaps dataset. This manual strategy might have similar latency as fetching from a known list of http servers.

    Installing lotus was not easy to try ingesting data into the Filecoin API. Would be great if there was a prepackaged executable. We had to use the Filecoin CLI, because of the size of the dataset. The commandline tool is called 'Lotus' and it needs to be built from source.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

tracking issueTracking issue for a grant proposal in reviewtype:open-grantOpen Grant Tracking (https://github.com/protocol/ipfs-grants/blob/master/open-grants/README.md)

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions