Repository navigation
Add open street map grant - #41
Conversation
255cfc3 to
abdc68b
Compare
|
Thanks! @mishmosh is my username for future reference. |
|
@mishmosh awesome, fixed! |
abdc68b to
f55c64a
Compare
|
|
||
| * A bash script can download the latest .pbf file from OpenStreetMap website and run the commandline tool to host the latest OpenStreetMap and Peermaps data on IPFS and Filecoin. | ||
| * A user can use an IPFS or Filecoin client to download both Peermaps and OpenStreetMap data. | ||
| * Peermaps team will maintain a mirror of this data. |
There was a problem hiding this comment.
Can you add deliverables / acceptance criteria for:
- Example code to embed map that uses the tiles published from IPFS (or HTTP gateway) by the Peermaps node
- Blog post written for the IPFS blog that introduces the project, and demos an embedded map view
There was a problem hiding this comment.
Or can collapse the deliverables and acceptance criteria sections, maybe clearer that way?
There was a problem hiding this comment.
These are already in there under deliverables -- I collapsed them so it would be clearer.
|
|
||
| Also include any relevant constraints that the implementer should be aware of before beginning this project.--> | ||
|
|
||
| We plan to run the script on a node as soon as we get the commandline tool |
There was a problem hiding this comment.
Can you add a sentence here that clarifies what the running node is? Eg, is it a go-ipfs node? Does it publish the processed/indexed data, and how often?
There was a problem hiding this comment.
Updated, thanks.
|
This is awesome! Great work @okdistribute @autonome and @mishmosh!! |
vmx
left a comment
There was a problem hiding this comment.
I'm excited to see this proposal. I wanted to look into Peermaps for ages, especially after Substack's talk that the DTN in Berlin. I guess it's time to finally have a closer look.
My background is not so much OSM, but rather open source GIS (e.g. OSGeo), hence my questions are rather GIS themed.
|
|
||
| We are building Peermaps, a distributed, offline-friendly alternative to commercial map providers such as Google Maps. Instead of fetching data from a centralized tile service, your computer fetches map data from other peers across the network. With the powerful inverse scaling dynamics of peer-to-peer, we can run a mapping platform at a fixed, modest cost, no matter how popular it becomes. | ||
|
|
||
| We've developed a new standard format for map data that is better for decentralized contexts. [This specification can be found here.](https://github.com/peermaps/docs/blob/master/bufferschema.md) The original OSM data format is XML and is not great for random access or space efficiency. But both of these features -- small footprint and efficient access -- will be critical for practical use on a distributed storage network. A typical size for data in XML format is at least 87GB, and the OSM pbf format reduces it to 50GB. The peermaps format should be much smaller and faster by using compact buffers, triangulation, and clever indexing developed in Rust. |
There was a problem hiding this comment.
I'd like to mention another promising data format called FlatGeobuf. It seems similar to your bufferschema. So far I've only read and spoke with people about FlatGeobuf that were all excited about it, so I cannot tell from personal experience how well it actually works (it's been on my todo list for ages). It also has a Rust implementation.
I don't propose moving away from your own format to move to another random other format. I just thought they share similar goals and perhaps there could be synergy effects from lessons learned with your format. The FlatGeobuf comes more from a GIS, rather than an OSM background and I always find it interesting bridging the gaps between those two worlds.
There was a problem hiding this comment.
Thanks for the link! We will look into this. The peermaps format right now is still in alpha, so we can look to adopting FlatGeobuf or at minimum learn from it.
There was a problem hiding this comment.
@substack says this might be too slow of a format to use at runtime for rendering the maps efficiently, but when we do a more formal release at the end of the year, we will write something up about why not FlatGeobuf, if we decide not to adopt it.
There was a problem hiding this comment.
The peermaps format is more about computer graphics, not GIS or even OSM. It's the minimal amount you need to render features with minimal linear processing on the data and only the attributes that are used for the graphics layer are used. FlatGeobuf appears to use many of the features native to OSM/geojson from what I can see https://github.com/bjornharrtell/flatgeobuf/blob/master/src/fbs/header.fbs#L5-L24 and that puts it more in the scope of tools like mapbox vector tiles as a more compact geojson. For what that is, it's useful and people use these formats to render tiles all the time, it's just not the approach that the peermaps graphics format takes and I think there are a lot of unique advantages to this graphics-first approach, particularly for making custom map shaders on the same large shared p2p dataset. The peermaps buffer schema records map relatively directly onto the elements and attributes you need to pass into a shader for GL_TRIANGLES and GL_TRIANGLE_STRIP rendering.
There was a problem hiding this comment.
Thanks for that information @substack. I suspected that it is for rendering (as it also mentions triangulation), it totally makes sense for peermaps. I think the Mapbox vector tiles also started as a format supposed to be rendered directly, but somehow it seems to have grown beyond that.
There was a problem hiding this comment.
only the attributes that are used for the graphics layer are used
So the focus is on keeping the minimum amount of data that can be rendered easily. If you would want additional information on a geometry (e.g. by clicking), then this would be a separate service that gets the data from somewhere else, wouldn't it (this is again a kind of out of scope question for this grant, but I'm just so interested in all this)?
|
|
||
| This grant will help us use IPFS as a "hot" peer-to-peer storage network for Peermaps data for near-real-time access. This will be a premiere application built on IPFS that will greatly improve digital mapping. There is between 3-4GB of XML updated per week on OpenStreetMap -- we (bits.coop peermaps team) will run a node will download, diffs, and pack those changes into the Peermaps format on a regular basis. | ||
|
|
||
| This script will also put OpenStreetMap data in the original (.pbf) format on the IPFS and Filecoin networks for cold storage, improving the overall resiliency and download speeds for the data. We figured this will be nice to do at the same time, since the processing node will have to downloading these large OSM files anyway, it would make sense to also upload the original (pre-processed) versions to IPFS and Filecoin. |
There was a problem hiding this comment.
- Do you have the "current state" dumps or the one with the full history in mind?
- Do you plan to store the official dumps as they are or will they be processed?
Both questions target at the required storage space and how much additional space will be needed every week. I ran a little experiment and looked into how a diff between two osm.pbf dumps looks like. I used dumps from my local area from geofabrik, so I don't know if the Planet dumps are the same. With those smaller dumps there would hardly be any de-duplication possible as lots in-place updates happen. Do you know if that's the same with the Planet dumps?
There was a problem hiding this comment.
It's the same with osm.pbf planet files. Due to the block level compression and sort order, there's almst zero de-duplication and very very little potential to reuse data from an earlier pbf file.
You may take a look at some discussion on twitter earlier this year (e.g. https://twitter.com/komzpa/status/1223405925068918784). PBF is not update friendly in any way.
There was a problem hiding this comment.
Thanks for linking the discussion. Improving this situation is surely out of scope for the proposal, but it's an interesting topic nonetheless. I wondered if tools expect that specific sorter order, or if a Planet file could contain the same information, just with an updater friendlier order. So that less blocks are change per update. Though probably a new format would be easier/make more sense then operating within the limitations of the existing one.
There was a problem hiding this comment.
Well, currently, osm.pbf planet files are sorted by object type / object id / object version. I believe most tools would actually depend on this particular order. As an example you would need to process all nodes in a first step, so that subsequent way processing can reference those nodes and calculate the way geoemtry.
As you probably know, changes on OSM objects occur by creating/modifying/deleting objects. While creation would add new objects with version number 1, all other operations create a new object version for an existing object. This way, a single new object version would change an existing compressed block, and probably would cause reassignment of a number of objects to different blocks due to changes in compression and object sizes.
There was a problem hiding this comment.
Compression blocks have a recommended size if I read the wiki page correctly, so this could be stretched a bit, so that blocks get bigger a bit before objects are push into a new block.
Anyway, I guess we are getting quite out of scope here :)
There was a problem hiding this comment.
The current text says:
This script will also put OpenStreetMap data in the original (.pbf) format on the IPFS and Filecoin networks for cold storage
And @substack says:
We are only planning to store the data produced by processing planet.osm:
So the peermaps team will take care (also infra wise) of the storage of the processes format. The scripts/pipeline will also make it possible to store the original data in IPFS/Filecoin, though those nodes will be operated by someone else (not the peermaps team). Is that a correct summary?
So the workflow is: OSM Planet -> custom file-based format which still contains all the data (i.e. all the tags) -> processing step that builds geometries etc. and outputs only the data that is needed for rendering purpose (and has a spatial index on top/is a spatial index).
The custom file-base format could also potentially be used by other projects.
There was a problem hiding this comment.
Yes, the plan is to look into the IPFS infra clusters and see what options there could be to have PL host it, or if that fails, we could try to get some free compute with some other place. It is pretty straightforward once the tool is built to work out the details on where it will be hosted, and since it is an open source public benefit project, we are confident we can get some support on this end.
There was a problem hiding this comment.
@ribasushi can you comment on whether chunker work could help out here?
There was a problem hiding this comment.
do you have cost estimates for Filecoin storage/pipping cost estimates for IPFS cold storage? is there budget to support this storage or will it need to be included as a follow-on grant?
There was a problem hiding this comment.
@parkan Not right away, but maybe in the very near future 😜
From previous discussions, and reading this thread it seems that presently it is out of scope to make any changes to the "master" pbf streams themselves, which is where deduplication can actually happen (diffs decidedly can not be de-duplicated based on their contents alone). So for the time being the answer is "not really"
However: the basis of the data stored seems to be an independent fileblock with stable references to "other things": this is the kind of data that is amenable to deduplication, provided the "edges" of each "block" can be discovered reliably.
If one goes down that path (trying to design and quickly test) the time-performance and space-performance of such a protocol: the framework which I have almost-but-not-yet finished would help tremendously, as it abstracts away the ingestion, stats counting, and even IPFS-export, so that all a user needs to do is experiment with a single function taking successive buffers and "finding" the cutpoints within.
At present I am racing to make the interface I described work in the presence of multiple "parsers", when this is ready (ETA ~4 days) I will drop links that better illustrate what I mean here.
|
|
||
| We plan to run the commandline tool to diff pbf files, convert to the | ||
| Peermaps format, and upload to IPFS. This will be run on a computer every week. | ||
| This is how often a new OSM diff file is published. Long-term goal of this |
There was a problem hiding this comment.
This description is not quite accurate, OSM publishes minutely diff updates: https://planet.openstreetmap.org/replication/minute/ - which are also being aggregated into daily diff updates (https://planet.openstreetmap.org/replication/day/).
What is being published on a weekly basis is the OSM planet file (https://planet.openstreetmap.org/pbf/). This format is not a delta from the previous week, it always contains all the most current data.
We usually recommended to download such a planet file only once, and keep it up to date locally by applying minutely/hourly/daily diff files. However, if you're redistributing a planet file, this may be a different story. Current Torrent-based planet download leverages existing Web feeds (provided by OSM mirrors) and other p2p hosts. In this case, the distribution is based on the same planet file as available on planet.openstreetmap.org (i.e. without resorting to diff file updates).
There was a problem hiding this comment.
Good point, we looked at this -- we originally had gotten the Weekly estimate based upon the main website, which says 'weekly dumps..' (https://planet.openstreetmap.org/).
After the first ingest, we can probably run the patches daily, since it would not take nearly as much compute time. Doing the patches even faster on a minute scale may be possible at some point as well, although it may not be practical depending on network performance -- this will be something that needs to be thought through and tested. As noted in the proposal, the aim is to have distributed indexing, which could speed things up.
Thanks for the clarification!
There was a problem hiding this comment.
After the first ingest, we can probably run the patches daily, since it would not take nearly as much compute time. Doing the patches even faster on a minute scale may be possible at some point as well, although it may not be practical depending on network performance -- this will be something that needs to be thought through and tested. As noted in the proposal, the aim is to have distributed indexing, which could speed things up.
Well, running minute patches is the way to go I think. Since daily patches would add peak loads to the network once a day.
With minute patches it would be possible to spread the load over the whole day, avoiding that there are any peaks.
There was a problem hiding this comment.
Good point. This is a parameter than can be easily tweaked in a cronjob once we have the ingest tool, so we'll look into that!
There was a problem hiding this comment.
For a minutely based update, you might end up doing more work and create more traffic, as every tiny change potentially invalidates a tile. Merging multiple changes over a period of time into a single tile update would be cheaper at the expense of having slightly more outdated data. You might have to experiment a bit here to find the sweet spot.
|
Thanks for the work on this - sounds really great! :) |
RubenKelevra
left a comment
There was a problem hiding this comment.
Some questions and recommendations
|
|
||
| We are building Peermaps, a distributed, offline-friendly alternative to commercial map providers such as Google Maps. Instead of fetching data from a centralized tile service, your computer fetches map data from other peers across the network. With the powerful inverse scaling dynamics of peer-to-peer, we can run a mapping platform at a fixed, modest cost, no matter how popular it becomes. | ||
|
|
||
| We've developed a new standard format for map data that is better for decentralized contexts. [This specification can be found here.](https://github.com/peermaps/docs/blob/master/bufferschema.md) The original OSM data format is XML and is not great for random access or space efficiency. But both of these features -- small footprint and efficient access -- will be critical for practical use on a distributed storage network. A typical size for data in XML format is at least 87GB, and the OSM pbf format reduces it to 50GB. The peermaps format should be much smaller and faster by using compact buffers, triangulation, and clever indexing developed in Rust. |
There was a problem hiding this comment.
A typical size for data in XML format is at least 87GB, and the OSM pbf format reduces it to 50GB.
The number for XML is off... the 87 GB referred to a multithreaded bzip2 compressed file. The uncompressed file size is way(!) bigger: 1205.8 GB (for 2020-04-01).
|
|
||
| <!-- Link any resources that might be helpful for an implementer who is working on this project.--> | ||
|
|
||
| * OpenStreetMap weekly data dump (planet.osm, 87GB): https://planet.openstreetmap.org/ |
There was a problem hiding this comment.
(planet.osm, 87GB)
planet.osm.bz2
| * MapBox Pricing: https://www.mapbox.com/pricing/ | ||
| * How to download OSM data: https://wiki.openstreetmap.org/wiki/Downloading_data | ||
| * Statistics of daily users: https://www.openstreetmap.org/stats/data_stats.html | ||
| * Diff size can range between 3-4GB of XML per week. |
There was a problem hiding this comment.
Please consider minute updates for this project. A minute is usually below 2 MB osm-XML.
| <!-- Please describe why the work that will come out of this Targeted Grant is valuable for the IPFS ecosystem. --> | ||
|
|
||
|
|
||
| We will be able to lower costs for developers who are currently using embedded Google Maps or Mapbox, which both cost money per impression after a certain limit. |
There was a problem hiding this comment.
Mapbox offers Vector tiles, which can be used to be rendered in a mobile application with a stylesheet with different styles, while still using the same data.
Since it's not stated directly anywhere, which formats this project aims to support, I gonna ask here if the plan is to offer vector tiles similarity to the Mapbox' ones.
|
|
||
|
|
||
| 1. `peermaps/ingest`: A library for injesting and diffing OpenStreetMap data, handling periodic changes. This tool will convert OpenStreetMap into the Peermaps format. This deliverable is co-funded by this devgrant and SamsungNEXT; | ||
| 2. Commandline tool using `peermaps/ingest` to diff .pbf files, convert to the Peermaps format, and pin both data formats on IPFS; |
There was a problem hiding this comment.
It would actually be nice if this project aims to support the OSM-XML as ingress and egress format as well, this way editors like JOSM could query the OSM data from it, edit the data and upload the changes to the main API.
pbf files, on the other hand, are boiled down representations of the OSM data, which can't be used to edit the OSM data, they are just meant for simple data consumption applications, like maps.
There was a problem hiding this comment.
PBF is only a transport format, you can store essentially exactly the same information in there as in XML (which is just another transport format).
There was a problem hiding this comment.
I would also concentrate on the OSM-PBF format only, I don't see a point spending time on the OSM-XML format (though that might be something that a third party could contribute).
| 2. Commandline tool using `peermaps/ingest` to diff .pbf files, convert to the Peermaps format, and pin both data formats on IPFS; | ||
| 3. Commandline tool using `peermaps/ingest` .pbf files, convert to the Peermaps format, and pin both data formats on Filecoin | ||
| 4. Post about the project for the IPFS blog | ||
| 5. Demo page with example code for map view in an HTML page using the Peermaps data published from IPFS. |
There was a problem hiding this comment.
Is this project aiming to support the Overpass API as well, for easy extraction of map features or POIs for example?
There was a problem hiding this comment.
That's not in our roadmap for 2020 but it would be possible as a module on top of it!
|
as an ex-GIS person I am overflowing with ❤️ for this proposal |
|
merging this given that remaining details are minor 🎉 |
After discussions with @autonome and @mishmosh, we have agreed to this grant proposal. Looking forward to getting started!