[gdal-dev] Vector Tiles in OGR

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
27 messages Options
12
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[gdal-dev] Vector Tiles in OGR

Petr Pridal - Klokan Technologies GmbH
Hi,

Is anybody working on an OGR driver for mapbox-like vector tiles (PBF blobs) stored in MBTiles or downloaded from a url via curl?

If this is done, all the GDAL-powered tools could access the vector features available in vector tiles directly - it means vector data in QGIS and Grass for any region in world without any extra effort. This would be damn cool!

It seems to be doable with an open-source code like:

Recently we have rendered the whole world from OpenStreetMap planet into vector tiles - and made these tiles downloadable for free (the whole world or country and city extracts).

Downloads are hosted on Swiss academical network and the project is cooperation between Klokan Technologies GmbH and students from HSR university in Switzerland.

Everything is done with open-source software, properly documented and forkable on github.


It practically means you can make your own OSM server running on your own hardware - usable even offline on your laptop or without firewalls within a few minutes.

No PostGIS needed and no processing of planet dump on a computer with 20GB+ RAM.
Any standard low-end computer or virtual machine can be used...

This including having your own raster tile server - where raster tiles are rendered on demand from vector tiles by Mapnik (we used tessera tilelive based server and docker - so the whole server can be up in a linux container with one command, if you don't want to install the software natively).

If you have WebGL in your browser then raw vector tiles are directly at: http://osm2vectortiles.tileserver.com/ with a style applied for example at: http://osm2vectortiles.org/maps/

Setup is quite easy: download vector tiles packed in MBTiles format (extracts has a few megabytes, countries gigabytes, whole world is about 60 GB) and use it.
Typical use would be a start of a basic raster tileserver or serving the vector tiles directly from any hosting (even a cheap PHP hosting can do this).

Everything is documented at: http://osm2vectortiles.org/docs/ and for free.
Patches and pull requests on the vector tiles rendering process and schema are welcome ;-)

If there would be OGR driver for vector tiles it would make a similar resource extremely useful also for other purposes in GIS tools - not only for tileservers and direct visualisation.

Any feedback or ideas are really welcome...

Petr
--
Petr Pridal, Ph.D.
CEO

Klokan Technologies GmbH
Hofnerstrasse 98, 6314 Unterageri, Switzerland
Tel: +41 (0)41 511 26 12
Email: [hidden email]
Web: http://www.klokantech.com/

_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Vector Tiles in OGR

flippmoke
Petr,

No one that I know of is currently working on such a tool. It could be a little difficult to write as well. 

Each Tile is its own discrete dataset therefore the hardest part would be merging of features into the original features across tiles. You could use some complicated merging technique, but that could be very expensive.

If you wanted to simply view vector tile data it could get confusing as well since most vector tiles need a buffer when they are used for visualization programs. This is important for things like labeling on maps etc. Therefore, you could get a large amount of overlapping data on tiles if you loaded them all in at once. 

I do not say this to discourage someone, but its a non-trivial problem. 

As far as writing a simple decoder -- the mapnik-vector-tile decoder isn't currently able to decode vector tiles into a form not supported by mapnik. Therefore, OGR would not be able to easily use this. Additionally Mapnik has a GDAL and OGR plugin, which creates an awesome circular dependency. We plan to write a generalized encoder and decoder eventually so that any other library can plugin with out the mapnik requirement. However, we have yet carved out time to do this properly. 

If you have any questions specifically about Mapnik Vector Tile or the Mapbox Vector Tile Specification feel free to ask me. I would note that you guys will want to update your vector tiles once the V2 specification changes are into Mapnik-Vector-Tile. 

Thanks,

Blake Thompson

_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Vector Tiles in OGR

Even Rouault-2
In reply to this post by Petr Pridal - Klokan Technologies GmbH
Le mardi 05 janvier 2016 19:33:32, Petr Pridal a écrit :
> Hi,
>
> Is anybody working on an OGR driver for mapbox-like vector tiles (PBF
> blobs) stored in MBTiles or downloaded from a url via curl?

Hi Petr,

I was involved in discussions a few months ago where Mapbox Vector Tiles
read/write support would have been part of a larger project, but I'm not sure
if/when this will come true.

I see you mention MVT in MBTiles. Is it a recognized practice ? I don't see  
mention of that neither in the MVT spec ( https://github.com/mapbox/vector-
tile-spec/tree/master/2.0 ) nor the MBTiles one (
https://github.com/mapbox/mbtiles-spec/blob/master/1.2/spec.md ), although I
guess it is just a matter of storing a MVT blob in the tile_data column.

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Vector Tiles in OGR

flippmoke
Even,

He is correct that MBTiles can store vector tiles. I know this is done in https://github.com/mapbox/tippecanoe . I believe there is talk about updating the MBTiles Spec as well. 

Thanks,

Blake Thompson

On Tue, Jan 5, 2016 at 2:21 PM, Even Rouault <[hidden email]> wrote:
Le mardi 05 janvier 2016 19:33:32, Petr Pridal a écrit :
> Hi,
>
> Is anybody working on an OGR driver for mapbox-like vector tiles (PBF
> blobs) stored in MBTiles or downloaded from a url via curl?

Hi Petr,

I was involved in discussions a few months ago where Mapbox Vector Tiles
read/write support would have been part of a larger project, but I'm not sure
if/when this will come true.

I see you mention MVT in MBTiles. Is it a recognized practice ? I don't see
mention of that neither in the MVT spec ( <a href="https://github.com/mapbox/vector- tile-spec/tree/master/2.0" rel="noreferrer" target="_blank">https://github.com/mapbox/vector-
tile-spec/tree/master/2.0 ) nor the MBTiles one (
https://github.com/mapbox/mbtiles-spec/blob/master/1.2/spec.md ), although I
guess it is just a matter of storing a MVT blob in the tile_data column.

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev


_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Vector Tiles in OGR

Stadin, Benjamin
I’ve commented on the MBTiles discussion [1] and suggested to make a change towards GeoPackage. There are in my opinion some reasons why it should rather be avoided to implement something towards MapBox’ current Tile spec or MBTiles, or any of the current vector tile formats in general:
  • Interoperability. Besides the need to define a custom extension for MapBox Vector tiles within a GPKG container (because vector tiles are not part of the spec) a standard based format with proper data field definitions would increase operability a lot. 
  • Conceptually. In my opinion, it should be avoided to mix rendering optimized data and geometry data. These should be separate concerns, but my understanding is that all vector tiles are currently optimized for rendering only, and not concepted as general purpose containers to convert to and from without anything breaking in practice. They are bound to a particular projection (in MapBox’ case always Web Mercator) and geometries cannot be exactly reconstructed after they’ve been tile-cut at the projection specific tile bounds. 
  • Tiling. When going for vector „tiles“ there should be no more need to stick to a certain projection. It should be possible to use unprojected data, and do the projection either in realtime or as part of a post processing step. In any case, the limitation to use just one projection (Web Mercator) may not be acceptable for general purpose application. As Blake said, merging features is a non-trivial problem. But once this is solved (maybe as part of a new vector tile spec), there is from a technical perspective no more need to produce tiles for rendering other than for performance reasons. Vector data can be pushed into a scene graph that contains all currently visible geometries. Tile data would still be sliced loosely in accordance to tile sizes and zoom levels, but not strictly so, and without need to produce projected data. I’m thinking of using an adaptation of the MODIS Grid [2] as tile storage (to store WGS84 datum, and reconstruct „partially complete“ geometries reproject and reproject to Web Mercator or UTM at runtime).
I think it would be of great benefit if common containers like GPKG would be adopted by more vendors. So that in the end more tools could operate on a data set first, and this data set could then be brought into an optimized form for specific rendering needs (which could also be an extension to GPKG as mentioned). 

Cheers
Ben



Von: gdal-dev <[hidden email]> on behalf of Blake Thompson <[hidden email]>
Datum: Dienstag, 5. Januar 2016 um 22:12
An: Even Rouault <[hidden email]>, "[hidden email]" <[hidden email]>
Cc: "[hidden email]" <[hidden email]>
Betreff: Re: [gdal-dev] Vector Tiles in OGR

Even,

He is correct that MBTiles can store vector tiles. I know this is done in https://github.com/mapbox/tippecanoe . I believe there is talk about updating the MBTiles Spec as well. 

Thanks,

Blake Thompson

On Tue, Jan 5, 2016 at 2:21 PM, Even Rouault <[hidden email]> wrote:
Le mardi 05 janvier 2016 19:33:32, Petr Pridal a écrit :
> Hi,
>
> Is anybody working on an OGR driver for mapbox-like vector tiles (PBF
> blobs) stored in MBTiles or downloaded from a url via curl?

Hi Petr,

I was involved in discussions a few months ago where Mapbox Vector Tiles
read/write support would have been part of a larger project, but I'm not sure
if/when this will come true.

I see you mention MVT in MBTiles. Is it a recognized practice ? I don't see
mention of that neither in the MVT spec ( <a href="https://github.com/mapbox/vector- tile-spec/tree/master/2.0" rel="noreferrer" target="_blank"> https://github.com/mapbox/vector-
tile-spec/tree/master/2.0 ) nor the MBTiles one (
https://github.com/mapbox/mbtiles-spec/blob/master/1.2/spec.md ), although I
guess it is just a matter of storing a MVT blob in the tile_data column.

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev


_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Vector Tiles in OGR

Even Rouault-2
In reply to this post by Even Rouault-2

> I was involved in discussions a few months ago where Mapbox Vector Tiles
> read/write support would have been part of a larger project, but I'm not
> sure if/when this will come true.

Update: just learned that this project has been won by another bidding group,
so I'm not sure if the winner has planned a OGR MVT development.

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Vector Tiles in OGR

Even Rouault-2
In reply to this post by flippmoke
Le mardi 05 janvier 2016 21:06:55, Blake Thompson a écrit :

> Petr,
>
> No one that I know of is currently working on such a tool. It could be a
> little difficult to write as well.
>
> Each Tile is its own discrete dataset therefore the hardest part would be
> merging of features into the original features across tiles. You could use
> some complicated merging technique, but that could be very expensive.
>
> If you wanted to simply view vector tile data it could get confusing as
> well since most vector tiles need a buffer when they are used for
> visualization programs.

Do you mean that the coordinates aren't all necessarily clipped to the
[0,extent-1] range ? If so, yes indeed the merging of consecutive tiles can be
"interesting".

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Vector Tiles in OGR

Petr Pridal - Klokan Technologies GmbH
In reply to this post by flippmoke
Hi everybody,

thanks for the comments. Regarding the technical details:

Each Tile is its own discrete dataset therefore the hardest part would be merging of features into the original features across tiles. You could use some complicated merging technique, but that could be very expensive.

The driver could require "id" parameter, which specifies name of an attribute with unique identifier on the features shared across the tiles. Both MapBox vector tiles and OSM2VectorTiles tiles have an unique non-zero "osm_id" in place. The first version of driver could read only features with such ID. The identifier does not have to be always filled, but often it is, especially in deeper zoom levels derived purely from OpenStreetMap - but yeh, a completely general stitching would be much harder.

The clipping + merging of shapes itself can be implemented with underlaying GEOS library functions.

BTW It is possible to examine vector tiles data in pure JavaScript X-Ray viewer (no WebGL support on browser required) made in OpenLayers3:

or with WebGL-enabled webbrowser with higher performance MapBox GL JS powered X-Ray: 

A debug viewer for each vector tile (made in OL3) decoded in pure JavaScript:

The OGR driver could be also practical for the unsimplified / ungeneralised MBTiles with OSM data available at: http://osmlab.github.io/osm-qa-tiles/
 
If you wanted to simply view vector tile data it could get confusing as well since most vector tiles need a buffer when they are used for visualization programs. This is important for things like labeling on maps etc. Therefore, you could get a large amount of overlapping data on tiles if you loaded them all in at once. 

Clipping on border of each tile must be applied before stitching the vector features together  - so the buffer is removed. Buffer is there only for visualisation and labeling.
 
As far as writing a simple decoder -- the mapnik-vector-tile decoder isn't currently able to decode vector tiles into a form not supported by mapnik. Therefore, OGR would not be able to easily use this.

Correct. It can't be used directly (GDAL can't link against), but the relevant code can be extracted and reused directly. For basic decoding critical is the Protobuf library (which is already in GDAL) and https://github.com/mapbox/mapnik-vector-tile/blob/master/proto/vector_tile.proto + the code for converting pixel coordinates in features to relevant geo coordinates and GEOS-based merging of features.

We plan to write a generalized encoder and decoder eventually so that any other library can plugin with out the mapnik requirement.

This would be amazing.

If you have any questions specifically about Mapnik Vector Tile or the Mapbox Vector Tile Specification feel free to ask me. I would note that you guys will want to update your vector tiles once the V2 specification changes are into Mapnik-Vector-Tile. 

Cool! Thanks. Our world tiles are generated via Docker containers on a cluster of computers - using Mapnik via Tilelive directly - so upgrades on Mapnik-Vector-Tile will propagate to the open-source rendering software.
I expect V2 spec will mean also new MapBox Streets V7, right?

I see you mention MVT in MBTiles. Is it a recognized practice ? I don't see
mention of that neither in the MVT spec ( https://github.com/mapbox/vector-
tile-spec/tree/master/2.0
 ) nor the MBTiles one (
https://github.com/mapbox/mbtiles-spec/blob/master/1.2/spec.md ), although I
guess it is just a matter of storing a MVT blob in the tile_data column.

Yes. MapBox Studio Classic generates such MBTiles with Vector Tiles inside - and Mapnik reads them directly when rasterising. Storing the vector tiles in SQLite make sense - and it simplifies deployment, and you can then easily have complete world on each node of a cluster of tileservers. If you don't want to upgrade your base map too often, this approach is pretty efficient and removes the centralised database as typical bottleneck.

I did some tests on vector tiles vs raster in MBTiles and documented these in README.md at and https://github.com/klokantech/vector-tiles-sample before we started to work on OSM2VectorTiles project.
There are also sample data and offline MapBox GL JS viewer in that repo.
Vector tiles can be hosted also "unpacked", just as files in folders on a standard web server - exactly like GDAL2Tiles or MapTiler.com does by default for raster tiles.

If OGR driver is implemented, the primary source should be probably reading from an URL via curl.

Reading PBFs from an MBTiles (or another SQLite) is just the most practical portable alternative to it.

Technically people may store the tiles in different ways - for example Wikimedia guys (http://maps.wikimedia.org/ and https://github.com/kartotherian/) use Cassandra for tile storage, I guess MapBox internally is also on a different storage for production servers ... for transfers and streaming tilesets there is the  - but tiles are ALWAYS exposed via HTTP.

Regards,

Petr
--
Petr Pridal, Ph.D.
CEO

Klokan Technologies GmbH
Hofnerstrasse 98, 6314 Unterageri, Switzerland
Tel: +41 (0)41 511 26 12
Email: [hidden email]
Web: http://www.klokantech.com/

_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Vector Tiles in OGR

Stefan Keller
Hi Petr, Blake, Even and everybody

I'm (besides Petr) the other maintainer of OSM2VectorTiles.
I'd already asked Even a similar question.
I'm advising now an intern to implement such a MVT Vector Tiles reader
client in Python if possible as part of a QGIS plugin.
And I'm interested not to do redundant implementations.
Are there any news on this matter?

:Stefan
Prof. at HSR and leader of Geometa Lab

@Even: Regarding OGR MVT development can you send to me (directly) a
contact name of the group which won the bidding?


2016-01-06 15:24 GMT+01:00 Petr Pridal <[hidden email]>:

> Hi everybody,
>
> thanks for the comments. Regarding the technical details:
>
>> Each Tile is its own discrete dataset therefore the hardest part would be
>> merging of features into the original features across tiles. You could use
>> some complicated merging technique, but that could be very expensive.
>
>
> The driver could require "id" parameter, which specifies name of an
> attribute with unique identifier on the features shared across the tiles.
> Both MapBox vector tiles and OSM2VectorTiles tiles have an unique non-zero
> "osm_id" in place. The first version of driver could read only features with
> such ID. The identifier does not have to be always filled, but often it is,
> especially in deeper zoom levels derived purely from OpenStreetMap - but
> yeh, a completely general stitching would be much harder.
>
> The clipping + merging of shapes itself can be implemented with underlaying
> GEOS library functions.
>
> BTW It is possible to examine vector tiles data in pure JavaScript X-Ray
> viewer (no WebGL support on browser required) made in OpenLayers3:
> http://klokantech.github.io/ol3-sandbox/vector/xray.html
>
> or with WebGL-enabled webbrowser with higher performance MapBox GL JS
> powered X-Ray:
> http://klokantech.github.io/mapbox-gl-js-offline-example/xray.html
>
> A debug viewer for each vector tile (made in OL3) decoded in pure
> JavaScript:
> http://klokantech.github.io/ol3-sandbox/vector/tile-inspector.html
>
> The OGR driver could be also practical for the unsimplified / ungeneralised
> MBTiles with OSM data available at: http://osmlab.github.io/osm-qa-tiles/
>
>>
>> If you wanted to simply view vector tile data it could get confusing as
>> well since most vector tiles need a buffer when they are used for
>> visualization programs. This is important for things like labeling on maps
>> etc. Therefore, you could get a large amount of overlapping data on tiles if
>> you loaded them all in at once.
>
>
> Clipping on border of each tile must be applied before stitching the vector
> features together  - so the buffer is removed. Buffer is there only for
> visualisation and labeling.
>
>>
>> As far as writing a simple decoder -- the mapnik-vector-tile decoder isn't
>> currently able to decode vector tiles into a form not supported by mapnik.
>> Therefore, OGR would not be able to easily use this.
>
>
> Correct. It can't be used directly (GDAL can't link against), but the
> relevant code can be extracted and reused directly. For basic decoding
> critical is the Protobuf library (which is already in GDAL) and
> https://github.com/mapbox/mapnik-vector-tile/blob/master/proto/vector_tile.proto
> + the code for converting pixel coordinates in features to relevant geo
> coordinates and GEOS-based merging of features.
>
>> We plan to write a generalized encoder and decoder eventually so that any
>> other library can plugin with out the mapnik requirement.
>
>
> This would be amazing.
>
>> If you have any questions specifically about Mapnik Vector Tile or the
>> Mapbox Vector Tile Specification feel free to ask me. I would note that you
>> guys will want to update your vector tiles once the V2 specification changes
>> are into Mapnik-Vector-Tile.
>
>
> Cool! Thanks. Our world tiles are generated via Docker containers on a
> cluster of computers - using Mapnik via Tilelive directly - so upgrades on
> Mapnik-Vector-Tile will propagate to the open-source rendering software.
> I expect V2 spec will mean also new MapBox Streets V7, right?
>
>> I see you mention MVT in MBTiles. Is it a recognized practice ? I don't
>> see
>> mention of that neither in the MVT spec (
>> https://github.com/mapbox/vector-
>> tile-spec/tree/master/2.0 ) nor the MBTiles one (
>> https://github.com/mapbox/mbtiles-spec/blob/master/1.2/spec.md ), although
>> I
>> guess it is just a matter of storing a MVT blob in the tile_data column.
>
>
> Yes. MapBox Studio Classic generates such MBTiles with Vector Tiles inside -
> and Mapnik reads them directly when rasterising. Storing the vector tiles in
> SQLite make sense - and it simplifies deployment, and you can then easily
> have complete world on each node of a cluster of tileservers. If you don't
> want to upgrade your base map too often, this approach is pretty efficient
> and removes the centralised database as typical bottleneck.
>
> I did some tests on vector tiles vs raster in MBTiles and documented these
> in README.md at and https://github.com/klokantech/vector-tiles-sample before
> we started to work on OSM2VectorTiles project.
> There are also sample data and offline MapBox GL JS viewer in that repo.
> Vector tiles can be hosted also "unpacked", just as files in folders on a
> standard web server - exactly like GDAL2Tiles or MapTiler.com does by
> default for raster tiles.
>
> If OGR driver is implemented, the primary source should be probably reading
> from an URL via curl.
>
> Reading PBFs from an MBTiles (or another SQLite) is just the most practical
> portable alternative to it.
>
> Technically people may store the tiles in different ways - for example
> Wikimedia guys (http://maps.wikimedia.org/ and
> https://github.com/kartotherian/) use Cassandra for tile storage, I guess
> MapBox internally is also on a different storage for production servers ...
> for transfers and streaming tilesets there is the  - but tiles are ALWAYS
> exposed via HTTP.
>
> Regards,
>
> Petr
> --
> Petr Pridal, Ph.D.
> CEO
>
> Klokan Technologies GmbH
> Hofnerstrasse 98, 6314 Unterageri, Switzerland
> Tel: +41 (0)41 511 26 12
> Email: [hidden email]
> Web: http://www.klokantech.com/
>
> _______________________________________________
> gdal-dev mailing list
> [hidden email]
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Vector Tiles in OGR

flippmoke
All,

The first thing that is going to be required likely is a good generic c++ library for encoding and decoding vector tiles IMO.

I have spent a lot of time working on developing the mapnik vector tile library to be a solid implementation, especially around all the work done for 2.0 of Mapbox vector tile specification. I would love to eventually help make this a generalized implementation so that it is portable to a library like GDAL.

We have talked about doing something like this at Mapbox but will require a good deal of time, so it hasn't been pushed forward yet. The other issues that come to mind to me is that some of the libraries that mapnik vector tile depends upon most likely need good custom implementations if we wanted to make it more generic.

The reason for this is that I would prefer for it to be a header only library with no dependencies. Adding more dependencies for GDAL seems like it could be a mess.

Blake Thompson

> On Jan 31, 2016, at 10:10 AM, Stefan Keller <[hidden email]> wrote:
>
> Hi Petr, Blake, Even and everybody
>
> I'm (besides Petr) the other maintainer of OSM2VectorTiles.
> I'd already asked Even a similar question.
> I'm advising now an intern to implement such a MVT Vector Tiles reader
> client in Python if possible as part of a QGIS plugin.
> And I'm interested not to do redundant implementations.
> Are there any news on this matter?
>
> :Stefan
> Prof. at HSR and leader of Geometa Lab
>
> @Even: Regarding OGR MVT development can you send to me (directly) a
> contact name of the group which won the bidding?
>
>
> 2016-01-06 15:24 GMT+01:00 Petr Pridal <[hidden email]>:
>> Hi everybody,
>>
>> thanks for the comments. Regarding the technical details:
>>
>>> Each Tile is its own discrete dataset therefore the hardest part would be
>>> merging of features into the original features across tiles. You could use
>>> some complicated merging technique, but that could be very expensive.
>>
>>
>> The driver could require "id" parameter, which specifies name of an
>> attribute with unique identifier on the features shared across the tiles.
>> Both MapBox vector tiles and OSM2VectorTiles tiles have an unique non-zero
>> "osm_id" in place. The first version of driver could read only features with
>> such ID. The identifier does not have to be always filled, but often it is,
>> especially in deeper zoom levels derived purely from OpenStreetMap - but
>> yeh, a completely general stitching would be much harder.
>>
>> The clipping + merging of shapes itself can be implemented with underlaying
>> GEOS library functions.
>>
>> BTW It is possible to examine vector tiles data in pure JavaScript X-Ray
>> viewer (no WebGL support on browser required) made in OpenLayers3:
>> http://klokantech.github.io/ol3-sandbox/vector/xray.html
>>
>> or with WebGL-enabled webbrowser with higher performance MapBox GL JS
>> powered X-Ray:
>> http://klokantech.github.io/mapbox-gl-js-offline-example/xray.html
>>
>> A debug viewer for each vector tile (made in OL3) decoded in pure
>> JavaScript:
>> http://klokantech.github.io/ol3-sandbox/vector/tile-inspector.html
>>
>> The OGR driver could be also practical for the unsimplified / ungeneralised
>> MBTiles with OSM data available at: http://osmlab.github.io/osm-qa-tiles/
>>
>>>
>>> If you wanted to simply view vector tile data it could get confusing as
>>> well since most vector tiles need a buffer when they are used for
>>> visualization programs. This is important for things like labeling on maps
>>> etc. Therefore, you could get a large amount of overlapping data on tiles if
>>> you loaded them all in at once.
>>
>>
>> Clipping on border of each tile must be applied before stitching the vector
>> features together  - so the buffer is removed. Buffer is there only for
>> visualisation and labeling.
>>
>>>
>>> As far as writing a simple decoder -- the mapnik-vector-tile decoder isn't
>>> currently able to decode vector tiles into a form not supported by mapnik.
>>> Therefore, OGR would not be able to easily use this.
>>
>>
>> Correct. It can't be used directly (GDAL can't link against), but the
>> relevant code can be extracted and reused directly. For basic decoding
>> critical is the Protobuf library (which is already in GDAL) and
>> https://github.com/mapbox/mapnik-vector-tile/blob/master/proto/vector_tile.proto
>> + the code for converting pixel coordinates in features to relevant geo
>> coordinates and GEOS-based merging of features.
>>
>>> We plan to write a generalized encoder and decoder eventually so that any
>>> other library can plugin with out the mapnik requirement.
>>
>>
>> This would be amazing.
>>
>>> If you have any questions specifically about Mapnik Vector Tile or the
>>> Mapbox Vector Tile Specification feel free to ask me. I would note that you
>>> guys will want to update your vector tiles once the V2 specification changes
>>> are into Mapnik-Vector-Tile.
>>
>>
>> Cool! Thanks. Our world tiles are generated via Docker containers on a
>> cluster of computers - using Mapnik via Tilelive directly - so upgrades on
>> Mapnik-Vector-Tile will propagate to the open-source rendering software.
>> I expect V2 spec will mean also new MapBox Streets V7, right?
>>
>>> I see you mention MVT in MBTiles. Is it a recognized practice ? I don't
>>> see
>>> mention of that neither in the MVT spec (
>>> https://github.com/mapbox/vector-
>>> tile-spec/tree/master/2.0 ) nor the MBTiles one (
>>> https://github.com/mapbox/mbtiles-spec/blob/master/1.2/spec.md ), although
>>> I
>>> guess it is just a matter of storing a MVT blob in the tile_data column.
>>
>>
>> Yes. MapBox Studio Classic generates such MBTiles with Vector Tiles inside -
>> and Mapnik reads them directly when rasterising. Storing the vector tiles in
>> SQLite make sense - and it simplifies deployment, and you can then easily
>> have complete world on each node of a cluster of tileservers. If you don't
>> want to upgrade your base map too often, this approach is pretty efficient
>> and removes the centralised database as typical bottleneck.
>>
>> I did some tests on vector tiles vs raster in MBTiles and documented these
>> in README.md at and https://github.com/klokantech/vector-tiles-sample before
>> we started to work on OSM2VectorTiles project.
>> There are also sample data and offline MapBox GL JS viewer in that repo.
>> Vector tiles can be hosted also "unpacked", just as files in folders on a
>> standard web server - exactly like GDAL2Tiles or MapTiler.com does by
>> default for raster tiles.
>>
>> If OGR driver is implemented, the primary source should be probably reading
>> from an URL via curl.
>>
>> Reading PBFs from an MBTiles (or another SQLite) is just the most practical
>> portable alternative to it.
>>
>> Technically people may store the tiles in different ways - for example
>> Wikimedia guys (http://maps.wikimedia.org/ and
>> https://github.com/kartotherian/) use Cassandra for tile storage, I guess
>> MapBox internally is also on a different storage for production servers ...
>> for transfers and streaming tilesets there is the  - but tiles are ALWAYS
>> exposed via HTTP.
>>
>> Regards,
>>
>> Petr
>> --
>> Petr Pridal, Ph.D.
>> CEO
>>
>> Klokan Technologies GmbH
>> Hofnerstrasse 98, 6314 Unterageri, Switzerland
>> Tel: +41 (0)41 511 26 12
>> Email: [hidden email]
>> Web: http://www.klokantech.com/
>>
>> _______________________________________________
>> gdal-dev mailing list
>> [hidden email]
>> http://lists.osgeo.org/mailman/listinfo/gdal-dev
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Vector Tiles in OGR

Stefan Keller
Hi Blake

It's clear to me that a good generic c++ library for encoding and
decoding vector tiles is needed.
On the other hand, I'd like to experiment with vector tiles and
related QGIS parallel loading issues for which Python is more
convenient.

It seems that Mapzen's [1] Python implementation is further along.
Or do you know if Jesse (or somebody) is still working on Mapbox's variant [2] ?

:Stefan

[1] https://github.com/mapzen/mapbox-vector-tile
[2] https://github.com/mapbox/vector-tile-py


2016-01-31 23:22 GMT+01:00 Flippmoke <[hidden email]>:

> All,
>
> The first thing that is going to be required likely is a good generic c++ library for encoding and decoding vector tiles IMO.
>
> I have spent a lot of time working on developing the mapnik vector tile library to be a solid implementation, especially around all the work done for 2.0 of Mapbox vector tile specification. I would love to eventually help make this a generalized implementation so that it is portable to a library like GDAL.
>
> We have talked about doing something like this at Mapbox but will require a good deal of time, so it hasn't been pushed forward yet. The other issues that come to mind to me is that some of the libraries that mapnik vector tile depends upon most likely need good custom implementations if we wanted to make it more generic.
>
> The reason for this is that I would prefer for it to be a header only library with no dependencies. Adding more dependencies for GDAL seems like it could be a mess.
>
> Blake Thompson
>
>> On Jan 31, 2016, at 10:10 AM, Stefan Keller <[hidden email]> wrote:
>>
>> Hi Petr, Blake, Even and everybody
>>
>> I'm (besides Petr) the other maintainer of OSM2VectorTiles.
>> I'd already asked Even a similar question.
>> I'm advising now an intern to implement such a MVT Vector Tiles reader
>> client in Python if possible as part of a QGIS plugin.
>> And I'm interested not to do redundant implementations.
>> Are there any news on this matter?
>>
>> :Stefan
>> Prof. at HSR and leader of Geometa Lab
>>
>> @Even: Regarding OGR MVT development can you send to me (directly) a
>> contact name of the group which won the bidding?
>>
>>
>> 2016-01-06 15:24 GMT+01:00 Petr Pridal <[hidden email]>:
>>> Hi everybody,
>>>
>>> thanks for the comments. Regarding the technical details:
>>>
>>>> Each Tile is its own discrete dataset therefore the hardest part would be
>>>> merging of features into the original features across tiles. You could use
>>>> some complicated merging technique, but that could be very expensive.
>>>
>>>
>>> The driver could require "id" parameter, which specifies name of an
>>> attribute with unique identifier on the features shared across the tiles.
>>> Both MapBox vector tiles and OSM2VectorTiles tiles have an unique non-zero
>>> "osm_id" in place. The first version of driver could read only features with
>>> such ID. The identifier does not have to be always filled, but often it is,
>>> especially in deeper zoom levels derived purely from OpenStreetMap - but
>>> yeh, a completely general stitching would be much harder.
>>>
>>> The clipping + merging of shapes itself can be implemented with underlaying
>>> GEOS library functions.
>>>
>>> BTW It is possible to examine vector tiles data in pure JavaScript X-Ray
>>> viewer (no WebGL support on browser required) made in OpenLayers3:
>>> http://klokantech.github.io/ol3-sandbox/vector/xray.html
>>>
>>> or with WebGL-enabled webbrowser with higher performance MapBox GL JS
>>> powered X-Ray:
>>> http://klokantech.github.io/mapbox-gl-js-offline-example/xray.html
>>>
>>> A debug viewer for each vector tile (made in OL3) decoded in pure
>>> JavaScript:
>>> http://klokantech.github.io/ol3-sandbox/vector/tile-inspector.html
>>>
>>> The OGR driver could be also practical for the unsimplified / ungeneralised
>>> MBTiles with OSM data available at: http://osmlab.github.io/osm-qa-tiles/
>>>
>>>>
>>>> If you wanted to simply view vector tile data it could get confusing as
>>>> well since most vector tiles need a buffer when they are used for
>>>> visualization programs. This is important for things like labeling on maps
>>>> etc. Therefore, you could get a large amount of overlapping data on tiles if
>>>> you loaded them all in at once.
>>>
>>>
>>> Clipping on border of each tile must be applied before stitching the vector
>>> features together  - so the buffer is removed. Buffer is there only for
>>> visualisation and labeling.
>>>
>>>>
>>>> As far as writing a simple decoder -- the mapnik-vector-tile decoder isn't
>>>> currently able to decode vector tiles into a form not supported by mapnik.
>>>> Therefore, OGR would not be able to easily use this.
>>>
>>>
>>> Correct. It can't be used directly (GDAL can't link against), but the
>>> relevant code can be extracted and reused directly. For basic decoding
>>> critical is the Protobuf library (which is already in GDAL) and
>>> https://github.com/mapbox/mapnik-vector-tile/blob/master/proto/vector_tile.proto
>>> + the code for converting pixel coordinates in features to relevant geo
>>> coordinates and GEOS-based merging of features.
>>>
>>>> We plan to write a generalized encoder and decoder eventually so that any
>>>> other library can plugin with out the mapnik requirement.
>>>
>>>
>>> This would be amazing.
>>>
>>>> If you have any questions specifically about Mapnik Vector Tile or the
>>>> Mapbox Vector Tile Specification feel free to ask me. I would note that you
>>>> guys will want to update your vector tiles once the V2 specification changes
>>>> are into Mapnik-Vector-Tile.
>>>
>>>
>>> Cool! Thanks. Our world tiles are generated via Docker containers on a
>>> cluster of computers - using Mapnik via Tilelive directly - so upgrades on
>>> Mapnik-Vector-Tile will propagate to the open-source rendering software.
>>> I expect V2 spec will mean also new MapBox Streets V7, right?
>>>
>>>> I see you mention MVT in MBTiles. Is it a recognized practice ? I don't
>>>> see
>>>> mention of that neither in the MVT spec (
>>>> https://github.com/mapbox/vector-
>>>> tile-spec/tree/master/2.0 ) nor the MBTiles one (
>>>> https://github.com/mapbox/mbtiles-spec/blob/master/1.2/spec.md ), although
>>>> I
>>>> guess it is just a matter of storing a MVT blob in the tile_data column.
>>>
>>>
>>> Yes. MapBox Studio Classic generates such MBTiles with Vector Tiles inside -
>>> and Mapnik reads them directly when rasterising. Storing the vector tiles in
>>> SQLite make sense - and it simplifies deployment, and you can then easily
>>> have complete world on each node of a cluster of tileservers. If you don't
>>> want to upgrade your base map too often, this approach is pretty efficient
>>> and removes the centralised database as typical bottleneck.
>>>
>>> I did some tests on vector tiles vs raster in MBTiles and documented these
>>> in README.md at and https://github.com/klokantech/vector-tiles-sample before
>>> we started to work on OSM2VectorTiles project.
>>> There are also sample data and offline MapBox GL JS viewer in that repo.
>>> Vector tiles can be hosted also "unpacked", just as files in folders on a
>>> standard web server - exactly like GDAL2Tiles or MapTiler.com does by
>>> default for raster tiles.
>>>
>>> If OGR driver is implemented, the primary source should be probably reading
>>> from an URL via curl.
>>>
>>> Reading PBFs from an MBTiles (or another SQLite) is just the most practical
>>> portable alternative to it.
>>>
>>> Technically people may store the tiles in different ways - for example
>>> Wikimedia guys (http://maps.wikimedia.org/ and
>>> https://github.com/kartotherian/) use Cassandra for tile storage, I guess
>>> MapBox internally is also on a different storage for production servers ...
>>> for transfers and streaming tilesets there is the  - but tiles are ALWAYS
>>> exposed via HTTP.
>>>
>>> Regards,
>>>
>>> Petr
>>> --
>>> Petr Pridal, Ph.D.
>>> CEO
>>>
>>> Klokan Technologies GmbH
>>> Hofnerstrasse 98, 6314 Unterageri, Switzerland
>>> Tel: +41 (0)41 511 26 12
>>> Email: [hidden email]
>>> Web: http://www.klokantech.com/
>>>
>>> _______________________________________________
>>> gdal-dev mailing list
>>> [hidden email]
>>> http://lists.osgeo.org/mailman/listinfo/gdal-dev
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Vector Tiles in OGR

flippmoke

Stefan,

I am not aware of any libraries besides mapnik vector tile that have been attempting to implement v2 specification. I think mapzen's implementation might be best? We have been wanting to add vector tiles to the python mapnik bindings for quite some time but we don't have the time to do it right now.

I will try to think this week about how we could make a generic C++ library quickly for Mapbox vector tiles.

Thanks,

Blake Thompson

> On Feb 1, 2016, at 1:06 AM, Stefan Keller <[hidden email]> wrote:
>
> Hi Blake
>
> It's clear to me that a good generic c++ library for encoding and
> decoding vector tiles is needed.
> On the other hand, I'd like to experiment with vector tiles and
> related QGIS parallel loading issues for which Python is more
> convenient.
>
> It seems that Mapzen's [1] Python implementation is further along.
> Or do you know if Jesse (or somebody) is still working on Mapbox's variant [2] ?
>
> :Stefan
>
> [1] https://github.com/mapzen/mapbox-vector-tile
> [2] https://github.com/mapbox/vector-tile-py
>
>
> 2016-01-31 23:22 GMT+01:00 Flippmoke <[hidden email]>:
>> All,
>>
>> The first thing that is going to be required likely is a good generic c++ library for encoding and decoding vector tiles IMO.
>>
>> I have spent a lot of time working on developing the mapnik vector tile library to be a solid implementation, especially around all the work done for 2.0 of Mapbox vector tile specification. I would love to eventually help make this a generalized implementation so that it is portable to a library like GDAL.
>>
>> We have talked about doing something like this at Mapbox but will require a good deal of time, so it hasn't been pushed forward yet. The other issues that come to mind to me is that some of the libraries that mapnik vector tile depends upon most likely need good custom implementations if we wanted to make it more generic.
>>
>> The reason for this is that I would prefer for it to be a header only library with no dependencies. Adding more dependencies for GDAL seems like it could be a mess.
>>
>> Blake Thompson
>>
>>> On Jan 31, 2016, at 10:10 AM, Stefan Keller <[hidden email]> wrote:
>>>
>>> Hi Petr, Blake, Even and everybody
>>>
>>> I'm (besides Petr) the other maintainer of OSM2VectorTiles.
>>> I'd already asked Even a similar question.
>>> I'm advising now an intern to implement such a MVT Vector Tiles reader
>>> client in Python if possible as part of a QGIS plugin.
>>> And I'm interested not to do redundant implementations.
>>> Are there any news on this matter?
>>>
>>> :Stefan
>>> Prof. at HSR and leader of Geometa Lab
>>>
>>> @Even: Regarding OGR MVT development can you send to me (directly) a
>>> contact name of the group which won the bidding?
>>>
>>>
>>> 2016-01-06 15:24 GMT+01:00 Petr Pridal <[hidden email]>:
>>>> Hi everybody,
>>>>
>>>> thanks for the comments. Regarding the technical details:
>>>>
>>>>> Each Tile is its own discrete dataset therefore the hardest part would be
>>>>> merging of features into the original features across tiles. You could use
>>>>> some complicated merging technique, but that could be very expensive.
>>>>
>>>>
>>>> The driver could require "id" parameter, which specifies name of an
>>>> attribute with unique identifier on the features shared across the tiles.
>>>> Both MapBox vector tiles and OSM2VectorTiles tiles have an unique non-zero
>>>> "osm_id" in place. The first version of driver could read only features with
>>>> such ID. The identifier does not have to be always filled, but often it is,
>>>> especially in deeper zoom levels derived purely from OpenStreetMap - but
>>>> yeh, a completely general stitching would be much harder.
>>>>
>>>> The clipping + merging of shapes itself can be implemented with underlaying
>>>> GEOS library functions.
>>>>
>>>> BTW It is possible to examine vector tiles data in pure JavaScript X-Ray
>>>> viewer (no WebGL support on browser required) made in OpenLayers3:
>>>> http://klokantech.github.io/ol3-sandbox/vector/xray.html
>>>>
>>>> or with WebGL-enabled webbrowser with higher performance MapBox GL JS
>>>> powered X-Ray:
>>>> http://klokantech.github.io/mapbox-gl-js-offline-example/xray.html
>>>>
>>>> A debug viewer for each vector tile (made in OL3) decoded in pure
>>>> JavaScript:
>>>> http://klokantech.github.io/ol3-sandbox/vector/tile-inspector.html
>>>>
>>>> The OGR driver could be also practical for the unsimplified / ungeneralised
>>>> MBTiles with OSM data available at: http://osmlab.github.io/osm-qa-tiles/
>>>>
>>>>>
>>>>> If you wanted to simply view vector tile data it could get confusing as
>>>>> well since most vector tiles need a buffer when they are used for
>>>>> visualization programs. This is important for things like labeling on maps
>>>>> etc. Therefore, you could get a large amount of overlapping data on tiles if
>>>>> you loaded them all in at once.
>>>>
>>>>
>>>> Clipping on border of each tile must be applied before stitching the vector
>>>> features together  - so the buffer is removed. Buffer is there only for
>>>> visualisation and labeling.
>>>>
>>>>>
>>>>> As far as writing a simple decoder -- the mapnik-vector-tile decoder isn't
>>>>> currently able to decode vector tiles into a form not supported by mapnik.
>>>>> Therefore, OGR would not be able to easily use this.
>>>>
>>>>
>>>> Correct. It can't be used directly (GDAL can't link against), but the
>>>> relevant code can be extracted and reused directly. For basic decoding
>>>> critical is the Protobuf library (which is already in GDAL) and
>>>> https://github.com/mapbox/mapnik-vector-tile/blob/master/proto/vector_tile.proto
>>>> + the code for converting pixel coordinates in features to relevant geo
>>>> coordinates and GEOS-based merging of features.
>>>>
>>>>> We plan to write a generalized encoder and decoder eventually so that any
>>>>> other library can plugin with out the mapnik requirement.
>>>>
>>>>
>>>> This would be amazing.
>>>>
>>>>> If you have any questions specifically about Mapnik Vector Tile or the
>>>>> Mapbox Vector Tile Specification feel free to ask me. I would note that you
>>>>> guys will want to update your vector tiles once the V2 specification changes
>>>>> are into Mapnik-Vector-Tile.
>>>>
>>>>
>>>> Cool! Thanks. Our world tiles are generated via Docker containers on a
>>>> cluster of computers - using Mapnik via Tilelive directly - so upgrades on
>>>> Mapnik-Vector-Tile will propagate to the open-source rendering software.
>>>> I expect V2 spec will mean also new MapBox Streets V7, right?
>>>>
>>>>> I see you mention MVT in MBTiles. Is it a recognized practice ? I don't
>>>>> see
>>>>> mention of that neither in the MVT spec (
>>>>> https://github.com/mapbox/vector-
>>>>> tile-spec/tree/master/2.0 ) nor the MBTiles one (
>>>>> https://github.com/mapbox/mbtiles-spec/blob/master/1.2/spec.md ), although
>>>>> I
>>>>> guess it is just a matter of storing a MVT blob in the tile_data column.
>>>>
>>>>
>>>> Yes. MapBox Studio Classic generates such MBTiles with Vector Tiles inside -
>>>> and Mapnik reads them directly when rasterising. Storing the vector tiles in
>>>> SQLite make sense - and it simplifies deployment, and you can then easily
>>>> have complete world on each node of a cluster of tileservers. If you don't
>>>> want to upgrade your base map too often, this approach is pretty efficient
>>>> and removes the centralised database as typical bottleneck.
>>>>
>>>> I did some tests on vector tiles vs raster in MBTiles and documented these
>>>> in README.md at and https://github.com/klokantech/vector-tiles-sample before
>>>> we started to work on OSM2VectorTiles project.
>>>> There are also sample data and offline MapBox GL JS viewer in that repo.
>>>> Vector tiles can be hosted also "unpacked", just as files in folders on a
>>>> standard web server - exactly like GDAL2Tiles or MapTiler.com does by
>>>> default for raster tiles.
>>>>
>>>> If OGR driver is implemented, the primary source should be probably reading
>>>> from an URL via curl.
>>>>
>>>> Reading PBFs from an MBTiles (or another SQLite) is just the most practical
>>>> portable alternative to it.
>>>>
>>>> Technically people may store the tiles in different ways - for example
>>>> Wikimedia guys (http://maps.wikimedia.org/ and
>>>> https://github.com/kartotherian/) use Cassandra for tile storage, I guess
>>>> MapBox internally is also on a different storage for production servers ...
>>>> for transfers and streaming tilesets there is the  - but tiles are ALWAYS
>>>> exposed via HTTP.
>>>>
>>>> Regards,
>>>>
>>>> Petr
>>>> --
>>>> Petr Pridal, Ph.D.
>>>> CEO
>>>>
>>>> Klokan Technologies GmbH
>>>> Hofnerstrasse 98, 6314 Unterageri, Switzerland
>>>> Tel: +41 (0)41 511 26 12
>>>> Email: [hidden email]
>>>> Web: http://www.klokantech.com/
>>>>
>>>> _______________________________________________
>>>> gdal-dev mailing list
>>>> [hidden email]
>>>> http://lists.osgeo.org/mailman/listinfo/gdal-dev
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Vector Tiles in OGR

Stefan Keller
That's good news!
We'll be happy to add Python bindings to such a generic C++ library
for Mapbox vector tiles.
We have still some time (say few weeks?) left here to decide to
contribute either to these bindings or to Mapzen's pure Python lib.

One reason why I'm insisting on Python (bindings) - besides easier
testing with QGIS - is the following:
AFAIK with vector tiles we have to deal with quite a new kind of
feature provider:
One which contains "zoom levels" and one which "suddenly" changes
feature class at different zoom levels.
If a client like QGIS consumes vector tiles from OGR in a one-shot
call, how should the client deal with such "zoom level" dependent
issues?
So my thinking is, to consume only the most appropriate zoom level,
probably only within current map canvas...

:Stefan

2016-02-01 18:27 GMT+01:00 Flippmoke <[hidden email]>:

>
> Stefan,
>
> I am not aware of any libraries besides mapnik vector tile that have been attempting to implement v2 specification. I think mapzen's implementation might be best? We have been wanting to add vector tiles to the python mapnik bindings for quite some time but we don't have the time to do it right now.
>
> I will try to think this week about how we could make a generic C++ library quickly for Mapbox vector tiles.
>
> Thanks,
>
> Blake Thompson
>
>> On Feb 1, 2016, at 1:06 AM, Stefan Keller <[hidden email]> wrote:
>>
>> Hi Blake
>>
>> It's clear to me that a good generic c++ library for encoding and
>> decoding vector tiles is needed.
>> On the other hand, I'd like to experiment with vector tiles and
>> related QGIS parallel loading issues for which Python is more
>> convenient.
>>
>> It seems that Mapzen's [1] Python implementation is further along.
>> Or do you know if Jesse (or somebody) is still working on Mapbox's variant [2] ?
>>
>> :Stefan
>>
>> [1] https://github.com/mapzen/mapbox-vector-tile
>> [2] https://github.com/mapbox/vector-tile-py
>>
>>
>> 2016-01-31 23:22 GMT+01:00 Flippmoke <[hidden email]>:
>>> All,
>>>
>>> The first thing that is going to be required likely is a good generic c++ library for encoding and decoding vector tiles IMO.
>>>
>>> I have spent a lot of time working on developing the mapnik vector tile library to be a solid implementation, especially around all the work done for 2.0 of Mapbox vector tile specification. I would love to eventually help make this a generalized implementation so that it is portable to a library like GDAL.
>>>
>>> We have talked about doing something like this at Mapbox but will require a good deal of time, so it hasn't been pushed forward yet. The other issues that come to mind to me is that some of the libraries that mapnik vector tile depends upon most likely need good custom implementations if we wanted to make it more generic.
>>>
>>> The reason for this is that I would prefer for it to be a header only library with no dependencies. Adding more dependencies for GDAL seems like it could be a mess.
>>>
>>> Blake Thompson
>>>
>>>> On Jan 31, 2016, at 10:10 AM, Stefan Keller <[hidden email]> wrote:
>>>>
>>>> Hi Petr, Blake, Even and everybody
>>>>
>>>> I'm (besides Petr) the other maintainer of OSM2VectorTiles.
>>>> I'd already asked Even a similar question.
>>>> I'm advising now an intern to implement such a MVT Vector Tiles reader
>>>> client in Python if possible as part of a QGIS plugin.
>>>> And I'm interested not to do redundant implementations.
>>>> Are there any news on this matter?
>>>>
>>>> :Stefan
>>>> Prof. at HSR and leader of Geometa Lab
>>>>
>>>> @Even: Regarding OGR MVT development can you send to me (directly) a
>>>> contact name of the group which won the bidding?
>>>>
>>>>
>>>> 2016-01-06 15:24 GMT+01:00 Petr Pridal <[hidden email]>:
>>>>> Hi everybody,
>>>>>
>>>>> thanks for the comments. Regarding the technical details:
>>>>>
>>>>>> Each Tile is its own discrete dataset therefore the hardest part would be
>>>>>> merging of features into the original features across tiles. You could use
>>>>>> some complicated merging technique, but that could be very expensive.
>>>>>
>>>>>
>>>>> The driver could require "id" parameter, which specifies name of an
>>>>> attribute with unique identifier on the features shared across the tiles.
>>>>> Both MapBox vector tiles and OSM2VectorTiles tiles have an unique non-zero
>>>>> "osm_id" in place. The first version of driver could read only features with
>>>>> such ID. The identifier does not have to be always filled, but often it is,
>>>>> especially in deeper zoom levels derived purely from OpenStreetMap - but
>>>>> yeh, a completely general stitching would be much harder.
>>>>>
>>>>> The clipping + merging of shapes itself can be implemented with underlaying
>>>>> GEOS library functions.
>>>>>
>>>>> BTW It is possible to examine vector tiles data in pure JavaScript X-Ray
>>>>> viewer (no WebGL support on browser required) made in OpenLayers3:
>>>>> http://klokantech.github.io/ol3-sandbox/vector/xray.html
>>>>>
>>>>> or with WebGL-enabled webbrowser with higher performance MapBox GL JS
>>>>> powered X-Ray:
>>>>> http://klokantech.github.io/mapbox-gl-js-offline-example/xray.html
>>>>>
>>>>> A debug viewer for each vector tile (made in OL3) decoded in pure
>>>>> JavaScript:
>>>>> http://klokantech.github.io/ol3-sandbox/vector/tile-inspector.html
>>>>>
>>>>> The OGR driver could be also practical for the unsimplified / ungeneralised
>>>>> MBTiles with OSM data available at: http://osmlab.github.io/osm-qa-tiles/
>>>>>
>>>>>>
>>>>>> If you wanted to simply view vector tile data it could get confusing as
>>>>>> well since most vector tiles need a buffer when they are used for
>>>>>> visualization programs. This is important for things like labeling on maps
>>>>>> etc. Therefore, you could get a large amount of overlapping data on tiles if
>>>>>> you loaded them all in at once.
>>>>>
>>>>>
>>>>> Clipping on border of each tile must be applied before stitching the vector
>>>>> features together  - so the buffer is removed. Buffer is there only for
>>>>> visualisation and labeling.
>>>>>
>>>>>>
>>>>>> As far as writing a simple decoder -- the mapnik-vector-tile decoder isn't
>>>>>> currently able to decode vector tiles into a form not supported by mapnik.
>>>>>> Therefore, OGR would not be able to easily use this.
>>>>>
>>>>>
>>>>> Correct. It can't be used directly (GDAL can't link against), but the
>>>>> relevant code can be extracted and reused directly. For basic decoding
>>>>> critical is the Protobuf library (which is already in GDAL) and
>>>>> https://github.com/mapbox/mapnik-vector-tile/blob/master/proto/vector_tile.proto
>>>>> + the code for converting pixel coordinates in features to relevant geo
>>>>> coordinates and GEOS-based merging of features.
>>>>>
>>>>>> We plan to write a generalized encoder and decoder eventually so that any
>>>>>> other library can plugin with out the mapnik requirement.
>>>>>
>>>>>
>>>>> This would be amazing.
>>>>>
>>>>>> If you have any questions specifically about Mapnik Vector Tile or the
>>>>>> Mapbox Vector Tile Specification feel free to ask me. I would note that you
>>>>>> guys will want to update your vector tiles once the V2 specification changes
>>>>>> are into Mapnik-Vector-Tile.
>>>>>
>>>>>
>>>>> Cool! Thanks. Our world tiles are generated via Docker containers on a
>>>>> cluster of computers - using Mapnik via Tilelive directly - so upgrades on
>>>>> Mapnik-Vector-Tile will propagate to the open-source rendering software.
>>>>> I expect V2 spec will mean also new MapBox Streets V7, right?
>>>>>
>>>>>> I see you mention MVT in MBTiles. Is it a recognized practice ? I don't
>>>>>> see
>>>>>> mention of that neither in the MVT spec (
>>>>>> https://github.com/mapbox/vector-
>>>>>> tile-spec/tree/master/2.0 ) nor the MBTiles one (
>>>>>> https://github.com/mapbox/mbtiles-spec/blob/master/1.2/spec.md ), although
>>>>>> I
>>>>>> guess it is just a matter of storing a MVT blob in the tile_data column.
>>>>>
>>>>>
>>>>> Yes. MapBox Studio Classic generates such MBTiles with Vector Tiles inside -
>>>>> and Mapnik reads them directly when rasterising. Storing the vector tiles in
>>>>> SQLite make sense - and it simplifies deployment, and you can then easily
>>>>> have complete world on each node of a cluster of tileservers. If you don't
>>>>> want to upgrade your base map too often, this approach is pretty efficient
>>>>> and removes the centralised database as typical bottleneck.
>>>>>
>>>>> I did some tests on vector tiles vs raster in MBTiles and documented these
>>>>> in README.md at and https://github.com/klokantech/vector-tiles-sample before
>>>>> we started to work on OSM2VectorTiles project.
>>>>> There are also sample data and offline MapBox GL JS viewer in that repo.
>>>>> Vector tiles can be hosted also "unpacked", just as files in folders on a
>>>>> standard web server - exactly like GDAL2Tiles or MapTiler.com does by
>>>>> default for raster tiles.
>>>>>
>>>>> If OGR driver is implemented, the primary source should be probably reading
>>>>> from an URL via curl.
>>>>>
>>>>> Reading PBFs from an MBTiles (or another SQLite) is just the most practical
>>>>> portable alternative to it.
>>>>>
>>>>> Technically people may store the tiles in different ways - for example
>>>>> Wikimedia guys (http://maps.wikimedia.org/ and
>>>>> https://github.com/kartotherian/) use Cassandra for tile storage, I guess
>>>>> MapBox internally is also on a different storage for production servers ...
>>>>> for transfers and streaming tilesets there is the  - but tiles are ALWAYS
>>>>> exposed via HTTP.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Petr
>>>>> --
>>>>> Petr Pridal, Ph.D.
>>>>> CEO
>>>>>
>>>>> Klokan Technologies GmbH
>>>>> Hofnerstrasse 98, 6314 Unterageri, Switzerland
>>>>> Tel: +41 (0)41 511 26 12
>>>>> Email: [hidden email]
>>>>> Web: http://www.klokantech.com/
>>>>>
>>>>> _______________________________________________
>>>>> gdal-dev mailing list
>>>>> [hidden email]
>>>>> http://lists.osgeo.org/mailman/listinfo/gdal-dev
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Vector Tiles in OGR

Stadin, Benjamin
In reply to this post by flippmoke
Hi Blake,

There is TileMaker [1]. It implements a MVT writer for v1 only as it
seems, but looks lightweight enough as a good base for a portable library.

Ben

[1]
https://github.com/systemed/tilemaker/blob/master/src/write_geometry.cpp


Am 01.02.16, 18:27 schrieb "gdal-dev on behalf of Flippmoke" unter
<[hidden email] on behalf of [hidden email]>:

>
>Stefan,
>
>I am not aware of any libraries besides mapnik vector tile that have been
>attempting to implement v2 specification. I think mapzen's implementation
>might be best? We have been wanting to add vector tiles to the python
>mapnik bindings for quite some time but we don't have the time to do it
>right now.
>
>I will try to think this week about how we could make a generic C++
>library quickly for Mapbox vector tiles.
>
>Thanks,
>
>Blake Thompson
>
>> On Feb 1, 2016, at 1:06 AM, Stefan Keller <[hidden email]> wrote:
>>
>> Hi Blake
>>
>> It's clear to me that a good generic c++ library for encoding and
>> decoding vector tiles is needed.
>> On the other hand, I'd like to experiment with vector tiles and
>> related QGIS parallel loading issues for which Python is more
>> convenient.
>>
>> It seems that Mapzen's [1] Python implementation is further along.
>> Or do you know if Jesse (or somebody) is still working on Mapbox's
>>variant [2] ?
>>
>> :Stefan
>>
>> [1] https://github.com/mapzen/mapbox-vector-tile
>> [2] https://github.com/mapbox/vector-tile-py
>>
>>
>> 2016-01-31 23:22 GMT+01:00 Flippmoke <[hidden email]>:
>>> All,
>>>
>>> The first thing that is going to be required likely is a good generic
>>>c++ library for encoding and decoding vector tiles IMO.
>>>
>>> I have spent a lot of time working on developing the mapnik vector
>>>tile library to be a solid implementation, especially around all the
>>>work done for 2.0 of Mapbox vector tile specification. I would love to
>>>eventually help make this a generalized implementation so that it is
>>>portable to a library like GDAL.
>>>
>>> We have talked about doing something like this at Mapbox but will
>>>require a good deal of time, so it hasn't been pushed forward yet. The
>>>other issues that come to mind to me is that some of the libraries that
>>>mapnik vector tile depends upon most likely need good custom
>>>implementations if we wanted to make it more generic.
>>>
>>> The reason for this is that I would prefer for it to be a header only
>>>library with no dependencies. Adding more dependencies for GDAL seems
>>>like it could be a mess.
>>>
>>> Blake Thompson
>>>
>>>> On Jan 31, 2016, at 10:10 AM, Stefan Keller <[hidden email]>
>>>>wrote:
>>>>
>>>> Hi Petr, Blake, Even and everybody
>>>>
>>>> I'm (besides Petr) the other maintainer of OSM2VectorTiles.
>>>> I'd already asked Even a similar question.
>>>> I'm advising now an intern to implement such a MVT Vector Tiles reader
>>>> client in Python if possible as part of a QGIS plugin.
>>>> And I'm interested not to do redundant implementations.
>>>> Are there any news on this matter?
>>>>
>>>> :Stefan
>>>> Prof. at HSR and leader of Geometa Lab
>>>>
>>>> @Even: Regarding OGR MVT development can you send to me (directly) a
>>>> contact name of the group which won the bidding?
>>>>
>>>>
>>>> 2016-01-06 15:24 GMT+01:00 Petr Pridal <[hidden email]>:
>>>>> Hi everybody,
>>>>>
>>>>> thanks for the comments. Regarding the technical details:
>>>>>
>>>>>> Each Tile is its own discrete dataset therefore the hardest part
>>>>>>would be
>>>>>> merging of features into the original features across tiles. You
>>>>>>could use
>>>>>> some complicated merging technique, but that could be very
>>>>>>expensive.
>>>>>
>>>>>
>>>>> The driver could require "id" parameter, which specifies name of an
>>>>> attribute with unique identifier on the features shared across the
>>>>>tiles.
>>>>> Both MapBox vector tiles and OSM2VectorTiles tiles have an unique
>>>>>non-zero
>>>>> "osm_id" in place. The first version of driver could read only
>>>>>features with
>>>>> such ID. The identifier does not have to be always filled, but often
>>>>>it is,
>>>>> especially in deeper zoom levels derived purely from OpenStreetMap -
>>>>>but
>>>>> yeh, a completely general stitching would be much harder.
>>>>>
>>>>> The clipping + merging of shapes itself can be implemented with
>>>>>underlaying
>>>>> GEOS library functions.
>>>>>
>>>>> BTW It is possible to examine vector tiles data in pure JavaScript
>>>>>X-Ray
>>>>> viewer (no WebGL support on browser required) made in OpenLayers3:
>>>>> http://klokantech.github.io/ol3-sandbox/vector/xray.html
>>>>>
>>>>> or with WebGL-enabled webbrowser with higher performance MapBox GL JS
>>>>> powered X-Ray:
>>>>> http://klokantech.github.io/mapbox-gl-js-offline-example/xray.html
>>>>>
>>>>> A debug viewer for each vector tile (made in OL3) decoded in pure
>>>>> JavaScript:
>>>>> http://klokantech.github.io/ol3-sandbox/vector/tile-inspector.html
>>>>>
>>>>> The OGR driver could be also practical for the unsimplified /
>>>>>ungeneralised
>>>>> MBTiles with OSM data available at:
>>>>>http://osmlab.github.io/osm-qa-tiles/
>>>>>
>>>>>>
>>>>>> If you wanted to simply view vector tile data it could get
>>>>>>confusing as
>>>>>> well since most vector tiles need a buffer when they are used for
>>>>>> visualization programs. This is important for things like labeling
>>>>>>on maps
>>>>>> etc. Therefore, you could get a large amount of overlapping data on
>>>>>>tiles if
>>>>>> you loaded them all in at once.
>>>>>
>>>>>
>>>>> Clipping on border of each tile must be applied before stitching the
>>>>>vector
>>>>> features together  - so the buffer is removed. Buffer is there only
>>>>>for
>>>>> visualisation and labeling.
>>>>>
>>>>>>
>>>>>> As far as writing a simple decoder -- the mapnik-vector-tile
>>>>>>decoder isn't
>>>>>> currently able to decode vector tiles into a form not supported by
>>>>>>mapnik.
>>>>>> Therefore, OGR would not be able to easily use this.
>>>>>
>>>>>
>>>>> Correct. It can't be used directly (GDAL can't link against), but the
>>>>> relevant code can be extracted and reused directly. For basic
>>>>>decoding
>>>>> critical is the Protobuf library (which is already in GDAL) and
>>>>>
>>>>>https://github.com/mapbox/mapnik-vector-tile/blob/master/proto/vector_
>>>>>tile.proto
>>>>> + the code for converting pixel coordinates in features to relevant
>>>>>geo
>>>>> coordinates and GEOS-based merging of features.
>>>>>
>>>>>> We plan to write a generalized encoder and decoder eventually so
>>>>>>that any
>>>>>> other library can plugin with out the mapnik requirement.
>>>>>
>>>>>
>>>>> This would be amazing.
>>>>>
>>>>>> If you have any questions specifically about Mapnik Vector Tile or
>>>>>>the
>>>>>> Mapbox Vector Tile Specification feel free to ask me. I would note
>>>>>>that you
>>>>>> guys will want to update your vector tiles once the V2
>>>>>>specification changes
>>>>>> are into Mapnik-Vector-Tile.
>>>>>
>>>>>
>>>>> Cool! Thanks. Our world tiles are generated via Docker containers on
>>>>>a
>>>>> cluster of computers - using Mapnik via Tilelive directly - so
>>>>>upgrades on
>>>>> Mapnik-Vector-Tile will propagate to the open-source rendering
>>>>>software.
>>>>> I expect V2 spec will mean also new MapBox Streets V7, right?
>>>>>
>>>>>> I see you mention MVT in MBTiles. Is it a recognized practice ? I
>>>>>>don't
>>>>>> see
>>>>>> mention of that neither in the MVT spec (
>>>>>> https://github.com/mapbox/vector-
>>>>>> tile-spec/tree/master/2.0 ) nor the MBTiles one (
>>>>>> https://github.com/mapbox/mbtiles-spec/blob/master/1.2/spec.md ),
>>>>>>although
>>>>>> I
>>>>>> guess it is just a matter of storing a MVT blob in the tile_data
>>>>>>column.
>>>>>
>>>>>
>>>>> Yes. MapBox Studio Classic generates such MBTiles with Vector Tiles
>>>>>inside -
>>>>> and Mapnik reads them directly when rasterising. Storing the vector
>>>>>tiles in
>>>>> SQLite make sense - and it simplifies deployment, and you can then
>>>>>easily
>>>>> have complete world on each node of a cluster of tileservers. If you
>>>>>don't
>>>>> want to upgrade your base map too often, this approach is pretty
>>>>>efficient
>>>>> and removes the centralised database as typical bottleneck.
>>>>>
>>>>> I did some tests on vector tiles vs raster in MBTiles and documented
>>>>>these
>>>>> in README.md at and
>>>>>https://github.com/klokantech/vector-tiles-sample before
>>>>> we started to work on OSM2VectorTiles project.
>>>>> There are also sample data and offline MapBox GL JS viewer in that
>>>>>repo.
>>>>> Vector tiles can be hosted also "unpacked", just as files in folders
>>>>>on a
>>>>> standard web server - exactly like GDAL2Tiles or MapTiler.com does by
>>>>> default for raster tiles.
>>>>>
>>>>> If OGR driver is implemented, the primary source should be probably
>>>>>reading
>>>>> from an URL via curl.
>>>>>
>>>>> Reading PBFs from an MBTiles (or another SQLite) is just the most
>>>>>practical
>>>>> portable alternative to it.
>>>>>
>>>>> Technically people may store the tiles in different ways - for
>>>>>example
>>>>> Wikimedia guys (http://maps.wikimedia.org/ and
>>>>> https://github.com/kartotherian/) use Cassandra for tile storage, I
>>>>>guess
>>>>> MapBox internally is also on a different storage for production
>>>>>servers ...
>>>>> for transfers and streaming tilesets there is the  - but tiles are
>>>>>ALWAYS
>>>>> exposed via HTTP.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Petr
>>>>> --
>>>>> Petr Pridal, Ph.D.
>>>>> CEO
>>>>>
>>>>> Klokan Technologies GmbH
>>>>> Hofnerstrasse 98, 6314 Unterageri, Switzerland
>>>>> Tel: +41 (0)41 511 26 12
>>>>> Email: [hidden email]
>>>>> Web: http://www.klokantech.com/
>>>>>
>>>>> _______________________________________________
>>>>> gdal-dev mailing list
>>>>> [hidden email]
>>>>> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>_______________________________________________
>gdal-dev mailing list
>[hidden email]
>http://lists.osgeo.org/mailman/listinfo/gdal-dev

_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Vector Tiles in OGR

Stadin, Benjamin
In reply to this post by Stefan Keller
All,

I¹d like to share a few thoughts about the current state of vector tiles.
I believe that, with vector data, there is a chance to get rid of some of
the problems that current mapping tools face in practice, especially in
regards to projections. And this chance should be used in my opinion,
instead of doing things the "old way³, as with MVT.

The current specification and implementation of MVT is both depending on a
certain projection when the tiles are created, and with the
relative-to-screen coordinates per tile it also depends on how the
renderer is implemented (e.g. Commonly in other OpenGL applications, as in
our OpenGL based map, input data, screen scale and perspective
transformation are mostly handled in the GL shaders).
The dependency on a specific projection also means that none of the issues
occurring in practice when dealing with map projections gets fixed. Tom
MacWright from MapBox gave his feedback in the comment section at [1]
about why MapBox has limited support for other projections. He nails it
mostly, but I draw different conclusions. It is well possible to separate
data from representation with vector tiles, and thus a general purpose
vector tile format must deal with.

This isn¹t a rant against MapBox Vector Tiles or Mapbox. This isn¹t an
easy thing. I just want to raise awareness that MVT is, currently, a
particular implementation for a particular rendering engine.
 It works well for Mapbox maps, and it would be nice to have conversion
support for MVT. But I think that, without major braking changes, this
should not be considered to be a general purpose vector format at all.
This seems to be what some are hoping for.

There is enough interest from our side that we¹d be interested in
contributing (maybe monetary or with code, or both) to define and
implement a better general purpose tile format.

~Ben

[1] https://vis4.net/blog/posts/no-more-mercator-tiles/



Am 01.02.16, 20:16 schrieb "gdal-dev on behalf of Stefan Keller" unter
<[hidden email] on behalf of [hidden email]>:

>That's good news!
>We'll be happy to add Python bindings to such a generic C++ library
>for Mapbox vector tiles.
>We have still some time (say few weeks?) left here to decide to
>contribute either to these bindings or to Mapzen's pure Python lib.
>
>One reason why I'm insisting on Python (bindings) - besides easier
>testing with QGIS - is the following:
>AFAIK with vector tiles we have to deal with quite a new kind of
>feature provider:
>One which contains "zoom levels" and one which "suddenly" changes
>feature class at different zoom levels.
>If a client like QGIS consumes vector tiles from OGR in a one-shot
>call, how should the client deal with such "zoom level" dependent
>issues?
>So my thinking is, to consume only the most appropriate zoom level,
>probably only within current map canvas...
>
>:Stefan
>
>2016-02-01 18:27 GMT+01:00 Flippmoke <[hidden email]>:
>>
>> Stefan,
>>
>> I am not aware of any libraries besides mapnik vector tile that have
>>been attempting to implement v2 specification. I think mapzen's
>>implementation might be best? We have been wanting to add vector tiles
>>to the python mapnik bindings for quite some time but we don't have the
>>time to do it right now.
>>
>> I will try to think this week about how we could make a generic C++
>>library quickly for Mapbox vector tiles.
>>
>> Thanks,
>>
>> Blake Thompson
>>
>>> On Feb 1, 2016, at 1:06 AM, Stefan Keller <[hidden email]> wrote:
>>>
>>> Hi Blake
>>>
>>> It's clear to me that a good generic c++ library for encoding and
>>> decoding vector tiles is needed.
>>> On the other hand, I'd like to experiment with vector tiles and
>>> related QGIS parallel loading issues for which Python is more
>>> convenient.
>>>
>>> It seems that Mapzen's [1] Python implementation is further along.
>>> Or do you know if Jesse (or somebody) is still working on Mapbox's
>>>variant [2] ?
>>>
>>> :Stefan
>>>
>>> [1] https://github.com/mapzen/mapbox-vector-tile
>>> [2] https://github.com/mapbox/vector-tile-py
>>>
>>>
>>> 2016-01-31 23:22 GMT+01:00 Flippmoke <[hidden email]>:
>>>> All,
>>>>
>>>> The first thing that is going to be required likely is a good generic
>>>>c++ library for encoding and decoding vector tiles IMO.
>>>>
>>>> I have spent a lot of time working on developing the mapnik vector
>>>>tile library to be a solid implementation, especially around all the
>>>>work done for 2.0 of Mapbox vector tile specification. I would love to
>>>>eventually help make this a generalized implementation so that it is
>>>>portable to a library like GDAL.
>>>>
>>>> We have talked about doing something like this at Mapbox but will
>>>>require a good deal of time, so it hasn't been pushed forward yet. The
>>>>other issues that come to mind to me is that some of the libraries
>>>>that mapnik vector tile depends upon most likely need good custom
>>>>implementations if we wanted to make it more generic.
>>>>
>>>> The reason for this is that I would prefer for it to be a header only
>>>>library with no dependencies. Adding more dependencies for GDAL seems
>>>>like it could be a mess.
>>>>
>>>> Blake Thompson
>>>>
>>>>> On Jan 31, 2016, at 10:10 AM, Stefan Keller <[hidden email]>
>>>>>wrote:
>>>>>
>>>>> Hi Petr, Blake, Even and everybody
>>>>>
>>>>> I'm (besides Petr) the other maintainer of OSM2VectorTiles.
>>>>> I'd already asked Even a similar question.
>>>>> I'm advising now an intern to implement such a MVT Vector Tiles
>>>>>reader
>>>>> client in Python if possible as part of a QGIS plugin.
>>>>> And I'm interested not to do redundant implementations.
>>>>> Are there any news on this matter?
>>>>>
>>>>> :Stefan
>>>>> Prof. at HSR and leader of Geometa Lab
>>>>>
>>>>> @Even: Regarding OGR MVT development can you send to me (directly) a
>>>>> contact name of the group which won the bidding?
>>>>>
>>>>>
>>>>> 2016-01-06 15:24 GMT+01:00 Petr Pridal <[hidden email]>:
>>>>>> Hi everybody,
>>>>>>
>>>>>> thanks for the comments. Regarding the technical details:
>>>>>>
>>>>>>> Each Tile is its own discrete dataset therefore the hardest part
>>>>>>>would be
>>>>>>> merging of features into the original features across tiles. You
>>>>>>>could use
>>>>>>> some complicated merging technique, but that could be very
>>>>>>>expensive.
>>>>>>
>>>>>>
>>>>>> The driver could require "id" parameter, which specifies name of an
>>>>>> attribute with unique identifier on the features shared across the
>>>>>>tiles.
>>>>>> Both MapBox vector tiles and OSM2VectorTiles tiles have an unique
>>>>>>non-zero
>>>>>> "osm_id" in place. The first version of driver could read only
>>>>>>features with
>>>>>> such ID. The identifier does not have to be always filled, but
>>>>>>often it is,
>>>>>> especially in deeper zoom levels derived purely from OpenStreetMap
>>>>>>- but
>>>>>> yeh, a completely general stitching would be much harder.
>>>>>>
>>>>>> The clipping + merging of shapes itself can be implemented with
>>>>>>underlaying
>>>>>> GEOS library functions.
>>>>>>
>>>>>> BTW It is possible to examine vector tiles data in pure JavaScript
>>>>>>X-Ray
>>>>>> viewer (no WebGL support on browser required) made in OpenLayers3:
>>>>>> http://klokantech.github.io/ol3-sandbox/vector/xray.html
>>>>>>
>>>>>> or with WebGL-enabled webbrowser with higher performance MapBox GL
>>>>>>JS
>>>>>> powered X-Ray:
>>>>>> http://klokantech.github.io/mapbox-gl-js-offline-example/xray.html
>>>>>>
>>>>>> A debug viewer for each vector tile (made in OL3) decoded in pure
>>>>>> JavaScript:
>>>>>> http://klokantech.github.io/ol3-sandbox/vector/tile-inspector.html
>>>>>>
>>>>>> The OGR driver could be also practical for the unsimplified /
>>>>>>ungeneralised
>>>>>> MBTiles with OSM data available at:
>>>>>>http://osmlab.github.io/osm-qa-tiles/
>>>>>>
>>>>>>>
>>>>>>> If you wanted to simply view vector tile data it could get
>>>>>>>confusing as
>>>>>>> well since most vector tiles need a buffer when they are used for
>>>>>>> visualization programs. This is important for things like labeling
>>>>>>>on maps
>>>>>>> etc. Therefore, you could get a large amount of overlapping data
>>>>>>>on tiles if
>>>>>>> you loaded them all in at once.
>>>>>>
>>>>>>
>>>>>> Clipping on border of each tile must be applied before stitching
>>>>>>the vector
>>>>>> features together  - so the buffer is removed. Buffer is there only
>>>>>>for
>>>>>> visualisation and labeling.
>>>>>>
>>>>>>>
>>>>>>> As far as writing a simple decoder -- the mapnik-vector-tile
>>>>>>>decoder isn't
>>>>>>> currently able to decode vector tiles into a form not supported by
>>>>>>>mapnik.
>>>>>>> Therefore, OGR would not be able to easily use this.
>>>>>>
>>>>>>
>>>>>> Correct. It can't be used directly (GDAL can't link against), but
>>>>>>the
>>>>>> relevant code can be extracted and reused directly. For basic
>>>>>>decoding
>>>>>> critical is the Protobuf library (which is already in GDAL) and
>>>>>>
>>>>>>https://github.com/mapbox/mapnik-vector-tile/blob/master/proto/vector
>>>>>>_tile.proto
>>>>>> + the code for converting pixel coordinates in features to relevant
>>>>>>geo
>>>>>> coordinates and GEOS-based merging of features.
>>>>>>
>>>>>>> We plan to write a generalized encoder and decoder eventually so
>>>>>>>that any
>>>>>>> other library can plugin with out the mapnik requirement.
>>>>>>
>>>>>>
>>>>>> This would be amazing.
>>>>>>
>>>>>>> If you have any questions specifically about Mapnik Vector Tile or
>>>>>>>the
>>>>>>> Mapbox Vector Tile Specification feel free to ask me. I would note
>>>>>>>that you
>>>>>>> guys will want to update your vector tiles once the V2
>>>>>>>specification changes
>>>>>>> are into Mapnik-Vector-Tile.
>>>>>>
>>>>>>
>>>>>> Cool! Thanks. Our world tiles are generated via Docker containers
>>>>>>on a
>>>>>> cluster of computers - using Mapnik via Tilelive directly - so
>>>>>>upgrades on
>>>>>> Mapnik-Vector-Tile will propagate to the open-source rendering
>>>>>>software.
>>>>>> I expect V2 spec will mean also new MapBox Streets V7, right?
>>>>>>
>>>>>>> I see you mention MVT in MBTiles. Is it a recognized practice ? I
>>>>>>>don't
>>>>>>> see
>>>>>>> mention of that neither in the MVT spec (
>>>>>>> https://github.com/mapbox/vector-
>>>>>>> tile-spec/tree/master/2.0 ) nor the MBTiles one (
>>>>>>> https://github.com/mapbox/mbtiles-spec/blob/master/1.2/spec.md ),
>>>>>>>although
>>>>>>> I
>>>>>>> guess it is just a matter of storing a MVT blob in the tile_data
>>>>>>>column.
>>>>>>
>>>>>>
>>>>>> Yes. MapBox Studio Classic generates such MBTiles with Vector Tiles
>>>>>>inside -
>>>>>> and Mapnik reads them directly when rasterising. Storing the vector
>>>>>>tiles in
>>>>>> SQLite make sense - and it simplifies deployment, and you can then
>>>>>>easily
>>>>>> have complete world on each node of a cluster of tileservers. If
>>>>>>you don't
>>>>>> want to upgrade your base map too often, this approach is pretty
>>>>>>efficient
>>>>>> and removes the centralised database as typical bottleneck.
>>>>>>
>>>>>> I did some tests on vector tiles vs raster in MBTiles and
>>>>>>documented these
>>>>>> in README.md at and
>>>>>>https://github.com/klokantech/vector-tiles-sample before
>>>>>> we started to work on OSM2VectorTiles project.
>>>>>> There are also sample data and offline MapBox GL JS viewer in that
>>>>>>repo.
>>>>>> Vector tiles can be hosted also "unpacked", just as files in
>>>>>>folders on a
>>>>>> standard web server - exactly like GDAL2Tiles or MapTiler.com does
>>>>>>by
>>>>>> default for raster tiles.
>>>>>>
>>>>>> If OGR driver is implemented, the primary source should be probably
>>>>>>reading
>>>>>> from an URL via curl.
>>>>>>
>>>>>> Reading PBFs from an MBTiles (or another SQLite) is just the most
>>>>>>practical
>>>>>> portable alternative to it.
>>>>>>
>>>>>> Technically people may store the tiles in different ways - for
>>>>>>example
>>>>>> Wikimedia guys (http://maps.wikimedia.org/ and
>>>>>> https://github.com/kartotherian/) use Cassandra for tile storage, I
>>>>>>guess
>>>>>> MapBox internally is also on a different storage for production
>>>>>>servers ...
>>>>>> for transfers and streaming tilesets there is the  - but tiles are
>>>>>>ALWAYS
>>>>>> exposed via HTTP.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Petr
>>>>>> --
>>>>>> Petr Pridal, Ph.D.
>>>>>> CEO
>>>>>>
>>>>>> Klokan Technologies GmbH
>>>>>> Hofnerstrasse 98, 6314 Unterageri, Switzerland
>>>>>> Tel: +41 (0)41 511 26 12
>>>>>> Email: [hidden email]
>>>>>> Web: http://www.klokantech.com/
>>>>>>
>>>>>> _______________________________________________
>>>>>> gdal-dev mailing list
>>>>>> [hidden email]
>>>>>> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>_______________________________________________
>gdal-dev mailing list
>[hidden email]
>http://lists.osgeo.org/mailman/listinfo/gdal-dev

_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Vector Tiles in OGR

Stadin, Benjamin
In reply to this post by Stefan Keller
Hi Stefan,

A vector tile may contain data for several zoom levels. But what data is
shown at which zoom level is defined by the styling, not by the data
itself. It is handled on client side, by the web (or native) tile
renderer.

The path is the same as with mercator tiles (zoomLevel/x/y):
https://example.com/17/65535/43602.mvt


The tile for zoom level 17 may contain features of class „building“, and
the style definition may contain a rule that filters type „building“ for
zoomLevels <= 18.

~Ben

Am 01.02.16, 20:16 schrieb "gdal-dev on behalf of Stefan Keller" unter
<[hidden email] on behalf of [hidden email]>:

>That's good news!
>We'll be happy to add Python bindings to such a generic C++ library
>for Mapbox vector tiles.
>We have still some time (say few weeks?) left here to decide to
>contribute either to these bindings or to Mapzen's pure Python lib.
>
>One reason why I'm insisting on Python (bindings) - besides easier
>testing with QGIS - is the following:
>AFAIK with vector tiles we have to deal with quite a new kind of
>feature provider:
>One which contains "zoom levels" and one which "suddenly" changes
>feature class at different zoom levels.
>If a client like QGIS consumes vector tiles from OGR in a one-shot
>call, how should the client deal with such "zoom level" dependent
>issues?
>So my thinking is, to consume only the most appropriate zoom level,
>probably only within current map canvas...
>
>:Stefan
>
>2016-02-01 18:27 GMT+01:00 Flippmoke <[hidden email]>:
>>
>> Stefan,
>>
>> I am not aware of any libraries besides mapnik vector tile that have
>>been attempting to implement v2 specification. I think mapzen's
>>implementation might be best? We have been wanting to add vector tiles
>>to the python mapnik bindings for quite some time but we don't have the
>>time to do it right now.
>>
>> I will try to think this week about how we could make a generic C++
>>library quickly for Mapbox vector tiles.
>>
>> Thanks,
>>
>> Blake Thompson
>>
>>> On Feb 1, 2016, at 1:06 AM, Stefan Keller <[hidden email]> wrote:
>>>
>>> Hi Blake
>>>
>>> It's clear to me that a good generic c++ library for encoding and
>>> decoding vector tiles is needed.
>>> On the other hand, I'd like to experiment with vector tiles and
>>> related QGIS parallel loading issues for which Python is more
>>> convenient.
>>>
>>> It seems that Mapzen's [1] Python implementation is further along.
>>> Or do you know if Jesse (or somebody) is still working on Mapbox's
>>>variant [2] ?
>>>
>>> :Stefan
>>>
>>> [1] https://github.com/mapzen/mapbox-vector-tile
>>> [2] https://github.com/mapbox/vector-tile-py
>>>
>>>
>>> 2016-01-31 23:22 GMT+01:00 Flippmoke <[hidden email]>:
>>>> All,
>>>>
>>>> The first thing that is going to be required likely is a good generic
>>>>c++ library for encoding and decoding vector tiles IMO.
>>>>
>>>> I have spent a lot of time working on developing the mapnik vector
>>>>tile library to be a solid implementation, especially around all the
>>>>work done for 2.0 of Mapbox vector tile specification. I would love to
>>>>eventually help make this a generalized implementation so that it is
>>>>portable to a library like GDAL.
>>>>
>>>> We have talked about doing something like this at Mapbox but will
>>>>require a good deal of time, so it hasn't been pushed forward yet. The
>>>>other issues that come to mind to me is that some of the libraries
>>>>that mapnik vector tile depends upon most likely need good custom
>>>>implementations if we wanted to make it more generic.
>>>>
>>>> The reason for this is that I would prefer for it to be a header only
>>>>library with no dependencies. Adding more dependencies for GDAL seems
>>>>like it could be a mess.
>>>>
>>>> Blake Thompson
>>>>
>>>>> On Jan 31, 2016, at 10:10 AM, Stefan Keller <[hidden email]>
>>>>>wrote:
>>>>>
>>>>> Hi Petr, Blake, Even and everybody
>>>>>
>>>>> I'm (besides Petr) the other maintainer of OSM2VectorTiles.
>>>>> I'd already asked Even a similar question.
>>>>> I'm advising now an intern to implement such a MVT Vector Tiles
>>>>>reader
>>>>> client in Python if possible as part of a QGIS plugin.
>>>>> And I'm interested not to do redundant implementations.
>>>>> Are there any news on this matter?
>>>>>
>>>>> :Stefan
>>>>> Prof. at HSR and leader of Geometa Lab
>>>>>
>>>>> @Even: Regarding OGR MVT development can you send to me (directly) a
>>>>> contact name of the group which won the bidding?
>>>>>
>>>>>
>>>>> 2016-01-06 15:24 GMT+01:00 Petr Pridal <[hidden email]>:
>>>>>> Hi everybody,
>>>>>>
>>>>>> thanks for the comments. Regarding the technical details:
>>>>>>
>>>>>>> Each Tile is its own discrete dataset therefore the hardest part
>>>>>>>would be
>>>>>>> merging of features into the original features across tiles. You
>>>>>>>could use
>>>>>>> some complicated merging technique, but that could be very
>>>>>>>expensive.
>>>>>>
>>>>>>
>>>>>> The driver could require "id" parameter, which specifies name of an
>>>>>> attribute with unique identifier on the features shared across the
>>>>>>tiles.
>>>>>> Both MapBox vector tiles and OSM2VectorTiles tiles have an unique
>>>>>>non-zero
>>>>>> "osm_id" in place. The first version of driver could read only
>>>>>>features with
>>>>>> such ID. The identifier does not have to be always filled, but
>>>>>>often it is,
>>>>>> especially in deeper zoom levels derived purely from OpenStreetMap
>>>>>>- but
>>>>>> yeh, a completely general stitching would be much harder.
>>>>>>
>>>>>> The clipping + merging of shapes itself can be implemented with
>>>>>>underlaying
>>>>>> GEOS library functions.
>>>>>>
>>>>>> BTW It is possible to examine vector tiles data in pure JavaScript
>>>>>>X-Ray
>>>>>> viewer (no WebGL support on browser required) made in OpenLayers3:
>>>>>> http://klokantech.github.io/ol3-sandbox/vector/xray.html
>>>>>>
>>>>>> or with WebGL-enabled webbrowser with higher performance MapBox GL
>>>>>>JS
>>>>>> powered X-Ray:
>>>>>> http://klokantech.github.io/mapbox-gl-js-offline-example/xray.html
>>>>>>
>>>>>> A debug viewer for each vector tile (made in OL3) decoded in pure
>>>>>> JavaScript:
>>>>>> http://klokantech.github.io/ol3-sandbox/vector/tile-inspector.html
>>>>>>
>>>>>> The OGR driver could be also practical for the unsimplified /
>>>>>>ungeneralised
>>>>>> MBTiles with OSM data available at:
>>>>>>http://osmlab.github.io/osm-qa-tiles/
>>>>>>
>>>>>>>
>>>>>>> If you wanted to simply view vector tile data it could get
>>>>>>>confusing as
>>>>>>> well since most vector tiles need a buffer when they are used for
>>>>>>> visualization programs. This is important for things like labeling
>>>>>>>on maps
>>>>>>> etc. Therefore, you could get a large amount of overlapping data
>>>>>>>on tiles if
>>>>>>> you loaded them all in at once.
>>>>>>
>>>>>>
>>>>>> Clipping on border of each tile must be applied before stitching
>>>>>>the vector
>>>>>> features together  - so the buffer is removed. Buffer is there only
>>>>>>for
>>>>>> visualisation and labeling.
>>>>>>
>>>>>>>
>>>>>>> As far as writing a simple decoder -- the mapnik-vector-tile
>>>>>>>decoder isn't
>>>>>>> currently able to decode vector tiles into a form not supported by
>>>>>>>mapnik.
>>>>>>> Therefore, OGR would not be able to easily use this.
>>>>>>
>>>>>>
>>>>>> Correct. It can't be used directly (GDAL can't link against), but
>>>>>>the
>>>>>> relevant code can be extracted and reused directly. For basic
>>>>>>decoding
>>>>>> critical is the Protobuf library (which is already in GDAL) and
>>>>>>
>>>>>>https://github.com/mapbox/mapnik-vector-tile/blob/master/proto/vector
>>>>>>_tile.proto
>>>>>> + the code for converting pixel coordinates in features to relevant
>>>>>>geo
>>>>>> coordinates and GEOS-based merging of features.
>>>>>>
>>>>>>> We plan to write a generalized encoder and decoder eventually so
>>>>>>>that any
>>>>>>> other library can plugin with out the mapnik requirement.
>>>>>>
>>>>>>
>>>>>> This would be amazing.
>>>>>>
>>>>>>> If you have any questions specifically about Mapnik Vector Tile or
>>>>>>>the
>>>>>>> Mapbox Vector Tile Specification feel free to ask me. I would note
>>>>>>>that you
>>>>>>> guys will want to update your vector tiles once the V2
>>>>>>>specification changes
>>>>>>> are into Mapnik-Vector-Tile.
>>>>>>
>>>>>>
>>>>>> Cool! Thanks. Our world tiles are generated via Docker containers
>>>>>>on a
>>>>>> cluster of computers - using Mapnik via Tilelive directly - so
>>>>>>upgrades on
>>>>>> Mapnik-Vector-Tile will propagate to the open-source rendering
>>>>>>software.
>>>>>> I expect V2 spec will mean also new MapBox Streets V7, right?
>>>>>>
>>>>>>> I see you mention MVT in MBTiles. Is it a recognized practice ? I
>>>>>>>don't
>>>>>>> see
>>>>>>> mention of that neither in the MVT spec (
>>>>>>> https://github.com/mapbox/vector-
>>>>>>> tile-spec/tree/master/2.0 ) nor the MBTiles one (
>>>>>>> https://github.com/mapbox/mbtiles-spec/blob/master/1.2/spec.md ),
>>>>>>>although
>>>>>>> I
>>>>>>> guess it is just a matter of storing a MVT blob in the tile_data
>>>>>>>column.
>>>>>>
>>>>>>
>>>>>> Yes. MapBox Studio Classic generates such MBTiles with Vector Tiles
>>>>>>inside -
>>>>>> and Mapnik reads them directly when rasterising. Storing the vector
>>>>>>tiles in
>>>>>> SQLite make sense - and it simplifies deployment, and you can then
>>>>>>easily
>>>>>> have complete world on each node of a cluster of tileservers. If
>>>>>>you don't
>>>>>> want to upgrade your base map too often, this approach is pretty
>>>>>>efficient
>>>>>> and removes the centralised database as typical bottleneck.
>>>>>>
>>>>>> I did some tests on vector tiles vs raster in MBTiles and
>>>>>>documented these
>>>>>> in README.md at and
>>>>>>https://github.com/klokantech/vector-tiles-sample before
>>>>>> we started to work on OSM2VectorTiles project.
>>>>>> There are also sample data and offline MapBox GL JS viewer in that
>>>>>>repo.
>>>>>> Vector tiles can be hosted also "unpacked", just as files in
>>>>>>folders on a
>>>>>> standard web server - exactly like GDAL2Tiles or MapTiler.com does
>>>>>>by
>>>>>> default for raster tiles.
>>>>>>
>>>>>> If OGR driver is implemented, the primary source should be probably
>>>>>>reading
>>>>>> from an URL via curl.
>>>>>>
>>>>>> Reading PBFs from an MBTiles (or another SQLite) is just the most
>>>>>>practical
>>>>>> portable alternative to it.
>>>>>>
>>>>>> Technically people may store the tiles in different ways - for
>>>>>>example
>>>>>> Wikimedia guys (http://maps.wikimedia.org/ and
>>>>>> https://github.com/kartotherian/) use Cassandra for tile storage, I
>>>>>>guess
>>>>>> MapBox internally is also on a different storage for production
>>>>>>servers ...
>>>>>> for transfers and streaming tilesets there is the  - but tiles are
>>>>>>ALWAYS
>>>>>> exposed via HTTP.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Petr
>>>>>> --
>>>>>> Petr Pridal, Ph.D.
>>>>>> CEO
>>>>>>
>>>>>> Klokan Technologies GmbH
>>>>>> Hofnerstrasse 98, 6314 Unterageri, Switzerland
>>>>>> Tel: +41 (0)41 511 26 12
>>>>>> Email: [hidden email]
>>>>>> Web: http://www.klokantech.com/
>>>>>>
>>>>>> _______________________________________________
>>>>>> gdal-dev mailing list
>>>>>> [hidden email]
>>>>>> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>_______________________________________________
>gdal-dev mailing list
>[hidden email]
>http://lists.osgeo.org/mailman/listinfo/gdal-dev

_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Vector Tiles in OGR

Stefan Keller
Hi Ben

Thanks for explaining - Petr and I are the maintainers of OSM2Vectortiles.
I like projections (and also GeoPackage :-) ).
But reprojection is solved here, since OGR/QGIS can transform on the fly.
This thread is about Petr's question about a reader "for mapbox-like
vector tiles stored in MBTiles or downloaded from a url...".

So I'd like to come back to the issue:
1. of a generic MVT decoder/enoder library (C++ or Python) => Blake?
2. and of a "feature provider" in an interactive environment (like
QGIS) or a file/stream converter (like OGR) => anybody?

:Stefan



2016-02-02 0:04 GMT+01:00 Stadin, Benjamin
<[hidden email]>:

> Hi Stefan,
>
> A vector tile may contain data for several zoom levels. But what data is
> shown at which zoom level is defined by the styling, not by the data
> itself. It is handled on client side, by the web (or native) tile
> renderer.
>
> The path is the same as with mercator tiles (zoomLevel/x/y):
> https://example.com/17/65535/43602.mvt
>
>
> The tile for zoom level 17 may contain features of class „building“, and
> the style definition may contain a rule that filters type „building“ for
> zoomLevels <= 18.
>
> ~Ben
>
> Am 01.02.16, 20:16 schrieb "gdal-dev on behalf of Stefan Keller" unter
> <[hidden email] on behalf of [hidden email]>:
>
>>That's good news!
>>We'll be happy to add Python bindings to such a generic C++ library
>>for Mapbox vector tiles.
>>We have still some time (say few weeks?) left here to decide to
>>contribute either to these bindings or to Mapzen's pure Python lib.
>>
>>One reason why I'm insisting on Python (bindings) - besides easier
>>testing with QGIS - is the following:
>>AFAIK with vector tiles we have to deal with quite a new kind of
>>feature provider:
>>One which contains "zoom levels" and one which "suddenly" changes
>>feature class at different zoom levels.
>>If a client like QGIS consumes vector tiles from OGR in a one-shot
>>call, how should the client deal with such "zoom level" dependent
>>issues?
>>So my thinking is, to consume only the most appropriate zoom level,
>>probably only within current map canvas...
>>
>>:Stefan
>>
>>2016-02-01 18:27 GMT+01:00 Flippmoke <[hidden email]>:
>>>
>>> Stefan,
>>>
>>> I am not aware of any libraries besides mapnik vector tile that have
>>>been attempting to implement v2 specification. I think mapzen's
>>>implementation might be best? We have been wanting to add vector tiles
>>>to the python mapnik bindings for quite some time but we don't have the
>>>time to do it right now.
>>>
>>> I will try to think this week about how we could make a generic C++
>>>library quickly for Mapbox vector tiles.
>>>
>>> Thanks,
>>>
>>> Blake Thompson
>>>
>>>> On Feb 1, 2016, at 1:06 AM, Stefan Keller <[hidden email]> wrote:
>>>>
>>>> Hi Blake
>>>>
>>>> It's clear to me that a good generic c++ library for encoding and
>>>> decoding vector tiles is needed.
>>>> On the other hand, I'd like to experiment with vector tiles and
>>>> related QGIS parallel loading issues for which Python is more
>>>> convenient.
>>>>
>>>> It seems that Mapzen's [1] Python implementation is further along.
>>>> Or do you know if Jesse (or somebody) is still working on Mapbox's
>>>>variant [2] ?
>>>>
>>>> :Stefan
>>>>
>>>> [1] https://github.com/mapzen/mapbox-vector-tile
>>>> [2] https://github.com/mapbox/vector-tile-py
>>>>
>>>>
>>>> 2016-01-31 23:22 GMT+01:00 Flippmoke <[hidden email]>:
>>>>> All,
>>>>>
>>>>> The first thing that is going to be required likely is a good generic
>>>>>c++ library for encoding and decoding vector tiles IMO.
>>>>>
>>>>> I have spent a lot of time working on developing the mapnik vector
>>>>>tile library to be a solid implementation, especially around all the
>>>>>work done for 2.0 of Mapbox vector tile specification. I would love to
>>>>>eventually help make this a generalized implementation so that it is
>>>>>portable to a library like GDAL.
>>>>>
>>>>> We have talked about doing something like this at Mapbox but will
>>>>>require a good deal of time, so it hasn't been pushed forward yet. The
>>>>>other issues that come to mind to me is that some of the libraries
>>>>>that mapnik vector tile depends upon most likely need good custom
>>>>>implementations if we wanted to make it more generic.
>>>>>
>>>>> The reason for this is that I would prefer for it to be a header only
>>>>>library with no dependencies. Adding more dependencies for GDAL seems
>>>>>like it could be a mess.
>>>>>
>>>>> Blake Thompson
>>>>>
>>>>>> On Jan 31, 2016, at 10:10 AM, Stefan Keller <[hidden email]>
>>>>>>wrote:
>>>>>>
>>>>>> Hi Petr, Blake, Even and everybody
>>>>>>
>>>>>> I'm (besides Petr) the other maintainer of OSM2VectorTiles.
>>>>>> I'd already asked Even a similar question.
>>>>>> I'm advising now an intern to implement such a MVT Vector Tiles
>>>>>>reader
>>>>>> client in Python if possible as part of a QGIS plugin.
>>>>>> And I'm interested not to do redundant implementations.
>>>>>> Are there any news on this matter?
>>>>>>
>>>>>> :Stefan
>>>>>> Prof. at HSR and leader of Geometa Lab
>>>>>>
>>>>>> @Even: Regarding OGR MVT development can you send to me (directly) a
>>>>>> contact name of the group which won the bidding?
>>>>>>
>>>>>>
>>>>>> 2016-01-06 15:24 GMT+01:00 Petr Pridal <[hidden email]>:
>>>>>>> Hi everybody,
>>>>>>>
>>>>>>> thanks for the comments. Regarding the technical details:
>>>>>>>
>>>>>>>> Each Tile is its own discrete dataset therefore the hardest part
>>>>>>>>would be
>>>>>>>> merging of features into the original features across tiles. You
>>>>>>>>could use
>>>>>>>> some complicated merging technique, but that could be very
>>>>>>>>expensive.
>>>>>>>
>>>>>>>
>>>>>>> The driver could require "id" parameter, which specifies name of an
>>>>>>> attribute with unique identifier on the features shared across the
>>>>>>>tiles.
>>>>>>> Both MapBox vector tiles and OSM2VectorTiles tiles have an unique
>>>>>>>non-zero
>>>>>>> "osm_id" in place. The first version of driver could read only
>>>>>>>features with
>>>>>>> such ID. The identifier does not have to be always filled, but
>>>>>>>often it is,
>>>>>>> especially in deeper zoom levels derived purely from OpenStreetMap
>>>>>>>- but
>>>>>>> yeh, a completely general stitching would be much harder.
>>>>>>>
>>>>>>> The clipping + merging of shapes itself can be implemented with
>>>>>>>underlaying
>>>>>>> GEOS library functions.
>>>>>>>
>>>>>>> BTW It is possible to examine vector tiles data in pure JavaScript
>>>>>>>X-Ray
>>>>>>> viewer (no WebGL support on browser required) made in OpenLayers3:
>>>>>>> http://klokantech.github.io/ol3-sandbox/vector/xray.html
>>>>>>>
>>>>>>> or with WebGL-enabled webbrowser with higher performance MapBox GL
>>>>>>>JS
>>>>>>> powered X-Ray:
>>>>>>> http://klokantech.github.io/mapbox-gl-js-offline-example/xray.html
>>>>>>>
>>>>>>> A debug viewer for each vector tile (made in OL3) decoded in pure
>>>>>>> JavaScript:
>>>>>>> http://klokantech.github.io/ol3-sandbox/vector/tile-inspector.html
>>>>>>>
>>>>>>> The OGR driver could be also practical for the unsimplified /
>>>>>>>ungeneralised
>>>>>>> MBTiles with OSM data available at:
>>>>>>>http://osmlab.github.io/osm-qa-tiles/
>>>>>>>
>>>>>>>>
>>>>>>>> If you wanted to simply view vector tile data it could get
>>>>>>>>confusing as
>>>>>>>> well since most vector tiles need a buffer when they are used for
>>>>>>>> visualization programs. This is important for things like labeling
>>>>>>>>on maps
>>>>>>>> etc. Therefore, you could get a large amount of overlapping data
>>>>>>>>on tiles if
>>>>>>>> you loaded them all in at once.
>>>>>>>
>>>>>>>
>>>>>>> Clipping on border of each tile must be applied before stitching
>>>>>>>the vector
>>>>>>> features together  - so the buffer is removed. Buffer is there only
>>>>>>>for
>>>>>>> visualisation and labeling.
>>>>>>>
>>>>>>>>
>>>>>>>> As far as writing a simple decoder -- the mapnik-vector-tile
>>>>>>>>decoder isn't
>>>>>>>> currently able to decode vector tiles into a form not supported by
>>>>>>>>mapnik.
>>>>>>>> Therefore, OGR would not be able to easily use this.
>>>>>>>
>>>>>>>
>>>>>>> Correct. It can't be used directly (GDAL can't link against), but
>>>>>>>the
>>>>>>> relevant code can be extracted and reused directly. For basic
>>>>>>>decoding
>>>>>>> critical is the Protobuf library (which is already in GDAL) and
>>>>>>>
>>>>>>>https://github.com/mapbox/mapnik-vector-tile/blob/master/proto/vector
>>>>>>>_tile.proto
>>>>>>> + the code for converting pixel coordinates in features to relevant
>>>>>>>geo
>>>>>>> coordinates and GEOS-based merging of features.
>>>>>>>
>>>>>>>> We plan to write a generalized encoder and decoder eventually so
>>>>>>>>that any
>>>>>>>> other library can plugin with out the mapnik requirement.
>>>>>>>
>>>>>>>
>>>>>>> This would be amazing.
>>>>>>>
>>>>>>>> If you have any questions specifically about Mapnik Vector Tile or
>>>>>>>>the
>>>>>>>> Mapbox Vector Tile Specification feel free to ask me. I would note
>>>>>>>>that you
>>>>>>>> guys will want to update your vector tiles once the V2
>>>>>>>>specification changes
>>>>>>>> are into Mapnik-Vector-Tile.
>>>>>>>
>>>>>>>
>>>>>>> Cool! Thanks. Our world tiles are generated via Docker containers
>>>>>>>on a
>>>>>>> cluster of computers - using Mapnik via Tilelive directly - so
>>>>>>>upgrades on
>>>>>>> Mapnik-Vector-Tile will propagate to the open-source rendering
>>>>>>>software.
>>>>>>> I expect V2 spec will mean also new MapBox Streets V7, right?
>>>>>>>
>>>>>>>> I see you mention MVT in MBTiles. Is it a recognized practice ? I
>>>>>>>>don't
>>>>>>>> see
>>>>>>>> mention of that neither in the MVT spec (
>>>>>>>> https://github.com/mapbox/vector-
>>>>>>>> tile-spec/tree/master/2.0 ) nor the MBTiles one (
>>>>>>>> https://github.com/mapbox/mbtiles-spec/blob/master/1.2/spec.md ),
>>>>>>>>although
>>>>>>>> I
>>>>>>>> guess it is just a matter of storing a MVT blob in the tile_data
>>>>>>>>column.
>>>>>>>
>>>>>>>
>>>>>>> Yes. MapBox Studio Classic generates such MBTiles with Vector Tiles
>>>>>>>inside -
>>>>>>> and Mapnik reads them directly when rasterising. Storing the vector
>>>>>>>tiles in
>>>>>>> SQLite make sense - and it simplifies deployment, and you can then
>>>>>>>easily
>>>>>>> have complete world on each node of a cluster of tileservers. If
>>>>>>>you don't
>>>>>>> want to upgrade your base map too often, this approach is pretty
>>>>>>>efficient
>>>>>>> and removes the centralised database as typical bottleneck.
>>>>>>>
>>>>>>> I did some tests on vector tiles vs raster in MBTiles and
>>>>>>>documented these
>>>>>>> in README.md at and
>>>>>>>https://github.com/klokantech/vector-tiles-sample before
>>>>>>> we started to work on OSM2VectorTiles project.
>>>>>>> There are also sample data and offline MapBox GL JS viewer in that
>>>>>>>repo.
>>>>>>> Vector tiles can be hosted also "unpacked", just as files in
>>>>>>>folders on a
>>>>>>> standard web server - exactly like GDAL2Tiles or MapTiler.com does
>>>>>>>by
>>>>>>> default for raster tiles.
>>>>>>>
>>>>>>> If OGR driver is implemented, the primary source should be probably
>>>>>>>reading
>>>>>>> from an URL via curl.
>>>>>>>
>>>>>>> Reading PBFs from an MBTiles (or another SQLite) is just the most
>>>>>>>practical
>>>>>>> portable alternative to it.
>>>>>>>
>>>>>>> Technically people may store the tiles in different ways - for
>>>>>>>example
>>>>>>> Wikimedia guys (http://maps.wikimedia.org/ and
>>>>>>> https://github.com/kartotherian/) use Cassandra for tile storage, I
>>>>>>>guess
>>>>>>> MapBox internally is also on a different storage for production
>>>>>>>servers ...
>>>>>>> for transfers and streaming tilesets there is the  - but tiles are
>>>>>>>ALWAYS
>>>>>>> exposed via HTTP.
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Petr
>>>>>>> --
>>>>>>> Petr Pridal, Ph.D.
>>>>>>> CEO
>>>>>>>
>>>>>>> Klokan Technologies GmbH
>>>>>>> Hofnerstrasse 98, 6314 Unterageri, Switzerland
>>>>>>> Tel: +41 (0)41 511 26 12
>>>>>>> Email: [hidden email]
>>>>>>> Web: http://www.klokantech.com/
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> gdal-dev mailing list
>>>>>>> [hidden email]
>>>>>>> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>>_______________________________________________
>>gdal-dev mailing list
>>[hidden email]
>>http://lists.osgeo.org/mailman/listinfo/gdal-dev
>
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Vector Tiles in OGR

Stadin, Benjamin
Hi Stefan,

"If there would be OGR driver for vector tiles it would make a similar
resource extremely useful also for other purposes in GIS tools - not only
for tileservers and direct visualisation.“

I was thinking you want to do more than just rendering. Then at least, as
Blake also mentioned, the features need to be merged. And I think some
degenerate geometries are to be expected for a few cases. You will also
have multiple polygons cut along tile bounds for what should be a single
polygon. And the introduced points to close a polygon’s path at the tile
bounds may add another inaccuracy when you later reproject the data.

It depends what your goal is. Serious map editing isn’t possible.

~Ben


Am 02.02.16, 00:27 schrieb "Stefan Keller" unter <[hidden email]>:

>Hi Ben
>
>Thanks for explaining - Petr and I are the maintainers of OSM2Vectortiles.
>I like projections (and also GeoPackage :-) ).
>But reprojection is solved here, since OGR/QGIS can transform on the fly.
>This thread is about Petr's question about a reader "for mapbox-like
>vector tiles stored in MBTiles or downloaded from a url...".
>
>So I'd like to come back to the issue:
>1. of a generic MVT decoder/enoder library (C++ or Python) => Blake?
>2. and of a "feature provider" in an interactive environment (like
>QGIS) or a file/stream converter (like OGR) => anybody?
>
>:Stefan
>
>
>
>2016-02-02 0:04 GMT+01:00 Stadin, Benjamin
><[hidden email]>:
>> Hi Stefan,
>>
>> A vector tile may contain data for several zoom levels. But what data is
>> shown at which zoom level is defined by the styling, not by the data
>> itself. It is handled on client side, by the web (or native) tile
>> renderer.
>>
>> The path is the same as with mercator tiles (zoomLevel/x/y):
>> https://example.com/17/65535/43602.mvt
>>
>>
>> The tile for zoom level 17 may contain features of class „building“, and
>> the style definition may contain a rule that filters type „building“ for
>> zoomLevels <= 18.
>>
>> ~Ben
>>
>> Am 01.02.16, 20:16 schrieb "gdal-dev on behalf of Stefan Keller" unter
>> <[hidden email] on behalf of [hidden email]>:
>>
>>>That's good news!
>>>We'll be happy to add Python bindings to such a generic C++ library
>>>for Mapbox vector tiles.
>>>We have still some time (say few weeks?) left here to decide to
>>>contribute either to these bindings or to Mapzen's pure Python lib.
>>>
>>>One reason why I'm insisting on Python (bindings) - besides easier
>>>testing with QGIS - is the following:
>>>AFAIK with vector tiles we have to deal with quite a new kind of
>>>feature provider:
>>>One which contains "zoom levels" and one which "suddenly" changes
>>>feature class at different zoom levels.
>>>If a client like QGIS consumes vector tiles from OGR in a one-shot
>>>call, how should the client deal with such "zoom level" dependent
>>>issues?
>>>So my thinking is, to consume only the most appropriate zoom level,
>>>probably only within current map canvas...
>>>
>>>:Stefan
>>>
>>>2016-02-01 18:27 GMT+01:00 Flippmoke <[hidden email]>:
>>>>
>>>> Stefan,
>>>>
>>>> I am not aware of any libraries besides mapnik vector tile that have
>>>>been attempting to implement v2 specification. I think mapzen's
>>>>implementation might be best? We have been wanting to add vector tiles
>>>>to the python mapnik bindings for quite some time but we don't have the
>>>>time to do it right now.
>>>>
>>>> I will try to think this week about how we could make a generic C++
>>>>library quickly for Mapbox vector tiles.
>>>>
>>>> Thanks,
>>>>
>>>> Blake Thompson
>>>>
>>>>> On Feb 1, 2016, at 1:06 AM, Stefan Keller <[hidden email]> wrote:
>>>>>
>>>>> Hi Blake
>>>>>
>>>>> It's clear to me that a good generic c++ library for encoding and
>>>>> decoding vector tiles is needed.
>>>>> On the other hand, I'd like to experiment with vector tiles and
>>>>> related QGIS parallel loading issues for which Python is more
>>>>> convenient.
>>>>>
>>>>> It seems that Mapzen's [1] Python implementation is further along.
>>>>> Or do you know if Jesse (or somebody) is still working on Mapbox's
>>>>>variant [2] ?
>>>>>
>>>>> :Stefan
>>>>>
>>>>> [1] https://github.com/mapzen/mapbox-vector-tile
>>>>> [2] https://github.com/mapbox/vector-tile-py
>>>>>
>>>>>
>>>>> 2016-01-31 23:22 GMT+01:00 Flippmoke <[hidden email]>:
>>>>>> All,
>>>>>>
>>>>>> The first thing that is going to be required likely is a good
>>>>>>generic
>>>>>>c++ library for encoding and decoding vector tiles IMO.
>>>>>>
>>>>>> I have spent a lot of time working on developing the mapnik vector
>>>>>>tile library to be a solid implementation, especially around all the
>>>>>>work done for 2.0 of Mapbox vector tile specification. I would love
>>>>>>to
>>>>>>eventually help make this a generalized implementation so that it is
>>>>>>portable to a library like GDAL.
>>>>>>
>>>>>> We have talked about doing something like this at Mapbox but will
>>>>>>require a good deal of time, so it hasn't been pushed forward yet.
>>>>>>The
>>>>>>other issues that come to mind to me is that some of the libraries
>>>>>>that mapnik vector tile depends upon most likely need good custom
>>>>>>implementations if we wanted to make it more generic.
>>>>>>
>>>>>> The reason for this is that I would prefer for it to be a header
>>>>>>only
>>>>>>library with no dependencies. Adding more dependencies for GDAL seems
>>>>>>like it could be a mess.
>>>>>>
>>>>>> Blake Thompson
>>>>>>
>>>>>>> On Jan 31, 2016, at 10:10 AM, Stefan Keller <[hidden email]>
>>>>>>>wrote:
>>>>>>>
>>>>>>> Hi Petr, Blake, Even and everybody
>>>>>>>
>>>>>>> I'm (besides Petr) the other maintainer of OSM2VectorTiles.
>>>>>>> I'd already asked Even a similar question.
>>>>>>> I'm advising now an intern to implement such a MVT Vector Tiles
>>>>>>>reader
>>>>>>> client in Python if possible as part of a QGIS plugin.
>>>>>>> And I'm interested not to do redundant implementations.
>>>>>>> Are there any news on this matter?
>>>>>>>
>>>>>>> :Stefan
>>>>>>> Prof. at HSR and leader of Geometa Lab
>>>>>>>
>>>>>>> @Even: Regarding OGR MVT development can you send to me (directly)
>>>>>>>a
>>>>>>> contact name of the group which won the bidding?
>>>>>>>
>>>>>>>
>>>>>>> 2016-01-06 15:24 GMT+01:00 Petr Pridal
>>>>>>><[hidden email]>:
>>>>>>>> Hi everybody,
>>>>>>>>
>>>>>>>> thanks for the comments. Regarding the technical details:
>>>>>>>>
>>>>>>>>> Each Tile is its own discrete dataset therefore the hardest part
>>>>>>>>>would be
>>>>>>>>> merging of features into the original features across tiles. You
>>>>>>>>>could use
>>>>>>>>> some complicated merging technique, but that could be very
>>>>>>>>>expensive.
>>>>>>>>
>>>>>>>>
>>>>>>>> The driver could require "id" parameter, which specifies name of
>>>>>>>>an
>>>>>>>> attribute with unique identifier on the features shared across the
>>>>>>>>tiles.
>>>>>>>> Both MapBox vector tiles and OSM2VectorTiles tiles have an unique
>>>>>>>>non-zero
>>>>>>>> "osm_id" in place. The first version of driver could read only
>>>>>>>>features with
>>>>>>>> such ID. The identifier does not have to be always filled, but
>>>>>>>>often it is,
>>>>>>>> especially in deeper zoom levels derived purely from OpenStreetMap
>>>>>>>>- but
>>>>>>>> yeh, a completely general stitching would be much harder.
>>>>>>>>
>>>>>>>> The clipping + merging of shapes itself can be implemented with
>>>>>>>>underlaying
>>>>>>>> GEOS library functions.
>>>>>>>>
>>>>>>>> BTW It is possible to examine vector tiles data in pure JavaScript
>>>>>>>>X-Ray
>>>>>>>> viewer (no WebGL support on browser required) made in OpenLayers3:
>>>>>>>> http://klokantech.github.io/ol3-sandbox/vector/xray.html
>>>>>>>>
>>>>>>>> or with WebGL-enabled webbrowser with higher performance MapBox GL
>>>>>>>>JS
>>>>>>>> powered X-Ray:
>>>>>>>> http://klokantech.github.io/mapbox-gl-js-offline-example/xray.html
>>>>>>>>
>>>>>>>> A debug viewer for each vector tile (made in OL3) decoded in pure
>>>>>>>> JavaScript:
>>>>>>>> http://klokantech.github.io/ol3-sandbox/vector/tile-inspector.html
>>>>>>>>
>>>>>>>> The OGR driver could be also practical for the unsimplified /
>>>>>>>>ungeneralised
>>>>>>>> MBTiles with OSM data available at:
>>>>>>>>http://osmlab.github.io/osm-qa-tiles/
>>>>>>>>
>>>>>>>>>
>>>>>>>>> If you wanted to simply view vector tile data it could get
>>>>>>>>>confusing as
>>>>>>>>> well since most vector tiles need a buffer when they are used for
>>>>>>>>> visualization programs. This is important for things like
>>>>>>>>>labeling
>>>>>>>>>on maps
>>>>>>>>> etc. Therefore, you could get a large amount of overlapping data
>>>>>>>>>on tiles if
>>>>>>>>> you loaded them all in at once.
>>>>>>>>
>>>>>>>>
>>>>>>>> Clipping on border of each tile must be applied before stitching
>>>>>>>>the vector
>>>>>>>> features together  - so the buffer is removed. Buffer is there
>>>>>>>>only
>>>>>>>>for
>>>>>>>> visualisation and labeling.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> As far as writing a simple decoder -- the mapnik-vector-tile
>>>>>>>>>decoder isn't
>>>>>>>>> currently able to decode vector tiles into a form not supported
>>>>>>>>>by
>>>>>>>>>mapnik.
>>>>>>>>> Therefore, OGR would not be able to easily use this.
>>>>>>>>
>>>>>>>>
>>>>>>>> Correct. It can't be used directly (GDAL can't link against), but
>>>>>>>>the
>>>>>>>> relevant code can be extracted and reused directly. For basic
>>>>>>>>decoding
>>>>>>>> critical is the Protobuf library (which is already in GDAL) and
>>>>>>>>
>>>>>>>>https://github.com/mapbox/mapnik-vector-tile/blob/master/proto/vect
>>>>>>>>or
>>>>>>>>_tile.proto
>>>>>>>> + the code for converting pixel coordinates in features to
>>>>>>>>relevant
>>>>>>>>geo
>>>>>>>> coordinates and GEOS-based merging of features.
>>>>>>>>
>>>>>>>>> We plan to write a generalized encoder and decoder eventually so
>>>>>>>>>that any
>>>>>>>>> other library can plugin with out the mapnik requirement.
>>>>>>>>
>>>>>>>>
>>>>>>>> This would be amazing.
>>>>>>>>
>>>>>>>>> If you have any questions specifically about Mapnik Vector Tile
>>>>>>>>>or
>>>>>>>>>the
>>>>>>>>> Mapbox Vector Tile Specification feel free to ask me. I would
>>>>>>>>>note
>>>>>>>>>that you
>>>>>>>>> guys will want to update your vector tiles once the V2
>>>>>>>>>specification changes
>>>>>>>>> are into Mapnik-Vector-Tile.
>>>>>>>>
>>>>>>>>
>>>>>>>> Cool! Thanks. Our world tiles are generated via Docker containers
>>>>>>>>on a
>>>>>>>> cluster of computers - using Mapnik via Tilelive directly - so
>>>>>>>>upgrades on
>>>>>>>> Mapnik-Vector-Tile will propagate to the open-source rendering
>>>>>>>>software.
>>>>>>>> I expect V2 spec will mean also new MapBox Streets V7, right?
>>>>>>>>
>>>>>>>>> I see you mention MVT in MBTiles. Is it a recognized practice ? I
>>>>>>>>>don't
>>>>>>>>> see
>>>>>>>>> mention of that neither in the MVT spec (
>>>>>>>>> https://github.com/mapbox/vector-
>>>>>>>>> tile-spec/tree/master/2.0 ) nor the MBTiles one (
>>>>>>>>> https://github.com/mapbox/mbtiles-spec/blob/master/1.2/spec.md ),
>>>>>>>>>although
>>>>>>>>> I
>>>>>>>>> guess it is just a matter of storing a MVT blob in the tile_data
>>>>>>>>>column.
>>>>>>>>
>>>>>>>>
>>>>>>>> Yes. MapBox Studio Classic generates such MBTiles with Vector
>>>>>>>>Tiles
>>>>>>>>inside -
>>>>>>>> and Mapnik reads them directly when rasterising. Storing the
>>>>>>>>vector
>>>>>>>>tiles in
>>>>>>>> SQLite make sense - and it simplifies deployment, and you can then
>>>>>>>>easily
>>>>>>>> have complete world on each node of a cluster of tileservers. If
>>>>>>>>you don't
>>>>>>>> want to upgrade your base map too often, this approach is pretty
>>>>>>>>efficient
>>>>>>>> and removes the centralised database as typical bottleneck.
>>>>>>>>
>>>>>>>> I did some tests on vector tiles vs raster in MBTiles and
>>>>>>>>documented these
>>>>>>>> in README.md at and
>>>>>>>>https://github.com/klokantech/vector-tiles-sample before
>>>>>>>> we started to work on OSM2VectorTiles project.
>>>>>>>> There are also sample data and offline MapBox GL JS viewer in that
>>>>>>>>repo.
>>>>>>>> Vector tiles can be hosted also "unpacked", just as files in
>>>>>>>>folders on a
>>>>>>>> standard web server - exactly like GDAL2Tiles or MapTiler.com does
>>>>>>>>by
>>>>>>>> default for raster tiles.
>>>>>>>>
>>>>>>>> If OGR driver is implemented, the primary source should be
>>>>>>>>probably
>>>>>>>>reading
>>>>>>>> from an URL via curl.
>>>>>>>>
>>>>>>>> Reading PBFs from an MBTiles (or another SQLite) is just the most
>>>>>>>>practical
>>>>>>>> portable alternative to it.
>>>>>>>>
>>>>>>>> Technically people may store the tiles in different ways - for
>>>>>>>>example
>>>>>>>> Wikimedia guys (http://maps.wikimedia.org/ and
>>>>>>>> https://github.com/kartotherian/) use Cassandra for tile storage,
>>>>>>>>I
>>>>>>>>guess
>>>>>>>> MapBox internally is also on a different storage for production
>>>>>>>>servers ...
>>>>>>>> for transfers and streaming tilesets there is the  - but tiles are
>>>>>>>>ALWAYS
>>>>>>>> exposed via HTTP.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Petr
>>>>>>>> --
>>>>>>>> Petr Pridal, Ph.D.
>>>>>>>> CEO
>>>>>>>>
>>>>>>>> Klokan Technologies GmbH
>>>>>>>> Hofnerstrasse 98, 6314 Unterageri, Switzerland
>>>>>>>> Tel: +41 (0)41 511 26 12
>>>>>>>> Email: [hidden email]
>>>>>>>> Web: http://www.klokantech.com/
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> gdal-dev mailing list
>>>>>>>> [hidden email]
>>>>>>>> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>>>_______________________________________________
>>>gdal-dev mailing list
>>>[hidden email]
>>>http://lists.osgeo.org/mailman/listinfo/gdal-dev
>>

_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Vector Tiles in OGR

flippmoke
In reply to this post by Stefan Keller
Benjamin Stadin,

As an author of the specification, I have to disagree with you on the support for other projections. There is no limitation on vector tiles being required to being in mercator. The specification in fact mentions that other projections may be used: 

https://github.com/mapbox/vector-tile-spec/tree/master/2.1#3-projection-and-bounds

While it is not accessible via the node-mapnik bindings, we do in fact support encoding as different projections in mapnik-vector-tile. We have implemented the decoder so that it could accept other projections, but have not added a mapnik datasource that would support them, but if there is demand for such a thing we could do it.

Your comments about the entire stack of Mapbox supporting mostly mercator could be very accurate, but this is not at all related to the Vector Tile Specification. We purposefully attempt to make our tools as projection-less as possible and hope that others find awesome ways to build upon the standard.

"But what data is shown at which zoom level is defined by the styling" -- This is not true because you could make your data not even provide data at certain zoom levels. At Mapbox we do this quite often, street data is never included at very low zoom levels. Therefore, the style isn't the only control here. In fact, I would encourage you to view styling and rendering of vector tiles as an entirely different topic. 

"There is enough interest from our side that we¹d be interested in contributing (maybe monetary or with code, or both) to define and implement a better general purpose tile format." -- Any issues you might have with the Mapbox Vector Tile Specification or any ideas you might have with the format are very much welcome. We want to make sure the spec migrates for community needs. Please see https://github.com/mapbox/vector-tile-spec/blob/master/CONTRIBUTING.md for how you might help!

Finally, I would like to stress that while Mapbox Vector Tiles were designed to be for rendering, it was well known during their design that they could be used for other applications. This was especially true when I was writing 2.0 of the specification as I tried to limit what would be considered even the most common uses of vector tiles.

Thanks,

Blake Thompson 

On Mon, Feb 1, 2016 at 4:27 PM, Stefan Keller <[hidden email]> wrote:
Hi Ben

Thanks for explaining - Petr and I are the maintainers of OSM2Vectortiles.
I like projections (and also GeoPackage :-) ).
But reprojection is solved here, since OGR/QGIS can transform on the fly.
This thread is about Petr's question about a reader "for mapbox-like
vector tiles stored in MBTiles or downloaded from a url...".

So I'd like to come back to the issue:
1. of a generic MVT decoder/enoder library (C++ or Python) => Blake?
2. and of a "feature provider" in an interactive environment (like
QGIS) or a file/stream converter (like OGR) => anybody?

:Stefan



2016-02-02 0:04 GMT+01:00 Stadin, Benjamin
<[hidden email]>:
> Hi Stefan,
>
> A vector tile may contain data for several zoom levels. But what data is
> shown at which zoom level is defined by the styling, not by the data
> itself. It is handled on client side, by the web (or native) tile
> renderer.
>
> The path is the same as with mercator tiles (zoomLevel/x/y):
> https://example.com/17/65535/43602.mvt
>
>
> The tile for zoom level 17 may contain features of class „building“, and
> the style definition may contain a rule that filters type „building“ for
> zoomLevels <= 18.
>
> ~Ben
>
> Am 01.02.16, 20:16 schrieb "gdal-dev on behalf of Stefan Keller" unter
> <[hidden email] on behalf of [hidden email]>:
>
>>That's good news!
>>We'll be happy to add Python bindings to such a generic C++ library
>>for Mapbox vector tiles.
>>We have still some time (say few weeks?) left here to decide to
>>contribute either to these bindings or to Mapzen's pure Python lib.
>>
>>One reason why I'm insisting on Python (bindings) - besides easier
>>testing with QGIS - is the following:
>>AFAIK with vector tiles we have to deal with quite a new kind of
>>feature provider:
>>One which contains "zoom levels" and one which "suddenly" changes
>>feature class at different zoom levels.
>>If a client like QGIS consumes vector tiles from OGR in a one-shot
>>call, how should the client deal with such "zoom level" dependent
>>issues?
>>So my thinking is, to consume only the most appropriate zoom level,
>>probably only within current map canvas...
>>
>>:Stefan
>>
>>2016-02-01 18:27 GMT+01:00 Flippmoke <[hidden email]>:
>>>
>>> Stefan,
>>>
>>> I am not aware of any libraries besides mapnik vector tile that have
>>>been attempting to implement v2 specification. I think mapzen's
>>>implementation might be best? We have been wanting to add vector tiles
>>>to the python mapnik bindings for quite some time but we don't have the
>>>time to do it right now.
>>>
>>> I will try to think this week about how we could make a generic C++
>>>library quickly for Mapbox vector tiles.
>>>
>>> Thanks,
>>>
>>> Blake Thompson
>>>
>>>> On Feb 1, 2016, at 1:06 AM, Stefan Keller <[hidden email]> wrote:
>>>>
>>>> Hi Blake
>>>>
>>>> It's clear to me that a good generic c++ library for encoding and
>>>> decoding vector tiles is needed.
>>>> On the other hand, I'd like to experiment with vector tiles and
>>>> related QGIS parallel loading issues for which Python is more
>>>> convenient.
>>>>
>>>> It seems that Mapzen's [1] Python implementation is further along.
>>>> Or do you know if Jesse (or somebody) is still working on Mapbox's
>>>>variant [2] ?
>>>>
>>>> :Stefan
>>>>
>>>> [1] https://github.com/mapzen/mapbox-vector-tile
>>>> [2] https://github.com/mapbox/vector-tile-py
>>>>
>>>>
>>>> 2016-01-31 23:22 GMT+01:00 Flippmoke <[hidden email]>:
>>>>> All,
>>>>>
>>>>> The first thing that is going to be required likely is a good generic
>>>>>c++ library for encoding and decoding vector tiles IMO.
>>>>>
>>>>> I have spent a lot of time working on developing the mapnik vector
>>>>>tile library to be a solid implementation, especially around all the
>>>>>work done for 2.0 of Mapbox vector tile specification. I would love to
>>>>>eventually help make this a generalized implementation so that it is
>>>>>portable to a library like GDAL.
>>>>>
>>>>> We have talked about doing something like this at Mapbox but will
>>>>>require a good deal of time, so it hasn't been pushed forward yet. The
>>>>>other issues that come to mind to me is that some of the libraries
>>>>>that mapnik vector tile depends upon most likely need good custom
>>>>>implementations if we wanted to make it more generic.
>>>>>
>>>>> The reason for this is that I would prefer for it to be a header only
>>>>>library with no dependencies. Adding more dependencies for GDAL seems
>>>>>like it could be a mess.
>>>>>
>>>>> Blake Thompson
>>>>>
>>>>>> On Jan 31, 2016, at 10:10 AM, Stefan Keller <[hidden email]>
>>>>>>wrote:
>>>>>>
>>>>>> Hi Petr, Blake, Even and everybody
>>>>>>
>>>>>> I'm (besides Petr) the other maintainer of OSM2VectorTiles.
>>>>>> I'd already asked Even a similar question.
>>>>>> I'm advising now an intern to implement such a MVT Vector Tiles
>>>>>>reader
>>>>>> client in Python if possible as part of a QGIS plugin.
>>>>>> And I'm interested not to do redundant implementations.
>>>>>> Are there any news on this matter?
>>>>>>
>>>>>> :Stefan
>>>>>> Prof. at HSR and leader of Geometa Lab
>>>>>>
>>>>>> @Even: Regarding OGR MVT development can you send to me (directly) a
>>>>>> contact name of the group which won the bidding?
>>>>>>
>>>>>>
>>>>>> 2016-01-06 15:24 GMT+01:00 Petr Pridal <[hidden email]>:
>>>>>>> Hi everybody,
>>>>>>>
>>>>>>> thanks for the comments. Regarding the technical details:
>>>>>>>
>>>>>>>> Each Tile is its own discrete dataset therefore the hardest part
>>>>>>>>would be
>>>>>>>> merging of features into the original features across tiles. You
>>>>>>>>could use
>>>>>>>> some complicated merging technique, but that could be very
>>>>>>>>expensive.
>>>>>>>
>>>>>>>
>>>>>>> The driver could require "id" parameter, which specifies name of an
>>>>>>> attribute with unique identifier on the features shared across the
>>>>>>>tiles.
>>>>>>> Both MapBox vector tiles and OSM2VectorTiles tiles have an unique
>>>>>>>non-zero
>>>>>>> "osm_id" in place. The first version of driver could read only
>>>>>>>features with
>>>>>>> such ID. The identifier does not have to be always filled, but
>>>>>>>often it is,
>>>>>>> especially in deeper zoom levels derived purely from OpenStreetMap
>>>>>>>- but
>>>>>>> yeh, a completely general stitching would be much harder.
>>>>>>>
>>>>>>> The clipping + merging of shapes itself can be implemented with
>>>>>>>underlaying
>>>>>>> GEOS library functions.
>>>>>>>
>>>>>>> BTW It is possible to examine vector tiles data in pure JavaScript
>>>>>>>X-Ray
>>>>>>> viewer (no WebGL support on browser required) made in OpenLayers3:
>>>>>>> http://klokantech.github.io/ol3-sandbox/vector/xray.html
>>>>>>>
>>>>>>> or with WebGL-enabled webbrowser with higher performance MapBox GL
>>>>>>>JS
>>>>>>> powered X-Ray:
>>>>>>> http://klokantech.github.io/mapbox-gl-js-offline-example/xray.html
>>>>>>>
>>>>>>> A debug viewer for each vector tile (made in OL3) decoded in pure
>>>>>>> JavaScript:
>>>>>>> http://klokantech.github.io/ol3-sandbox/vector/tile-inspector.html
>>>>>>>
>>>>>>> The OGR driver could be also practical for the unsimplified /
>>>>>>>ungeneralised
>>>>>>> MBTiles with OSM data available at:
>>>>>>>http://osmlab.github.io/osm-qa-tiles/
>>>>>>>
>>>>>>>>
>>>>>>>> If you wanted to simply view vector tile data it could get
>>>>>>>>confusing as
>>>>>>>> well since most vector tiles need a buffer when they are used for
>>>>>>>> visualization programs. This is important for things like labeling
>>>>>>>>on maps
>>>>>>>> etc. Therefore, you could get a large amount of overlapping data
>>>>>>>>on tiles if
>>>>>>>> you loaded them all in at once.
>>>>>>>
>>>>>>>
>>>>>>> Clipping on border of each tile must be applied before stitching
>>>>>>>the vector
>>>>>>> features together  - so the buffer is removed. Buffer is there only
>>>>>>>for
>>>>>>> visualisation and labeling.
>>>>>>>
>>>>>>>>
>>>>>>>> As far as writing a simple decoder -- the mapnik-vector-tile
>>>>>>>>decoder isn't
>>>>>>>> currently able to decode vector tiles into a form not supported by
>>>>>>>>mapnik.
>>>>>>>> Therefore, OGR would not be able to easily use this.
>>>>>>>
>>>>>>>
>>>>>>> Correct. It can't be used directly (GDAL can't link against), but
>>>>>>>the
>>>>>>> relevant code can be extracted and reused directly. For basic
>>>>>>>decoding
>>>>>>> critical is the Protobuf library (which is already in GDAL) and
>>>>>>>
>>>>>>>https://github.com/mapbox/mapnik-vector-tile/blob/master/proto/vector
>>>>>>>_tile.proto
>>>>>>> + the code for converting pixel coordinates in features to relevant
>>>>>>>geo
>>>>>>> coordinates and GEOS-based merging of features.
>>>>>>>
>>>>>>>> We plan to write a generalized encoder and decoder eventually so
>>>>>>>>that any
>>>>>>>> other library can plugin with out the mapnik requirement.
>>>>>>>
>>>>>>>
>>>>>>> This would be amazing.
>>>>>>>
>>>>>>>> If you have any questions specifically about Mapnik Vector Tile or
>>>>>>>>the
>>>>>>>> Mapbox Vector Tile Specification feel free to ask me. I would note
>>>>>>>>that you
>>>>>>>> guys will want to update your vector tiles once the V2
>>>>>>>>specification changes
>>>>>>>> are into Mapnik-Vector-Tile.
>>>>>>>
>>>>>>>
>>>>>>> Cool! Thanks. Our world tiles are generated via Docker containers
>>>>>>>on a
>>>>>>> cluster of computers - using Mapnik via Tilelive directly - so
>>>>>>>upgrades on
>>>>>>> Mapnik-Vector-Tile will propagate to the open-source rendering
>>>>>>>software.
>>>>>>> I expect V2 spec will mean also new MapBox Streets V7, right?
>>>>>>>
>>>>>>>> I see you mention MVT in MBTiles. Is it a recognized practice ? I
>>>>>>>>don't
>>>>>>>> see
>>>>>>>> mention of that neither in the MVT spec (
>>>>>>>> https://github.com/mapbox/vector-
>>>>>>>> tile-spec/tree/master/2.0 ) nor the MBTiles one (
>>>>>>>> https://github.com/mapbox/mbtiles-spec/blob/master/1.2/spec.md ),
>>>>>>>>although
>>>>>>>> I
>>>>>>>> guess it is just a matter of storing a MVT blob in the tile_data
>>>>>>>>column.
>>>>>>>
>>>>>>>
>>>>>>> Yes. MapBox Studio Classic generates such MBTiles with Vector Tiles
>>>>>>>inside -
>>>>>>> and Mapnik reads them directly when rasterising. Storing the vector
>>>>>>>tiles in
>>>>>>> SQLite make sense - and it simplifies deployment, and you can then
>>>>>>>easily
>>>>>>> have complete world on each node of a cluster of tileservers. If
>>>>>>>you don't
>>>>>>> want to upgrade your base map too often, this approach is pretty
>>>>>>>efficient
>>>>>>> and removes the centralised database as typical bottleneck.
>>>>>>>
>>>>>>> I did some tests on vector tiles vs raster in MBTiles and
>>>>>>>documented these
>>>>>>> in README.md at and
>>>>>>>https://github.com/klokantech/vector-tiles-sample before
>>>>>>> we started to work on OSM2VectorTiles project.
>>>>>>> There are also sample data and offline MapBox GL JS viewer in that
>>>>>>>repo.
>>>>>>> Vector tiles can be hosted also "unpacked", just as files in
>>>>>>>folders on a
>>>>>>> standard web server - exactly like GDAL2Tiles or MapTiler.com does
>>>>>>>by
>>>>>>> default for raster tiles.
>>>>>>>
>>>>>>> If OGR driver is implemented, the primary source should be probably
>>>>>>>reading
>>>>>>> from an URL via curl.
>>>>>>>
>>>>>>> Reading PBFs from an MBTiles (or another SQLite) is just the most
>>>>>>>practical
>>>>>>> portable alternative to it.
>>>>>>>
>>>>>>> Technically people may store the tiles in different ways - for
>>>>>>>example
>>>>>>> Wikimedia guys (http://maps.wikimedia.org/ and
>>>>>>> https://github.com/kartotherian/) use Cassandra for tile storage, I
>>>>>>>guess
>>>>>>> MapBox internally is also on a different storage for production
>>>>>>>servers ...
>>>>>>> for transfers and streaming tilesets there is the  - but tiles are
>>>>>>>ALWAYS
>>>>>>> exposed via HTTP.
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Petr
>>>>>>> --
>>>>>>> Petr Pridal, Ph.D.
>>>>>>> CEO
>>>>>>>
>>>>>>> Klokan Technologies GmbH
>>>>>>> Hofnerstrasse 98, 6314 Unterageri, Switzerland
>>>>>>> Tel: <a href="tel:%2B41%20%280%2941%20511%2026%2012" value="+41415112612">+41 (0)41 511 26 12
>>>>>>> Email: [hidden email]
>>>>>>> Web: http://www.klokantech.com/
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> gdal-dev mailing list
>>>>>>> [hidden email]
>>>>>>> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>>_______________________________________________
>>gdal-dev mailing list
>>[hidden email]
>>http://lists.osgeo.org/mailman/listinfo/gdal-dev
>


_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Vector Tiles in OGR

Stefan Keller
Blake,

Now the vector tiles spec. and the Mapbox approach became clearer to me.

I'm still pondering about realizing a vector tiles reader (especially
for QGIS) and for vector tiles uses cases like 1. base maps and 2.
spatial analysis:

Regarding clipped linestrings and polygons arriving at reader, there
seems to exist no hint (clipped) nor id mentioned in the spec. to help
the reader merging them?

While there's overzoom, what's about leaving out data in intermediate
zoom levels to save space? Does the spec. regulate that and who's
responsible for handling this: server or client?

Finally and back to my implementation concerns: How should an
interactive client deal occurring/disappearing feature classes in a
single "layer"? Is it forced to read all features on all zoom levels
in order to announce all feature classes?

:Stefan


2016-02-02 1:03 GMT+01:00 Blake Thompson <[hidden email]>:

> Benjamin Stadin,
>
> As an author of the specification, I have to disagree with you on the
> support for other projections. There is no limitation on vector tiles being
> required to being in mercator. The specification in fact mentions that other
> projections may be used:
>
> https://github.com/mapbox/vector-tile-spec/tree/master/2.1#3-projection-and-bounds
>
> While it is not accessible via the node-mapnik bindings, we do in fact
> support encoding as different projections in mapnik-vector-tile. We have
> implemented the decoder so that it could accept other projections, but have
> not added a mapnik datasource that would support them, but if there is
> demand for such a thing we could do it.
>
> Your comments about the entire stack of Mapbox supporting mostly mercator
> could be very accurate, but this is not at all related to the Vector Tile
> Specification. We purposefully attempt to make our tools as projection-less
> as possible and hope that others find awesome ways to build upon the
> standard.
>
> "But what data is shown at which zoom level is defined by the styling" --
> This is not true because you could make your data not even provide data at
> certain zoom levels. At Mapbox we do this quite often, street data is never
> included at very low zoom levels. Therefore, the style isn't the only
> control here. In fact, I would encourage you to view styling and rendering
> of vector tiles as an entirely different topic.
>
> "There is enough interest from our side that we¹d be interested in
> contributing (maybe monetary or with code, or both) to define and implement
> a better general purpose tile format." -- Any issues you might have with the
> Mapbox Vector Tile Specification or any ideas you might have with the format
> are very much welcome. We want to make sure the spec migrates for community
> needs. Please see
> https://github.com/mapbox/vector-tile-spec/blob/master/CONTRIBUTING.md for
> how you might help!
>
> Finally, I would like to stress that while Mapbox Vector Tiles were designed
> to be for rendering, it was well known during their design that they could
> be used for other applications. This was especially true when I was writing
> 2.0 of the specification as I tried to limit what would be considered even
> the most common uses of vector tiles.
>
> Thanks,
>
> Blake Thompson
>
> On Mon, Feb 1, 2016 at 4:27 PM, Stefan Keller <[hidden email]> wrote:
>>
>> Hi Ben
>>
>> Thanks for explaining - Petr and I are the maintainers of OSM2Vectortiles.
>> I like projections (and also GeoPackage :-) ).
>> But reprojection is solved here, since OGR/QGIS can transform on the fly.
>> This thread is about Petr's question about a reader "for mapbox-like
>> vector tiles stored in MBTiles or downloaded from a url...".
>>
>> So I'd like to come back to the issue:
>> 1. of a generic MVT decoder/enoder library (C++ or Python) => Blake?
>> 2. and of a "feature provider" in an interactive environment (like
>> QGIS) or a file/stream converter (like OGR) => anybody?
>>
>> :Stefan
>>
>>
>>
>> 2016-02-02 0:04 GMT+01:00 Stadin, Benjamin
>> <[hidden email]>:
>> > Hi Stefan,
>> >
>> > A vector tile may contain data for several zoom levels. But what data is
>> > shown at which zoom level is defined by the styling, not by the data
>> > itself. It is handled on client side, by the web (or native) tile
>> > renderer.
>> >
>> > The path is the same as with mercator tiles (zoomLevel/x/y):
>> > https://example.com/17/65535/43602.mvt
>> >
>> >
>> > The tile for zoom level 17 may contain features of class „building“, and
>> > the style definition may contain a rule that filters type „building“ for
>> > zoomLevels <= 18.
>> >
>> > ~Ben
>> >
>> > Am 01.02.16, 20:16 schrieb "gdal-dev on behalf of Stefan Keller" unter
>> > <[hidden email] on behalf of [hidden email]>:
>> >
>> >>That's good news!
>> >>We'll be happy to add Python bindings to such a generic C++ library
>> >>for Mapbox vector tiles.
>> >>We have still some time (say few weeks?) left here to decide to
>> >>contribute either to these bindings or to Mapzen's pure Python lib.
>> >>
>> >>One reason why I'm insisting on Python (bindings) - besides easier
>> >>testing with QGIS - is the following:
>> >>AFAIK with vector tiles we have to deal with quite a new kind of
>> >>feature provider:
>> >>One which contains "zoom levels" and one which "suddenly" changes
>> >>feature class at different zoom levels.
>> >>If a client like QGIS consumes vector tiles from OGR in a one-shot
>> >>call, how should the client deal with such "zoom level" dependent
>> >>issues?
>> >>So my thinking is, to consume only the most appropriate zoom level,
>> >>probably only within current map canvas...
>> >>
>> >>:Stefan
>> >>
>> >>2016-02-01 18:27 GMT+01:00 Flippmoke <[hidden email]>:
>> >>>
>> >>> Stefan,
>> >>>
>> >>> I am not aware of any libraries besides mapnik vector tile that have
>> >>>been attempting to implement v2 specification. I think mapzen's
>> >>>implementation might be best? We have been wanting to add vector tiles
>> >>>to the python mapnik bindings for quite some time but we don't have the
>> >>>time to do it right now.
>> >>>
>> >>> I will try to think this week about how we could make a generic C++
>> >>>library quickly for Mapbox vector tiles.
>> >>>
>> >>> Thanks,
>> >>>
>> >>> Blake Thompson
>> >>>
>> >>>> On Feb 1, 2016, at 1:06 AM, Stefan Keller <[hidden email]> wrote:
>> >>>>
>> >>>> Hi Blake
>> >>>>
>> >>>> It's clear to me that a good generic c++ library for encoding and
>> >>>> decoding vector tiles is needed.
>> >>>> On the other hand, I'd like to experiment with vector tiles and
>> >>>> related QGIS parallel loading issues for which Python is more
>> >>>> convenient.
>> >>>>
>> >>>> It seems that Mapzen's [1] Python implementation is further along.
>> >>>> Or do you know if Jesse (or somebody) is still working on Mapbox's
>> >>>>variant [2] ?
>> >>>>
>> >>>> :Stefan
>> >>>>
>> >>>> [1] https://github.com/mapzen/mapbox-vector-tile
>> >>>> [2] https://github.com/mapbox/vector-tile-py
>> >>>>
>> >>>>
>> >>>> 2016-01-31 23:22 GMT+01:00 Flippmoke <[hidden email]>:
>> >>>>> All,
>> >>>>>
>> >>>>> The first thing that is going to be required likely is a good
>> >>>>> generic
>> >>>>>c++ library for encoding and decoding vector tiles IMO.
>> >>>>>
>> >>>>> I have spent a lot of time working on developing the mapnik vector
>> >>>>>tile library to be a solid implementation, especially around all the
>> >>>>>work done for 2.0 of Mapbox vector tile specification. I would love
>> >>>>> to
>> >>>>>eventually help make this a generalized implementation so that it is
>> >>>>>portable to a library like GDAL.
>> >>>>>
>> >>>>> We have talked about doing something like this at Mapbox but will
>> >>>>>require a good deal of time, so it hasn't been pushed forward yet.
>> >>>>> The
>> >>>>>other issues that come to mind to me is that some of the libraries
>> >>>>>that mapnik vector tile depends upon most likely need good custom
>> >>>>>implementations if we wanted to make it more generic.
>> >>>>>
>> >>>>> The reason for this is that I would prefer for it to be a header
>> >>>>> only
>> >>>>>library with no dependencies. Adding more dependencies for GDAL seems
>> >>>>>like it could be a mess.
>> >>>>>
>> >>>>> Blake Thompson
>> >>>>>
>> >>>>>> On Jan 31, 2016, at 10:10 AM, Stefan Keller <[hidden email]>
>> >>>>>>wrote:
>> >>>>>>
>> >>>>>> Hi Petr, Blake, Even and everybody
>> >>>>>>
>> >>>>>> I'm (besides Petr) the other maintainer of OSM2VectorTiles.
>> >>>>>> I'd already asked Even a similar question.
>> >>>>>> I'm advising now an intern to implement such a MVT Vector Tiles
>> >>>>>>reader
>> >>>>>> client in Python if possible as part of a QGIS plugin.
>> >>>>>> And I'm interested not to do redundant implementations.
>> >>>>>> Are there any news on this matter?
>> >>>>>>
>> >>>>>> :Stefan
>> >>>>>> Prof. at HSR and leader of Geometa Lab
>> >>>>>>
>> >>>>>> @Even: Regarding OGR MVT development can you send to me (directly)
>> >>>>>> a
>> >>>>>> contact name of the group which won the bidding?
>> >>>>>>
>> >>>>>>
>> >>>>>> 2016-01-06 15:24 GMT+01:00 Petr Pridal
>> >>>>>> <[hidden email]>:
>> >>>>>>> Hi everybody,
>> >>>>>>>
>> >>>>>>> thanks for the comments. Regarding the technical details:
>> >>>>>>>
>> >>>>>>>> Each Tile is its own discrete dataset therefore the hardest part
>> >>>>>>>>would be
>> >>>>>>>> merging of features into the original features across tiles. You
>> >>>>>>>>could use
>> >>>>>>>> some complicated merging technique, but that could be very
>> >>>>>>>>expensive.
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> The driver could require "id" parameter, which specifies name of
>> >>>>>>> an
>> >>>>>>> attribute with unique identifier on the features shared across the
>> >>>>>>>tiles.
>> >>>>>>> Both MapBox vector tiles and OSM2VectorTiles tiles have an unique
>> >>>>>>>non-zero
>> >>>>>>> "osm_id" in place. The first version of driver could read only
>> >>>>>>>features with
>> >>>>>>> such ID. The identifier does not have to be always filled, but
>> >>>>>>>often it is,
>> >>>>>>> especially in deeper zoom levels derived purely from OpenStreetMap
>> >>>>>>>- but
>> >>>>>>> yeh, a completely general stitching would be much harder.
>> >>>>>>>
>> >>>>>>> The clipping + merging of shapes itself can be implemented with
>> >>>>>>>underlaying
>> >>>>>>> GEOS library functions.
>> >>>>>>>
>> >>>>>>> BTW It is possible to examine vector tiles data in pure JavaScript
>> >>>>>>>X-Ray
>> >>>>>>> viewer (no WebGL support on browser required) made in OpenLayers3:
>> >>>>>>> http://klokantech.github.io/ol3-sandbox/vector/xray.html
>> >>>>>>>
>> >>>>>>> or with WebGL-enabled webbrowser with higher performance MapBox GL
>> >>>>>>>JS
>> >>>>>>> powered X-Ray:
>> >>>>>>> http://klokantech.github.io/mapbox-gl-js-offline-example/xray.html
>> >>>>>>>
>> >>>>>>> A debug viewer for each vector tile (made in OL3) decoded in pure
>> >>>>>>> JavaScript:
>> >>>>>>> http://klokantech.github.io/ol3-sandbox/vector/tile-inspector.html
>> >>>>>>>
>> >>>>>>> The OGR driver could be also practical for the unsimplified /
>> >>>>>>>ungeneralised
>> >>>>>>> MBTiles with OSM data available at:
>> >>>>>>>http://osmlab.github.io/osm-qa-tiles/
>> >>>>>>>
>> >>>>>>>>
>> >>>>>>>> If you wanted to simply view vector tile data it could get
>> >>>>>>>>confusing as
>> >>>>>>>> well since most vector tiles need a buffer when they are used for
>> >>>>>>>> visualization programs. This is important for things like
>> >>>>>>>> labeling
>> >>>>>>>>on maps
>> >>>>>>>> etc. Therefore, you could get a large amount of overlapping data
>> >>>>>>>>on tiles if
>> >>>>>>>> you loaded them all in at once.
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> Clipping on border of each tile must be applied before stitching
>> >>>>>>>the vector
>> >>>>>>> features together  - so the buffer is removed. Buffer is there
>> >>>>>>> only
>> >>>>>>>for
>> >>>>>>> visualisation and labeling.
>> >>>>>>>
>> >>>>>>>>
>> >>>>>>>> As far as writing a simple decoder -- the mapnik-vector-tile
>> >>>>>>>>decoder isn't
>> >>>>>>>> currently able to decode vector tiles into a form not supported
>> >>>>>>>> by
>> >>>>>>>>mapnik.
>> >>>>>>>> Therefore, OGR would not be able to easily use this.
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> Correct. It can't be used directly (GDAL can't link against), but
>> >>>>>>>the
>> >>>>>>> relevant code can be extracted and reused directly. For basic
>> >>>>>>>decoding
>> >>>>>>> critical is the Protobuf library (which is already in GDAL) and
>> >>>>>>>
>>
>> >>>>>>> >>>>>>>https://github.com/mapbox/mapnik-vector-tile/blob/master/proto/vector
>> >>>>>>>_tile.proto
>> >>>>>>> + the code for converting pixel coordinates in features to
>> >>>>>>> relevant
>> >>>>>>>geo
>> >>>>>>> coordinates and GEOS-based merging of features.
>> >>>>>>>
>> >>>>>>>> We plan to write a generalized encoder and decoder eventually so
>> >>>>>>>>that any
>> >>>>>>>> other library can plugin with out the mapnik requirement.
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> This would be amazing.
>> >>>>>>>
>> >>>>>>>> If you have any questions specifically about Mapnik Vector Tile
>> >>>>>>>> or
>> >>>>>>>>the
>> >>>>>>>> Mapbox Vector Tile Specification feel free to ask me. I would
>> >>>>>>>> note
>> >>>>>>>>that you
>> >>>>>>>> guys will want to update your vector tiles once the V2
>> >>>>>>>>specification changes
>> >>>>>>>> are into Mapnik-Vector-Tile.
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> Cool! Thanks. Our world tiles are generated via Docker containers
>> >>>>>>>on a
>> >>>>>>> cluster of computers - using Mapnik via Tilelive directly - so
>> >>>>>>>upgrades on
>> >>>>>>> Mapnik-Vector-Tile will propagate to the open-source rendering
>> >>>>>>>software.
>> >>>>>>> I expect V2 spec will mean also new MapBox Streets V7, right?
>> >>>>>>>
>> >>>>>>>> I see you mention MVT in MBTiles. Is it a recognized practice ? I
>> >>>>>>>>don't
>> >>>>>>>> see
>> >>>>>>>> mention of that neither in the MVT spec (
>> >>>>>>>> https://github.com/mapbox/vector-
>> >>>>>>>> tile-spec/tree/master/2.0 ) nor the MBTiles one (
>> >>>>>>>> https://github.com/mapbox/mbtiles-spec/blob/master/1.2/spec.md ),
>> >>>>>>>>although
>> >>>>>>>> I
>> >>>>>>>> guess it is just a matter of storing a MVT blob in the tile_data
>> >>>>>>>>column.
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> Yes. MapBox Studio Classic generates such MBTiles with Vector
>> >>>>>>> Tiles
>> >>>>>>>inside -
>> >>>>>>> and Mapnik reads them directly when rasterising. Storing the
>> >>>>>>> vector
>> >>>>>>>tiles in
>> >>>>>>> SQLite make sense - and it simplifies deployment, and you can then
>> >>>>>>>easily
>> >>>>>>> have complete world on each node of a cluster of tileservers. If
>> >>>>>>>you don't
>> >>>>>>> want to upgrade your base map too often, this approach is pretty
>> >>>>>>>efficient
>> >>>>>>> and removes the centralised database as typical bottleneck.
>> >>>>>>>
>> >>>>>>> I did some tests on vector tiles vs raster in MBTiles and
>> >>>>>>>documented these
>> >>>>>>> in README.md at and
>> >>>>>>>https://github.com/klokantech/vector-tiles-sample before
>> >>>>>>> we started to work on OSM2VectorTiles project.
>> >>>>>>> There are also sample data and offline MapBox GL JS viewer in that
>> >>>>>>>repo.
>> >>>>>>> Vector tiles can be hosted also "unpacked", just as files in
>> >>>>>>>folders on a
>> >>>>>>> standard web server - exactly like GDAL2Tiles or MapTiler.com does
>> >>>>>>>by
>> >>>>>>> default for raster tiles.
>> >>>>>>>
>> >>>>>>> If OGR driver is implemented, the primary source should be
>> >>>>>>> probably
>> >>>>>>>reading
>> >>>>>>> from an URL via curl.
>> >>>>>>>
>> >>>>>>> Reading PBFs from an MBTiles (or another SQLite) is just the most
>> >>>>>>>practical
>> >>>>>>> portable alternative to it.
>> >>>>>>>
>> >>>>>>> Technically people may store the tiles in different ways - for
>> >>>>>>>example
>> >>>>>>> Wikimedia guys (http://maps.wikimedia.org/ and
>> >>>>>>> https://github.com/kartotherian/) use Cassandra for tile storage,
>> >>>>>>> I
>> >>>>>>>guess
>> >>>>>>> MapBox internally is also on a different storage for production
>> >>>>>>>servers ...
>> >>>>>>> for transfers and streaming tilesets there is the  - but tiles are
>> >>>>>>>ALWAYS
>> >>>>>>> exposed via HTTP.
>> >>>>>>>
>> >>>>>>> Regards,
>> >>>>>>>
>> >>>>>>> Petr
>> >>>>>>> --
>> >>>>>>> Petr Pridal, Ph.D.
>> >>>>>>> CEO
>> >>>>>>>
>> >>>>>>> Klokan Technologies GmbH
>> >>>>>>> Hofnerstrasse 98, 6314 Unterageri, Switzerland
>> >>>>>>> Tel: +41 (0)41 511 26 12
>> >>>>>>> Email: [hidden email]
>> >>>>>>> Web: http://www.klokantech.com/
>> >>>>>>>
>> >>>>>>> _______________________________________________
>> >>>>>>> gdal-dev mailing list
>> >>>>>>> [hidden email]
>> >>>>>>> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>> >>_______________________________________________
>> >>gdal-dev mailing list
>> >>[hidden email]
>> >>http://lists.osgeo.org/mailman/listinfo/gdal-dev
>> >
>
>
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
12
Loading...