RPC and MVT Layers

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

RPC and MVT Layers

Alexander W. Rolek
Greetings OSGeo - 

For the last few months I have been working on a MVT server written in Go called Tegola (http://tegola.io). One of the design goals for Tegola has been to support multiple data providers (i.e. PostGIS, Geopackage, etc.). As anyone who works with geospatial data knows, a good chunk of time is dedicated to the extract, transform and load (ETL) pipeline. For many situations it's unnecessary to replicate a data layer from a data authority so you can have it in your own database. This unnecessary data duplication got us thinking about the possibility of a RPC data provider that would return MVT Layers instead of MVT Tiles.

The MVT spec basically consists of a Tile, which is made up of Layers, which are made up of Features. A typical request to a MVT server includes Z, X and Y values (Slippy map tilenames) with an expected response being a protocol buffer encoded MVT tile. At a high level, a RPC data provider would still expect ZXY values in the request, but instead of returning an encoded MVT Tile it would return an encoded MVT Layer.

With this approach, a MVT server configuration could point to various remote data providers and make requests for MVT Layers that it would then combine into a MVT Tile and deliver to the client. This would allow MVT servers to combine numerous data sources without needing to host the actual data!

The intent here is to minimize data duplication and fragmentation while also simplifying data mashups and "getting started" time for MVT servers.

The possibilities are pretty wild and there is still quite a bit to think through. I'm hoping we can use this thread to discuss the strengths and weaknesses of the concept. 

Looking forward to the discussion.

--
Alexander W. Rolek

_______________________________________________
Vector-Tiles mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/vector-tiles
Reply | Threaded
Open this post in threaded view
|

Re: RPC and MVT Layers

Tobias Wendorff
Dear Alexander,

good work! I'll test the server in the next days.

One annotation to the example map shown on the project website:
Does OpenLayers3 support map rotation, like MapboxGL does? For some
people it's hard to distinguish between a map being created of rasters
or vectors. Rotating the map and keeping the labels upright is always
a good indicator ;)

Best regards,
Tobias


Am Di, 13.09.2016, 04:57 schrieb Alexander W. Rolek:
> Looking forward to the discussion.



_______________________________________________
Vector-Tiles mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/vector-tiles
Reply | Threaded
Open this post in threaded view
|

Re: RPC and MVT Layers

Rory McCann
In reply to this post by Alexander W. Rolek
Hi,

This is a very interesting idea, and something I had thought would be
good to have.

Rather than inventing a new file format, you could write something which
"merges" vector tiles together. Take all the layers from tile A, then
append (or overwrite) the layer(s) from tile B (etc).

This way you could (say) easily add contour (vector tile) data to a data
source which didn't have that.

On 13/09/16 04:57, Alexander W. Rolek wrote:

> Greetings OSGeo -
>
> For the last few months I have been working on a MVT server written in
> Go called Tegola (http://tegola.io <http://tegola.io/>). One of the
> design goals for Tegola has been to support multiple data providers
> (i.e. PostGIS, Geopackage, etc.). As anyone who works with geospatial
> data knows, a good chunk of time is dedicated to the extract, transform
> and load (ETL) pipeline. For many situations it's unnecessary to
> replicate a data layer from a data authority so you can have it in your
> own database. This unnecessary data duplication got us thinking about
> the possibility of a RPC data provider that would return MVT Layers
> instead of MVT Tiles.
>
> The MVT spec basically consists of a Tile, which is made up of Layers,
> which are made up of Features. A typical request to a MVT server
> includes Z, X and Y values (Slippy map tilenames) with an expected
> response being a protocol buffer encoded MVT tile. At a high level, a
> RPC data provider would still expect ZXY values in the request, but
> instead of returning an encoded MVT Tile it would return an encoded MVT
> Layer.
>
> With this approach, a MVT server configuration could point to various
> remote data providers and make requests for MVT Layers that it would
> then combine into a MVT Tile and deliver to the client. This would allow
> MVT servers to combine numerous data sources without needing to host the
> actual data!
>
> The intent here is to minimize data duplication and fragmentation while
> also simplifying data mashups and "getting started" time for MVT servers.
>
> The possibilities are pretty wild and there is still quite a bit to
> think through. I'm hoping we can use this thread to discuss the
> strengths and weaknesses of the concept.
>
> Looking forward to the discussion.
>
> --
> Alexander W. Rolek
>
>
> _______________________________________________
> Vector-Tiles mailing list
> [hidden email]
> http://lists.osgeo.org/mailman/listinfo/vector-tiles
>

_______________________________________________
Vector-Tiles mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/vector-tiles

signature.asc (501 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: RPC and MVT Layers

Andreas Hocevar
In reply to this post by Tobias Wendorff
Hey Tobias,

in OpenLayers, you can rotate the map by dragging with the Alt+Shift
keys pressed.

Andreas.

On Tue, Sep 13, 2016 at 2:00 PM, Tobias Wendorff
<[hidden email]> wrote:

> Dear Alexander,
>
> good work! I'll test the server in the next days.
>
> One annotation to the example map shown on the project website:
> Does OpenLayers3 support map rotation, like MapboxGL does? For some
> people it's hard to distinguish between a map being created of rasters
> or vectors. Rotating the map and keeping the labels upright is always
> a good indicator ;)
>
> Best regards,
> Tobias
>
>
> Am Di, 13.09.2016, 04:57 schrieb Alexander W. Rolek:
>> Looking forward to the discussion.
>
>
>
> _______________________________________________
> Vector-Tiles mailing list
> [hidden email]
> http://lists.osgeo.org/mailman/listinfo/vector-tiles
_______________________________________________
Vector-Tiles mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/vector-tiles
Reply | Threaded
Open this post in threaded view
|

Re: RPC and MVT Layers

Alexander W. Rolek
Thanks for the positive feedback! We're working on adding labels and hope to have something to show this week. I agree, that's an obvious indicator that the data is vector vs raster. 

Let me know if you have any feedback running Tegola. We just released v0.3.0 which comes with a /capabilities endpoint (i.e. http://tegola.cfapps.io/capabilities) and the ability to fetch individual layers of a map. I will update the docs this week with the new features.

On Tue, Sep 13, 2016 at 7:42 AM, Andreas Hocevar <[hidden email]> wrote:
Hey Tobias,

in OpenLayers, you can rotate the map by dragging with the Alt+Shift
keys pressed.

Andreas.

On Tue, Sep 13, 2016 at 2:00 PM, Tobias Wendorff
<[hidden email]> wrote:
> Dear Alexander,
>
> good work! I'll test the server in the next days.
>
> One annotation to the example map shown on the project website:
> Does OpenLayers3 support map rotation, like MapboxGL does? For some
> people it's hard to distinguish between a map being created of rasters
> or vectors. Rotating the map and keeping the labels upright is always
> a good indicator ;)
>
> Best regards,
> Tobias
>
>
> Am Di, 13.09.2016, 04:57 schrieb Alexander W. Rolek:
>> Looking forward to the discussion.
>
>
>
> _______________________________________________
> Vector-Tiles mailing list
> [hidden email]
> http://lists.osgeo.org/mailman/listinfo/vector-tiles



--
Alexander W. Rolek
303-829-9989

_______________________________________________
Vector-Tiles mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/vector-tiles
Reply | Threaded
Open this post in threaded view
|

Re: RPC and MVT Layers

Alexander W. Rolek
In reply to this post by Rory McCann
Roy - 

If I understand what you're saying, instead of expecting a RPC data provider to return a MVT Layer, just fetch MVT Tiles from already implemented providers, decode them and extract whatever you want for your mashup. I like this idea. The pros and cons I see:

Pros 
- Does not require any additional implementation from an MVT server. This is pretty big as MVT servers are already in the wild.
- User could easily define which layers they want to extract so they don't have to merge entire tiles together. 

Cons
- Possibly unreliable if a data provider changes the name of a layer in a tile or removes that layer. This could also happen when returning MVT Layers, but the explicit implementation of the MVT Layer response might result in more integrity from the data provider.
- Larger payloads.
- As you mentioned, this might result in layer name collisions during merging which could cause more complex server config rulesets. 

Thoughts on other pros / cons?


On Tue, Sep 13, 2016 at 5:38 AM, Rory McCann <[hidden email]> wrote:
Hi,

This is a very interesting idea, and something I had thought would be
good to have.

Rather than inventing a new file format, you could write something which
"merges" vector tiles together. Take all the layers from tile A, then
append (or overwrite) the layer(s) from tile B (etc).

This way you could (say) easily add contour (vector tile) data to a data
source which didn't have that.

On 13/09/16 04:57, Alexander W. Rolek wrote:
> Greetings OSGeo -
>
> For the last few months I have been working on a MVT server written in
> Go called Tegola (http://tegola.io <http://tegola.io/>). One of the
> design goals for Tegola has been to support multiple data providers
> (i.e. PostGIS, Geopackage, etc.). As anyone who works with geospatial
> data knows, a good chunk of time is dedicated to the extract, transform
> and load (ETL) pipeline. For many situations it's unnecessary to
> replicate a data layer from a data authority so you can have it in your
> own database. This unnecessary data duplication got us thinking about
> the possibility of a RPC data provider that would return MVT Layers
> instead of MVT Tiles.
>
> The MVT spec basically consists of a Tile, which is made up of Layers,
> which are made up of Features. A typical request to a MVT server
> includes Z, X and Y values (Slippy map tilenames) with an expected
> response being a protocol buffer encoded MVT tile. At a high level, a
> RPC data provider would still expect ZXY values in the request, but
> instead of returning an encoded MVT Tile it would return an encoded MVT
> Layer.
>
> With this approach, a MVT server configuration could point to various
> remote data providers and make requests for MVT Layers that it would
> then combine into a MVT Tile and deliver to the client. This would allow
> MVT servers to combine numerous data sources without needing to host the
> actual data!
>
> The intent here is to minimize data duplication and fragmentation while
> also simplifying data mashups and "getting started" time for MVT servers.
>
> The possibilities are pretty wild and there is still quite a bit to
> think through. I'm hoping we can use this thread to discuss the
> strengths and weaknesses of the concept.
>
> Looking forward to the discussion.
>
> --
> Alexander W. Rolek
>
>
> _______________________________________________
> Vector-Tiles mailing list
> [hidden email]
> http://lists.osgeo.org/mailman/listinfo/vector-tiles
>




--
Alexander W. Rolek

_______________________________________________
Vector-Tiles mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/vector-tiles