Status of MVT vector tiles?

Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Status of MVT vector tiles?

Erik H
Hello

I'm very interested in MapBox Vector Tiles, because I'd like to do client-side rendering. I found a branch in several github clones (tbonfort,sdlime) but it's not clear to me if/how I could try them out.

My setup is perhaps a little unusual: I run a small cluster of Java-based web servers, each of which links to the MapServer shared library. I therefore do not use WMS or MapCache.

How would I configure a .map file to generate a .mvt file? Is this even possible? In Steve's mvt-demo, I see a parameter 'map.imagetype=mvt' hidden in a JSON file, but nothing in the .map file seems to indicate that a .mvt should be created.

Thanks,

Erik


_______________________________________________
mapserver-users mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapserver-users
Reply | Threaded
Open this post in threaded view
|

Re: Status of MVT vector tiles?

Lime, Steve D (MNIT)

My branch (https://github.com/sdlime/mapserver/tree/vector-tiles) is the one to try. You should be able build it as you would any other branch – using cmake. You need protobuf and protobuf-c installed to be able to compile things though.

 

Configuring MVT output is really quite similar to configuring WFS or WMS output. The MVT renderer relies on the WFS/WMS/OWS/GML metadata items to define what fields should be output and what they should be named. In my demo https://github.com/sdlime/mapserver/tree/vector-tiles the MVT related configuration is pretty much just the gml_* metadata items associated with each layer. I’m mapping shapefile fields to standard names in the MVT output for consistency – so every feature has a name and a unique id. The id is referenced in the MVT specs (https://github.com/mapbox/vector-tile-spec) and must be unique within a layer.

 

From there it’s just a matter of setting the image type. In the case of the demo I do that remotely – as a parameter in the service URL but I could just as easily just set IMAGETYPE ‘mvt’ in the mapfile.

 

How are you computing tile bounds, etc…? The demo (https://github.com/sdlime/mvt-demo) runs without map cache and uses MapServer mode=tile to handle extent computation and so forth.

 

Steve

 

From: mapserver-users [mailto:[hidden email]] On Behalf Of Erik H
Sent: Sunday, February 04, 2018 2:04 PM
To: [hidden email]
Subject: [mapserver-users] Status of MVT vector tiles?

 

Hello

 

I'm very interested in MapBox Vector Tiles, because I'd like to do client-side rendering. I found a branch in several github clones (tbonfort,sdlime) but it's not clear to me if/how I could try them out.

 

My setup is perhaps a little unusual: I run a small cluster of Java-based web servers, each of which links to the MapServer shared library. I therefore do not use WMS or MapCache.

 

How would I configure a .map file to generate a .mvt file? Is this even possible? In Steve's mvt-demo, I see a parameter 'map.imagetype=mvt' hidden in a JSON file, but nothing in the .map file seems to indicate that a .mvt should be created.

 

Thanks,

 

Erik

 


_______________________________________________
mapserver-users mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapserver-users
Reply | Threaded
Open this post in threaded view
|

Re: Status of MVT vector tiles?

mp-surv
Hey Steve.
I have a working demo using your data and when zooming out not all vector
tiles load and browser becomes unresponsive (image below). To avoid
overloading my computer when zooming out and causing my browser to crash I
would like to use mapcache, but I'm struggling to set it upt. Can you please
give a working example using mapache?

<http://osgeo-org.1560.x6.nabble.com/file/t368111/Capture.png>

I can see there are also a lot of canceled requests but they are not causing
my browser to crash. Do you know what produces them?

Thanks.







--
Sent from: http://osgeo-org.1560.x6.nabble.com/Mapserver-User-f4226646.html
_______________________________________________
mapserver-users mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapserver-users
Reply | Threaded
Open this post in threaded view
|

Re: Status of MVT vector tiles?

Lime, Steve D (MNIT)
Will do. I have a working MapCache example on my dev box that I can share pieces from. First step is to get MapCache working with the pull-request I submitted some time ago. It relies on WMS requests to MapServer for the MVT tiles. Pull request is at:

  https://github.com/mapserver/mapcache/pull/166

Regarding the canceled requests:

 I'm not sure what up with those.  However, I don't believe those are a server issue, rather a client-based. You can see the same behavior on other sites using vector tiles from other services (https://openmaptiles.com/downloads/dataset/osm/#0.23/0/-26) in Chrome DevTools. Could be related to pre-fetching resources.

Regarding a unresponsive browser:

There's likely a secret sauce in terms of the matching min/max scales you should offer a layer at, whether you should manage multiple resolutions of your data on the server and other filtering. The demo is very naïve in that regard so you can end up with big tiles. At this point it's not something I can offer a ton of help with - I don’t have a production MVT application yet.

Steve

-----Original Message-----
From: mapserver-users [mailto:[hidden email]] On Behalf Of mp-surv
Sent: Saturday, June 23, 2018 6:28 PM
To: [hidden email]
Subject: Re: [mapserver-users] Status of MVT vector tiles?

Hey Steve.
I have a working demo using your data and when zooming out not all vector
tiles load and browser becomes unresponsive (image below). To avoid
overloading my computer when zooming out and causing my browser to crash I
would like to use mapcache, but I'm struggling to set it upt. Can you please
give a working example using mapache?

<http://osgeo-org.1560.x6.nabble.com/file/t368111/Capture.png>

I can see there are also a lot of canceled requests but they are not causing
my browser to crash. Do you know what produces them?

Thanks.







--
Sent from: http://osgeo-org.1560.x6.nabble.com/Mapserver-User-f4226646.html
_______________________________________________
mapserver-users mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapserver-users
_______________________________________________
mapserver-users mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapserver-users
Reply | Threaded
Open this post in threaded view
|

Re: Status of MVT vector tiles?

mp-surv
I found that pull request before thanks, will play around with it some more. When you say it relies on WMS requests does that mean it generates raster tiles or vector tiles in PBD file format? I don't have mine mapcache example working yet so i don't know what to expect.

About canceled requests i found related "issue" discussed on leaflet forum:
But i don't see a problem with that because all entities are loaded on my map.

I will have to find that secret souce yes, will try to optimize my map file for different scales. Pregenerated vector tiles would also help.

Thanks for your answers. Will track your github for further updates.

Miha 


V V pon., 25. jun. 2018 ob 23:57 je oseba Lime, Steve D (MNIT) <[hidden email]> napisala:
Will do. I have a working MapCache example on my dev box that I can share pieces from. First step is to get MapCache working with the pull-request I submitted some time ago. It relies on WMS requests to MapServer for the MVT tiles. Pull request is at:

  https://github.com/mapserver/mapcache/pull/166

Regarding the canceled requests:

 I'm not sure what up with those.  However, I don't believe those are a server issue, rather a client-based. You can see the same behavior on other sites using vector tiles from other services (https://openmaptiles.com/downloads/dataset/osm/#0.23/0/-26) in Chrome DevTools. Could be related to pre-fetching resources.

Regarding a unresponsive browser:

There's likely a secret sauce in terms of the matching min/max scales you should offer a layer at, whether you should manage multiple resolutions of your data on the server and other filtering. The demo is very naïve in that regard so you can end up with big tiles. At this point it's not something I can offer a ton of help with - I don’t have a production MVT application yet.

Steve

-----Original Message-----
From: mapserver-users [mailto:[hidden email]] On Behalf Of mp-surv
Sent: Saturday, June 23, 2018 6:28 PM
To: [hidden email]
Subject: Re: [mapserver-users] Status of MVT vector tiles?

Hey Steve.
I have a working demo using your data and when zooming out not all vector
tiles load and browser becomes unresponsive (image below). To avoid
overloading my computer when zooming out and causing my browser to crash I
would like to use mapcache, but I'm struggling to set it upt. Can you please
give a working example using mapache?

<http://osgeo-org.1560.x6.nabble.com/file/t368111/Capture.png>

I can see there are also a lot of canceled requests but they are not causing
my browser to crash. Do you know what produces them?

Thanks.







--
Sent from: http://osgeo-org.1560.x6.nabble.com/Mapserver-User-f4226646.html
_______________________________________________
mapserver-users mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapserver-users


--
Lp,
Miha

_______________________________________________
mapserver-users mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapserver-users
Reply | Threaded
Open this post in threaded view
|

TMS layer in mapfile

Hans Wapenaar (GIS Innovations)

Hi,

 

Is it possible to add a TMS layer to a mapfile? I have a connection to an external TMS server. I can connect to that server with Leaflet or OpenLayers. But I want to render maps serverside with mapserver/mapfiles and store these maps directly on the server as images. With almost all layer types that works including WMS, but I cannot find a way to do that with TMS layers.

 

Any suggestions?

 

Hans Wapenaar

 


_______________________________________________
mapserver-users mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapserver-users
Reply | Threaded
Open this post in threaded view
|

Re: TMS layer in mapfile

Eichner, Andreas - SID-NLKM
MapServer uses GDAL to access rasters, so you can use a GDAL virtual raster:
http://www.gdal.org/frmt_wms.html


> -----Original Message-----
> From: mapserver-users [mailto:[hidden email]]
> On Behalf Of Hans Wapenaar (GIS Innovations)
> Sent: Tuesday, June 26, 2018 10:59 AM
> To: [hidden email]
> Subject: [mapserver-users] TMS layer in mapfile
>
> Hi,
>
>
>
> Is it possible to add a TMS layer to a mapfile? I have a connection to
> an external TMS server. I can connect to that server with Leaflet or
> OpenLayers. But I want to render maps serverside with mapserver/mapfiles
> and store these maps directly on the server as images. With almost all
> layer types that works including WMS, but I cannot find a way to do that
> with TMS layers.
>
>
>
> Any suggestions?
>
>
>
> Hans Wapenaar
>
>

_______________________________________________
mapserver-users mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapserver-users
Reply | Threaded
Open this post in threaded view
|

Re: Status of MVT vector tiles?

Lime, Steve D (MNIT)
In reply to this post by mp-surv

It generates vector tiles based on WMS requests. MapCache computes the proper extent based on the requested tile and then kicks off a WMS call for vector tile. The pull request creates support for a new generic “raw” format. MapCache doesn’t know much about files in this format, just a mime-type, so things like metatiling, image processing or tile assembly aren’t supported. You do get the cache management though. The raw format is not specific to vector tiles so you could use it for other formats like GeoJSON or UTFGrid.

 

Anyway, in my mapcache.xml file I have the following setup:

 

<!—define a new raw format to use for MapBox vector tiles -->

<format name="MVT" type="RAW">

   <extension>mvt</extension>

   <mime_type>application/vnd.mapbox-vector-tile</mime_type>

</format>

 

<!—this WMS source fetches MapBox vector tiles (.mvt) -->

<source name="wms_mvt" type="wms">

  <http>

    <url>http://mapserver.localhost/cgi-bin/mapserv?</url>

  </http>

  <getmap>

    <params>

      <LAYERS>sna,wma,lakes,major_roads,places</LAYERS>

      <FORMAT>application/vnd.mapbox-vector-tile</FORMAT>

      <MAP>/Users/sdlime/work/mvt-demo/demo.map </MAP>

    </params>

  </getmap>

</source>

 

<!—and finally the tileset -->

<tileset name="mvt">

  <source>wms_mvt</source>

  <grid>g</grid>

  <cache>disk</cache>

  <format>MVT</format>

</tileset>

 

Then, if you look at the simple.json file in my demo repo you just change the source for the vector tiles layer to reference MapCache (I was running as FastCGI) instead of MapServer (mode=tile), so from:

 

"tiles": ["http://mapserver.localhost/cgi-bin/mapserv?map=/Users/sdlime/work/mvt-demo/demo.map&mode=tile&tilemode=gmap&tile={x}+{y}+{z}&layers=all&map.imagetype=mvt"],

 

to:

 

"tiles": ["http://mapserver.localhost/cgi-bin/mapcache.fcgi/gmaps/mvt@g/{z}/{x}/{y}.mvt"]

 

One thing to note is that you can control the resolution that each tile is created at. Each tile has a grid associated with it and that is used to convert the features to a grid-based coordinate system. The higher the resolution of that grid the more detail is represented in the tile. The default is 4096x4096. You can change that value via the MVT output format, for example:

 

OUTPUTFORMAT

  NAME “mvt”

  DRIVER MVT

  FORMATOPTION “EXTENT=512”

END

 

A smaller resolution or extent will generally result in smaller tiles.

 

--Steve

 

From: Miha Požauko [mailto:[hidden email]]
Sent: Tuesday, June 26, 2018 3:09 AM
To: Lime, Steve D (MNIT) <[hidden email]>
Cc: [hidden email]
Subject: Re: [mapserver-users] Status of MVT vector tiles?

 

I found that pull request before thanks, will play around with it some more. When you say it relies on WMS requests does that mean it generates raster tiles or vector tiles in PBD file format? I don't have mine mapcache example working yet so i don't know what to expect.

 

About canceled requests i found related "issue" discussed on leaflet forum:

But i don't see a problem with that because all entities are loaded on my map.

 

I will have to find that secret souce yes, will try to optimize my map file for different scales. Pregenerated vector tiles would also help.

 

Thanks for your answers. Will track your github for further updates.

 

Miha 

 

 

V V pon., 25. jun. 2018 ob 23:57 je oseba Lime, Steve D (MNIT) <[hidden email]> napisala:

Will do. I have a working MapCache example on my dev box that I can share pieces from. First step is to get MapCache working with the pull-request I submitted some time ago. It relies on WMS requests to MapServer for the MVT tiles. Pull request is at:

  https://github.com/mapserver/mapcache/pull/166

Regarding the canceled requests:

 I'm not sure what up with those.  However, I don't believe those are a server issue, rather a client-based. You can see the same behavior on other sites using vector tiles from other services (https://openmaptiles.com/downloads/dataset/osm/#0.23/0/-26) in Chrome DevTools. Could be related to pre-fetching resources.

Regarding a unresponsive browser:

There's likely a secret sauce in terms of the matching min/max scales you should offer a layer at, whether you should manage multiple resolutions of your data on the server and other filtering. The demo is very naïve in that regard so you can end up with big tiles. At this point it's not something I can offer a ton of help with - I don’t have a production MVT application yet.

Steve

-----Original Message-----
From: mapserver-users [mailto:[hidden email]] On Behalf Of mp-surv
Sent: Saturday, June 23, 2018 6:28 PM
To: [hidden email]
Subject: Re: [mapserver-users] Status of MVT vector tiles?

Hey Steve.
I have a working demo using your data and when zooming out not all vector
tiles load and browser becomes unresponsive (image below). To avoid
overloading my computer when zooming out and causing my browser to crash I
would like to use mapcache, but I'm struggling to set it upt. Can you please
give a working example using mapache?

<http://osgeo-org.1560.x6.nabble.com/file/t368111/Capture.png>

I can see there are also a lot of canceled requests but they are not causing
my browser to crash. Do you know what produces them?

Thanks.







--
Sent from: http://osgeo-org.1560.x6.nabble.com/Mapserver-User-f4226646.html
_______________________________________________
mapserver-users mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapserver-users


 

--

Lp,

Miha


_______________________________________________
mapserver-users mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapserver-users
Reply | Threaded
Open this post in threaded view
|

Re: Status of MVT vector tiles?

mp-surv
It took me some time to configure everything because I migrated everything on a fresh ubuntu machine where i compiled and installed mapserver and mapacache for the first time ever (you can imagine the strugle). But following your instructions i have a working mapserver+mapcache with MVT support now. Thanks!

Miha

V V čet., 28. jun. 2018 ob 21:57 je oseba Lime, Steve D (MNIT) <[hidden email]> napisala:

It generates vector tiles based on WMS requests. MapCache computes the proper extent based on the requested tile and then kicks off a WMS call for vector tile. The pull request creates support for a new generic “raw” format. MapCache doesn’t know much about files in this format, just a mime-type, so things like metatiling, image processing or tile assembly aren’t supported. You do get the cache management though. The raw format is not specific to vector tiles so you could use it for other formats like GeoJSON or UTFGrid.

 

Anyway, in my mapcache.xml file I have the following setup:

 

<!—define a new raw format to use for MapBox vector tiles -->

<format name="MVT" type="RAW">

   <extension>mvt</extension>

   <mime_type>application/vnd.mapbox-vector-tile</mime_type>

</format>

 

<!—this WMS source fetches MapBox vector tiles (.mvt) -->

<source name="wms_mvt" type="wms">

  <http>

    <url>http://mapserver.localhost/cgi-bin/mapserv?</url>

  </http>

  <getmap>

    <params>

      <LAYERS>sna,wma,lakes,major_roads,places</LAYERS>

      <FORMAT>application/vnd.mapbox-vector-tile</FORMAT>

      <MAP>/Users/sdlime/work/mvt-demo/demo.map </MAP>

    </params>

  </getmap>

</source>

 

<!—and finally the tileset -->

<tileset name="mvt">

  <source>wms_mvt</source>

  <grid>g</grid>

  <cache>disk</cache>

  <format>MVT</format>

</tileset>

 

Then, if you look at the simple.json file in my demo repo you just change the source for the vector tiles layer to reference MapCache (I was running as FastCGI) instead of MapServer (mode=tile), so from:

 

"tiles": ["http://mapserver.localhost/cgi-bin/mapserv?map=/Users/sdlime/work/mvt-demo/demo.map&mode=tile&tilemode=gmap&tile={x}+{y}+{z}&layers=all&map.imagetype=mvt"],

 

to:

 

"tiles": ["http://mapserver.localhost/cgi-bin/mapcache.fcgi/gmaps/mvt@g/{z}/{x}/{y}.mvt"]

 

One thing to note is that you can control the resolution that each tile is created at. Each tile has a grid associated with it and that is used to convert the features to a grid-based coordinate system. The higher the resolution of that grid the more detail is represented in the tile. The default is 4096x4096. You can change that value via the MVT output format, for example:

 

OUTPUTFORMAT

  NAME “mvt”

  DRIVER MVT

  FORMATOPTION “EXTENT=512”

END

 

A smaller resolution or extent will generally result in smaller tiles.

 

--Steve

 

From: Miha Požauko [mailto:[hidden email]]
Sent: Tuesday, June 26, 2018 3:09 AM
To: Lime, Steve D (MNIT) <[hidden email]>
Cc: [hidden email]
Subject: Re: [mapserver-users] Status of MVT vector tiles?

 

I found that pull request before thanks, will play around with it some more. When you say it relies on WMS requests does that mean it generates raster tiles or vector tiles in PBD file format? I don't have mine mapcache example working yet so i don't know what to expect.

 

About canceled requests i found related "issue" discussed on leaflet forum:

But i don't see a problem with that because all entities are loaded on my map.

 

I will have to find that secret souce yes, will try to optimize my map file for different scales. Pregenerated vector tiles would also help.

 

Thanks for your answers. Will track your github for further updates.

 

Miha 

 

 

V V pon., 25. jun. 2018 ob 23:57 je oseba Lime, Steve D (MNIT) <[hidden email]> napisala:

Will do. I have a working MapCache example on my dev box that I can share pieces from. First step is to get MapCache working with the pull-request I submitted some time ago. It relies on WMS requests to MapServer for the MVT tiles. Pull request is at:

  https://github.com/mapserver/mapcache/pull/166

Regarding the canceled requests:

 I'm not sure what up with those.  However, I don't believe those are a server issue, rather a client-based. You can see the same behavior on other sites using vector tiles from other services (https://openmaptiles.com/downloads/dataset/osm/#0.23/0/-26) in Chrome DevTools. Could be related to pre-fetching resources.

Regarding a unresponsive browser:

There's likely a secret sauce in terms of the matching min/max scales you should offer a layer at, whether you should manage multiple resolutions of your data on the server and other filtering. The demo is very naïve in that regard so you can end up with big tiles. At this point it's not something I can offer a ton of help with - I don’t have a production MVT application yet.

Steve

-----Original Message-----
From: mapserver-users [mailto:[hidden email]] On Behalf Of mp-surv
Sent: Saturday, June 23, 2018 6:28 PM
To: [hidden email]
Subject: Re: [mapserver-users] Status of MVT vector tiles?

Hey Steve.
I have a working demo using your data and when zooming out not all vector
tiles load and browser becomes unresponsive (image below). To avoid
overloading my computer when zooming out and causing my browser to crash I
would like to use mapcache, but I'm struggling to set it upt. Can you please
give a working example using mapache?

<http://osgeo-org.1560.x6.nabble.com/file/t368111/Capture.png>

I can see there are also a lot of canceled requests but they are not causing
my browser to crash. Do you know what produces them?

Thanks.







--
Sent from: http://osgeo-org.1560.x6.nabble.com/Mapserver-User-f4226646.html
_______________________________________________
mapserver-users mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapserver-users


 

--

Lp,

Miha



--
Lp,
Miha

_______________________________________________
mapserver-users mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapserver-users