[gdal-dev] Missing layers with GeoPDF to MVT conversion with ogr2ogr/gdal 2.3.0

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] Missing layers with GeoPDF to MVT conversion with ogr2ogr/gdal 2.3.0

Max Ogden
Hi, I am trying to convert USFS FSTopo GeoPDFs into Mapbox Vector Tiles like this:

$ ogr2ogr -f MBTILES test.mbtiles 340711752_Azusa_FSTopo.pdf

I'm using gdal git master (GDAL 2.4.0dev-1f6d597, released 2018/06/28) with poppler 0.64.0. 

My issue is that when I view the mbtiles in a webgl map (using $ tileserver-gl-light test.mbtiles) I only see 13 layers, not the 59 I was expecting. Is it possible to include the vector information in all the GeoPDF metadata layers when converting to MVT? I tried adding `--config OGR_PDF_READ_NON_STRUCTURED YES` to the end of my ogr2ogr command but it had no effect.

ogrinfo shows 59 metadata layers and 13 top level named layers:

```

$ ogrinfo -mdd layers 340711752_Azusa_FSTopo.pdf

INFO: Open of `340711752_Azusa_FSTopo.pdf'

      using driver `PDF' successful.

Metadata:

  CREATION_DATE=D:20171221174439Z

  CREATOR=Esri ArcSOC 10.3.1.4959

  NEATLINE=POLYGON ((407183.394216688 3775784.59268212,407183.480495386 3790905.61434883,420034.534067471 3790905.80204647,420034.842839467 3775784.26947838,407183.394216688 3775784.59268212))

Metadata (layers):

  LAYER_00_NAME=Other_5

  LAYER_01_NAME=Quadrangle_Extent

  LAYER_02_NAME=Quadrangle_Extent.Labels

  LAYER_03_NAME=Quadrangle_Extent.Labels.State_Outline_-_Default

  LAYER_04_NAME=Quadrangle_Extent.Other_4

  LAYER_05_NAME=Quadrangle_Extent.State_Outline

  LAYER_06_NAME=Adjacent_Quadrangle_Diagram

  LAYER_07_NAME=Adjacent_Quadrangle_Diagram.Other_3

  LAYER_08_NAME=Adjacent_Quadrangle_Diagram.Labels

  LAYER_09_NAME=Adjacent_Quadrangle_Diagram.Labels.Quad_Outlines_-_Default

  LAYER_10_NAME=Adjacent_Quadrangle_Diagram.Quadrangle_Extent

  LAYER_11_NAME=Adjacent_Quadrangle_Diagram.Quad_Outlines

  LAYER_12_NAME=Other_2

  LAYER_13_NAME=Quadrangle

  LAYER_14_NAME=Quadrangle.Other

  LAYER_15_NAME=Quadrangle.Labels

  LAYER_16_NAME=Quadrangle.Labels.Unplaced_Labels

  LAYER_17_NAME=Quadrangle.Labels.Unplaced_Labels.Road_Shields_-_Vertical

  LAYER_18_NAME=Quadrangle.Labels.Road_Shields_-_Horizontal

  LAYER_19_NAME=Quadrangle.Labels.Road_Shields_-_Vertical

  LAYER_20_NAME=Quadrangle.Labels.Road_Shields_-_Default

  LAYER_21_NAME=Quadrangle.Neatline/Mask

  LAYER_22_NAME=Quadrangle.Neatline/Mask.Grid_Labels

  LAYER_23_NAME=Quadrangle.Neatline/Mask.Neatline

  LAYER_24_NAME=Quadrangle.Neatline/Mask.Mask

  LAYER_25_NAME=Quadrangle.Text

  LAYER_26_NAME=Quadrangle.Road_Shields

  LAYER_27_NAME=Quadrangle.Buildings

  LAYER_28_NAME=Quadrangle.Culture_Features

  LAYER_29_NAME=Quadrangle.Drainage_Features

  LAYER_30_NAME=Quadrangle.Large_Tanks

  LAYER_31_NAME=Quadrangle.Survey_Points

  LAYER_32_NAME=Quadrangle.Geodetic_Control

  LAYER_33_NAME=Quadrangle.Recreation/Facility_Features

  LAYER_34_NAME=Quadrangle.Transportation_Features

  LAYER_35_NAME=Quadrangle.Linear_Transportation_Features

  LAYER_36_NAME=Quadrangle.Railroads_

  LAYER_37_NAME=Quadrangle.Linear_Culture_Features

  LAYER_38_NAME=Quadrangle.Linear_Landform_Features

  LAYER_39_NAME=Quadrangle.Boundaries

  LAYER_40_NAME=Quadrangle.Boundaries

  LAYER_41_NAME=Quadrangle.PLSS

  LAYER_42_NAME=Quadrangle.Survey_Lines

  LAYER_43_NAME=Quadrangle.Linear_Drainage_Features

  LAYER_44_NAME=Quadrangle.Contour_Labels

  LAYER_45_NAME=Quadrangle.Contours

  LAYER_46_NAME=Quadrangle.2_5`_Tics

  LAYER_47_NAME=Quadrangle.2_5`_Tics.Interior_Grid_Intersections

  LAYER_48_NAME=Quadrangle.2_5`_Tics.Grid_Tics_along_Neatline

  LAYER_49_NAME=Quadrangle.UTM_Grid

  LAYER_50_NAME=Quadrangle.UTM_Grid.Interior_Grid_Intersections

  LAYER_51_NAME=Quadrangle.UTM_Grid.Grid_Tics_along_Neatline

  LAYER_52_NAME=Quadrangle.UTM_Grid.UTM_Grid_Lines

  LAYER_53_NAME=Quadrangle.Large_Buildings

  LAYER_54_NAME=Quadrangle.Culture_Features

  LAYER_55_NAME=Quadrangle.Landforms

  LAYER_56_NAME=Quadrangle.Drainage_Polygons

  LAYER_57_NAME=Quadrangle.Ownership

  LAYER_58_NAME=Quadrangle.Builtup_Areas

  LAYER_59_NAME=Quadrangle.WoodlandUSGS_P

1: Quadrangle_Culture_Features

2: Quadrangle_Contour_Labels

3: Quadrangle_Contours

4: Quadrangle_2_5`_Tics_Interior_Grid_Intersections

5: Quadrangle_2_5`_Tics_Grid_Tics_along_Neatline

6: Quadrangle_UTM_Grid_Interior_Grid_Intersections

7: Quadrangle_UTM_Grid_Grid_Tics_along_Neatline

8: Quadrangle_UTM_Grid_UTM_Grid_Lines

9: Quadrangle_Large_Buildings

10: Quadrangle_Drainage_Polygons

11: Quadrangle_Ownership

12: Quadrangle_Builtup_Areas

13: Quadrangle_WoodlandUSGS_P

```

Thank you,
Max


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Missing layers with GeoPDF to MVT conversion with ogr2ogr/gdal 2.3.0

Even Rouault-2
Max,

I've pushed a fix (in master and release/2.3 branches) in the drawing
instruction parser of the PDF driver that enables to read that file

$ ogrinfo ~/gdal/data/pdf/340711752_Azusa_FSTopo.pdf
[...]
1: Other_5
2: Quadrangle_Extent_Other_4
3: Quadrangle_Extent_State_Outline
4: Adjacent_Quadrangle_Diagram_Other_3
5: Adjacent_Quadrangle_Diagram_Quadrangle_Extent
6: Adjacent_Quadrangle_Diagram_Quad_Outlines
7: Quadrangle_Other
8: Quadrangle_Labels_Unplaced_Labels_Road_Shields_-_Vertical
9: Quadrangle_Labels_Road_Shields_-_Horizontal
10: Quadrangle_Labels_Road_Shields_-_Vertical
11: Quadrangle_Neatline/Mask_Neatline
12: Quadrangle_Neatline/Mask_Mask
13: Quadrangle_Culture_Features
14: Quadrangle_Large_Tanks
15: Quadrangle_Linear_Transportation_Features
16: Quadrangle_Railroads_
17: Quadrangle_Linear_Culture_Features
18: Quadrangle_Linear_Landform_Features
19: Quadrangle_Boundaries
20: Quadrangle_PLSS
21: Quadrangle_Survey_Lines
22: Quadrangle_Linear_Drainage_Features
23: Quadrangle_Contour_Labels
24: Quadrangle_Contours
25: Quadrangle_2_5`_Tics_Interior_Grid_Intersections
26: Quadrangle_2_5`_Tics_Grid_Tics_along_Neatline
27: Quadrangle_UTM_Grid_Interior_Grid_Intersections
28: Quadrangle_UTM_Grid_Grid_Tics_along_Neatline
29: Quadrangle_UTM_Grid_UTM_Grid_Lines
30: Quadrangle_Large_Buildings
31: Quadrangle_Drainage_Polygons
32: Quadrangle_Ownership
33: Quadrangle_Builtup_Areas
34: Quadrangle_WoodlandUSGS_P

Some of the layers are missing, but either they are not terminal, or contain
only objects that OGR cannot handle (labels e.g.)

Even


> Hi, I am trying to convert USFS FSTopo GeoPDFs into Mapbox Vector Tiles
> like this:
>
> $ wget
> https://data.fs.usda.gov/geodata/rastergateway/data/34117/fstopo/340711752_A
> zusa_FSTopo.pdf $ ogr2ogr -f MBTILES test.mbtiles 340711752_Azusa_FSTopo.pdf
>
> I'm using gdal git master (GDAL 2.4.0dev-1f6d597, released 2018/06/28) with
> poppler 0.64.0.
>
> My issue is that when I view the mbtiles in a webgl map (using $
> tileserver-gl-light test.mbtiles) I only see 13 layers, not the 59 I was
> expecting. Is it possible to include the vector information in all the
> GeoPDF metadata layers when converting to MVT? I tried adding `--config
> OGR_PDF_READ_NON_STRUCTURED YES` to the end of my ogr2ogr command but it
> had no effect.
>
> ogrinfo shows 59 metadata layers and 13 top level named layers:
>
> ```
>
> $ ogrinfo -mdd layers 340711752_Azusa_FSTopo.pdf
>
> INFO: Open of `340711752_Azusa_FSTopo.pdf'
>
>       using driver `PDF' successful.
>
> Metadata:
>
>   CREATION_DATE=D:20171221174439Z
>
>   CREATOR=Esri ArcSOC 10.3.1.4959
>
>   NEATLINE=POLYGON ((407183.394216688 3775784.59268212,407183.480495386
> 3790905.61434883,420034.534067471 3790905.80204647,420034.842839467
> 3775784.26947838,407183.394216688 3775784.59268212))
>
> Metadata (layers):
>
>   LAYER_00_NAME=Other_5
>
>   LAYER_01_NAME=Quadrangle_Extent
>
>   LAYER_02_NAME=Quadrangle_Extent.Labels
>
>   LAYER_03_NAME=Quadrangle_Extent.Labels.State_Outline_-_Default
>
>   LAYER_04_NAME=Quadrangle_Extent.Other_4
>
>   LAYER_05_NAME=Quadrangle_Extent.State_Outline
>
>   LAYER_06_NAME=Adjacent_Quadrangle_Diagram
>
>   LAYER_07_NAME=Adjacent_Quadrangle_Diagram.Other_3
>
>   LAYER_08_NAME=Adjacent_Quadrangle_Diagram.Labels
>
>   LAYER_09_NAME=Adjacent_Quadrangle_Diagram.Labels.Quad_Outlines_-_Default
>
>   LAYER_10_NAME=Adjacent_Quadrangle_Diagram.Quadrangle_Extent
>
>   LAYER_11_NAME=Adjacent_Quadrangle_Diagram.Quad_Outlines
>
>   LAYER_12_NAME=Other_2
>
>   LAYER_13_NAME=Quadrangle
>
>   LAYER_14_NAME=Quadrangle.Other
>
>   LAYER_15_NAME=Quadrangle.Labels
>
>   LAYER_16_NAME=Quadrangle.Labels.Unplaced_Labels
>
>   LAYER_17_NAME=Quadrangle.Labels.Unplaced_Labels.Road_Shields_-_Vertical
>
>   LAYER_18_NAME=Quadrangle.Labels.Road_Shields_-_Horizontal
>
>   LAYER_19_NAME=Quadrangle.Labels.Road_Shields_-_Vertical
>
>   LAYER_20_NAME=Quadrangle.Labels.Road_Shields_-_Default
>
>   LAYER_21_NAME=Quadrangle.Neatline/Mask
>
>   LAYER_22_NAME=Quadrangle.Neatline/Mask.Grid_Labels
>
>   LAYER_23_NAME=Quadrangle.Neatline/Mask.Neatline
>
>   LAYER_24_NAME=Quadrangle.Neatline/Mask.Mask
>
>   LAYER_25_NAME=Quadrangle.Text
>
>   LAYER_26_NAME=Quadrangle.Road_Shields
>
>   LAYER_27_NAME=Quadrangle.Buildings
>
>   LAYER_28_NAME=Quadrangle.Culture_Features
>
>   LAYER_29_NAME=Quadrangle.Drainage_Features
>
>   LAYER_30_NAME=Quadrangle.Large_Tanks
>
>   LAYER_31_NAME=Quadrangle.Survey_Points
>
>   LAYER_32_NAME=Quadrangle.Geodetic_Control
>
>   LAYER_33_NAME=Quadrangle.Recreation/Facility_Features
>
>   LAYER_34_NAME=Quadrangle.Transportation_Features
>
>   LAYER_35_NAME=Quadrangle.Linear_Transportation_Features
>
>   LAYER_36_NAME=Quadrangle.Railroads_
>
>   LAYER_37_NAME=Quadrangle.Linear_Culture_Features
>
>   LAYER_38_NAME=Quadrangle.Linear_Landform_Features
>
>   LAYER_39_NAME=Quadrangle.Boundaries
>
>   LAYER_40_NAME=Quadrangle.Boundaries
>
>   LAYER_41_NAME=Quadrangle.PLSS
>
>   LAYER_42_NAME=Quadrangle.Survey_Lines
>
>   LAYER_43_NAME=Quadrangle.Linear_Drainage_Features
>
>   LAYER_44_NAME=Quadrangle.Contour_Labels
>
>   LAYER_45_NAME=Quadrangle.Contours
>
>   LAYER_46_NAME=Quadrangle.2_5`_Tics
>
>   LAYER_47_NAME=Quadrangle.2_5`_Tics.Interior_Grid_Intersections
>
>   LAYER_48_NAME=Quadrangle.2_5`_Tics.Grid_Tics_along_Neatline
>
>   LAYER_49_NAME=Quadrangle.UTM_Grid
>
>   LAYER_50_NAME=Quadrangle.UTM_Grid.Interior_Grid_Intersections
>
>   LAYER_51_NAME=Quadrangle.UTM_Grid.Grid_Tics_along_Neatline
>
>   LAYER_52_NAME=Quadrangle.UTM_Grid.UTM_Grid_Lines
>
>   LAYER_53_NAME=Quadrangle.Large_Buildings
>
>   LAYER_54_NAME=Quadrangle.Culture_Features
>
>   LAYER_55_NAME=Quadrangle.Landforms
>
>   LAYER_56_NAME=Quadrangle.Drainage_Polygons
>
>   LAYER_57_NAME=Quadrangle.Ownership
>
>   LAYER_58_NAME=Quadrangle.Builtup_Areas
>
>   LAYER_59_NAME=Quadrangle.WoodlandUSGS_P
>
> 1: Quadrangle_Culture_Features
>
> 2: Quadrangle_Contour_Labels
>
> 3: Quadrangle_Contours
>
> 4: Quadrangle_2_5`_Tics_Interior_Grid_Intersections
>
> 5: Quadrangle_2_5`_Tics_Grid_Tics_along_Neatline
>
> 6: Quadrangle_UTM_Grid_Interior_Grid_Intersections
>
> 7: Quadrangle_UTM_Grid_Grid_Tics_along_Neatline
>
> 8: Quadrangle_UTM_Grid_UTM_Grid_Lines
>
> 9: Quadrangle_Large_Buildings
>
> 10: Quadrangle_Drainage_Polygons
>
> 11: Quadrangle_Ownership
>
> 12: Quadrangle_Builtup_Areas
>
> 13: Quadrangle_WoodlandUSGS_P
> ```
>
> Thank you,
> Max


--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Missing layers with GeoPDF to MVT conversion with ogr2ogr/gdal 2.3.0

Max Ogden
Thank you, Even! Your fix works as described. The MVT MBTiles version of the Azusa GeoPDF rendered in WebGL looks like this now, much more complete: https://i.imgur.com/Nnq6Brp.jpg

I took the layers that got skipped and did a raster render of them for reference: https://i.imgur.com/9MjQ9hk.jpg

Any ideas on how to extract the rest of the information? I suppose as a transparent png overlay tile layer the text labels would be much smaller than the full topo tiles... so rendering the raster labels on top of the vector terrain might work OK. But I would love to render everything in WebGL :D

Max

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev