[gdal-dev] Extracting keys & values from OSM Planet

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] Extracting keys & values from OSM Planet

NikosAlexandris
Dear OSM experts,

a reproducible workflow that extracts specific tags from OSM's Planet
data set, includes:

1. downloading OSM Planet in form of a pbf file

2. convert pbf to o5m using `osmconvert`

3  extract areas of interest in separate `aoi_*.o5m` files, based on boundaries in form of .poly
files, using `osmconvert`

4. filtering for specific tags (i.e. highways), parallel tasks in 80 cores

```bash
# custom function for 'highway' tags
function osmfilterhighway { osmfilter $1 --drop-version --keep="highway=track =footway =bridleway =path" > $(basename $1 .o5m)_highway.osm; }
export -f osmfilterhighway

# filter
find . -type f -name \*aoi*.o5m |parallel -j80 osmfilterhighway {}
```

1. Is it any better using GDAL's `osmconf.ini` "method" [0],
over `osmconvert` and `osmfilter`, in terms of reliability/speed etc.?

2. Is OGR handling well the conversion from .osm to ESRI Shapefiles?

3. What is the way to split "other_tags" in multiple new fields *when*
knowing exactly which tags are contained and should be obtained?

Thank you for any help,

Nikos


[0] https://svn.osgeo.org/gdal/trunk/gdal/data/osmconf.ini

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev

signature.asc (235 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Extracting keys & values from OSM Planet

Even Rouault-2

> 2. Is OGR handling well the conversion from .osm to ESRI Shapefiles?

 

Yes, but within the limits ot the shapefile, and particularly .dbf format: limitation to 254 characters for field values, 10 characters for field names... which are easily violated by OSM extracts.

 

Spatialite, GeoPackage, PostGIS etc. would be better choices as output format

 

>

> 3. What is the way to split "other_tags" in multiple new fields *when*

> knowing exactly which tags are contained and should be obtained?

 

Edit osmconf.ini to add in the attributes= settings the tag names you're interested in.

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Extracting keys & values from OSM Planet

Frank Broniewski-5
In reply to this post by NikosAlexandris

Hi Nikos,

 

I am using a similar approach to yours to extract data from OSM files to a GIS format. I chose Spatiallite as a format since it is superior to the Shapefile format in all areas.

FYI here’s what I do:

osmconvert -b="5.5,49,8,50.5" -o=saarland.osm.pbf "europe-latest.osm.pbf"

ogr2ogr --config OSM_CONFIG_FILE /home/frank/gdal/osmconf.ini -f SQLite -dsco SPATIALITE=YES -progress -gt 65536 -t_srs "EPSG:31466" /tmp/saarland.sqlite /tmp/saarland.osm.pbf

 

Please note the “–config” switch for a custom osmconf.ini. With that you can copy the config to a custom location and have several configs for many use cases.

 

HTH

Frank

 

Dipl. Geogr. Frank Broniewski
Waldhölzbacher Str. 51
66679 Losheim am See
06872 5090684
www.frankbroniewski.com

 

Von: [hidden email]
Gesendet: Dienstag, 7. November 2017 16:15
An: [hidden email]
Betreff: [gdal-dev] Extracting keys & values from OSM Planet

 

Dear OSM experts,

 

a reproducible workflow that extracts specific tags from OSM's Planet

data set, includes:

 

1. downloading OSM Planet in form of a pbf file

 

2. convert pbf to o5m using `osmconvert`

 

3  extract areas of interest in separate `aoi_*.o5m` files, based on boundaries in form of .poly

files, using `osmconvert`

 

4. filtering for specific tags (i.e. highways), parallel tasks in 80 cores

 

```bash

# custom function for 'highway' tags

function osmfilterhighway { osmfilter $1 --drop-version --keep="highway=track =footway =bridleway =path" > $(basename $1 .o5m)_highway.osm; }

export -f osmfilterhighway

 

# filter

find . -type f -name \*aoi*.o5m |parallel -j80 osmfilterhighway {}

```

 

1. Is it any better using GDAL's `osmconf.ini` "method" [0],

over `osmconvert` and `osmfilter`, in terms of reliability/speed etc.?

 

2. Is OGR handling well the conversion from .osm to ESRI Shapefiles?

 

3. What is the way to split "other_tags" in multiple new fields *when*

knowing exactly which tags are contained and should be obtained?

 

Thank you for any help,

 

Nikos

 

 

[0] https://svn.osgeo.org/gdal/trunk/gdal/data/osmconf.ini

 


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Extracting keys & values from OSM Planet

NikosAlexandris
Frank Broniewski:

>Hi Nikos,
>
>I am using a similar approach to yours to extract data from OSM files
>to a GIS format. I chose Spatiallite as a format since it is superior
>to the Shapefile format in all areas.  FYI here’s what I do:

>osmconvert -b="5.5,49,8,50.5" -o=saarland.osm.pbf "europe-latest.osm.pbf"

>ogr2ogr --config OSM_CONFIG_FILE /home/frank/gdal/osmconf.ini -f SQLite -dsco SPATIALITE=YES -progress -gt 65536 -t_srs "EPSG:31466" >/tmp/saarland.sqlite /tmp/saarland.osm.pbf

>Please note the “–config” switch for a custom osmconf.ini. With that
>you can copy the config to a custom location and have several configs
>for many use cases.

Thank you Even and Frank!

I will likely adopt the `osmconf.ini` method.

@Frank, I would go a step further if time permits. A function or script that
takes something like `layer=lines` and
`keys="track,footway,bridleway,path"` as input parameters and outputs the
custom configuration file.

From reading the driver's manual [0], there is a hint about the spatial,
filtering resulting in lines or polygons missing vertices. Is this a/the
reason, to stick to `osmconvert` for extracting areas of interest?

Thank you for your (invaluable) time, Nikos

[0] http://www.gdal.org/drv_osm.html

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev

signature.asc (235 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Extracting keys & values from OSM Planet

Frank Broniewski-5

Hi Nikos,

 

I think it would be quite easy to hack up a python script that generates a osmconf.ini with the configparser module and calls ogr2ogr via subprocess to process the data, but I don’t have done anything in that regard yet.

I don’t remember exactly anymore why I chose osmconvert over ogr2ogr for extracting a subregion from a larger PBF.  But as far as I remember osmconvert was superior in speed and accuracy for that task than ogr2ogr.

 

Dipl. Geogr. Frank Broniewski
Waldhölzbacher Str. 51
66679 Losheim am See
06872 5090684
www.frankbroniewski.com

 

Von: [hidden email]
Gesendet: Mittwoch, 8. November 2017 11:38
An: [hidden email]
Cc: [hidden email]
Betreff: Re: AW: [gdal-dev] Extracting keys & values from OSM Planet

 

Frank Broniewski:

 

>Hi Nikos,

> 

>I am using a similar approach to yours to extract data from OSM files

>to a GIS format. I chose Spatiallite as a format since it is superior

>to the Shapefile format in all areas.  FYI here’s what I do:

 

>osmconvert -b="5.5,49,8,50.5" -o=saarland.osm.pbf "europe-latest.osm.pbf"

 

>ogr2ogr --config OSM_CONFIG_FILE /home/frank/gdal/osmconf.ini -f SQLite -dsco SPATIALITE=YES -progress -gt 65536 -t_srs "EPSG:31466" >/tmp/saarland.sqlite /tmp/saarland.osm.pbf

 

>Please note the “–config” switch for a custom osmconf.ini. With that

>you can copy the config to a custom location and have several configs

>for many use cases.

 

Thank you Even and Frank!

 

I will likely adopt the `osmconf.ini` method.

 

@Frank, I would go a step further if time permits. A function or script that

takes something like `layer=lines` and

`keys="track,footway,bridleway,path"` as input parameters and outputs the

custom configuration file.

 

From reading the driver's manual [0], there is a hint about the spatial,

filtering resulting in lines or polygons missing vertices. Is this a/the

reason, to stick to `osmconvert` for extracting areas of interest?

 

Thank you for your (invaluable) time, Nikos

 

[0] http://www.gdal.org/drv_osm.html

 


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Extracting keys & values from OSM Planet

NikosAlexandris
In reply to this post by Even Rouault-2
* Even Rouault <[hidden email]> [2017-11-07 22:37:02 +0100]:

>> 2. Is OGR handling well the conversion from .osm to ESRI Shapefiles?
>
>Yes, but within the limits ot the shapefile, and particularly .dbf format: limitation to 254
>characters for field values, 10 characters for field names... which are easily violated by
>OSM extracts.
>
>Spatialite, GeoPackage, PostGIS etc. would be better choices as output format
>
>>
>> 3. What is the way to split "other_tags" in multiple new fields *when*
>> knowing exactly which tags are contained and should be obtained?
>
>Edit osmconf.ini to add in the attributes= settings the tag names you're interested in.
>
>Even
Dear List,

in a custom .ini file, under the [points] layer, an `attributes=`
instruction set of values that includes `natural,cave_entrance` (key and
value under OSM's Natural key), well extracts for the points layer some
existing nodes which bear the tag `natural:cave_entrance`.

It is not required, as far as I understand the documentation, to include both
the parent key and any values (under it) of interest. Perhaps it may be
practical to have the attributes of interest split in a separate column.

The command I tested, for example, is:

```
ogr2ogr --config OSM_CONFIG_FILE custom.ini -f SQLite -dsco SPATIALITE=YES -gt 65536 -progress output.sqlite source.osm
```

While this is sufficient for the work I am doing (I can filter
`cave_entrance`s out from the `natural` column in the `output.sqlite` file), I
wonder why the entries for some nodes correctly bear the string
`cave_entrance` in the `natural` column, yet the `cave_entrance` column
itself is empty for the same entries.

Thank you for any hints, Nikos

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev

signature.asc (235 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Extracting keys & values from OSM Planet

Even Rouault-2

On lundi 20 novembre 2017 15:48:20 CET Nikos Alexandris wrote:

> * Even Rouault <[hidden email]> [2017-11-07 22:37:02 +0100]:

> >> 2. Is OGR handling well the conversion from .osm to ESRI Shapefiles?

> >

> >Yes, but within the limits ot the shapefile, and particularly .dbf format:

> >limitation to 254 characters for field values, 10 characters for field

> >names... which are easily violated by OSM extracts.

> >

> >Spatialite, GeoPackage, PostGIS etc. would be better choices as output

> >format>

> >> 3. What is the way to split "other_tags" in multiple new fields *when*

> >> knowing exactly which tags are contained and should be obtained?

> >

> >Edit osmconf.ini to add in the attributes= settings the tag names you're

> >interested in.

> >

> >Even

>

> Dear List,

>

> in a custom .ini file, under the [points] layer, an `attributes=`

> instruction set of values that includes `natural,cave_entrance` (key and

> value under OSM's Natural key), well extracts for the points layer some

> existing nodes which bear the tag `natural:cave_entrance`.

>

> It is not required, as far as I understand the documentation, to include

> both the parent key and any values (under it) of interest. Perhaps it may

> be practical to have the attributes of interest split in a separate column.

>

> The command I tested, for example, is:

>

> ```

> ogr2ogr --config OSM_CONFIG_FILE custom.ini -f SQLite -dsco SPATIALITE=YES

> -gt 65536 -progress output.sqlite source.osm ```

>

> While this is sufficient for the work I am doing (I can filter

> `cave_entrance`s out from the `natural` column in the `output.sqlite` file),

> I wonder why the entries for some nodes correctly bear the string

> `cave_entrance` in the `natural` column, yet the `cave_entrance` column

> itself is empty for the same entries.

 

The attributes keyword in osmconf.ini will select OSM tags whose *key* is one of the item specified in attributes.

If you want to filter by value, you may add a -where or -sql clause with something like "natural = 'cave_entrance'"

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Extracting keys & values from OSM Planet

NikosAlexandris
* Even Rouault <[hidden email]> [2017-11-20 15:59:51 +0100]:

>On lundi 20 novembre 2017 15:48:20 CET Nikos Alexandris wrote:
>> * Even Rouault <[hidden email]> [2017-11-07 22:37:02 +0100]:
>> >> 2. Is OGR handling well the conversion from .osm to ESRI Shapefiles?
>> >
>> >Yes, but within the limits ot the shapefile, and particularly .dbf format:
>> >limitation to 254 characters for field values, 10 characters for field
>> >names... which are easily violated by OSM extracts.
>> >
>> >Spatialite, GeoPackage, PostGIS etc. would be better choices as output
>> >format>
>> >> 3. What is the way to split "other_tags" in multiple new fields *when*
>> >> knowing exactly which tags are contained and should be obtained?
>> >
>> >Edit osmconf.ini to add in the attributes= settings the tag names you're
>> >interested in.
>> >
>> >Even
>>
>> Dear List,
>>
>> in a custom .ini file, under the [points] layer, an `attributes=`
>> instruction set of values that includes `natural,cave_entrance` (key and
>> value under OSM's Natural key), well extracts for the points layer some
>> existing nodes which bear the tag `natural:cave_entrance`.
>>
>> It is not required, as far as I understand the documentation, to include
>> both the parent key and any values (under it) of interest. Perhaps it may
>> be practical to have the attributes of interest split in a separate column.
>>
>> The command I tested, for example, is:
>>
>> ```
>> ogr2ogr --config OSM_CONFIG_FILE custom.ini -f SQLite -dsco SPATIALITE=YES
>> -gt 65536 -progress output.sqlite source.osm ```
>>
>> While this is sufficient for the work I am doing (I can filter
>> `cave_entrance`s out from the `natural` column in the `output.sqlite` file),
>> I wonder why the entries for some nodes correctly bear the string
>> `cave_entrance` in the `natural` column, yet the `cave_entrance` column
>> itself is empty for the same entries.
>
>The attributes keyword in osmconf.ini will select OSM tags whose *key* is one of the item
>specified in attributes.
>If you want to filter by value, you may add a -where or -sql clause  with something like
>"natural = 'cave_entrance'"
>
>Even
Merci Even.

How does the driver handle the "too many tags in node/relation" errors?
Throw them away?

Nikos

>--
>Spatialys - Geospatial professional services
>http://www.spatialys.com

--
Nikos Alexandris | Remote Sensing & Geomatics
GPG Key Fingerprint 6F9D4506F3CA28380974D31A9053534B693C4FB3

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev

signature.asc (235 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Extracting keys & values from OSM Planet

NikosAlexandris
In reply to this post by Even Rouault-2
* Even Rouault <[hidden email]> [2017-11-20 15:59:51 +0100]:

>On lundi 20 novembre 2017 15:48:20 CET Nikos Alexandris wrote:
>> * Even Rouault <[hidden email]> [2017-11-07 22:37:02 +0100]:
>> >> 2. Is OGR handling well the conversion from .osm to ESRI Shapefiles?
>> >
>> >Yes, but within the limits ot the shapefile, and particularly .dbf format:
>> >limitation to 254 characters for field values, 10 characters for field
>> >names... which are easily violated by OSM extracts.
>> >
>> >Spatialite, GeoPackage, PostGIS etc. would be better choices as output
>> >format>
>> >> 3. What is the way to split "other_tags" in multiple new fields *when*
>> >> knowing exactly which tags are contained and should be obtained?
>> >
>> >Edit osmconf.ini to add in the attributes= settings the tag names you're
>> >interested in.
>> >
>> >Even
>>
>> Dear List,
>>
>> in a custom .ini file, under the [points] layer, an `attributes=`
>> instruction set of values that includes `natural,cave_entrance` (key and
>> value under OSM's Natural key), well extracts for the points layer some
>> existing nodes which bear the tag `natural:cave_entrance`.
>>
>> It is not required, as far as I understand the documentation, to include
>> both the parent key and any values (under it) of interest. Perhaps it may
>> be practical to have the attributes of interest split in a separate column.
>>
>> The command I tested, for example, is:
>>
>> ```
>> ogr2ogr --config OSM_CONFIG_FILE custom.ini -f SQLite -dsco SPATIALITE=YES
>> -gt 65536 -progress output.sqlite source.osm ```
>>
>> While this is sufficient for the work I am doing (I can filter
>> `cave_entrance`s out from the `natural` column in the `output.sqlite` file),
>> I wonder why the entries for some nodes correctly bear the string
>> `cave_entrance` in the `natural` column, yet the `cave_entrance` column
>> itself is empty for the same entries.
>
>The attributes keyword in osmconf.ini will select OSM tags whose *key* is one of the item
>specified in attributes.
>If you want to filter by value, you may add a -where or -sql clause  with something like
>"natural = 'cave_entrance'"
Apologies for yet another message on this. OSM needs its time to
understand. What is somewhat confusing is the fact that some values,
e.g. 'drinking_water' for keys, i.e. 'natural', are also keys
themselves.  Anothr example is 'parking'. It can be the value in
the 'amenity=parking' tag or itself, being a key, "contains"
(sub-)values such as 'surface', 'multi-storey' and more.

It really takes to go through each and every key in OSM and see what
is of interest.

Nikos

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev

signature.asc (235 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Extracting keys & values from OSM Planet

NikosAlexandris
In reply to this post by Even Rouault-2
* Even Rouault <[hidden email]> [2017-11-20 15:59:51 +0100]:

>On lundi 20 novembre 2017 15:48:20 CET Nikos Alexandris wrote:
>> * Even Rouault <[hidden email]> [2017-11-07 22:37:02 +0100]:
>> >> 2. Is OGR handling well the conversion from .osm to ESRI Shapefiles?
>> >
>> >Yes, but within the limits ot the shapefile, and particularly .dbf format:
>> >limitation to 254 characters for field values, 10 characters for field
>> >names... which are easily violated by OSM extracts.
>> >
>> >Spatialite, GeoPackage, PostGIS etc. would be better choices as output
>> >format>
>> >> 3. What is the way to split "other_tags" in multiple new fields *when*
>> >> knowing exactly which tags are contained and should be obtained?
>> >
>> >Edit osmconf.ini to add in the attributes= settings the tag names you're
>> >interested in.
>> >
>> >Even
>>
>> Dear List,
>>
>> in a custom .ini file, under the [points] layer, an `attributes=`
>> instruction set of values that includes `natural,cave_entrance` (key and
>> value under OSM's Natural key), well extracts for the points layer some
>> existing nodes which bear the tag `natural:cave_entrance`.
>>
>> It is not required, as far as I understand the documentation, to include
>> both the parent key and any values (under it) of interest. Perhaps it may
>> be practical to have the attributes of interest split in a separate column.
>>
>> The command I tested, for example, is:
>>
>> ```
>> ogr2ogr --config OSM_CONFIG_FILE custom.ini -f SQLite -dsco SPATIALITE=YES
>> -gt 65536 -progress output.sqlite source.osm ```
>>
>> While this is sufficient for the work I am doing (I can filter
>> `cave_entrance`s out from the `natural` column in the `output.sqlite` file),
>> I wonder why the entries for some nodes correctly bear the string
>> `cave_entrance` in the `natural` column, yet the `cave_entrance` column
>> itself is empty for the same entries.
>
>The attributes keyword in osmconf.ini will select OSM tags whose *key* is one of the item
>specified in attributes.
>If you want to filter by value, you may add a -where or -sql clause  with something like
>"natural = 'cave_entrance'"
This works!

Yet, I can't seem to find an answer on the web, nor a solution by trying. I'd
like to perform SQL queries in one go: one querying the points layer and another
one the lines layer.

This is certainly not a JOIN operation, as far as I understand. And
UNION only works in data sourced from the same table/layer. Is it doable
to merge an

SELECT *
FROM points
WHERE ...

and an

SELECT *
FROM lines
WHERE ...

query?

Thank you, Nikos

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev

signature.asc (235 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Extracting keys & values from OSM Planet

Even Rouault-2

On mercredi 22 novembre 2017 14:14:38 CET Nikos Alexandris wrote:

> * Even Rouault <[hidden email]> [2017-11-20 15:59:51 +0100]:

> >On lundi 20 novembre 2017 15:48:20 CET Nikos Alexandris wrote:

> >> * Even Rouault <[hidden email]> [2017-11-07 22:37:02 +0100]:

> >> >> 2. Is OGR handling well the conversion from .osm to ESRI Shapefiles?

> >> >

> >> >Yes, but within the limits ot the shapefile, and particularly .dbf

> >> >format:

> >> >limitation to 254 characters for field values, 10 characters for field

> >> >names... which are easily violated by OSM extracts.

> >> >

> >> >Spatialite, GeoPackage, PostGIS etc. would be better choices as output

> >> >format>

> >> >

> >> >> 3. What is the way to split "other_tags" in multiple new fields *when*

> >> >> knowing exactly which tags are contained and should be obtained?

> >> >

> >> >Edit osmconf.ini to add in the attributes= settings the tag names you're

> >> >interested in.

> >> >

> >> >Even

> >>

> >> Dear List,

> >>

> >> in a custom .ini file, under the [points] layer, an `attributes=`

> >> instruction set of values that includes `natural,cave_entrance` (key and

> >> value under OSM's Natural key), well extracts for the points layer some

> >> existing nodes which bear the tag `natural:cave_entrance`.

> >>

> >> It is not required, as far as I understand the documentation, to include

> >> both the parent key and any values (under it) of interest. Perhaps it may

> >> be practical to have the attributes of interest split in a separate

> >> column.

> >>

> >> The command I tested, for example, is:

> >>

> >> ```

> >> ogr2ogr --config OSM_CONFIG_FILE custom.ini -f SQLite -dsco

> >> SPATIALITE=YES

> >> -gt 65536 -progress output.sqlite source.osm ```

> >>

> >> While this is sufficient for the work I am doing (I can filter

> >> `cave_entrance`s out from the `natural` column in the `output.sqlite`

> >> file), I wonder why the entries for some nodes correctly bear the string

> >> `cave_entrance` in the `natural` column, yet the `cave_entrance` column

> >> itself is empty for the same entries.

> >

> >The attributes keyword in osmconf.ini will select OSM tags whose *key* is

> >one of the item specified in attributes.

> >If you want to filter by value, you may add a -where or -sql clause with

> >something like "natural = 'cave_entrance'"

>

> This works!

>

> Yet, I can't seem to find an answer on the web, nor a solution by trying.

> I'd like to perform SQL queries in one go: one querying the points layer

> and another one the lines layer.

>

> This is certainly not a JOIN operation, as far as I understand. And

> UNION only works in data sourced from the same table/layer. Is it doable

> to merge an

>

> SELECT *

> FROM points

> WHERE ...

>

> and an

>

> SELECT *

> FROM lines

> WHERE ...

>

> query?

 

You can't do that in a single operation with ogr2ogr. You may use ogr2ogr for each layer, with -append if you need to append to an existing output file

 

Or if you want to do in a single operation. You can use a OGR VRT

( http://gdal.org/drv_vrt.html ) file like the following one (customize the name of the datasource and the SQL expressions) as the input file of ogr2ogr

 

<OGRVRTDataSource>

<OGRVRTLayer name="points">

<SrcDataSource relativeToVRT="1" shared="0">in.osm</SrcDataSource>

<SrcSQL>SELECT * FROM points</SrcSQL>

<GeometryType>wkbPoint</GeometryType>

<LayerSRS>WGS84</LayerSRS>

</OGRVRTLayer>

<OGRVRTLayer name="lines">

<SrcDataSource relativeToVRT="1" shared="0">in.osm</SrcDataSource>

<SrcSQL>SELECT * FROM lines</SrcSQL>

<GeometryType>wkbLineString</GeometryType>

<LayerSRS>WGS84</LayerSRS>

</OGRVRTLayer>

<OGRVRTLayer name="multilinestrings">

<SrcDataSource relativeToVRT="1" shared="0">in.osm</SrcDataSource>

<SrcSQL>SELECT * FROM multilinestrings</SrcSQL>

<GeometryType>wkbMultiLineString</GeometryType>

<LayerSRS>WGS84</LayerSRS>

</OGRVRTLayer>

<OGRVRTLayer name="multipolygons">

<SrcDataSource relativeToVRT="1" shared="0">in.osm</SrcDataSource>

<SrcSQL>SELECT * FROM multipolygons</SrcSQL>

<GeometryType>wkbMultiPolygon</GeometryType>

<LayerSRS>WGS84</LayerSRS>

</OGRVRTLayer>

<OGRVRTLayer name="other_relations">

<SrcDataSource relativeToVRT="1" shared="0">in.osm</SrcDataSource>

<SrcSQL>SELECT * FROM other_relations</SrcSQL>

<GeometryType>wkbGeometryCollection</GeometryType>

<LayerSRS>WGS84</LayerSRS>

</OGRVRTLayer>

</OGRVRTDataSource>

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Extracting keys & values from OSM Planet

NikosAlexandris
* Even Rouault <[hidden email]> [2017-11-22 14:30:23 +0100]:

>On mercredi 22 novembre 2017 14:14:38 CET Nikos Alexandris wrote:
>> * Even Rouault <[hidden email]> [2017-11-20 15:59:51 +0100]:
>> >On lundi 20 novembre 2017 15:48:20 CET Nikos Alexandris wrote:
>> >> * Even Rouault <[hidden email]> [2017-11-07 22:37:02 +0100]:
>> >> >> 2. Is OGR handling well the conversion from .osm to ESRI Shapefiles?
>> >> >
>> >> >Yes, but within the limits ot the shapefile, and particularly .dbf
>> >> >format:
>> >> >limitation to 254 characters for field values, 10 characters for field
>> >> >names... which are easily violated by OSM extracts.
>> >> >
>> >> >Spatialite, GeoPackage, PostGIS etc. would be better choices as output
>> >> >format>
>> >> >
>> >> >> 3. What is the way to split "other_tags" in multiple new fields *when*
>> >> >> knowing exactly which tags are contained and should be obtained?
>> >> >
>> >> >Edit osmconf.ini to add in the attributes= settings the tag names you're
>> >> >interested in.
>> >> >
>> >> >Even
>> >>
>> >> Dear List,
>> >>
>> >> in a custom .ini file, under the [points] layer, an `attributes=`
>> >> instruction set of values that includes `natural,cave_entrance` (key and
>> >> value under OSM's Natural key), well extracts for the points layer some
>> >> existing nodes which bear the tag `natural:cave_entrance`.
>> >>
>> >> It is not required, as far as I understand the documentation, to include
>> >> both the parent key and any values (under it) of interest. Perhaps it may
>> >> be practical to have the attributes of interest split in a separate
>> >> column.
>> >>
>> >> The command I tested, for example, is:
>> >>
>> >> ```
>> >> ogr2ogr --config OSM_CONFIG_FILE custom.ini -f SQLite -dsco
>> >> SPATIALITE=YES
>> >> -gt 65536 -progress output.sqlite source.osm ```
>> >>
>> >> While this is sufficient for the work I am doing (I can filter
>> >> `cave_entrance`s out from the `natural` column in the `output.sqlite`
>> >> file), I wonder why the entries for some nodes correctly bear the string
>> >> `cave_entrance` in the `natural` column, yet the `cave_entrance` column
>> >> itself is empty for the same entries.
>> >
>> >The attributes keyword in osmconf.ini will select OSM tags whose *key* is
>> >one of the item specified in attributes.
>> >If you want to filter by value, you may add a -where or -sql clause  with
>> >something like "natural = 'cave_entrance'"
>>
>> This works!
>>
>> Yet, I can't seem to find an answer on the web, nor a solution by trying.
>> I'd like to perform SQL queries in one go: one querying the points layer
>> and another one the lines layer.
>>
>> This is certainly not a JOIN operation, as far as I understand. And
>> UNION only works in data sourced from the same table/layer. Is it doable
>> to merge an
>>
>> SELECT *
>> FROM points
>> WHERE ...
>>
>> and an
>>
>> SELECT *
>> FROM lines
>> WHERE ...
>>
>> query?
>
>You can't do that in a single operation with ogr2ogr. You may use ogr2ogr for each layer, with
>-append if you need to append to an existing output file
>
>Or if you want to do in a single operation. You can use a OGR VRT
>( http://gdal.org/drv_vrt.html ) file like the following one (customize the name of the
>datasource and the SQL expressions) as the input file of ogr2ogr
>
><OGRVRTDataSource>
>  <OGRVRTLayer name="points">
>    <SrcDataSource relativeToVRT="1" shared="0">in.osm</SrcDataSource>
>    <SrcSQL>SELECT * FROM points</SrcSQL>
>    <GeometryType>wkbPoint</GeometryType>
>    <LayerSRS>WGS84</LayerSRS>
>  </OGRVRTLayer>
>  <OGRVRTLayer name="lines">
>    <SrcDataSource relativeToVRT="1" shared="0">in.osm</SrcDataSource>
>    <SrcSQL>SELECT * FROM lines</SrcSQL>
>    <GeometryType>wkbLineString</GeometryType>
>    <LayerSRS>WGS84</LayerSRS>
>  </OGRVRTLayer>
>  <OGRVRTLayer name="multilinestrings">
>    <SrcDataSource relativeToVRT="1" shared="0">in.osm</SrcDataSource>
>    <SrcSQL>SELECT * FROM multilinestrings</SrcSQL>
>    <GeometryType>wkbMultiLineString</GeometryType>
>    <LayerSRS>WGS84</LayerSRS>
>  </OGRVRTLayer>
>  <OGRVRTLayer name="multipolygons">
>    <SrcDataSource relativeToVRT="1" shared="0">in.osm</SrcDataSource>
>    <SrcSQL>SELECT * FROM multipolygons</SrcSQL>
>    <GeometryType>wkbMultiPolygon</GeometryType>
>    <LayerSRS>WGS84</LayerSRS>
>  </OGRVRTLayer>
>  <OGRVRTLayer name="other_relations">
>    <SrcDataSource relativeToVRT="1" shared="0">in.osm</SrcDataSource>
>    <SrcSQL>SELECT * FROM other_relations</SrcSQL>
>    <GeometryType>wkbGeometryCollection</GeometryType>
>    <LayerSRS>WGS84</LayerSRS>
>  </OGRVRTLayer>
></OGRVRTDataSource>
Magnifique!  Both work as expected.


Some bash functions:

For a single operation, given a custom osmconf.ini file

# Syntax: osm2sqlite  INI-FILE  SQL-CLAUSE  SOURCE.OSM
# Reusing parts from INI and SQL filenames for the output file
function osm2sqlite { ogr2ogr --config OSM_CONFIG_FILE $1 -sql @"$2" -f SQLite -dsco SPATIALITE=YES -gt 65536 -progress $(basename $3 .osm)_$(basename $1 .ini)_${2#*.}.sqlite $3; };


For two operations

# First filtering
# Syntax: osm2sqlite  INI-FILE  SQL-CLAUSE  SOURCE.OSM
function osm2sqlite { ogr2ogr --config OSM_CONFIG_FILE $1 -sql @"$2" -f SQLite -dsco SPATIALITE=YES -gt 65536 -progress $(basename $3 .osm)_$(basename $1 .ini).sqlite $3; };

# Filter and append to output file of the above command, given the same source file is used
# Syntax: osm2sqlite.append  INI-FILE  SQL-CLAUSE  SOURCE.OSM
function osm2sqlite.append { ogr2ogr -append --config OSM_CONFIG_FILE $1 -sql @"$2" -f SQLite -dsco SPATIALITE=YES -gt 65536 -progress $(basename $3 .osm)_$(basename $1 .ini).sqlite $3; };


Similar for the VRT file, just skipping the '-sql' part. For
completeness, I kept only points and lines in the VRT file, since I am
not interested, at the moment, for the rest of the features.

Merci Even u. Danke Frank,

Nikos

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev

signature.asc (235 bytes) Download Attachment