[gdal-dev] Question about command line options -append

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] Question about command line options -append

Victor Chernetsky
Hello,

How do I get value/state of a command line option of ogr2ogr. For instance I want to handle options like -append, -update inside of my DataSource class? Is there a documentation about it?

Thanks,
Victor Chernetsky

+1 (408) 368-4607

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Question about command line options -append

Even Rouault-2

On mardi 28 novembre 2017 15:00:11 CET Victor Chernetsky wrote:

> Hello,

>

> How do I get value/state of a command line option of ogr2ogr. For instance

> I want to handle options like -append, -update inside of my DataSource

> class? Is there a documentation about it?

 

Victor,

 

-append and -update are just ways of operating of ogr2ogr and the drivers are not aware of them. The consequence are just how ogr2ogr use the driver API.

 

* -update: open the dataset in update mode

 

* -append imply -update, and instruct ogr2ogr to reuse exisisting layers instead of erroring out if they exist.

 

In both cases, features will get inserted through ICreateFeature()

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Question about command line options -append

Victor Chernetsky
Hi Even,

Then in my driver, how do I know if I have to append data to existing dataset, or overwrite existing data, or create a new dataset all together?

What about -overwrite option?

Thanks,

Victor Chernetsky


+1 (408) 368-4607

On Tue, Nov 28, 2017 at 3:15 PM, Even Rouault <[hidden email]> wrote:

On mardi 28 novembre 2017 15:00:11 CET Victor Chernetsky wrote:

> Hello,

>

> How do I get value/state of a command line option of ogr2ogr. For instance

> I want to handle options like -append, -update inside of my DataSource

> class? Is there a documentation about it?

 

Victor,

 

-append and -update are just ways of operating of ogr2ogr and the drivers are not aware of them. The consequence are just how ogr2ogr use the driver API.

 

* -update: open the dataset in update mode

 

* -append imply -update, and instruct ogr2ogr to reuse exisisting layers instead of erroring out if they exist.

 

In both cases, features will get inserted through ICreateFeature()

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com



_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Question about command line options -append

Even Rouault-2

On mardi 28 novembre 2017 15:20:20 CET Victor Chernetsky wrote:

> Hi Even,

>

> Then in my driver, how do I know if I have to append data to existing

> dataset,

> or overwrite existing data, or create a new dataset all together?

 

That should work automagically if you correctly implement the semantics of the

relevant dataset and layer virtual methods

 

If the user doesn't specify -append or -overwrite, but just -update, and the layer doesn't exist, then ogr2ogr will call GDALDataset::ICreateLayer()

 

>

> What about -overwrite option?

 

If the user specify -overwrite, then ogr2ogr will call GDALDataset::DeleteLayer(), and then ICreateLayer()

 

Looking quickly at the amigocloud driver, everything looks in place already.

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Question about command line options -append

Victor Chernetsky
Here is my example:
ogr2ogr -overwrite -f AmigoCloud "AmigoCloud:15276 datasets=100056" ~/Downloads/Visits.geojson

I am trying to append, or overwrite data into existing dataset id:100056. But it still created a new dataset, and it did not delete/clean old dataset. I am thinking I need to do something in my OGRAmigoCloudDataSource::ICreateLayer(). I am not checking if dataset exist. I always create a new one. 

How should I handle this case?


Victor Chernetsky


+1 (408) 368-4607

On Tue, Nov 28, 2017 at 3:30 PM, Even Rouault <[hidden email]> wrote:

On mardi 28 novembre 2017 15:20:20 CET Victor Chernetsky wrote:

> Hi Even,

>

> Then in my driver, how do I know if I have to append data to existing

> dataset,

> or overwrite existing data, or create a new dataset all together?

 

That should work automagically if you correctly implement the semantics of the

relevant dataset and layer virtual methods

 

If the user doesn't specify -append or -overwrite, but just -update, and the layer doesn't exist, then ogr2ogr will call GDALDataset::ICreateLayer()

 

>

> What about -overwrite option?

 

If the user specify -overwrite, then ogr2ogr will call GDALDataset::DeleteLayer(), and then ICreateLayer()

 

Looking quickly at the amigocloud driver, everything looks in place already.

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com



_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Question about command line options -append

Even Rouault-2

On mardi 28 novembre 2017 15:40:46 CET Victor Chernetsky wrote:

> Here is my example:

> ogr2ogr -overwrite -f AmigoCloud "AmigoCloud:15276 datasets=100056"

> ~/Downloads/Visits.geojson

>

> I am trying to append, or overwrite data into existing dataset id:100056.

> But it still created a new dataset, and it did not delete/clean old

> dataset.

 

-overwrite overwrite *layers* not datasets

Here it will probably create a new layer because the layer name of Visits.geojson

must be "Visits" and probably doesn't exist yet in your dataset.

 

If you want to overwrite an existing layer foo with the geojson, you must add "-nln foo"

 

> I am thinking I need to do something in

> my OGRAmigoCloudDataSource::ICreateLayer(). I am not checking if dataset

> exist. I always create a new one.

 

You can have a look at the ogr2ogr logic to decide how to overwrite a layer:

https://github.com/OSGeo/gdal/blob/trunk/gdal/apps/ogr2ogr_lib.cpp#L3180

 

For non-ogr2ogr clients, you could indeed add a check if a layer of same name already exists

A number of drivers also support a OVERWRITE=YES layer creation option

See

https://github.com/OSGeo/gdal/blob/trunk/gdal/ogr/ogrsf_frmts/carto/ogrcartodatasource.cpp#L412

 

The advantage when you write an example is that you have tens of examples to get inspiration from ;-)

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Question about command line options -append

Victor Chernetsky
Thanks, this helps! 

The thing is I cannot really use layer name to identify an AmigoCloud dataset. In AmigoCloud we have names, but they are not guarantied to be unique. We use dataset Id to identify a dataset (ogr layer) instead. In my example "Visits" it's just a name of the source data file, which gdal uses as a layer name. This is fine, but for destination AmigoCloud dataset I specify a dataset id explicitly anyway. So I know my destination dataset, and if it exists or not. I think with the info I have now I could do few small changes to handle these cases.



Victor Chernetsky


+1 (408) 368-4607

On Tue, Nov 28, 2017 at 4:01 PM, Even Rouault <[hidden email]> wrote:

On mardi 28 novembre 2017 15:40:46 CET Victor Chernetsky wrote:

> Here is my example:

> ogr2ogr -overwrite -f AmigoCloud "AmigoCloud:15276 datasets=100056"

> ~/Downloads/Visits.geojson

>

> I am trying to append, or overwrite data into existing dataset id:100056.

> But it still created a new dataset, and it did not delete/clean old

> dataset.

 

-overwrite overwrite *layers* not datasets

Here it will probably create a new layer because the layer name of Visits.geojson

must be "Visits" and probably doesn't exist yet in your dataset.

 

If you want to overwrite an existing layer foo with the geojson, you must add "-nln foo"

 

> I am thinking I need to do something in

> my OGRAmigoCloudDataSource::ICreateLayer(). I am not checking if dataset

> exist. I always create a new one.

 

You can have a look at the ogr2ogr logic to decide how to overwrite a layer:

https://github.com/OSGeo/gdal/blob/trunk/gdal/apps/ogr2ogr_lib.cpp#L3180

 

For non-ogr2ogr clients, you could indeed add a check if a layer of same name already exists

A number of drivers also support a OVERWRITE=YES layer creation option

See

https://github.com/OSGeo/gdal/blob/trunk/gdal/ogr/ogrsf_frmts/carto/ogrcartodatasource.cpp#L412

 

The advantage when you write an example is that you have tens of examples to get inspiration from ;-)

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com



_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Question about command line options -append

Victor Chernetsky
Hey Even,

How do I set OVERWRITE option? 
I tried all these:
ogr2ogr -overwrite -f AmigoCloud "AmigoCloud:15276 datasets=100322" ~/Downloads/Visits.geojson
ogr2ogr --config OVERWRITE YES -f AmigoCloud "AmigoCloud:15276 datasets=100322" ~/Downloads/Visits.geojson
ogr2ogr -oo OVERWRITE=YES -f AmigoCloud "AmigoCloud:15276 datasets=100322" ~/Downloads/Visits.geojson

In my driver CSLFetchNameValue( papszOptions, "OVERWRITE" ) is always NULL, and DeleteLayer(int iLayer) never gets called.

How do switch ogr2ogr onto overwrite mode? Or how do I catch it in my driver?

Thanks,


Victor Chernetsky


+1 (408) 368-4607

On Tue, Nov 28, 2017 at 4:26 PM, Victor Chernetsky <[hidden email]> wrote:
Thanks, this helps! 

The thing is I cannot really use layer name to identify an AmigoCloud dataset. In AmigoCloud we have names, but they are not guarantied to be unique. We use dataset Id to identify a dataset (ogr layer) instead. In my example "Visits" it's just a name of the source data file, which gdal uses as a layer name. This is fine, but for destination AmigoCloud dataset I specify a dataset id explicitly anyway. So I know my destination dataset, and if it exists or not. I think with the info I have now I could do few small changes to handle these cases.



Victor Chernetsky


+1 <a href="tel:(408)%20368-4607" value="+14083684607" target="_blank">(408) 368-4607

On Tue, Nov 28, 2017 at 4:01 PM, Even Rouault <[hidden email]> wrote:

On mardi 28 novembre 2017 15:40:46 CET Victor Chernetsky wrote:

> Here is my example:

> ogr2ogr -overwrite -f AmigoCloud "AmigoCloud:15276 datasets=100056"

> ~/Downloads/Visits.geojson

>

> I am trying to append, or overwrite data into existing dataset id:100056.

> But it still created a new dataset, and it did not delete/clean old

> dataset.

 

-overwrite overwrite *layers* not datasets

Here it will probably create a new layer because the layer name of Visits.geojson

must be "Visits" and probably doesn't exist yet in your dataset.

 

If you want to overwrite an existing layer foo with the geojson, you must add "-nln foo"

 

> I am thinking I need to do something in

> my OGRAmigoCloudDataSource::ICreateLayer(). I am not checking if dataset

> exist. I always create a new one.

 

You can have a look at the ogr2ogr logic to decide how to overwrite a layer:

https://github.com/OSGeo/gdal/blob/trunk/gdal/apps/ogr2ogr_lib.cpp#L3180

 

For non-ogr2ogr clients, you could indeed add a check if a layer of same name already exists

A number of drivers also support a OVERWRITE=YES layer creation option

See

https://github.com/OSGeo/gdal/blob/trunk/gdal/ogr/ogrsf_frmts/carto/ogrcartodatasource.cpp#L412

 

The advantage when you write an example is that you have tens of examples to get inspiration from ;-)

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com




_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Question about command line options -append

Even Rouault-2

On mercredi 29 novembre 2017 12:00:17 CET Victor Chernetsky wrote:

> Hey Even,

>

> How do I set OVERWRITE option?

 

With -lco OVERWRITE=YES since this is a layer creation option

 

> I tried all these:

> ogr2ogr -overwrite -f AmigoCloud "AmigoCloud:15276 datasets=100322"

 

This will trigger the logic at

https://github.com/OSGeo/gdal/blob/trunk/gdal/apps/ogr2ogr_lib.cpp#L3180

that first try to see if a layer has the same name as the input layer name, and if so call DeleteLayer() on it

 

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Question about command line options -append

Victor Chernetsky
Thanks, this worked! 

I have a followup question: how does gdal/ogr2ogr knows that my destination dataset/layer exists?
For instance, -append option suppose to do different things if layer exists, or not. I don't see any isExists() APIs that would check it.

Thanks,
Victor Chernetsky


+1 (408) 368-4607

On Wed, Nov 29, 2017 at 12:02 PM, Even Rouault <[hidden email]> wrote:

On mercredi 29 novembre 2017 12:00:17 CET Victor Chernetsky wrote:

> Hey Even,

>

> How do I set OVERWRITE option?

 

With -lco OVERWRITE=YES since this is a layer creation option

 

> I tried all these:

> ogr2ogr -overwrite -f AmigoCloud "AmigoCloud:15276 datasets=100322"

 

This will trigger the logic at

https://github.com/OSGeo/gdal/blob/trunk/gdal/apps/ogr2ogr_lib.cpp#L3180

that first try to see if a layer has the same name as the input layer name, and if so call DeleteLayer() on it

 

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com



_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Question about command line options -append

Even Rouault-2

On mercredi 29 novembre 2017 13:33:17 CET Victor Chernetsky wrote:

> Thanks, this worked!

>

> I have a followup question: how does gdal/ogr2ogr knows that my destination

> dataset/layer exists?

 

For the dataset, it tries to GDALOpen() it.

 

For the layer, by iterating over layers with GetLayer() and checking their names against the target layer name. See the portion of ogr2ogr I pointed to.

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev