Memory data provider persistence

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
32 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Memory data provider persistence

Chris Crook
Hi All

I'm looking for ideas/suggestions/comments on the memory data provider.

A while ago I raised a ticket http://trac.osgeo.org/qgis/ticket/2487, the essence of which is that if you save and reload a project nothing should have changed, but if you have a layer supported by a memory data provider then its contents get lost.

In the ticket I suggested that the user should be warned on saving, and the layers not included in the project file.  That way at least you don't end up with empty layers.

I'm now coming to a different point of view, which is that it would make much more sense to save the data for the memory provider into the project file.  Basically this would involve the following:

1) On saving a project, the data would be serialized into an element, probably in the map layer element in the project file.  For example it could be a <data...> node.  This could include the attribute definitions as well as each feature, or perhaps better the attribute definitions could be encoded into the datasourceUri.

2) On loading the data node would be read to repopulate the provider.  

Also useful, but not directly related to this, would be to enhance the layer "Save as" function so that when it is invoked on a memory layer the memory layer is replaced with the saved version, maybe as an option.

In terms of implementation I think this could involve the following components:

1) Add virtual functions to the QgsDataProvider writeDataXML(QDomNode &layerNode) and readDataXml( QDomNode &layerNode) with default empty implementations.  Call these functions in the QgsVectorLayer writeXml and readXml functions.

2) Add a signal dataSectionDirty() to the QgsDataProvider, so that edits to the data in a memory provider could render the project dirty.  This signal would ultimately need to be linked to the project dirty() function.

2) Add serialization/deserialisation functions to QgsFeature, and QgsAttributeMap to read and write the data for a memory provider.  There would be a question with this as to how much to structure the data for "human" consumption.  Should the geometry be WKT or WKB?  Should the attributes be simply serialized into a QDataStream and then encoded into XML, or should they be dumped more intelligently as structured XML elements.

3) Implement the readDataXml(), writeDataXml() functions in QgsMemoryProvider, and fire the dataSectionDirty() signal whenever the data is changed.

I guess the main issue with this is that the project file would potentially get a lot larger when large memory data sets are added (eg Contour plugin).  But I think that is better than silently losing the work held in a memory data provider.

Any thoughts on these suggestions??

Thanks
Chris
______________________________________________________________________________________________________

This message contains information, which is confidential and may be subject to legal privilege.
If you are not the intended recipient, you must not peruse, use, disseminate, distribute or copy this message.
If you have received this message in error, please notify us immediately (Phone 0800 665 463 or [hidden email]) and destroy the original message.
LINZ accepts no responsibility for changes to this email, or for any attachments, after its transmission from LINZ.

Thank you.
______________________________________________________________________________________________________
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Memory data provider persistence

Martin Dobias
Hi Chris

On Mon, Nov 29, 2010 at 8:54 PM, Chris Crook <[hidden email]> wrote:
> Hi All
>
> I'm looking for ideas/suggestions/comments on the memory data provider.
>
> A while ago I raised a ticket http://trac.osgeo.org/qgis/ticket/2487, the essence of which is that if you save and reload a project nothing should have changed, but if you have a layer supported by a memory data provider then its contents get lost.
>
> In the ticket I suggested that the user should be warned on saving, and the layers not included in the project file.  That way at least you don't end up with empty layers.
>
> I'm now coming to a different point of view, which is that it would make much more sense to save the data for the memory provider into the project file.

Having a possibility to save the data into a project would be a nice
feature. However I think that saving the features directly into the
XML project file wouldn't scale well with the amount of data. It can
work well for small datasets, though I'm afraid that it will fail with
a lot of data. One reason is that the project file would get pretty
big due the verbosity of XML, another problem would be with the memory
footprint and parsing speed - we're using DOM.

I have started to think about a more radical step: to allow the
projects to be not only XML files, but additionally they could be e.g.
ZIP files with project XML file inside and optionally some more files.
AFAIK OpenDocument format (used in OpenOffice) does this too.
Internally QGIS could ask the user and convert his data from memory
provider (or any other provider) to e.g. spatialite database that
would be included in that project ZIP file.

This would involve the following changes:
- implement reading and writing of compressed project files (using zlib)
- write a generic QgsVectorLayer -> SpatiaLite export class (similar
to QgsVectorFileWriter or an extension of it)
- add support to read/write "embedded" layers

This would also enable a nice feature to let QGIS pack all data into
one (compressed) file and pass just that file around!

What do you (and others) think of such solution?

Regards
Martin
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

RE: Memory data provider persistence

Chris Crook
Hi Martin and list readers

Interesting idea!  This would certainly achieve my main goal, which is to have the project reopen with the same information in as when I closed it.  

I agree that saving the the XML file could be cumbersome - even though these are only memory provider layers I guess they could get quite big.  And very good point about loading from the DOM.

Another way to achieve a similar result could be to embed the project file in the spatialite database?  That would make the memory provider data easier to use - basically it just becomes a spatialite provided layer.  The project itself becomes harder to manually edit, if you wanted to do so.  However it could easily be exported to a separate file if that is required.  And the whole thing could still be zipped if you wanted to.

I don't see these options as exclusive.  

Look forward to comments,
Chris  

________________________________________
From: Martin Dobias [[hidden email]]
Sent: 03 December 2010 11:01
To: Chris Crook
Cc: [hidden email]
Subject: Re: [Qgis-developer] Memory data provider persistence

Hi Chris

On Mon, Nov 29, 2010 at 8:54 PM, Chris Crook <[hidden email]> wrote:
> Hi All
>
> I'm looking for ideas/suggestions/comments on the memory data provider.
>
> A while ago I raised a ticket http://trac.osgeo.org/qgis/ticket/2487, the essence of which is that if you save and reload a project nothing should have changed, but if you have a layer supported by a memory data provider then its contents get lost.
>
> In the ticket I suggested that the user should be warned on saving, and the layers not included in the project file.  That way at least you don't end up with empty layers.
>
> I'm now coming to a different point of view, which is that it would make much more sense to save the data for the memory provider into the project file.

Having a possibility to save the data into a project would be a nice
feature. However I think that saving the features directly into the
XML project file wouldn't scale well with the amount of data. It can
work well for small datasets, though I'm afraid that it will fail with
a lot of data. One reason is that the project file would get pretty
big due the verbosity of XML, another problem would be with the memory
footprint and parsing speed - we're using DOM.

I have started to think about a more radical step: to allow the
projects to be not only XML files, but additionally they could be e.g.
ZIP files with project XML file inside and optionally some more files.
AFAIK OpenDocument format (used in OpenOffice) does this too.
Internally QGIS could ask the user and convert his data from memory
provider (or any other provider) to e.g. spatialite database that
would be included in that project ZIP file.

This would involve the following changes:
- implement reading and writing of compressed project files (using zlib)
- write a generic QgsVectorLayer -> SpatiaLite export class (similar
to QgsVectorFileWriter or an extension of it)
- add support to read/write "embedded" layers

This would also enable a nice feature to let QGIS pack all data into
one (compressed) file and pass just that file around!

What do you (and others) think of such solution?

Regards
Martin
______________________________________________________________________________________________________

This message contains information, which is confidential and may be subject to legal privilege.
If you are not the intended recipient, you must not peruse, use, disseminate, distribute or copy this message.
If you have received this message in error, please notify us immediately (Phone 0800 665 463 or [hidden email]) and destroy the original message.
LINZ accepts no responsibility for changes to this email, or for any attachments, after its transmission from LINZ.

Thank you.
______________________________________________________________________________________________________
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Memory data provider persistence

Nathan Woodrow
Hi all,

This is a very interesting idea indeed.  My personal preference would be a zip file with the xml file inside for the project file, and other files for the extra stuff eg memory layers, styles, images.

So:

MyProject.zqgs <- zip file with custom extension.
        |
        |> MyProject.qgs <- Project file
        |
        |> MyProectMemory.sqllte <- spatialite database
        |
        |> Other files, maybe some project specific qml files, images etc

My reason for using just a plan zip file with everything in it would be; it makes it easier to get to the project file if needed; allows for as many items as needed in the future; don't need anything special to view the content. 

As far as I am aware Steam, the gaming engine, does the same kind of thing with its game files.

Just my 2 cents worth.  

- Nathan

On Sat, Dec 4, 2010 at 1:35 PM, Chris Crook <[hidden email]> wrote:
Hi Martin and list readers

Interesting idea!  This would certainly achieve my main goal, which is to have the project reopen with the same information in as when I closed it.

I agree that saving the the XML file could be cumbersome - even though these are only memory provider layers I guess they could get quite big.  And very good point about loading from the DOM.

Another way to achieve a similar result could be to embed the project file in the spatialite database?  That would make the memory provider data easier to use - basically it just becomes a spatialite provided layer.  The project itself becomes harder to manually edit, if you wanted to do so.  However it could easily be exported to a separate file if that is required.  And the whole thing could still be zipped if you wanted to.

I don't see these options as exclusive.

Look forward to comments,
Chris

________________________________________
From: Martin Dobias [wonder.sk@gmail.com]
Sent: 03 December 2010 11:01
To: Chris Crook
Cc: [hidden email]
Subject: Re: [Qgis-developer] Memory data provider persistence

Hi Chris

On Mon, Nov 29, 2010 at 8:54 PM, Chris Crook <[hidden email]> wrote:
> Hi All
>
> I'm looking for ideas/suggestions/comments on the memory data provider.
>
> A while ago I raised a ticket http://trac.osgeo.org/qgis/ticket/2487, the essence of which is that if you save and reload a project nothing should have changed, but if you have a layer supported by a memory data provider then its contents get lost.
>
> In the ticket I suggested that the user should be warned on saving, and the layers not included in the project file.  That way at least you don't end up with empty layers.
>
> I'm now coming to a different point of view, which is that it would make much more sense to save the data for the memory provider into the project file.

Having a possibility to save the data into a project would be a nice
feature. However I think that saving the features directly into the
XML project file wouldn't scale well with the amount of data. It can
work well for small datasets, though I'm afraid that it will fail with
a lot of data. One reason is that the project file would get pretty
big due the verbosity of XML, another problem would be with the memory
footprint and parsing speed - we're using DOM.

I have started to think about a more radical step: to allow the
projects to be not only XML files, but additionally they could be e.g.
ZIP files with project XML file inside and optionally some more files.
AFAIK OpenDocument format (used in OpenOffice) does this too.
Internally QGIS could ask the user and convert his data from memory
provider (or any other provider) to e.g. spatialite database that
would be included in that project ZIP file.

This would involve the following changes:
- implement reading and writing of compressed project files (using zlib)
- write a generic QgsVectorLayer -> SpatiaLite export class (similar
to QgsVectorFileWriter or an extension of it)
- add support to read/write "embedded" layers

This would also enable a nice feature to let QGIS pack all data into
one (compressed) file and pass just that file around!

What do you (and others) think of such solution?

Regards
Martin
______________________________________________________________________________________________________

This message contains information, which is confidential and may be subject to legal privilege.
If you are not the intended recipient, you must not peruse, use, disseminate, distribute or copy this message.
If you have received this message in error, please notify us immediately (Phone 0800 665 463 or [hidden email]) and destroy the original message.
LINZ accepts no responsibility for changes to this email, or for any attachments, after its transmission from LINZ.

Thank you.
______________________________________________________________________________________________________
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer


_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Memory data provider persistence

Martin Dobias
In reply to this post by Chris Crook
Hi Chris

On Sat, Dec 4, 2010 at 3:35 AM, Chris Crook <[hidden email]> wrote:
> Another way to achieve a similar result could be to embed the project file in the spatialite database?  That would make the memory provider data easier to use - basically it just becomes a spatialite provided layer.  The project itself becomes harder to manually edit, if you wanted to do so.  However it could easily be exported to a separate file if that is required.  And the whole thing could still be zipped if you wanted to.

This would be also a solution, though I would prefer the former
approach with a ZIP file containing the project XML and other files.
The advantages I see here is that it would be simpler to store various
other types of data - with the project we could store also raster
layers, SVGs for symbols or any other blobs. Loading and saving these
data from/to sqlite database would be a bit cumbersome compared to an
archive.

Regards
Martin
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

RE: Memory data provider persistence

Chris Crook
Yes, I can see that there definitely are advantages to a ZIP format from the point of view of accessing the contents from other programs or a command line interface.

I think either would be as easy from within QGIS - an Sqlite blob can be used pretty much as a file stream so any arbitrary content could be embedded in the Sqlite database.  I guess it depends whether you want to get at the spatial data more easily, by using the Sqlite option, or the other contents, by using the zip option.  I think that you are right, that the zip option is probably more useful for a project file.  And as you point out, it is an approach used by other software too.

If we are going the zip route, I wonder if it would be a better idea to serialize the memory data to a QDataStream represented as a file in the zip archive, rather than as a spatialite database.  The thought here is that it could then be read directly from the zip archive (as it would be read sequentially to reload), whereas an Sqlite database would probably need to be extracted to a temporary file to use.  And the saved layer would remain a memory provided layer that way too, rather than an Sqlite layer, so the reinstated project would be a more accurate copy of the one originally saved.

Cheers
Chris
________________________________________
From: Martin Dobias [[hidden email]]
Sent: 05 December 2010 06:07
To: Chris Crook
Cc: [hidden email]
Subject: Re: [Qgis-developer] Memory data provider persistence

Hi Chris

On Sat, Dec 4, 2010 at 3:35 AM, Chris Crook <[hidden email]> wrote:
> Another way to achieve a similar result could be to embed the project file in the spatialite database?  That would make the memory provider data easier to use - basically it just becomes a spatialite provided layer.  The project itself becomes harder to manually edit, if you wanted to do so.  However it could easily be exported to a separate file if that is required.  And the whole thing could still be zipped if you wanted to.

This would be also a solution, though I would prefer the former
approach with a ZIP file containing the project XML and other files.
The advantages I see here is that it would be simpler to store various
other types of data - with the project we could store also raster
layers, SVGs for symbols or any other blobs. Loading and saving these
data from/to sqlite database would be a bit cumbersome compared to an
archive.

Regards
Martin
______________________________________________________________________________________________________

This message contains information, which is confidential and may be subject to legal privilege.
If you are not the intended recipient, you must not peruse, use, disseminate, distribute or copy this message.
If you have received this message in error, please notify us immediately (Phone 0800 665 463 or [hidden email]) and destroy the original message.
LINZ accepts no responsibility for changes to this email, or for any attachments, after its transmission from LINZ.

Thank you.
______________________________________________________________________________________________________
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

RE: Memory data provider persistence

Chris Crook
In reply to this post by Martin Dobias
... And still further thoughts on this ...

Basically I think there are two issues here.  One is to do with the project file, and the exciting option of using a zipped project file with multiple documents in it, including the project XML file, and whatever resources it uses.

The other is storing and reloading memory provider data, so that the reloaded project is represents what was saved.  

What I've come to realize is that these are independent issues.  I agree with Martin that embedding in the XML file is a bad idea, as the data could become too big to reside comfortably there. Reloading the memory provider data requires that the data exists in a well known location (eg projectFile + ".data") from which it is reloaded.  If the project is a zip file project, then this is included in the project.  If the project is an XML project, then it sits alongside it.

Either way, the implementation of the memory provider persistance is independent of the zipping or otherwise of the project file.

>From my point of view this is good, as I need the memory provider, whereas the zip project file is a nice to have.

So, for my requirement for memory provider persistance, the only real question is what is the right way to do it.  Although Martin had suggested a spatialite database, my leaning is towards a simple QDataStream.  A simple implementation could be (in crude pseudo code)

Qstring header("Qgis data file");
Int version = 1;
stream << header << version;

Foreach memory_provider
  if provider < persist
      stream << layer_id
      stream << attribute_count
      foreach attribute
         stream << attribute_definition
      stream << feature_count
      foreach feature
         stream << feature

This could readily be reloaded after the XML project file is read.

Because it is processed sequentially it would sit comfortably in a ZIP file and be sequentially read from it without needing to be extracted and then processed (as a spatialite database would need to be).  It is portable between OS etc.

If the user actually wants a spatialite database, or any other format, then they can save the layer to that.

How does this sound to people?

Cheers
Chris

-----Original Message-----
From: Martin Dobias [mailto:[hidden email]]
Sent: Sunday, 5 December 2010 6:08 a.m.
To: Chris Crook
Cc: [hidden email]
Subject: Re: [Qgis-developer] Memory data provider persistence

Hi Chris

On Sat, Dec 4, 2010 at 3:35 AM, Chris Crook <[hidden email]> wrote:
> Another way to achieve a similar result could be to embed the project file in the spatialite database?  That would make the memory provider data easier to use - basically it just becomes a spatialite provided layer.  The project itself becomes harder to manually edit, if you wanted to do so.  However it could easily be exported to a separate file if that is required.  And the whole thing could still be zipped if you wanted to.

This would be also a solution, though I would prefer the former approach with a ZIP file containing the project XML and other files.
The advantages I see here is that it would be simpler to store various other types of data - with the project we could store also raster layers, SVGs for symbols or any other blobs. Loading and saving these data from/to sqlite database would be a bit cumbersome compared to an archive.

Regards
Martin
______________________________________________________________________________________________________

This message contains information, which is confidential and may be subject to legal privilege.
If you are not the intended recipient, you must not peruse, use, disseminate, distribute or copy this message.
If you have received this message in error, please notify us immediately (Phone 0800 665 463 or [hidden email]) and destroy the original message.
LINZ accepts no responsibility for changes to this email, or for any attachments, after its transmission from LINZ.

Thank you.
______________________________________________________________________________________________________
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Memory data provider persistence

Barry Rowlingson
On Mon, Dec 6, 2010 at 2:04 AM, Chris Crook <[hidden email]> wrote:

> So, for my requirement for memory provider persistance, the only real question is what is the right way to do it.  Although Martin had suggested a spatialite database, my leaning is towards a simple QDataStream.  A simple implementation could be (in crude pseudo code)
>
> Qstring header("Qgis data file");
> Int version = 1;
> stream << header << version;
>
> Foreach memory_provider
>  if provider < persist
>      stream << layer_id
>      stream << attribute_count
>      foreach attribute
>         stream << attribute_definition
>      stream << feature_count
>      foreach feature
>         stream << feature
>
> This could readily be reloaded after the XML project file is read.
>
> Because it is processed sequentially it would sit comfortably in a ZIP file and be sequentially read from it without needing to be extracted and then processed (as a spatialite database would need to be).  It is portable between OS etc.
>
> If the user actually wants a spatialite database, or any other format, then they can save the layer to that.
>
> How does this sound to people?

 It sounds like you're just creating another spatial data format. Why
not, on project exit, just go "You have unsaved memory data layers -
would you like them converted to [gml|shapefile|spatialite|whatever]
and reloaded before saving/qutting or do you want to lose them
forever?".

Barry
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Memory data provider persistence

Martin Dobias
On Mon, Dec 6, 2010 at 10:45 AM, Barry Rowlingson
<[hidden email]> wrote:

> On Mon, Dec 6, 2010 at 2:04 AM, Chris Crook <[hidden email]> wrote:
>
>> So, for my requirement for memory provider persistance, the only real question is what is the right way to do it.  Although Martin had suggested a spatialite database, my leaning is towards a simple QDataStream.  A simple implementation could be (in crude pseudo code)
>>
>> Qstring header("Qgis data file");
>> Int version = 1;
>> stream << header << version;
>>
>> Foreach memory_provider
>>  if provider < persist
>>      stream << layer_id
>>      stream << attribute_count
>>      foreach attribute
>>         stream << attribute_definition
>>      stream << feature_count
>>      foreach feature
>>         stream << feature
>>
>> This could readily be reloaded after the XML project file is read.
>>
>> Because it is processed sequentially it would sit comfortably in a ZIP file and be sequentially read from it without needing to be extracted and then processed (as a spatialite database would need to be).  It is portable between OS etc.
>>
>> If the user actually wants a spatialite database, or any other format, then they can save the layer to that.
>>
>> How does this sound to people?
>
>  It sounds like you're just creating another spatial data format. Why
> not, on project exit, just go "You have unsaved memory data layers -
> would you like them converted to [gml|shapefile|spatialite|whatever]
> and reloaded before saving/qutting or do you want to lose them
> forever?".

I completely agree with Barry - to me it looks like a waste of time to
invent a new data format - exclusively readable just by QGIS - and
intended just to load/store some data from/to memory provider.
Additionally, memory provider was thought to represent just temporary
(nonpersistent) data - so asking the user to convert it to some other
format when exiting seems reasonable to me.

Regards
Martin
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

RE: Memory data provider persistence

Chris Crook
Hi All

Fair comment I guess.  In fact Barry's suggestion is pretty much back at my original ticket on this issue.  This had two components, one was the warning about memory provider layers, and the other was making it easier to save them (or more accurately, to replace the memory layer with the saved layer in the project).  Ideally the legend would also have some sort of visual clue as to which layers were not yet saved.

Nonetheless I do think there is value in having the memory provided data saved more or less transparently to the user.  For example the contour plugin builds its contours in a memory layer.  I don't want them to disappear when I save and reopen the project.  Yet sometimes  I want to save the project quickly without having to decide where and in what format I want bits of it saved.  For example when I have to exit the train on the way to work!!  I just want to be able to save the project and then start up later and carry on working.

While the memory provider is intended for transient data, I think that it is a valid user requirement that it persist beyond the user session.

In terms of the QDataStream suggestion, the idea is not to create a format, it is to persist the memory layer efficiently and transparently.  I don't think that is the same thing.  

I guess that the requirements for the format are that it is reasonably compact and can be efficiently read from and written to a sequential data stream (to facilitate possible writing to a zip file or other compressed format in the future).  Also it would be nice if it didn't generate a plethora of files.  Any other format that meets that requirement would be good, and I certainly agree that using an open format is preferable.  Other suggestions??

Cheers
Chris
________________________________________
From: Martin Dobias [[hidden email]]
Sent: 06 December 2010 22:57
To: Barry Rowlingson
Cc: Chris Crook; [hidden email]
Subject: Re: [Qgis-developer] Memory data provider persistence

On Mon, Dec 6, 2010 at 10:45 AM, Barry Rowlingson
<[hidden email]> wrote:

> On Mon, Dec 6, 2010 at 2:04 AM, Chris Crook <[hidden email]> wrote:
>
>> So, for my requirement for memory provider persistance, the only real question is what is the right way to do it.  Although Martin had suggested a spatialite database, my leaning is towards a simple QDataStream.  A simple implementation could be (in crude pseudo code)
>>
>> Qstring header("Qgis data file");
>> Int version = 1;
>> stream << header << version;
>>
>> Foreach memory_provider
>>  if provider < persist
>>      stream << layer_id
>>      stream << attribute_count
>>      foreach attribute
>>         stream << attribute_definition
>>      stream << feature_count
>>      foreach feature
>>         stream << feature
>>
>> This could readily be reloaded after the XML project file is read.
>>
>> Because it is processed sequentially it would sit comfortably in a ZIP file and be sequentially read from it without needing to be extracted and then processed (as a spatialite database would need to be).  It is portable between OS etc.
>>
>> If the user actually wants a spatialite database, or any other format, then they can save the layer to that.
>>
>> How does this sound to people?
>
>  It sounds like you're just creating another spatial data format. Why
> not, on project exit, just go "You have unsaved memory data layers -
> would you like them converted to [gml|shapefile|spatialite|whatever]
> and reloaded before saving/qutting or do you want to lose them
> forever?".

I completely agree with Barry - to me it looks like a waste of time to
invent a new data format - exclusively readable just by QGIS - and
intended just to load/store some data from/to memory provider.
Additionally, memory provider was thought to represent just temporary
(nonpersistent) data - so asking the user to convert it to some other
format when exiting seems reasonable to me.

Regards
Martin
______________________________________________________________________________________________________

This message contains information, which is confidential and may be subject to legal privilege.
If you are not the intended recipient, you must not peruse, use, disseminate, distribute or copy this message.
If you have received this message in error, please notify us immediately (Phone 0800 665 463 or [hidden email]) and destroy the original message.
LINZ accepts no responsibility for changes to this email, or for any attachments, after its transmission from LINZ.

Thank you.
______________________________________________________________________________________________________
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Memory data provider persistence

Barry Rowlingson
On Mon, Dec 6, 2010 at 6:05 PM, Chris Crook <[hidden email]> wrote:
> Hi All
>
> Fair comment I guess.  In fact Barry's suggestion is pretty much back at my original ticket on this issue.  This had two components, one was the warning about memory provider layers, and the other was making it easier to save them (or more accurately, to replace the memory layer with the saved layer in the project).  Ideally the legend would also have some sort of visual clue as to which layers were not yet saved.
>
> Nonetheless I do think there is value in having the memory provided data saved more or less transparently to the user.  For example the contour plugin builds its contours in a memory layer.  I don't want them to disappear when I save and reopen the project.  Yet sometimes  I want to save the project quickly without having to decide where and in what format I want bits of it saved.  For example when I have to exit the train on the way to work!!  I just want to be able to save the project and then start up later and carry on working.

 So as long as the saving of memory layers is quick, you're happy? So
what about if it happened transparently that memory layers were saved
to /tmp/something in Shapefile format, and the location and format
were configurable from a settings dialog and saved (either per-project
or in user settings), and that .qgs files added the converted layers
into the project? Sounds like a piece of cake, meets the 'train
suddenly arriving at station' criterion (although in this country you
get an extra twenty minutes past the expected arrival time with
trains), and could possibly be implemented as a plugin. You may never
lose memory data provider data again...

 I'm not sure serializing the object to binary is a good idea, you
will one day want to load it into another GIS... Standards are a good
thing, that's why there are so many of them.


Barry
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

RE: Memory data provider persistence

Chris Crook
Hi Barry et al

Good point about a plugin implementation .. I hadn't thought of that obvious option - definitely one to follow up on.  It might still need a few tweaks in terms of Qgis.  The QgisInterface has a projectRead() signal that could be used to trigger reloading the data, but I don't see a projectSaved() signal to trigger writing.  Easy to add that though, and probably generally useful.

Do you know of an alternative signal I could use to trigger writing?

Given that there is a signal to hook in to this is, as you say, a piece of cake.  I'm kicking myself for not having thought of this months ago!!

In terms of file location my preference would be something directly related to the project file .. That is obviously easy.  And shapefile is not my choice, just because of the messy bunch of files it creates and having so many of them and because the DBF format is limiting and pretty scungy!  But that is personal preference I guess.

Brilliant ..

Cheers
Chris



-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Barry Rowlingson
Sent: Tuesday, 7 December 2010 7:40 a.m.
To: Chris Crook
Cc: Martin Dobias; [hidden email]
Subject: Re: [Qgis-developer] Memory data provider persistence

On Mon, Dec 6, 2010 at 6:05 PM, Chris Crook <[hidden email]> wrote:
> Hi All
>
> Fair comment I guess.  In fact Barry's suggestion is pretty much back at my original ticket on this issue.  This had two components, one was the warning about memory provider layers, and the other was making it easier to save them (or more accurately, to replace the memory layer with the saved layer in the project).  Ideally the legend would also have some sort of visual clue as to which layers were not yet saved.
>
> Nonetheless I do think there is value in having the memory provided data saved more or less transparently to the user.  For example the contour plugin builds its contours in a memory layer.  I don't want them to disappear when I save and reopen the project.  Yet sometimes  I want to save the project quickly without having to decide where and in what format I want bits of it saved.  For example when I have to exit the train on the way to work!!  I just want to be able to save the project and then start up later and carry on working.

 So as long as the saving of memory layers is quick, you're happy? So what about if it happened transparently that memory layers were saved to /tmp/something in Shapefile format, and the location and format were configurable from a settings dialog and saved (either per-project or in user settings), and that .qgs files added the converted layers into the project? Sounds like a piece of cake, meets the 'train suddenly arriving at station' criterion (although in this country you get an extra twenty minutes past the expected arrival time with trains), and could possibly be implemented as a plugin. You may never lose memory data provider data again...

 I'm not sure serializing the object to binary is a good idea, you will one day want to load it into another GIS... Standards are a good thing, that's why there are so many of them.


Barry
______________________________________________________________________________________________________

This message contains information, which is confidential and may be subject to legal privilege.
If you are not the intended recipient, you must not peruse, use, disseminate, distribute or copy this message.
If you have received this message in error, please notify us immediately (Phone 0800 665 463 or [hidden email]) and destroy the original message.
LINZ accepts no responsibility for changes to this email, or for any attachments, after its transmission from LINZ.

Thank you.
______________________________________________________________________________________________________
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

RE: Memory data provider persistence

Chris Crook
In reply to this post by Barry Rowlingson
Further to this .. I've raised a ticket #3297 with a patch to add a projectSaved() signal matching the projectRead().

http://trac.osgeo.org/qgis/ticket/3297

This should make building a memory provider saving plugin trivial.

Could this be committed if it looks Ok.

Thanks
Chris

______________________________________________________________________________________________________

This message contains information, which is confidential and may be subject to legal privilege.
If you are not the intended recipient, you must not peruse, use, disseminate, distribute or copy this message.
If you have received this message in error, please notify us immediately (Phone 0800 665 463 or [hidden email]) and destroy the original message.
LINZ accepts no responsibility for changes to this email, or for any attachments, after its transmission from LINZ.

Thank you.
______________________________________________________________________________________________________
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Memory data provider persistence

Barry Rowlingson
In reply to this post by Chris Crook
On Mon, Dec 6, 2010 at 7:12 PM, Chris Crook <[hidden email]> wrote:
> Hi Barry et al
>
> Good point about a plugin implementation .. I hadn't thought of that obvious option - definitely one to follow up on.  It might still need a few tweaks in terms of Qgis.  The QgisInterface has a projectRead() signal that could be used to trigger reloading the data, but I don't see a projectSaved() signal to trigger writing.  Easy to add that though, and probably generally useful.

 There's a projectWrite signal:

http://www.qgis.org/api/classQgsProject.html#a935c811d84c51e908c73868be9b69c4

 which seems to be emitted before the layers are saved, so if anything
hooked into that wants to make changes it should be able to. Haven't
tried, but will that do the job?

> Given that there is a signal to hook in to this is, as you say, a piece of cake.  I'm kicking myself for not having thought of this months ago!!

 The only problem might be if the memory provider doesn't set the
project as dirty when it changes itself. Otherwise users could do a
'quit', and then qgis quits without chance to save the project.

> In terms of file location my preference would be something directly related to the project file .. That is obviously easy.  And shapefile is not my choice, just because of the messy bunch of files it creates and having so many of them and because the DBF format is limiting and pretty scungy!  But that is personal preference I guess.

 Agreed! I've been saving things as GML lately.

Barry
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

RE: Memory data provider persistence

Chris Crook
Yes .. I got carried away and put up a patch implementing the projectWrite signal in the QgisInterface before I saw this alternative.  Though in fact I do think that is makes sense to have a QgsInterface projectSaved() signal matching the projectRead().  And maybe a savingProject() that happens before it is saved.

I forgot about needing to catch changes in the memory provider data to set the dirty state .. This was part of my original plan for implementing the memory provider data into the project file itself but I forgot it in my excitement about a plug in option.  (The excitement is about being to do it now - I still think this really belongs in the core system).

Maybe if I can trap an event relating to saving edits, then if the edited layer provider is a memory provider I can set the project state to dirty.  And plugins will have to set the dirty state themselves (this may happen anyway, eg when the contour plugin generates a new contour layer adding the layer sets the state).  This is getting a bit messy though!

So maybe there is still some work required to provide the signals for changes to vector provider data.  In fact the project does need to take ownership of the data.  For it all to make sense it needs a projectDataDirty() flag.  Bother!

Chris

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Barry Rowlingson
Sent: Tuesday, 7 December 2010 12:21 p.m.
To: Chris Crook
Cc: Martin Dobias; [hidden email]
Subject: Re: [Qgis-developer] Memory data provider persistence

On Mon, Dec 6, 2010 at 7:12 PM, Chris Crook <[hidden email]> wrote:
> Hi Barry et al
>
> Good point about a plugin implementation .. I hadn't thought of that obvious option - definitely one to follow up on.  It might still need a few tweaks in terms of Qgis.  The QgisInterface has a projectRead() signal that could be used to trigger reloading the data, but I don't see a projectSaved() signal to trigger writing.  Easy to add that though, and probably generally useful.

 There's a projectWrite signal:

http://www.qgis.org/api/classQgsProject.html#a935c811d84c51e908c73868be9b69c4

 which seems to be emitted before the layers are saved, so if anything hooked into that wants to make changes it should be able to. Haven't tried, but will that do the job?

> Given that there is a signal to hook in to this is, as you say, a piece of cake.  I'm kicking myself for not having thought of this months ago!!

 The only problem might be if the memory provider doesn't set the project as dirty when it changes itself. Otherwise users could do a 'quit', and then qgis quits without chance to save the project.

> In terms of file location my preference would be something directly related to the project file .. That is obviously easy.  And shapefile is not my choice, just because of the messy bunch of files it creates and having so many of them and because the DBF format is limiting and pretty scungy!  But that is personal preference I guess.

 Agreed! I've been saving things as GML lately.

Barry
______________________________________________________________________________________________________

This message contains information, which is confidential and may be subject to legal privilege.
If you are not the intended recipient, you must not peruse, use, disseminate, distribute or copy this message.
If you have received this message in error, please notify us immediately (Phone 0800 665 463 or [hidden email]) and destroy the original message.
LINZ accepts no responsibility for changes to this email, or for any attachments, after its transmission from LINZ.

Thank you.
______________________________________________________________________________________________________
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Memory data provider persistence

Régis Haubourg
In reply to this post by Chris Crook
Hi all,
one up for this topic. Since QGIS 1.8, memory layer saver is not working the same way as in 1.7.4.
I raised a tickets here http://hub.qgis.org/issues/6075, but I think something changed in memory layer provider or OGR GML provider, and it maybe isn't a plugin issue:

In fact, FID field was previously hidden to end user in QGIS but was present in GML.  It was filled with "F"+id . It now is filled with "projectName."+id. FID field is now shown to end user whic alterates table structure when saving.
Reopening project, saving and reoping a second time add a second FID field.

Anyone has an idea of what's going on ?

BTW I'am really interested to have a stable way of saving memory layers along with project.Anyone still working on it ? Having it part of the Core would be great too. I am ready to sponsor it since it.  
What solution should we keep? gml+xsd along with project? A dialog asking what memory layers to save, and explaining what are thoses gml and xsd files would be nice too. I have seen users cleaning up those messy files since they did not understand what they were..
Reply | Threaded
Open this post in threaded view
|

Re: Memory data provider persistence

Chris Crook
Hi

I think the change in behaviour is due to the change in the gdal 1.9 implementation of the GML driver, which generates the fid field when the layer is reloaded.  I am looking at a fix that basically removes the field after it is reloaded.
Not an ideal solution (especially if you want a field called "fid").

On you other points on the memory layer saving, it may be worth reviewing the earlier discussions about the memory layer saver.

http://lists.osgeo.org/pipermail/qgis-developer/2010-December/011876.html

I have a couple of times proposed that memory layers get saved alongside the project or within the project file.  Inside the project file is not a good idea - it could easily get too big.  

So the main question for the ideal solution is how to store the information.  I am using the GML format as a preferrred open format (preferred in this case based on responese on the qgis dev list).  I'm inclined to modify this to a format that uses a single file for all the layers as a cleaner implementation from the users perspective (eg if the project file is test.qgs then the persisted data is in test.qgs.db, or something like that).  The format could be spatialite perhaps, or a zip file of the gml files?  It would also get rid of the horrible long file names.

I see in your bug post that you have found that sometimes the layers don't get saved at all.  I haven't experienced this, but I did notice that if you start editing a layer, add some features, and then quit qgis without saving the edits, then there is no prompt for saving the edits.  If the edits don't get saved then the memory saver plugin doesn't flag the project as dirty, and so the changes don't get saved.  Could this match your example?

Cheers
Chris Crook


-----Original Message-----
From: haubourg [mailto:[hidden email]]
Sent: Saturday, 21 July 2012 1:55 a.m.
To: [hidden email]
Subject: Re: [Qgis-developer] Memory data provider persistence

Hi all,
one up for this topic. Since QGIS 1.8, memory layer saver is not working the same way as in 1.7.4.
I raised a tickets here  http://hub.qgis.org/issues/6075
http://hub.qgis.org/issues/6075 , but I think something changed in memory layer provider or OGR GML provider, and it maybe isn't a plugin issue:

In fact, FID field was previously hidden to end user in QGIS but was present in GML.  It was filled with "F"+id . It now is filled with "projectName."+id. FID field is now shown to end user whic alterates table structure when saving.
Reopening project, saving and reoping a second time add a second FID field.

Anyone has an idea of what's going on ?

BTW *I'am really interested to have a stable way of saving memory layers along with project.Anyone still working on it ? Having it part of the Core would be great too. I am ready to sponsor it since it. * What solution should we keep? gml+xsd along with project? A dialog asking what memory layers to save, and explaining what are thoses gml and xsd files would be nice too. I have seen users cleaning up those messy files since they did not understand what they were..




--
View this message in context: http://osgeo-org.1560.n6.nabble.com/Memory-data-provider-persistence-tp4108012p4989667.html
Sent from the Quantum GIS - Developer mailing list archive at Nabble.com.

This message contains information, which is confidential and may be subject to legal privilege. If you are not the intended recipient, you must not peruse, use, disseminate, distribute or copy this message. If you have received this message in error, please notify us immediately (Phone 0800 665 463 or [hidden email]) and destroy the original message. LINZ accepts no responsibility for changes to this email, or for any attachments, after its transmission from LINZ. Thank You.
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Memory data provider persistence

Régis Haubourg
Hi Chris, thanks for your feedback. I start thinking using a spatialite is a great idea. I prefer this than having rules to mask FID field. This can generate very hard-to-find bugs..

Having a complex Qgs zipfile containing either a spatialite DB or xml + data would be great but is a big evolution. IMHO, such a break should be kept for a major version change in QGIS (2.0?).

Is that a work you are on ? Need contributors?


Reply | Threaded
Open this post in threaded view
|

Re: Memory data provider persistence

Chris Crook
Hi .. yes I don't like masking the FID field and I'm not thinking this is a long term solution, even before the problem in my implementation with the field index issue (which I think I will be able to fix).  I'm looking at a couple of options over the next few days .. and open to any contributions, suggestions, code, etc :-)

Cheers
Chris
________________________________________
From: haubourg [[hidden email]]
Sent: 23 July 2012 20:58
To: [hidden email]
Subject: Re: [Qgis-developer] Memory data provider persistence

Hi Chris, thanks for your feedback. I start thinking using a spatialite is a
great idea. I prefer this than having rules to mask FID field. This can
generate very hard-to-find bugs..

Having a complex Qgs zipfile containing either a spatialite DB or xml + data
would be great but is a big evolution. IMHO, such a break should be kept for
a major version change in QGIS (2.0?).

Is that a work you are on ? Need contributors?






--
View this message in context: http://osgeo-org.1560.n6.nabble.com/Memory-data-provider-persistence-tp4108012p4989995.html
Sent from the Quantum GIS - Developer mailing list archive at Nabble.com.

This message contains information, which is confidential and may be subject to legal privilege. If you are not the intended recipient, you must not peruse, use, disseminate, distribute or copy this message. If you have received this message in error, please notify us immediately (Phone 0800 665 463 or [hidden email]) and destroy the original message. LINZ accepts no responsibility for changes to this email, or for any attachments, after its transmission from LINZ. Thank You.
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Memory data provider persistence

Even Rouault
Le mercredi 25 juillet 2012 20:03:16, Chris Crook a écrit :
> Hi .. yes I don't like masking the FID field and I'm not thinking this is a
> long term solution, even before the problem in my implementation with the
> field index issue (which I think I will be able to fix).  I'm looking at a
> couple of options over the next few days .. and open to any contributions,
> suggestions, code, etc :-)

Chris,

did you try setting GML_EXPOSE_FID to NO before opening the GML as we
discussed on the GDAL trac system ? This should fix your issue without much
code, basically calling CPLSetConfigOption("GML_EXPOSE_FID", "NO") (or
gdal.SetConfigOption("GML_EXPOSE_FID", "NO") if you're coding in Python) before
doing the OGROpen() call with the GML file.

Actually, to avoid "polluting" other portions of code, you should save the
original value and restore it afterwards, so :

in C/C++ :

char* pszOldVal = CPLGetConfigOption("GML_EXPOSE_FID") ?
strdup(CPLGetConfigOption("GML_EXPOSE_FID")) : NULL;
CPLSetConfigOption("GML_EXPOSE_FID", "NO");
OGRDataSource hDS  = OGROpen(....);
CPLSetConfigOption("GML_EXPOSE_FID", pszOldVal);
free(pszOldVal);

in Python :

old_val = gdal.GetConfigOption("GML_EXPOSE_FID")
gdal.SetConfigOption("GML_EXPOSE_FID", "NO")
ds = ogr.Open(....)
gdal.SetConfigOption("GML_EXPOSE_FID",old_val)

And to avoid confusion, yes, you should also avoid that a user creates a 'FID'
field.

>
> Cheers
> Chris
> ________________________________________
> From: haubourg [[hidden email]]
> Sent: 23 July 2012 20:58
> To: [hidden email]
> Subject: Re: [Qgis-developer] Memory data provider persistence
>
> Hi Chris, thanks for your feedback. I start thinking using a spatialite is
> a great idea. I prefer this than having rules to mask FID field. This can
> generate very hard-to-find bugs..
>
> Having a complex Qgs zipfile containing either a spatialite DB or xml +
> data would be great but is a big evolution. IMHO, such a break should be
> kept for a major version change in QGIS (2.0?).
>
> Is that a work you are on ? Need contributors?
>
>
>
>
>
>
> --
> View this message in context:
> http://osgeo-org.1560.n6.nabble.com/Memory-data-provider-persistence-tp410
> 8012p4989995.html Sent from the Quantum GIS - Developer mailing list
> archive at Nabble.com.
>
> This message contains information, which is confidential and may be subject
> to legal privilege. If you are not the intended recipient, you must not
> peruse, use, disseminate, distribute or copy this message. If you have
> received this message in error, please notify us immediately (Phone 0800
> 665 463 or [hidden email]) and destroy the original message. LINZ
> accepts no responsibility for changes to this email, or for any
> attachments, after its transmission from LINZ. Thank You.
> _______________________________________________
> Qgis-developer mailing list
> [hidden email]
> http://lists.osgeo.org/mailman/listinfo/qgis-developer
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
12