Memory data provider persistence

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
32 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: Memory data provider persistence

Chris Crook
Hi again

When I originally was proposing saving the memory layer (as a built in function of QGIS rather than as a plugin), I was thinking of streaming the data out and back using a QDataStream.  It would have minimal overhead, unlike say converting to GML, or storing in an SQLite database.  Also it would mean loading data into another format and then copying back into the memory layer, as is currently done, as it would just be read sequentially into the memory layer.

On the down side it would have no use other than for reloading the memory layer data.  I got some feedback suggesting that this was not a good thing ... why was I creating another spatial data format.

However I'm now wondering if that is a better way to go, since in addition to the minimal overhead, it also makes the plugin independent of potential changes in other components, such as the OGR GML driver in this case, which could change and break the functionality.  Also it removes any concerns about the data being altered as it gets passed into another format and back again.  The QDataStream is a versioned format, so it should be easy to ensure compatibility with future implementations of the Qt framework.

If the user actually wants to use the data outside of QGIS, then they can save it the format they want from QGIS.  The purpose of the memory layer persistence is to allow the user to close the project (possibly move it), open it again, and not lose their data - basically to be able to carry on working without having to worry whether or not their data is in memory layers or other forms.  (This is how I think QGIS should behave by default).  The plugin is not intended to save the data for other uses.  Given that the default behaviour of QGIS (without the plugin), is to discard the data altogether, I think it is safe to assume that this is not seen as a requirement!

So what I'm considering is rebuilding the plugin to read and write the memory layer data into a single file, say project.qgis.mldata, instead of the current multiple gml files.  

Thoughts?

Chris

-----Original Message-----
From: Chris Crook
Sent: Thursday, 26 July 2012 6:03 a.m.
To: haubourg; [hidden email]
Subject: RE: [Qgis-developer] Memory data provider persistence

Hi .. yes I don't like masking the FID field and I'm not thinking this is a long term solution, even before the problem in my implementation with the field index issue (which I think I will be able to fix).  I'm looking at a couple of options over the next few days .. and open to any contributions, suggestions, code, etc :-)

Cheers
Chris
________________________________________
From: haubourg [[hidden email]]
Sent: 23 July 2012 20:58
To: [hidden email]
Subject: Re: [Qgis-developer] Memory data provider persistence

Hi Chris, thanks for your feedback. I start thinking using a spatialite is a great idea. I prefer this than having rules to mask FID field. This can generate very hard-to-find bugs..

Having a complex Qgs zipfile containing either a spatialite DB or xml + data would be great but is a big evolution. IMHO, such a break should be kept for a major version change in QGIS (2.0?).

Is that a work you are on ? Need contributors?






--
View this message in context: http://osgeo-org.1560.n6.nabble.com/Memory-data-provider-persistence-tp4108012p4989995.html
Sent from the Quantum GIS - Developer mailing list archive at Nabble.com.

This message contains information, which is confidential and may be subject to legal privilege. If you are not the intended recipient, you must not peruse, use, disseminate, distribute or copy this message. If you have received this message in error, please notify us immediately (Phone 0800 665 463 or [hidden email]) and destroy the original message. LINZ accepts no responsibility for changes to this email, or for any attachments, after its transmission from LINZ. Thank You.
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Memory data provider persistence

Jürgen E. Fischer
Hi,

On Thu, 26. Jul 2012 at 16:31:46 +1200, Chris Crook wrote:
> Thoughts?

I didn't really follow this thread.  But the subject made me wonder if anybody
ever mentioned/explored sqlite/spatialite in-memory databases[1] and their
backup facility[2]?   Would that be an option?  Maybe even as replacement for
the memory provider itself.


Jürgen

[1] http://www.sqlite.org/inmemorydb.html
[2] http://www.sqlite.org/backup.html
 
--
Jürgen E. Fischer         norBIT GmbH               Tel. +49-4931-918175-31
Dipl.-Inf. (FH)           Rheinstraße 13            Fax. +49-4931-918175-50
Software Engineer         D-26506 Norden               http://www.norbit.de
committ(ed|ing) to Quantum GIS                         IRC: jef on FreeNode                        

--
norBIT Gesellschaft fuer Unternehmensberatung und Informationssysteme mbH
Rheinstrasse 13, 26506 Norden
GF: Jelto Buurman, HR: Amtsgericht Emden, HRB 5502

_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Memory data provider persistence

pcav
In reply to this post by Chris Crook
Il 26/07/2012 06:31, Chris Crook ha scritto:
> So what I'm considering is rebuilding the plugin to read and write the memory layer data into a single file, say project.qgis.mldata, instead of the current multiple gml files.  
>
> Thoughts?
Hi Chris,
please consider that much the same problem applies to other plugins,
especially Sextante and GDALTools. Now that we are going to have a Save
as... facility also for rasters, this could also be included.
So a solution designed only for fTools seems suboptimal.
All the best.

--
Paolo Cavallini - Faunalia
www.faunalia.eu
Full contact details at www.faunalia.eu/pc
Nuovi corsi QGIS e PostGIS: http://www.faunalia.it/calendario

_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Memory data provider persistence

Régis Haubourg
Having a specific memory storage have pros and cons that you explained very well Chris.
 From a user point of view, one drawback is that if something goes wrong in memory layer loading whe project opens, or on memory layer save, data is lost. So User will for sure ask for a "Add memory layer" provider.. because they will try to reopen that dataset they see with their project.  
That's why I prefer SQLlite, already easy to open. It stores vector or raster, is compact, no install...  BUT, we will be dependent of that tool and associated libraries, that's true.
My bet is that Spatialite will be a MANDATORY tool for people who don't have postgis access or skills. One simple argument: it store long field names, and is fast.  So QGIS should stick close to spatialite as much as it does for postgis. If we all agree on that roadmap, it would not be so risquy to choose spatialite ..
Hope this helps..
Reply | Threaded
Open this post in threaded view
|

Re: Memory data provider persistence

Etienne Tourigny-3
In reply to this post by Jürgen E. Fischer
I don't know much about the memory layer handling - but here are some thoughts:

I think Jürgen's idea makes a lot of sense: keep in-memory spatialite
database, and save/restore when the project is saved/opened. And the
user can export the layer to a preferred format if he/she wishes.

Perhaps this could be added to qgis core, as a plugin or c++ ?

I also think having spatialite as a required dependency is not a problem.

Etienne

On Thu, Jul 26, 2012 at 1:51 AM, Jürgen E. <[hidden email]> wrote:

> Hi,
>
> On Thu, 26. Jul 2012 at 16:31:46 +1200, Chris Crook wrote:
>> Thoughts?
>
> I didn't really follow this thread.  But the subject made me wonder if anybody
> ever mentioned/explored sqlite/spatialite in-memory databases[1] and their
> backup facility[2]?   Would that be an option?  Maybe even as replacement for
> the memory provider itself.
>
>
> Jürgen
>
> [1] http://www.sqlite.org/inmemorydb.html
> [2] http://www.sqlite.org/backup.html
>
> --
> Jürgen E. Fischer         norBIT GmbH               Tel. +49-4931-918175-31
> Dipl.-Inf. (FH)           Rheinstraße 13            Fax. +49-4931-918175-50
> Software Engineer         D-26506 Norden               http://www.norbit.de
> committ(ed|ing) to Quantum GIS                         IRC: jef on FreeNode
>
> --
> norBIT Gesellschaft fuer Unternehmensberatung und Informationssysteme mbH
> Rheinstrasse 13, 26506 Norden
> GF: Jelto Buurman, HR: Amtsgericht Emden, HRB 5502
>
> _______________________________________________
> Qgis-developer mailing list
> [hidden email]
> http://lists.osgeo.org/mailman/listinfo/qgis-developer
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Memory data provider persistence

Etienne Tourigny-3
In reply to this post by pcav
We are designing the raster save as in a way that could be re-used
elsewhere, perhaps vector saving could follow the same route -
although not much (nothing) can be shared between the vector and
raster dialogs.

Etienne

On Thu, Jul 26, 2012 at 2:29 AM, Paolo Cavallini <[hidden email]> wrote:

> Il 26/07/2012 06:31, Chris Crook ha scritto:
>> So what I'm considering is rebuilding the plugin to read and write the memory layer data into a single file, say project.qgis.mldata, instead of the current multiple gml files.
>>
>> Thoughts?
> Hi Chris,
> please consider that much the same problem applies to other plugins,
> especially Sextante and GDALTools. Now that we are going to have a Save
> as... facility also for rasters, this could also be included.
> So a solution designed only for fTools seems suboptimal.
> All the best.
>
> --
> Paolo Cavallini - Faunalia
> www.faunalia.eu
> Full contact details at www.faunalia.eu/pc
> Nuovi corsi QGIS e PostGIS: http://www.faunalia.it/calendario
>
> _______________________________________________
> Qgis-developer mailing list
> [hidden email]
> http://lists.osgeo.org/mailman/listinfo/qgis-developer
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Memory data provider persistence

Chris Crook
In reply to this post by Even Rouault
Hi Even

Thanks for the suggestion and code examples - very helpful. I haven't followed up on this yet partly because this has prompted me to look at other storage formats for the data (something I'd been putting off).

If I understand the GML driver documentation properly then I would have to manage the possibility of the user having a field called FID, as after I've saved the data, then reloaded with GMS_EXPOSE_FID set to NO, that field will be lost? As you suggest, renaming the fields before saving and then after reloading would fix handle this.

Cheers
Chris

-----Original Message-----
From: Even Rouault [mailto:[hidden email]]
Sent: Thursday, 26 July 2012 6:47 a.m.
To: [hidden email]
Cc: Chris Crook; haubourg
Subject: Re: [Qgis-developer] Memory data provider persistence

Le mercredi 25 juillet 2012 20:03:16, Chris Crook a écrit :
> Hi .. yes I don't like masking the FID field and I'm not thinking this
> is a long term solution, even before the problem in my implementation
> with the field index issue (which I think I will be able to fix).  I'm
> looking at a couple of options over the next few days .. and open to
> any contributions, suggestions, code, etc :-)

Chris,

did you try setting GML_EXPOSE_FID to NO before opening the GML as we discussed on the GDAL trac system ? This should fix your issue without much code, basically calling CPLSetConfigOption("GML_EXPOSE_FID", "NO") (or gdal.SetConfigOption("GML_EXPOSE_FID", "NO") if you're coding in Python) before doing the OGROpen() call with the GML file.

Actually, to avoid "polluting" other portions of code, you should save the original value and restore it afterwards, so :

in C/C++ :

char* pszOldVal = CPLGetConfigOption("GML_EXPOSE_FID") ?
strdup(CPLGetConfigOption("GML_EXPOSE_FID")) : NULL; CPLSetConfigOption("GML_EXPOSE_FID", "NO"); OGRDataSource hDS  = OGROpen(....); CPLSetConfigOption("GML_EXPOSE_FID", pszOldVal); free(pszOldVal);

in Python :

old_val = gdal.GetConfigOption("GML_EXPOSE_FID")
gdal.SetConfigOption("GML_EXPOSE_FID", "NO") ds = ogr.Open(....)
gdal.SetConfigOption("GML_EXPOSE_FID",old_val)

And to avoid confusion, yes, you should also avoid that a user creates a 'FID'
field.

>
> Cheers
> Chris
> ________________________________________
> From: haubourg [[hidden email]]
> Sent: 23 July 2012 20:58
> To: [hidden email]
> Subject: Re: [Qgis-developer] Memory data provider persistence
>
> Hi Chris, thanks for your feedback. I start thinking using a
> spatialite is a great idea. I prefer this than having rules to mask
> FID field. This can generate very hard-to-find bugs..
>
> Having a complex Qgs zipfile containing either a spatialite DB or xml
> + data would be great but is a big evolution. IMHO, such a break
> should be kept for a major version change in QGIS (2.0?).
>
> Is that a work you are on ? Need contributors?
>
>
>
>
>
>
> --
> View this message in context:
> http://osgeo-org.1560.n6.nabble.com/Memory-data-provider-persistence-t
> p410 8012p4989995.html Sent from the Quantum GIS - Developer mailing
> list archive at Nabble.com.
>
> This message contains information, which is confidential and may be
> subject to legal privilege. If you are not the intended recipient, you
> must not peruse, use, disseminate, distribute or copy this message. If
> you have received this message in error, please notify us immediately
> (Phone 0800
> 665 463 or [hidden email]) and destroy the original message. LINZ
> accepts no responsibility for changes to this email, or for any
> attachments, after its transmission from LINZ. Thank You.
> _______________________________________________
> Qgis-developer mailing list
> [hidden email]
> http://lists.osgeo.org/mailman/listinfo/qgis-developer
This message contains information, which is confidential and may be subject to legal privilege. If you are not the intended recipient, you must not peruse, use, disseminate, distribute or copy this message. If you have received this message in error, please notify us immediately (Phone 0800 665 463 or [hidden email]) and destroy the original message. LINZ accepts no responsibility for changes to this email, or for any attachments, after its transmission from LINZ. Thank You.
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Memory data provider persistence

Even Rouault
Le jeudi 26 juillet 2012 21:41:57, Chris Crook a écrit :

> Hi Even
>
> Thanks for the suggestion and code examples - very helpful. I haven't
> followed up on this yet partly because this has prompted me to look at
> other storage formats for the data (something I'd been putting off).
>
> If I understand the GML driver documentation properly then I would have to
> manage the possibility of the user having a field called FID, as after
> I've saved the data, then reloaded with GMS_EXPOSE_FID set to NO, that
> field will be lost?

If a user creates a 'fid' field, then that one will be used to set the fid
attribute of the element. It will not be "lost" strictly speaking, but indeed
if GML_EXPOSE_FID is set to NO, then it will not be visible.

> As you suggest, renaming the fields before saving and
> then after reloading would fix handle this.

Not sure to understand you here : are you speaking about the mechanism for
saving the old value of GML_EXPOSE_FID, setting GML_EXPOSE_FID=OLD, calling
OGROpen() and restoring the original value of GML_EXPOSE_FID ? If so, this
will not change anything about the behaviour with a user field called 'fid'.

The intention here was not to interfere with other parts of QGIS that might
manipulate GML files. For example, if a user opens a GML file through "Add
Vector Layer", he might want to see the fid reported.

_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Memory data provider persistence

Chris Crook
In reply to this post by Régis Haubourg
Good points.  I'm thinking perhaps the solution is to use Sqlite/spatialite (don't get me wrong - I like and use spatialite a lot).  But perhaps use the python spatialite driver directly, rather than the QgsVectorFileWriter class as I am currently doing, as I cannot see any OGR options for writing multiple layers to a single sqlite db.

I have a version working using the QDataStream, which goes well .. It shouldn't be too hard to modify that to use spatialite.

You also mentioned losing data completely .. Is is possible that this was data that hadn't been saved before QGIS was closed (ie you were still in editing mode).  I think this may be a QGIS bug in 1.8.  On ubuntu I am finding that QGIS does not prompt for for uncommitted edits.  I'm sure that it used to do that..

Cheers
Chris



-----Original Message-----
From: haubourg [mailto:[hidden email]]
Sent: Thursday, 26 July 2012 7:55 p.m.
To: [hidden email]
Subject: Re: [Qgis-developer] Memory data provider persistence

Having a specific memory storage have pros and cons that you explained very well Chris.
 From a user point of view, one drawback is that if something goes wrong in memory layer loading whe project opens, or on memory layer save, data is lost. So User will for sure ask for a "Add memory layer" provider.. because they will try to reopen that dataset they see with their project.  
That's why I prefer SQLlite, already easy to open. It stores vector or raster, is compact, no install...  BUT, we will be dependent of that tool and associated libraries, that's true.
My bet is that Spatialite will be a MANDATORY tool for people who don't have postgis access or skills. One simple argument: it store long field names, and is fast.  So QGIS should stick close to spatialite as much as it does for postgis. If we all agree on that roadmap, it would not be so risquy to choose spatialite ..
Hope this helps..



--
View this message in context: http://osgeo-org.1560.n6.nabble.com/Memory-data-provider-persistence-tp4108012p4990975.html
Sent from the Quantum GIS - Developer mailing list archive at Nabble.com.

This message contains information, which is confidential and may be subject to legal privilege. If you are not the intended recipient, you must not peruse, use, disseminate, distribute or copy this message. If you have received this message in error, please notify us immediately (Phone 0800 665 463 or [hidden email]) and destroy the original message. LINZ accepts no responsibility for changes to this email, or for any attachments, after its transmission from LINZ. Thank You.
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Memory data provider persistence

Chris Crook
In reply to this post by Even Rouault
Hi Even

Sorry .. lost wasn't quite correct.  The point is that my code for saving some users data and then reloading needs to take account of whether they have a field called fid or not if I want the list of fields unchanged after saving and reloading the data.

Either I have to record somewhere that the fid field is real (ie was in the original data set), and set the GML_EXPOSE_FID accordingly so that it appears in the reloaded data, or alternatively rename the users fields an a reversible way before I save the data so that none of the saved fields can be called fid (eg prefix all fields with "u_" for example), and set GML_EXPOSE_FID to NO.

At least that is my understanding of the way the driver works now?

Cheers
Chris

-----Original Message-----
From: Even Rouault [mailto:[hidden email]]
Sent: Friday, 27 July 2012 7:59 a.m.
To: Chris Crook
Cc: [hidden email]; haubourg
Subject: Re: [Qgis-developer] Memory data provider persistence

Le jeudi 26 juillet 2012 21:41:57, Chris Crook a écrit :

> Hi Even
>
> Thanks for the suggestion and code examples - very helpful. I haven't
> followed up on this yet partly because this has prompted me to look at
> other storage formats for the data (something I'd been putting off).
>
> If I understand the GML driver documentation properly then I would
> have to manage the possibility of the user having a field called FID,
> as after I've saved the data, then reloaded with GMS_EXPOSE_FID set to
> NO, that field will be lost?

If a user creates a 'fid' field, then that one will be used to set the fid attribute of the element. It will not be "lost" strictly speaking, but indeed if GML_EXPOSE_FID is set to NO, then it will not be visible.

> As you suggest, renaming the fields before saving and then after
> reloading would fix handle this.

Not sure to understand you here : are you speaking about the mechanism for saving the old value of GML_EXPOSE_FID, setting GML_EXPOSE_FID=OLD, calling
OGROpen() and restoring the original value of GML_EXPOSE_FID ? If so, this will not change anything about the behaviour with a user field called 'fid'.

The intention here was not to interfere with other parts of QGIS that might manipulate GML files. For example, if a user opens a GML file through "Add Vector Layer", he might want to see the fid reported.

This message contains information, which is confidential and may be subject to legal privilege. If you are not the intended recipient, you must not peruse, use, disseminate, distribute or copy this message. If you have received this message in error, please notify us immediately (Phone 0800 665 463 or [hidden email]) and destroy the original message. LINZ accepts no responsibility for changes to this email, or for any attachments, after its transmission from LINZ. Thank You.
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Memory data provider persistence

Even Rouault
Le jeudi 26 juillet 2012 22:52:47, Chris Crook a écrit :

> Hi Even
>
> Sorry .. lost wasn't quite correct.  The point is that my code for saving
> some users data and then reloading needs to take account of whether they
> have a field called fid or not if I want the list of fields unchanged
> after saving and reloading the data.
>
> Either I have to record somewhere that the fid field is real (ie was in the
> original data set), and set the GML_EXPOSE_FID accordingly so that it
> appears in the reloaded data, or alternatively rename the users fields an
> a reversible way before I save the data so that none of the saved fields
> can be called fid (eg prefix all fields with "u_" for example), and set
> GML_EXPOSE_FID to NO.
>
> At least that is my understanding of the way the driver works now?

Well, you could do it, but I think you're infliciting yourself too much pain,
but it's up to you as you're the coder ;-) Personnaly, I'd just prevent the
user from creating a field called "fid", saying it is a reserved name. But your
solutions would certainly work.
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: Memory data provider persistence

Chris Crook
In reply to this post by pcav
Hi Paolo

This is about saving the persisting the data in memory data provider layers, which by default is discarded when the project is closed (although the layer definition is in the project file, all the data is lost).  It is not related to fTools.  It is specific to the memory data provider.

However having a common storage area for project related data does sound a very good idea.  At the moment the solution for the memory layers is leaning towards an Sqlite storage, say in a file called project.qgis.mldb.  However this could be held in a common sqlite database, provided we respect a table naming convention.  That would certainly make sense for users wanting to move projects around.

Is that the sort of thing you are thinking of?

Cheers
Chris

-----Original Message-----
From: Paolo Cavallini [mailto:[hidden email]]
Sent: Thursday, 26 July 2012 5:29 p.m.
To: [hidden email]
Subject: Re: [Qgis-developer] Memory data provider persistence

Il 26/07/2012 06:31, Chris Crook ha scritto:
> So what I'm considering is rebuilding the plugin to read and write the memory layer data into a single file, say project.qgis.mldata, instead of the current multiple gml files.  
>
> Thoughts?
Hi Chris,
please consider that much the same problem applies to other plugins, especially Sextante and GDALTools. Now that we are going to have a Save as... facility also for rasters, this could also be included.
So a solution designed only for fTools seems suboptimal.
All the best.

--
Paolo Cavallini - Faunalia
www.faunalia.eu
Full contact details at www.faunalia.eu/pc Nuovi corsi QGIS e PostGIS: http://www.faunalia.it/calendario


This message contains information, which is confidential and may be subject to legal privilege. If you are not the intended recipient, you must not peruse, use, disseminate, distribute or copy this message. If you have received this message in error, please notify us immediately (Phone 0800 665 463 or [hidden email]) and destroy the original message. LINZ accepts no responsibility for changes to this email, or for any attachments, after its transmission from LINZ. Thank You.
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
12