New Vector API

classic Classic list List threaded Threaded
39 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: New Vector API

Martin Dobias
On Mon, Jan 7, 2013 at 12:42 PM, Matthias Kuhn <[hidden email]> wrote:
> Hi Martin,
>
> as already mentioned by Marco, there seems to be a problem with the latest
> fix. I have multiple postgres layers and after every redraw only one layer
> (random) is drawn (sometimes not even one layer).
>
> When I quit QGIS I get a segmentation fault in the postgres provider
> (QgsPostgresConn, member methods called on a NULL pointer), backtrace
> attached.

Matthias,

the latest fix should solve both problems. In the previous commit I
have removed automatic closing of postgres cursor when iterating over
features finishes and this caused that a new cursor (with the same
name) was being opened while the old one still existed.

Martin
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: New Vector API

Matthias Kuhn
Hi Martin,

On 01/07/2013 11:04 PM, Martin Dobias wrote:

> On Mon, Jan 7, 2013 at 12:42 PM, Matthias Kuhn <[hidden email]> wrote:
>> Hi Martin,
>>
>> as already mentioned by Marco, there seems to be a problem with the latest
>> fix. I have multiple postgres layers and after every redraw only one layer
>> (random) is drawn (sometimes not even one layer).
>>
>> When I quit QGIS I get a segmentation fault in the postgres provider
>> (QgsPostgresConn, member methods called on a NULL pointer), backtrace
>> attached.
> Matthias,
>
> the latest fix should solve both problems. In the previous commit I
> have removed automatic closing of postgres cursor when iterating over
> features finishes and this caused that a new cursor (with the same
> name) was being opened while the old one still existed.
>
> Martin
Layers are rendered correctly again.

However, dataProvider.getFeatures() doesn't return any features in my
plugin ( the same instructions used to work before your last two commits ).

The postgres provider logs the following:

-------------
2 cursor states lost.
SQL: DECLARE qgisf6 BINARY CURSOR FOR SELECT
asbinary(force_2d("situation_geometry"),'NDR'),"gid","obj_id"::text,"type"::text,"node_type"::text,"level"::text,"usage_current"::text,"cover_level"::text,"description"::text,asewkt("detail_geometry")
FROM "qgep"."vw_network_node"
Result: 7 (ERROR:  cursor "qgisf6" already exists
)
-------------
Query: move absolute 0 in qgisf6 returned 7 [ERROR:  cursor "qgisf6"
does not exist
]
-------------

Regards,
Matthias
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: New Vector API

Matthias Kuhn
In reply to this post by Martin Dobias
Hi Martin,

This mail concerns the vectorlayercache outlined in section 8) of your mail.

I'm currently working on the attribute table which has some internal
caching already. As I'm currently rewriting the attribute table, I moved
this code outside and generalized it as good as possible. This work is
currently available in my branch vectorlayercache [0]. Do you think this
could be used instead of your geometry cache? It should be able to cache
geometries of features (although I only tested attributes so far) and
update and invalidate them if required.

I have also the idea to create a QgsVectorDataProviderCacheProxy class
which inherits QgsVectorDataProvider and which you can use as a proxy
between your app and the DataProvider. I've some unfinished code here
which provides such functionality. But for now, the cache in front of
the VectorLayer should already help a lot.

[0] https://github.com/matthias-kuhn/Quantum-GIS/tree/vectorlayercache

Regards,
Matthias

On 12/28/2012 05:03 PM, Martin Dobias wrote:

> Hi everyone,
>
> it seems that now it is a good time to ask for a broader review of the
> work I have been doing during recent months: essentially making QGIS
> vector API more flexible and more ready for introduction of threading
> for rendering. That resulted in a greater refactoring of some parts of
> QgsVectorLayer class and QgsVectorDataProvider implementations.
> Everything is in new_vector_api branch in QGIS repository on GitHub
> [1].
>
> There are few things that are not finished, but should not take too
> much work to sort out:
> - disabled providers - mssql, osm, sqlanywhere, wfs - not yet updated to new API
> - disabled mapserver - there are few things to update
>
> If no serious problems will be found, I would like to merge the branch
> to master and continue working on that on master branch to avoid the
> possibility to drift (again) too much from the quick development that
> happens on master. In short term I plan to do some polishing and
> fixing bugs, then eventually start looking at the threading again.
>
> For more details about what happened in the branch, see the text below
> (long read!). A great help from early testers would be to compile the
> branch, try to do some "usual" stuff with vectors and see if things
> break apart (crashes? data corruption?).
>
> Looking forward to your comments!
>
> Regards
> Martin
>
> [1] https://github.com/qgis/Quantum-GIS/tree/new_vector_api
>
>
>
> QGIS VECTOR API CHANGES
>
> 1. QgsFeature (changes)
>
> a) access to attributes by name. Vector data providers and vector
> layer assign pointer to the fields in QgsFeature, so it is possible to
> set/get attributes by their name. This is not as efficient as with
> direct access by index, but often this convenience is useful for
> users.
>
> b) attributes are stored in a QVector container rather than a QMap.
> The major advantage is simplification of logic related to handling of
> attributes: it's not possible to have "holes" in the array of
> fields/attributes. Currently it is possible that for example layer
> with three fields returns indexes 0,1,3 - but it is not so common nor
> obvious, so it's a common source of errors. After refactoring there
> must not be any holes, so a layer with three fields always returns
> indexes 0,1,2. When iterating over layer's features, QgsFeature always
> contains a vector of all layer's attributes. In case the client has
> not requested attributes to be fetched, the attributes contain invalid
> values.
>
>
> 2. QgsFields (new class)
>
> Just like attributes within a feature are stored in a vector instead
> of map, also layer's fields are now stored in a vector. QgsFields is a
> new class that mimics QVector API and adds two more pieces of
> functionality:
>
> a) fast lookup of field index from name. When QgsFields is populated
> with fields, it creates a map of fields that facilitates fast lookups
>
> b) detection of field origin. When working with a vector layer, it is
> sometimes useful to find out origin of the field - whether it comes
> from provider, from a join or whether it is a newly added field (not
> committed). In the future we could add also expression-based fields,
> creating a field calculator that calculates the values on the fly.
>
>
> 3. QgsFeatureRequest (new class)
>
> Class that encapsulates requests for features to a vector layer or
> vector data provider. Right now in master branch, the request is
> defined by arguments to select() method. That's not very flexible nor
> simple to use. Feature request class allows easier extensibility in
> future (support generic expression filter, native provider's SQL
> filter...).
>
> Without any customization, the request will ask for all features with
> geometries and attributes - somehow better default that the current
> one that does not fetch attributes if their list is not explicitly
> given.
>
> (I'm not yet completely happy with the API of this class, so it may be
> changed to some degree. Suggestions are welcome.)
>
> Examples:
> - fetch all features:
>      QgsFeatureRequest()
> - fetch all features, only one attribute
>      QgsFeatureRequest().setSubsetOfAttributes(QStringList("myfield"),
> provider->fields())
> - fetch all features, without geometries
>      QgsFeatureRequest().setFlags(QgsFeatureRequest::NoGeometry)
> - fetch only features from particular extent
>      QgsFeatureRequest().setFilterRect(QgsRectangle(0,0,1,1))
> - fetch only one feature
>      QgsFeatureRequest().setFilterFid(45)
>
>
>
> 4. QgsFeatureIterator (new class)
>
> The iterator class allows iteration over features of a vector layer or
> a provider. It contains the usual nextFeature() method that fills
> given QgsFeature class with data.
>
> Internally, this class is a wrapper around QgsAbstractFeatureIterator
> interface that is implemented by each vector data provider and by
> vector layer. In theory we could use directly
> QgsAbstractFeatureIterator pointers, but this wrapper allows us to use
> it as a local variable, so it gets destroyed automatically when it
> gets out of scope, not requiring an explicit "delete" call that would
> be necessary with a pointer (easy to forget, right?)
>
>
> 5. Access to features
>
> Vector layer and providers implement one important method call:
> getFeatures(). It takes single argument (QgsFeatureRequest) and
> returns QgsFeatureIterator. This replaces select(), nextFeature(),
> rewind() and featureAtId() methods.
>
> This approach allows users to create multiple iterators over a vector
> layer (or a vector data provider). But currently that's not supported
> because this needs support in the providers. At some point, we could
> add support for simultaneous iterators - something that old API does
> not allow at all.
>
> When QgsFeatureIterator instance of a particular layer is closed (by
> calling close() or destructed), then a new getFeatures() call may be
> issued.
>
>
> 6. QgsVectorLayerEditBuffer (new class)
>
> All editing functionality of vector layer has been moved out of
> QgsVectorLayer into a new class that stores everything related to
> editing: added features, removed features, changed geometries, changed
> attribute values, added attributes, removed attributes. This class is
> accessible from QgsVectorLayer, but it exists only when the layer is
> in editing mode - it is deleted once features are rolled back or
> committed.
>
> The undo/redo support has been completely rewritten and greatly
> simplified: instead of one undo command that gathers changes within
> the edit buffer, there is now one undo command implementation for each
> action. The undo commands can be grouped together with QUndoStack
> macros. The new implementation ensures that the undo stack does not
> get out of sync, because all editing changes are stored onto undo
> stack (unlike current implementation in master branch where
> begin/end-UndoCommand() methods must be called before/after edit
> operations to ensure correct behavior of undo).
>
>
> 7. QgsVectorLayerEditUtils (new class)
>
> Advanced editing operations that operate on vector layers (e.g. add
> part) that can be composed from one or more simple edit operations
> have been moved out of QgsVectorLayer and its edit buffer, so that the
> vector layer API is not polluted by methods that do not need access to
> its private members.
>
>
> 8. QgsVectorLayerCache (new class)
>
> To speed up geometry editing, QGIS keeps a cache of geometries
> currently shown in map canvas. This cache has been decoupled from
> editing operations and moved to a separate class to keep things tidy.
> In the future we should be able to add more general caching scheme to
> allow faster rendering of vectors. Also, this might be a good place to
> store index of feature ID -> row number that is calculated every time
> when attribute table is opened.
>
>
> 9. Python
>
> Python API has been updated to support functionality from the points
> above, that is access by attribute name and support for python
> iterators, e.g.:
>
> for feature in layer.getFeatures(request):
>    print feature["road_id"]
> _______________________________________________
> Qgis-developer mailing list
> [hidden email]
> http://lists.osgeo.org/mailman/listinfo/qgis-developer

_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: New Vector API

Martin Dobias
Hi Matthias

On Tue, Jan 15, 2013 at 11:46 PM, Matthias Kuhn <[hidden email]> wrote:

> Hi Martin,
>
> This mail concerns the vectorlayercache outlined in section 8) of your mail.
>
> I'm currently working on the attribute table which has some internal caching
> already. As I'm currently rewriting the attribute table, I moved this code
> outside and generalized it as good as possible. This work is currently
> available in my branch vectorlayercache [0]. Do you think this could be used
> instead of your geometry cache? It should be able to cache geometries of
> features (although I only tested attributes so far) and update and
> invalidate them if required.
>
> I have also the idea to create a QgsVectorDataProviderCacheProxy class which
> inherits QgsVectorDataProvider and which you can use as a proxy between your
> app and the DataProvider. I've some unfinished code here which provides such
> functionality. But for now, the cache in front of the VectorLayer should
> already help a lot.

Just out of curiosity, what are you rewriting in the attribute table -
is it just the cache or does it involve more changes?

Your idea of having a cache proxy vector data provider is interesting.
Such transparent approach would enable us to remove caching from
vector data provider implementations (e.g. wfs) and transparently add
caching to the ones where caching would be useful (e.g.
delimitedtext). I need to give it some more thoughts what impact that
would have.

The code for cache in your github repository implements cache with
feature ids as keys. That is useful for attribute table queries,
however for rendering it does not help much. In the cache we will need
also a spatial index for fast spatial queries used for map rendering,
identification and geometry editing. Without spatial index, the cache
is quite slow if the whole cache need to be traversed in order to find
features in a specified rectangle. As of now, the cache you've done is
bound to a one particular query it gets: I would be most interested in
a general cache to avoid having situations where cache would be filled
by results of a first query and then flushed and filled with results
of a following query.

The caching behavior should be ideally configurable: e.g. which
layers/providers to cache (probably we don't want to cache huge
layers), what to cache (everything / only geometries / only attributes
/ geometries and some attributes), cache size limit.

The next question is how to invalidate the cache, so that if data have
been changed in the backend, the user will not end up with a cached
version forever. In some providers I think it's possible to detect the
updates automatically, but some providers do not have any such
notifications. Maybe we should allow users to set a timeout - after
specified interval the cache would need to check again with the
backend whether the features have not been altered.

Back to your original question whether such improved cache could be
used instead of the current geometry cache: yes, it could, in case it
will include a spatial index. We are really in a need of a general
caching mechanism, so I would welcome any work in this area. The
speedup of rendering from cache should be significant. The current
caching of geometries is there just to preserve the functionality of
moveVertex() and other functions from vector layer that were using
cached geometries before.

Regards
Martin
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: New Vector API

Martin Dobias
In reply to this post by Matthias Kuhn
On Tue, Jan 8, 2013 at 9:46 AM, Matthias Kuhn <[hidden email]> wrote:
>
> Layers are rendered correctly again.
>
> However, dataProvider.getFeatures() doesn't return any features in my plugin
> ( the same instructions used to work before your last two commits ).

Matthias,
my recent update to feature iterators should fix this and some more issues, see:

https://github.com/qgis/Quantum-GIS/commit/a6c5fd875bc73e4775b825640a7cedcc2f5a8832

The default behavior for providers is now that they close any previous
active iterator. Later I will be adding a new provider capability
where the provider could state that it supports concurrent iterators.

Regards
Martin
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: New Vector API

Matthias Kuhn
In reply to this post by Martin Dobias
Hi Martin,

see answers below

On 01/16/2013 10:13 PM, Martin Dobias wrote:

> Hi Matthias
>
> On Tue, Jan 15, 2013 at 11:46 PM, Matthias Kuhn <[hidden email]> wrote:
>> Hi Martin,
>>
>> This mail concerns the vectorlayercache outlined in section 8) of your mail.
>>
>> I'm currently working on the attribute table which has some internal caching
>> already. As I'm currently rewriting the attribute table, I moved this code
>> outside and generalized it as good as possible. This work is currently
>> available in my branch vectorlayercache [0]. Do you think this could be used
>> instead of your geometry cache? It should be able to cache geometries of
>> features (although I only tested attributes so far) and update and
>> invalidate them if required.
>>
>> I have also the idea to create a QgsVectorDataProviderCacheProxy class which
>> inherits QgsVectorDataProvider and which you can use as a proxy between your
>> app and the DataProvider. I've some unfinished code here which provides such
>> functionality. But for now, the cache in front of the VectorLayer should
>> already help a lot.
> Just out of curiosity, what are you rewriting in the attribute table -
> is it just the cache or does it involve more changes?

Have a look at the dual view discussion on this list. It's explained
more there.
>
> Your idea of having a cache proxy vector data provider is interesting.
> Such transparent approach would enable us to remove caching from
> vector data provider implementations (e.g. wfs) and transparently add
> caching to the ones where caching would be useful (e.g.
> delimitedtext). I need to give it some more thoughts what impact that
> would have.
Let me know when you do. I have now implemented such a thing in my
plugin and would love to have this functionality in core.

>
> The code for cache in your github repository implements cache with
> feature ids as keys. That is useful for attribute table queries,
> however for rendering it does not help much. In the cache we will need
> also a spatial index for fast spatial queries used for map rendering,
> identification and geometry editing. Without spatial index, the cache
> is quite slow if the whole cache need to be traversed in order to find
> features in a specified rectangle. As of now, the cache you've done is
> bound to a one particular query it gets: I would be most interested in
> a general cache to avoid having situations where cache would be filled
> by results of a first query and then flushed and filled with results
> of a following query.
Yes it is bound to feature ids by now. My idea was, that you will keep
track of keys of features of interest anyway. Let's say, the keys of
features of a particular rectangle.

> The caching behavior should be ideally configurable: e.g. which
> layers/providers to cache (probably we don't want to cache huge
> layers), what to cache (everything / only geometries / only attributes
> / geometries and some attributes), cache size limit.
My cache approach does most of this: only geometries, only attributes or
both (A subset of attributes is still solved unsatisfactory, but that
wouldn't be too hard to add) and cache limits. The only thing missing is
the possibility to configure which layers to cache (Needs to be a higher
level approach maybe? Layer settings?)
>
> The next question is how to invalidate the cache, so that if data have
> been changed in the backend, the user will not end up with a cached
> version forever. In some providers I think it's possible to detect the
> updates automatically, but some providers do not have any such
> notifications. Maybe we should allow users to set a timeout - after
> specified interval the cache would need to check again with the
> backend whether the features have not been altered.
It would be nice to get this information from providers which do have
the possibility, for others, see below.
>
> Back to your original question whether such improved cache could be
> used instead of the current geometry cache: yes, it could, in case it
> will include a spatial index. We are really in a need of a general
> caching mechanism, so I would welcome any work in this area. The
> speedup of rendering from cache should be significant. The current
> caching of geometries is there just to preserve the functionality of
> moveVertex() and other functions from vector layer that were using
> cached geometries before.

To solve these problems I was thinking about adding an abstract
interface ( QgsCachItemManager or something) which provides at least the
method
* bool requiresCaching( QgsFeature& )

It could then also be extended to check if the cache is able to answer a
certain select operation (in which case it would return a QgsFeatureIds
list). So it could keep track of a spatial index, cache age and so on.
Speak custom tailored caching depending on the special use-case.
Accomplishing that would require some small internal changes, as we need
to keep track of removed items. It's using QCache now, which as far as I
know does not communicate, when a certain item is removed (it cleans
silently). So either we'd have to replace QCache by some other mechanism
or let the cache-item destructor emit this signal.

I have to say, that I'm a bit limited in time. At the moment, the
attribute editor has a higher priority for me (I would really like to
have this ready by 2.0) so the caching was more of a side-effect.

>
> Regards
> Martin
Regards
Matthias
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: New Vector API

Matthias Kuhn
In reply to this post by Martin Dobias
Martin,

Works for me. My plugin is ready for the new vector API then. Thank you
for your work.

What are your plans for merging?

Regards,
Matthias

On 01/17/2013 12:18 AM, Martin Dobias wrote:

> On Tue, Jan 8, 2013 at 9:46 AM, Matthias Kuhn <[hidden email]> wrote:
>> Layers are rendered correctly again.
>>
>> However, dataProvider.getFeatures() doesn't return any features in my plugin
>> ( the same instructions used to work before your last two commits ).
> Matthias,
> my recent update to feature iterators should fix this and some more issues, see:
>
> https://github.com/qgis/Quantum-GIS/commit/a6c5fd875bc73e4775b825640a7cedcc2f5a8832
>
> The default behavior for providers is now that they close any previous
> active iterator. Later I will be adding a new provider capability
> where the provider could state that it supports concurrent iterators.
>
> Regards
> Martin

_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: New Vector API

Martin Dobias
Hi Matthias

On Mon, Jan 21, 2013 at 10:17 AM, Matthias Kuhn <[hidden email]> wrote:
> Martin,
>
> Works for me. My plugin is ready for the new vector API then. Thank you for
> your work.
>
> What are your plans for merging?

I will be merging new vector API in next few days.

Martin
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: New Vector API

Larry_S
Hi Martin,

What do you think about tagging the repository, right before merging your branch? That way there would be a reference point for building a pre-merge state, useful for nightly build maintainers who can offer an additional download of that tagged, semi-stable build.

Currently tagging seems to only be done for releases:
https://github.com/qgis/Quantum-GIS/tags

Larry


On Wed, Jan 23, 2013 at 3:37 PM, Martin Dobias <[hidden email]> wrote:
Hi Matthias

On Mon, Jan 21, 2013 at 10:17 AM, Matthias Kuhn <[hidden email]> wrote:
> Martin,
>
> Works for me. My plugin is ready for the new vector API then. Thank you for
> your work.
>
> What are your plans for merging?

I will be merging new vector API in next few days.

Martin
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer


_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: New Vector API

Martin Dobias
Hi Larry

On Wed, Jan 23, 2013 at 11:52 PM, Larry Shaffer <[hidden email]> wrote:
> Hi Martin,
>
> What do you think about tagging the repository, right before merging your
> branch? That way there would be a reference point for building a pre-merge
> state, useful for nightly build maintainers who can offer an additional
> download of that tagged, semi-stable build.

No problem to tag master branch before the merge.

Regards
Martin
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: New Vector API

Matthias Kuhn
In reply to this post by Martin Dobias
Hi Martin, hi List

On 04.01.2013 22:19, Martin Dobias wrote:

> Hi Matthias
>
> thanks for your mail.
>
> On Thu, Jan 3, 2013 at 10:37 AM, Matthias Kuhn <[hidden email]> wrote:
>> You've been talking about updating to PyQt4 SIP version 2. I noticed this
>> change did not yet happen. What are your plans regarding this update?
> Yes, the plans are still there. I'm only wondering what would be the
> best time to switch - right now within the branch or a bit later once
> the branch is merged and the new code settles down a bit? I can see a
> slight advantage of the latter approach of having a greater chance of
> separating problems caused by the merge and problems caused by the SIP
> API change, but I do not have a strong opinion here.
>
>
Despite feature freeze, I'd like to ask what happened to that plan. As
far as I know, this change has not been made, and if this change is not
made before QGIS 2.0 it will have to wait for 3.0 (This is not something
to be done for 2.1).
It will probably be a small change (to switch in the core) but with a
huge impace (concerning plugins).

Matthias
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: New Vector API

Nathan Woodrow
Yeah if we could flick it on it would make a load of difference to how clean the code is.

- Nathan


On Tue, Apr 2, 2013 at 11:10 PM, Matthias Kuhn <[hidden email]> wrote:
Hi Martin, hi List

On <a href="tel:04.01.2013%2022" value="+61401201322">04.01.2013 22:19, Martin Dobias wrote:
> Hi Matthias
>
> thanks for your mail.
>
> On Thu, Jan 3, 2013 at 10:37 AM, Matthias Kuhn <[hidden email]> wrote:
>> You've been talking about updating to PyQt4 SIP version 2. I noticed this
>> change did not yet happen. What are your plans regarding this update?
> Yes, the plans are still there. I'm only wondering what would be the
> best time to switch - right now within the branch or a bit later once
> the branch is merged and the new code settles down a bit? I can see a
> slight advantage of the latter approach of having a greater chance of
> separating problems caused by the merge and problems caused by the SIP
> API change, but I do not have a strong opinion here.
>
>
Despite feature freeze, I'd like to ask what happened to that plan. As
far as I know, this change has not been made, and if this change is not
made before QGIS 2.0 it will have to wait for 3.0 (This is not something
to be done for 2.1).
It will probably be a small change (to switch in the core) but with a
huge impace (concerning plugins).

Matthias
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer


_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: New Vector API

Alexander Bruy
In reply to this post by Matthias Kuhn
Hi,

I would vote to switch now, because later we will have much more
plugins to update. Also have two breaks (first — new API, second
— PyQt API update) is not good IMO.

On Tue, 02 Apr 2013 15:10:05 +0200
Matthias Kuhn <[hidden email]> wrote:

> Despite feature freeze, I'd like to ask what happened to that plan. As
> far as I know, this change has not been made, and if this change is not
> made before QGIS 2.0 it will have to wait for 3.0 (This is not something
> to be done for 2.1).
> It will probably be a small change (to switch in the core) but with a
> huge impace (concerning plugins).


--
Alexander Bruy
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: New Vector API

Andreas Neumann-4
Hi,

+1 to switch now to the new PyQT API. We already a big break - so lets
break now (even more) and then stay stable for a while. Plugin
developers will have to rework their plugins anyway because of the new
Vector API and the switch to the new PyQT API can be done in the same step.

Andreas

Am 02.04.2013 15:38, schrieb Alexander Bruy:

> Hi,
>
> I would vote to switch now, because later we will have much more
> plugins to update. Also have two breaks (first — new API, second
> — PyQt API update) is not good IMO.
>
> On Tue, 02 Apr 2013 15:10:05 +0200
> Matthias Kuhn <[hidden email]> wrote:
>
>> Despite feature freeze, I'd like to ask what happened to that plan. As
>> far as I know, this change has not been made, and if this change is not
>> made before QGIS 2.0 it will have to wait for 3.0 (This is not something
>> to be done for 2.1).
>> It will probably be a small change (to switch in the core) but with a
>> huge impace (concerning plugins).
>
>

_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: New Vector API

Ivan Mincik
In reply to this post by Matthias Kuhn

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 04/02/2013 03:10 PM, Matthias Kuhn wrote:
> Hi Martin, hi List
>
> On 04.01.2013 22:19, Martin Dobias wrote:
>> Hi Matthias
>>
>> thanks for your mail.
>>
>> On Thu, Jan 3, 2013 at 10:37 AM, Matthias Kuhn <[hidden email]>
wrote:
>>> You've been talking about updating to PyQt4 SIP version 2. I noticed
this

>>> change did not yet happen. What are your plans regarding this update?
>> Yes, the plans are still there. I'm only wondering what would be the
>> best time to switch - right now within the branch or a bit later once
>> the branch is merged and the new code settles down a bit? I can see a
>> slight advantage of the latter approach of having a greater chance of
>> separating problems caused by the merge and problems caused by the SIP
>> API change, but I do not have a strong opinion here.
>>
>>
> Despite feature freeze, I'd like to ask what happened to that plan. As
> far as I know, this change has not been made, and if this change is not
> made before QGIS 2.0 it will have to wait for 3.0 (This is not something
> to be done for 2.1).
> It will probably be a small change (to switch in the core) but with a
> huge impace (concerning plugins).

I would also vote to switch now when API is broken.


- --
Ivan Mincik
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iQEcBAEBAgAGBQJRWypZAAoJEPfdLsR5UpoeXIMIAKw/IIaDGdRGlF8Q2tKpbvnR
VrjteXlFQB+u2g+X6DCgSnbQKUEdpS+HflFJjVVNcTIUMOarMx0qn8nMe8MUiSVE
C/IpKvysW4FlF/bhSdpER/yj42oVNBfzcc+eX6rfZl3qNoidNGN6U+2hvpMDKLX0
oPr6YqPUNwjj9L+MlaV70jBM3kZSPs9dBx9cvktBZeXWH0n9iJpB4vpsSgb6SwDa
CYL6IJZSwCohVdQ7pWOUnb0eQ/Te2GKx9sv22Tlh1rahr7VBWBrKCXssyrkDNiDL
ZjACCof+DxkUOGASDOpFCVbLQDWZsbC2s6eeZxlXNNswycxGch+p2idKeozPluA=
=spqX
-----END PGP SIGNATURE-----

_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: New Vector API

Martin Dobias
In reply to this post by Matthias Kuhn
Hi

On Tue, Apr 2, 2013 at 3:10 PM, Matthias Kuhn <[hidden email]> wrote:
> Despite feature freeze, I'd like to ask what happened to that plan. As
> far as I know, this change has not been made, and if this change is not
> made before QGIS 2.0 it will have to wait for 3.0 (This is not something
> to be done for 2.1).
> It will probably be a small change (to switch in the core) but with a
> huge impace (concerning plugins).

I have a patch pending on my machine, but I have found out it needs
more work than just running few "sip.setapi(...)" calls. In PyQt4
QVariant API v2 there is basically no QVariant anymore. Given that we
use QVariant.Type for things like defining QgsField types, we need to
find a workaround (in PyQt4 they had a bit similar situation with
QSettings).

I still would like to get it into 2.0, but recently I have been too
busy with real life to do anything with it. In the following weeks I
should be able to find more time, so hopefully we will make a switch.

Martin
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: New Vector API

Matthias Kuhn
Hi

On Mit 03 Apr 2013 11:00:15 CEST, Martin Dobias wrote:

> Hi
>
> On Tue, Apr 2, 2013 at 3:10 PM, Matthias Kuhn <[hidden email]> wrote:
>> Despite feature freeze, I'd like to ask what happened to that plan. As
>> far as I know, this change has not been made, and if this change is not
>> made before QGIS 2.0 it will have to wait for 3.0 (This is not something
>> to be done for 2.1).
>> It will probably be a small change (to switch in the core) but with a
>> huge impace (concerning plugins).
>
> I have a patch pending on my machine, but I have found out it needs
> more work than just running few "sip.setapi(...)" calls. In PyQt4
> QVariant API v2 there is basically no QVariant anymore. Given that we
> use QVariant.Type for things like defining QgsField types, we need to
> find a workaround (in PyQt4 they had a bit similar situation with
> QSettings).
>
> I still would like to get it into 2.0, but recently I have been too
> busy with real life to do anything with it. In the following weeks I
> should be able to find more time, so hopefully we will make a switch.
>
> Martin

According to this discussion, most people would appreciate the switch.

I would like to have this change included as fast as possible, to give
plugin developers as much time as possible to make the necessary
changes before 2.0 is released.

Martin, if you are still too busy, would it be possible to push your
branch and create a ToDo list, so others could jump in if they find
some time?

_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: New Vector API

Martin Dobias
Hi Matthias

On Wed, Apr 3, 2013 at 11:06 AM, Matthias Kuhn <[hidden email]> wrote:

>
> According to this discussion, most people would appreciate the switch.
>
> I would like to have this change included as fast as possible, to give
> plugin developers as much time as possible to make the necessary
> changes before 2.0 is released.
>
> Martin, if you are still too busy, would it be possible to push your
> branch and create a ToDo list, so others could jump in if they find
> some time?

I will have a look at that during the weekend to make see what else is
missing - if that will be too much work, I will just put the initial
work somewhere so that others can help.

Martin
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: New Vector API

Matthias Kuhn
In reply to this post by Martin Dobias
Hi List, Hi Martin,

TLDR; New QgsFeatureRequest filters
- FilterExpression
- FilterFids
Available here [1]

For another project I'm currently working on, I need additional filter
possibilities in QgsFeatureRequest to filter with a QgsExpression.
As I
- did not want to repeat the code inside each iterator
- but need the possibility to override the behavior (e.g.
vectorlayerfeatureiterator only needs to filter changed features bot
does not need to refilter features already filtered by the provider)
I extended QgsAbstractFeatureIterator to handle this but allow derived
classes to override the method if they want. I introduced a small API
change such that, the nextFeature method calls
nextFeatureFilterExpression which can be overridden by subclasses. Main
filtering is done within a new method fetchFeature which gets called by
nextFeature and should not perform any filtering as long as
FilterExpression is not implemented.
Same goes for FilterFids which takes multiple featureids but is now
implemented to iterate over all features, but can easily be
reimplemented inside providers to be executed on the server side.
The change updates the API, but only for the implementation of iterators
and not for any module which uses the iterator (same nextFeature call as
before).

Anyway, before spending more time on this, I'd like to hear your opinion.

Matthias

[1] https://github.com/matthias-kuhn/Quantum-GIS/tree/request-filters

On 28.12.2012 17:03, Martin Dobias wrote:

> Hi everyone,
>
> it seems that now it is a good time to ask for a broader review of the
> work I have been doing during recent months: essentially making QGIS
> vector API more flexible and more ready for introduction of threading
> for rendering. That resulted in a greater refactoring of some parts of
> QgsVectorLayer class and QgsVectorDataProvider implementations.
> Everything is in new_vector_api branch in QGIS repository on GitHub
> [1].
>
> There are few things that are not finished, but should not take too
> much work to sort out:
> - disabled providers - mssql, osm, sqlanywhere, wfs - not yet updated
to new API

> - disabled mapserver - there are few things to update
>
> If no serious problems will be found, I would like to merge the branch
> to master and continue working on that on master branch to avoid the
> possibility to drift (again) too much from the quick development that
> happens on master. In short term I plan to do some polishing and
> fixing bugs, then eventually start looking at the threading again.
>
> For more details about what happened in the branch, see the text below
> (long read!). A great help from early testers would be to compile the
> branch, try to do some "usual" stuff with vectors and see if things
> break apart (crashes? data corruption?).
>
> Looking forward to your comments!
>
> Regards
> Martin
>
> [1] https://github.com/qgis/Quantum-GIS/tree/new_vector_api
>
>
>
> QGIS VECTOR API CHANGES
>
> 1. QgsFeature (changes)
>
> a) access to attributes by name. Vector data providers and vector
> layer assign pointer to the fields in QgsFeature, so it is possible to
> set/get attributes by their name. This is not as efficient as with
> direct access by index, but often this convenience is useful for
> users.
>
> b) attributes are stored in a QVector container rather than a QMap.
> The major advantage is simplification of logic related to handling of
> attributes: it's not possible to have "holes" in the array of
> fields/attributes. Currently it is possible that for example layer
> with three fields returns indexes 0,1,3 - but it is not so common nor
> obvious, so it's a common source of errors. After refactoring there
> must not be any holes, so a layer with three fields always returns
> indexes 0,1,2. When iterating over layer's features, QgsFeature always
> contains a vector of all layer's attributes. In case the client has
> not requested attributes to be fetched, the attributes contain invalid
> values.
>
>
> 2. QgsFields (new class)
>
> Just like attributes within a feature are stored in a vector instead
> of map, also layer's fields are now stored in a vector. QgsFields is a
> new class that mimics QVector API and adds two more pieces of
> functionality:
>
> a) fast lookup of field index from name. When QgsFields is populated
> with fields, it creates a map of fields that facilitates fast lookups
>
> b) detection of field origin. When working with a vector layer, it is
> sometimes useful to find out origin of the field - whether it comes
> from provider, from a join or whether it is a newly added field (not
> committed). In the future we could add also expression-based fields,
> creating a field calculator that calculates the values on the fly.
>
>
> 3. QgsFeatureRequest (new class)
>
> Class that encapsulates requests for features to a vector layer or
> vector data provider. Right now in master branch, the request is
> defined by arguments to select() method. That's not very flexible nor
> simple to use. Feature request class allows easier extensibility in
> future (support generic expression filter, native provider's SQL
> filter...).
>
> Without any customization, the request will ask for all features with
> geometries and attributes - somehow better default that the current
> one that does not fetch attributes if their list is not explicitly
> given.
>
> (I'm not yet completely happy with the API of this class, so it may be
> changed to some degree. Suggestions are welcome.)
>
> Examples:
> - fetch all features:
>     QgsFeatureRequest()
> - fetch all features, only one attribute
>     QgsFeatureRequest().setSubsetOfAttributes(QStringList("myfield"),
> provider->fields())
> - fetch all features, without geometries
>     QgsFeatureRequest().setFlags(QgsFeatureRequest::NoGeometry)
> - fetch only features from particular extent
>     QgsFeatureRequest().setFilterRect(QgsRectangle(0,0,1,1))
> - fetch only one feature
>     QgsFeatureRequest().setFilterFid(45)
>
>
>
> 4. QgsFeatureIterator (new class)
>
> The iterator class allows iteration over features of a vector layer or
> a provider. It contains the usual nextFeature() method that fills
> given QgsFeature class with data.
>
> Internally, this class is a wrapper around QgsAbstractFeatureIterator
> interface that is implemented by each vector data provider and by
> vector layer. In theory we could use directly
> QgsAbstractFeatureIterator pointers, but this wrapper allows us to use
> it as a local variable, so it gets destroyed automatically when it
> gets out of scope, not requiring an explicit "delete" call that would
> be necessary with a pointer (easy to forget, right?)
>
>
> 5. Access to features
>
> Vector layer and providers implement one important method call:
> getFeatures(). It takes single argument (QgsFeatureRequest) and
> returns QgsFeatureIterator. This replaces select(), nextFeature(),
> rewind() and featureAtId() methods.
>
> This approach allows users to create multiple iterators over a vector
> layer (or a vector data provider). But currently that's not supported
> because this needs support in the providers. At some point, we could
> add support for simultaneous iterators - something that old API does
> not allow at all.
>
> When QgsFeatureIterator instance of a particular layer is closed (by
> calling close() or destructed), then a new getFeatures() call may be
> issued.
>
>
> 6. QgsVectorLayerEditBuffer (new class)
>
> All editing functionality of vector layer has been moved out of
> QgsVectorLayer into a new class that stores everything related to
> editing: added features, removed features, changed geometries, changed
> attribute values, added attributes, removed attributes. This class is
> accessible from QgsVectorLayer, but it exists only when the layer is
> in editing mode - it is deleted once features are rolled back or
> committed.
>
> The undo/redo support has been completely rewritten and greatly
> simplified: instead of one undo command that gathers changes within
> the edit buffer, there is now one undo command implementation for each
> action. The undo commands can be grouped together with QUndoStack
> macros. The new implementation ensures that the undo stack does not
> get out of sync, because all editing changes are stored onto undo
> stack (unlike current implementation in master branch where
> begin/end-UndoCommand() methods must be called before/after edit
> operations to ensure correct behavior of undo).
>
>
> 7. QgsVectorLayerEditUtils (new class)
>
> Advanced editing operations that operate on vector layers (e.g. add
> part) that can be composed from one or more simple edit operations
> have been moved out of QgsVectorLayer and its edit buffer, so that the
> vector layer API is not polluted by methods that do not need access to
> its private members.
>
>
> 8. QgsVectorLayerCache (new class)
>
> To speed up geometry editing, QGIS keeps a cache of geometries
> currently shown in map canvas. This cache has been decoupled from
> editing operations and moved to a separate class to keep things tidy.
> In the future we should be able to add more general caching scheme to
> allow faster rendering of vectors. Also, this might be a good place to
> store index of feature ID -> row number that is calculated every time
> when attribute table is opened.
>
>
> 9. Python
>
> Python API has been updated to support functionality from the points
> above, that is access by attribute name and support for python
> iterators, e.g.:
>
> for feature in layer.getFeatures(request):
>   print feature["road_id"]
> _______________________________________________
> Qgis-developer mailing list
> [hidden email]
> http://lists.osgeo.org/mailman/listinfo/qgis-developer


_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer
12