QGIS Server 3.0 roadmap

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

QGIS Server 3.0 roadmap

Alessandro Pasotti-2
Hi,

I'm glad that Matthias's semi-serious threats to remove the server from source tree and Nyall's complaints have shaken the souls of the server people!

I insist that the server remains a first class citizen and a full featured tool within the QGIS project, but I understand that it should not be a brake for the core refactoring ,slowing down the amazing development that QGIS dev team is doing on the core and gui libraries.

 I've just started a small WIP contribution aimed to solve a first issue (https://github.com/qgis/QGIS/pull/3528).

I wonder what would be the better way to coordinate the efforts, gain in efficience and avoid duplication.

This is a possible roadmap (suggestions welcome!):

- start a QEP for the new server architecture (C++ plugins for the core services etc.) (@rldhont, can you take care of this?)
- a list of prioritized issues (on the qgis hub?, on github? no idea here) ( @nyall, @mkuhn: you know the new and old API very well,  if you could start creating a list of must-have, should-have, nice-to-have issues it would be very helpful)
- create a formal (but completely open) dedicated team of developers and assign issues
- possibly, meet periodically to coordinate the development

Am I dreaming?

--
Alessandro Pasotti
w3:   www.itopen.it

_______________________________________________
Qgis-developer mailing list
[hidden email]
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: QGIS Server 3.0 roadmap

René-Luc Dhont
Hi Alessandro,

I'm working with my colleague David on the QEP about new server architecture.

Regards,

Le 25/09/2016 à 10:06, Alessandro Pasotti a écrit :
Hi,

I'm glad that Matthias's semi-serious threats to remove the server from source tree and Nyall's complaints have shaken the souls of the server people!

I insist that the server remains a first class citizen and a full featured tool within the QGIS project, but I understand that it should not be a brake for the core refactoring ,slowing down the amazing development that QGIS dev team is doing on the core and gui libraries.

 I've just started a small WIP contribution aimed to solve a first issue (https://github.com/qgis/QGIS/pull/3528).

I wonder what would be the better way to coordinate the efforts, gain in efficience and avoid duplication.

This is a possible roadmap (suggestions welcome!):

- start a QEP for the new server architecture (C++ plugins for the core services etc.) (@rldhont, can you take care of this?)
- a list of prioritized issues (on the qgis hub?, on github? no idea here) ( @nyall, @mkuhn: you know the new and old API very well,  if you could start creating a list of must-have, should-have, nice-to-have issues it would be very helpful)
- create a formal (but completely open) dedicated team of developers and assign issues
- possibly, meet periodically to coordinate the development

Am I dreaming?

--
Alessandro Pasotti
w3:   www.itopen.it


_______________________________________________
Qgis-developer mailing list
[hidden email]
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: QGIS Server 3.0 roadmap

giohappy

It sounds great. I agree that QGIS Server needs a dedicated effort and coordination.
We (gis3w) will try to contribute. We are not C++ devs but we have summed a significative experience with QGIS Servers's strengths and weeknesses. We will partecipate to the discussions on the "high level" roadmap ;)

giovanni


Il 26 set 2016 15:35, "René-Luc Dhont" <[hidden email]> ha scritto:
Hi Alessandro,

I'm working with my colleague David on the QEP about new server architecture.

Regards,

Le 25/09/2016 à 10:06, Alessandro Pasotti a écrit :
Hi,

I'm glad that Matthias's semi-serious threats to remove the server from source tree and Nyall's complaints have shaken the souls of the server people!

I insist that the server remains a first class citizen and a full featured tool within the QGIS project, but I understand that it should not be a brake for the core refactoring ,slowing down the amazing development that QGIS dev team is doing on the core and gui libraries.

 I've just started a small WIP contribution aimed to solve a first issue (https://github.com/qgis/QGIS/pull/3528).

I wonder what would be the better way to coordinate the efforts, gain in efficience and avoid duplication.

This is a possible roadmap (suggestions welcome!):

- start a QEP for the new server architecture (C++ plugins for the core services etc.) (@rldhont, can you take care of this?)
- a list of prioritized issues (on the qgis hub?, on github? no idea here) ( @nyall, @mkuhn: you know the new and old API very well,  if you could start creating a list of must-have, should-have, nice-to-have issues it would be very helpful)
- create a formal (but completely open) dedicated team of developers and assign issues
- possibly, meet periodically to coordinate the development

Am I dreaming?

--
Alessandro Pasotti
w3:   www.itopen.it


_______________________________________________
Qgis-developer mailing list
[hidden email]
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

_______________________________________________
Qgis-developer mailing list
[hidden email]
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: QGIS Server 3.0 roadmap

Nyall Dawson
In reply to this post by Alessandro Pasotti-2
On 25 September 2016 at 18:06, Alessandro Pasotti <[hidden email]> wrote:
> Hi,
>
> I'm glad that Matthias's semi-serious threats to remove the server from
> source tree and Nyall's complaints have shaken the souls of the server
> people!

Apologies if I've caused any offence here - I was trying to give
constructive feedback and share some explanation as to the frustration
against maintaining server. I want to stress that i am VERY
appreciative of everyone who has put in effort into server & who has
funded time for devs to work on it.

I also applaud how well everyone has reacted to this, and all the
offers of time/development to improve server for 3.0!

Based on this reaction I think I'll go file a QEP to "rm -rf src/core"
and see if we can shake up some more developers ;)

>
> I insist that the server remains a first class citizen and a full featured
> tool within the QGIS project, but I understand that it should not be a brake
> for the core refactoring ,slowing down the amazing development that QGIS dev
> team is doing on the core and gui libraries.

I agree with this - I don't really want to see server pulled out of
core either.

As I mentioned over on the 3.0 api tracker, the biggest issue I see is
the way server loads projects and messes with the layer registry. I
understand this was probably a design decision made due to api
constraints, but now we've got the opportunity to fix the underlying
issues in core and avoid the messy workarounds in server.

I'd really appreciate it if someone more familiar with the server
architecture could summarise what's missing in core's project/layer
handling. I may be missing something, but would a something like this
help?:

- add a QgsProjectRegistry, with an instance attached to
QgsApplication. Change all the use of QgsProject::instance to
QgsProjectRegistry::currentProject().
- move QgsMapLayerRegistry from an instance to a member within
QgsProject. Core code would then use
currentProject()->mapLayerRegistry() instead of
QgsMapLayerRegistry::instance()
- server could directly pull projects from the registry (via some form
of generated project id) as required and access their map layer
registries directly

Nyall











>
>  I've just started a small WIP contribution aimed to solve a first issue
> (https://github.com/qgis/QGIS/pull/3528).
>
> I wonder what would be the better way to coordinate the efforts, gain in
> efficience and avoid duplication.
>
> This is a possible roadmap (suggestions welcome!):
>
> - start a QEP for the new server architecture (C++ plugins for the core
> services etc.) (@rldhont, can you take care of this?)
> - a list of prioritized issues (on the qgis hub?, on github? no idea here) (
> @nyall, @mkuhn: you know the new and old API very well,  if you could start
> creating a list of must-have, should-have, nice-to-have issues it would be
> very helpful)
> - create a formal (but completely open) dedicated team of developers and
> assign issues
> - possibly, meet periodically to coordinate the development
>
> Am I dreaming?
>
> --
> Alessandro Pasotti
> w3:   www.itopen.it
_______________________________________________
Qgis-developer mailing list
[hidden email]
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: QGIS Server 3.0 roadmap

Luca Manganelli
On Tue, Sep 27, 2016 at 9:19 AM, Nyall Dawson <[hidden email]> wrote:
> As I mentioned over on the 3.0 api tracker, the biggest issue I see is
> the way server loads projects and messes with the layer registry. I
> understand this was probably a design decision made due to api
> constraints, but now we've got the opportunity to fix the underlying
> issues in core and avoid the messy workarounds in server.

Not only this; but the way in which qgis server does the same PostGis
query (for extent or bounding box?) all times for each layer. It woud
be useful to store these informations on the project file, to optimize
times, with a configuration option in the Server tab?
_______________________________________________
Qgis-developer mailing list
[hidden email]
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: QGIS Server 3.0 roadmap

Régis Haubourg
Le mardi 27 septembre 2016 à 09:25 +0200, Luca Manganelli a écrit :
> way in which qgis server does the same PostGis
> query (for extent or bounding box?) all times for each layer. It woud
> be useful to store these informations on the project file, to optimize
> times, with a configuration option in the Server tab?

Hi, on this part, improvements have been made recently since this was a blocker in my previous employer's architecture and we
funded 3Liz to improve things.
René Luc, can you summarize what is the current status on postgis extent and unique id's queries? 

Cheers
Régis
_______________________________________________
Qgis-developer mailing list
[hidden email]
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: QGIS Server 3.0 roadmap

René-Luc Dhont
In reply to this post by Luca Manganelli
Hi Luca


Le 27/09/2016 à 09:25, Luca Manganelli a écrit :

> On Tue, Sep 27, 2016 at 9:19 AM, Nyall Dawson <[hidden email]> wrote:
>> As I mentioned over on the 3.0 api tracker, the biggest issue I see is
>> the way server loads projects and messes with the layer registry. I
>> understand this was probably a design decision made due to api
>> constraints, but now we've got the opportunity to fix the underlying
>> issues in core and avoid the messy workarounds in server.
> Not only this; but the way in which qgis server does the same PostGis
> query (for extent or bounding box?) all times for each layer. It woud
> be useful to store these informations on the project file, to optimize
> times, with a configuration option in the Server tab?

This has been already fixed:
* firstly layer has been cached to avoid reloading it
* the layer description stored Extent and used it

The problem you point is the way Layer is build by QGIS. In the Desktop,
QGIS can build it at each project loading, but server side is could be a
mess. If the cache is not conserved by the system QGIS Server is used,
the layers are reloaded at each request.

> _______________________________________________
> Qgis-developer mailing list
> [hidden email]
> List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

_______________________________________________
Qgis-developer mailing list
[hidden email]
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: QGIS Server 3.0 roadmap

Luca Manganelli
On Tue, Sep 27, 2016 at 9:30 AM, René-Luc Dhont <[hidden email]> wrote:
> This has been already fixed:
> * firstly layer has been cached to avoid reloading it
> * the layer description stored Extent and used it

Fixed in which QGIS version? We are using the 2.14 version

> The problem you point is the way Layer is build by QGIS. In the Desktop,
> QGIS can build it at each project loading, but server side is could be a
> mess. If the cache is not conserved by the system QGIS Server is used, the
> layers are reloaded at each request.

How can QGIS Server conserve the cache?
_______________________________________________
Qgis-developer mailing list
[hidden email]
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: QGIS Server 3.0 roadmap

René-Luc Dhont
In reply to this post by Régis Haubourg
Hi Régis and Luca,

The other point, unique's id, is due to the fact that QGIS and QGIS
Server does not truth the QGIS Layer Description and verifying the
unicity of ids. So to avoid id, the layer has to be added with
'estimated metadata'.
If a PostGIS layer has not been opened with this option, the user has to
update the datasource to add it.

Regards,


Le 27/09/2016 à 09:29, Régis Haubourg a écrit :

> Le mardi 27 septembre 2016 à 09:25 +0200, Luca Manganelli a écrit :
>> way in which qgis server does the same PostGis
>> query (for extent or bounding box?) all times for each layer. It woud
>> be useful to store these informations on the project file, to optimize
>> times, with a configuration option in the Server tab?
> Hi, on this part, improvements have been made recently since this was a blocker in my previous employer's architecture and we
> funded 3Liz to improve things.
> René Luc, can you summarize what is the current status on postgis extent and unique id's queries?
>
> Cheers
> Régis
> _______________________________________________
> Qgis-developer mailing list
> [hidden email]
> List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

_______________________________________________
Qgis-developer mailing list
[hidden email]
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: QGIS Server 3.0 roadmap

René-Luc Dhont
In reply to this post by Luca Manganelli

Le 27/09/2016 à 09:33, Luca Manganelli a écrit :
> On Tue, Sep 27, 2016 at 9:30 AM, René-Luc Dhont <[hidden email]> wrote:
>> This has been already fixed:
>> * firstly layer has been cached to avoid reloading it
>> * the layer description stored Extent and used it
> Fixed in which QGIS version? We are using the 2.14 version
I fixed it during Gerona Hackfest and backport it in 2.14
>
>> The problem you point is the way Layer is build by QGIS. In the Desktop,
>> QGIS can build it at each project loading, but server side is could be a
>> mess. If the cache is not conserved by the system QGIS Server is used, the
>> layers are reloaded at each request.
> How can QGIS Server conserve the cache?

We use NGINX and Supervisor to conserve QGIS Server instances.
Apache with FCGI is failed to conserve instances. It regularly reload
instances and clear the cache.

> _______________________________________________
> Qgis-developer mailing list
> [hidden email]
> List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

_______________________________________________
Qgis-developer mailing list
[hidden email]
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: QGIS Server 3.0 roadmap

René-Luc Dhont
In reply to this post by Nyall Dawson
Hi Devs,

To enhance the discussion, David Marteau @dmarteau, has updated the QEP:
QGIS Server Services as plugins like providers with Proposed solution.
https://github.com/qgis/QGIS-Enhancement-Proposals/issues/74

Feedback is welcomed.

René-Luc

Le 27/09/2016 à 09:19, Nyall Dawson a écrit :

> On 25 September 2016 at 18:06, Alessandro Pasotti <[hidden email]> wrote:
>> Hi,
>>
>> I'm glad that Matthias's semi-serious threats to remove the server from
>> source tree and Nyall's complaints have shaken the souls of the server
>> people!
> Apologies if I've caused any offence here - I was trying to give
> constructive feedback and share some explanation as to the frustration
> against maintaining server. I want to stress that i am VERY
> appreciative of everyone who has put in effort into server & who has
> funded time for devs to work on it.
>
> I also applaud how well everyone has reacted to this, and all the
> offers of time/development to improve server for 3.0!
>
> Based on this reaction I think I'll go file a QEP to "rm -rf src/core"
> and see if we can shake up some more developers ;)
>
>> I insist that the server remains a first class citizen and a full featured
>> tool within the QGIS project, but I understand that it should not be a brake
>> for the core refactoring ,slowing down the amazing development that QGIS dev
>> team is doing on the core and gui libraries.
> I agree with this - I don't really want to see server pulled out of
> core either.
>
> As I mentioned over on the 3.0 api tracker, the biggest issue I see is
> the way server loads projects and messes with the layer registry. I
> understand this was probably a design decision made due to api
> constraints, but now we've got the opportunity to fix the underlying
> issues in core and avoid the messy workarounds in server.
>
> I'd really appreciate it if someone more familiar with the server
> architecture could summarise what's missing in core's project/layer
> handling. I may be missing something, but would a something like this
> help?:
>
> - add a QgsProjectRegistry, with an instance attached to
> QgsApplication. Change all the use of QgsProject::instance to
> QgsProjectRegistry::currentProject().
> - move QgsMapLayerRegistry from an instance to a member within
> QgsProject. Core code would then use
> currentProject()->mapLayerRegistry() instead of
> QgsMapLayerRegistry::instance()
> - server could directly pull projects from the registry (via some form
> of generated project id) as required and access their map layer
> registries directly
>
> Nyall
>
>
>
>
>
>
>
>
>
>
>
>>   I've just started a small WIP contribution aimed to solve a first issue
>> (https://github.com/qgis/QGIS/pull/3528).
>>
>> I wonder what would be the better way to coordinate the efforts, gain in
>> efficience and avoid duplication.
>>
>> This is a possible roadmap (suggestions welcome!):
>>
>> - start a QEP for the new server architecture (C++ plugins for the core
>> services etc.) (@rldhont, can you take care of this?)
>> - a list of prioritized issues (on the qgis hub?, on github? no idea here) (
>> @nyall, @mkuhn: you know the new and old API very well,  if you could start
>> creating a list of must-have, should-have, nice-to-have issues it would be
>> very helpful)
>> - create a formal (but completely open) dedicated team of developers and
>> assign issues
>> - possibly, meet periodically to coordinate the development
>>
>> Am I dreaming?
>>
>> --
>> Alessandro Pasotti
>> w3:   www.itopen.it

_______________________________________________
Qgis-developer mailing list
[hidden email]
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: QGIS Server 3.0 roadmap

René-Luc Dhont
In reply to this post by Nyall Dawson
Hi Nyall,

I like your proposition on QGIS Project and Layer Registry. Will you
write a QEP about it ? Do you wait the grant vote ?

About the server part, we used QgsMapLayerRegistry because some widgets
rely on map layer ids and not map layer pointers. We create our own
Project parser because the QgsProject load segfault if the
QgsApplication is instanciate without interface (GUIenabled to false).
And QgsProject is too close to QGIS gui.

René-Luc

Le 27/09/2016 à 09:19, Nyall Dawson a écrit :

> I'd really appreciate it if someone more familiar with the server
> architecture could summarise what's missing in core's project/layer
> handling. I may be missing something, but would a something like this
> help?:
>
> - add a QgsProjectRegistry, with an instance attached to
> QgsApplication. Change all the use of QgsProject::instance to
> QgsProjectRegistry::currentProject().
> - move QgsMapLayerRegistry from an instance to a member within
> QgsProject. Core code would then use
> currentProject()->mapLayerRegistry() instead of
> QgsMapLayerRegistry::instance()
> - server could directly pull projects from the registry (via some form
> of generated project id) as required and access their map layer
> registries directly
>
> Nyall

_______________________________________________
Qgis-developer mailing list
[hidden email]
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Reply | Threaded
Open this post in threaded view
|

Re: QGIS Server 3.0 roadmap

Richard Duivenvoorde
On 28-09-16 15:12, René-Luc Dhont wrote:

> Hi Nyall,
>
> I like your proposition on QGIS Project and Layer Registry. Will you
> write a QEP about it ? Do you wait the grant vote ?
>
> About the server part, we used QgsMapLayerRegistry because some widgets
> rely on map layer ids and not map layer pointers. We create our own
> Project parser because the QgsProject load segfault if the
> QgsApplication is instanciate without interface (GUIenabled to false).
> And QgsProject is too close to QGIS gui.

May I point (as part of 3.0 changes, second try ;-) ) to:

https://hub.qgis.org/issues/13494#note-19

Where Martin Dobias says:

"In general, the QgsApplication API is extremely confusing - and I hope
we can fix it for QGIS 3.0. It derives from QApplication (IMHO bad
idea), it also handles init/exit and paths. The init/exit handling
should be ideally completely automatic with PyQGIS, hopefully avoiding
different possible ways how people init PyQGIS in the future."

I hit this also, so it would be nice to get this fixed for 3.0, as both
pyqgis would benefit from it and apparently QGIS server :-)

Regards,

Richard Duivenvoorde



_______________________________________________
Qgis-developer mailing list
[hidden email]
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer