Qgis crashes while loading WMTS layers

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Qgis crashes while loading WMTS layers

Patrick Dunford-2
Good day to all.

Not so long ago I wrote a post about issues handling large numbers of
raster files. This results from what appears to be an architectural
design limitation of Qgis in that it only has enough resources available
for a certain (unknown) number of raster layers and does not have any
effective system for dealing with the exhaustion of these resources.

I also noted that the software does not appear to have the ability to
use the swap (virtual memory) resources within a system to deal with the
apparent exhaustion of resources, as other software would. For example,
Gimp which I use for graphical editing is able to handle certain
projects using a swap space of 200 GB, which consists of 32 GB of
physical RAM and the rest in SSD. It is not unusual with some of these
graphics projects to have 100 GB or more of swap space in use, without
crashing the system. (This being possible since Gimp 2.10 with its
ability to write files of more than 4 GB)

Whilst it is possible to manage the number of layers with file based
rasters, this cannot be done with a WMTS server because there is no way
I am aware of to manage the number of layers that can be downloaded from
the server.

Version 2.18 and earlier almost never crashed with excess layers,
usually all that would happen is a layer would be displayed with invalid
data. However, crashes have become the norm with 3.x versions of software.

It seems to me there is a big architectural problem in the software with
its inability to utilise the full resources of the system in order to
process the number of layers that a WMTS server is capable of issuing.

_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: Qgis crashes while loading WMTS layers

Bo Victor Thomsen

Hi Patrick -

Swap space is an internal component the underlying operating system (OS) uses for handling memory request that exceeds the ram resources of the computer. A normal user program like QGIS has no "knowledge" about this facility and can't directly manipulate the swap system.

As Jonathan describes, "swap" space for for other program is temporary file storage, usually placed in a specific directory (in windows designated by the environment variable %TEMP%). QGIS uses this directory too.

A QGIS crash can be caused by running out of memory resources and/or temp file storage. However, that is result of a bad setup of your operating system, which can limit how much swap space, that can be allocated to memory hungry programs and the size of hard-disk used for swap space and/or temp storage

So what to do:

  1. Check memory consumption when using QGIS. Is it really using all the ram - memory and starting to swap ?
  2. Check the setup of the swap space parameters. It it using a hard-disk which is running out of storage space.
  3. Check the setup of the temp storage parameters. It it using a hard-disk which  is running out of storage space ? Are the user allowed to allocate enough storage?

(I can't be more specific not knowing the operating system you use)

If the above checks out OK and you still have issues with QGIS crashing..

The shift from QGIS 2.x to QGIS 3.x meant a complete change of the underlying graphics subsystem QT from ver 4.x to ver. 5.x (and a huge amount of other changes).

One or more of those changes  might cause a failure as you described. However, you have to check the above and then write an error issue (https://github.com/qgis/QGIS/issues) where you describe error messages, your specific setup like operating system and version, QGIS version and - if possible - test-data and project to replicate the issue.  

-- 
Med venlig hilsen / Kind regards

Bo Victor Thomsen


 





Den 26-01-2020 kl. 13:19 skrev Patrick Dunford:
Good day to all.

Not so long ago I wrote a post about issues handling large numbers of raster files. This results from what appears to be an architectural design limitation of Qgis in that it only has enough resources available for a certain (unknown) number of raster layers and does not have any effective system for dealing with the exhaustion of these resources.

I also noted that the software does not appear to have the ability to use the swap (virtual memory) resources within a system to deal with the apparent exhaustion of resources, as other software would. For example, Gimp which I use for graphical editing is able to handle certain projects using a swap space of 200 GB, which consists of 32 GB of physical RAM and the rest in SSD. It is not unusual with some of these graphics projects to have 100 GB or more of swap space in use, without crashing the system. (This being possible since Gimp 2.10 with its ability to write files of more than 4 GB)

Whilst it is possible to manage the number of layers with file based rasters, this cannot be done with a WMTS server because there is no way I am aware of to manage the number of layers that can be downloaded from the server.

Version 2.18 and earlier almost never crashed with excess layers, usually all that would happen is a layer would be displayed with invalid data. However, crashes have become the norm with 3.x versions of software.

It seems to me there is a big architectural problem in the software with its inability to utilise the full resources of the system in order to process the number of layers that a WMTS server is capable of issuing.

_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

  


_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: Qgis crashes while loading WMTS layers

Patrick Dunford-2
In reply to this post by Patrick Dunford-2
Further investigation and comment

Since recently installing Qgis 3.4.15 or 3.4.14 on some of my computers
running Debian 10, I immediately began to see a trend of the software
exhibiting an increased tendency to crash when working with certain WMTS
layers that I had previously worked with extensively over recent weeks
for mapping purposes.

After attempting numerous changes including special versions of projects
containing no file based raster layers and only containing one WMTS
layer at a time and being unable to use the software regardless I
created a virtual machine running Debian 9 which has the latest version
available for that platform, 3.4.4. This has resulted in immediate major
improvement in the reliability of the software to the point I can resume
using it without the numerous crashes of the software that have been
noted recently, especially in the last week approximately.

On 27/01/20 1:19 am, Patrick Dunford wrote:

> Good day to all.
>
> Not so long ago I wrote a post about issues handling large numbers of
> raster files. This results from what appears to be an architectural
> design limitation of Qgis in that it only has enough resources
> available for a certain (unknown) number of raster layers and does not
> have any effective system for dealing with the exhaustion of these
> resources.
>
> I also noted that the software does not appear to have the ability to
> use the swap (virtual memory) resources within a system to deal with
> the apparent exhaustion of resources, as other software would. For
> example, Gimp which I use for graphical editing is able to handle
> certain projects using a swap space of 200 GB, which consists of 32 GB
> of physical RAM and the rest in SSD. It is not unusual with some of
> these graphics projects to have 100 GB or more of swap space in use,
> without crashing the system. (This being possible since Gimp 2.10 with
> its ability to write files of more than 4 GB)
>
> Whilst it is possible to manage the number of layers with file based
> rasters, this cannot be done with a WMTS server because there is no
> way I am aware of to manage the number of layers that can be
> downloaded from the server.
>
> Version 2.18 and earlier almost never crashed with excess layers,
> usually all that would happen is a layer would be displayed with
> invalid data. However, crashes have become the norm with 3.x versions
> of software.
>
> It seems to me there is a big architectural problem in the software
> with its inability to utilise the full resources of the system in
> order to process the number of layers that a WMTS server is capable of
> issuing.
>
_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: Qgis crashes while loading WMTS layers

Jonathan Moules-4
Hi Patrick,

I'd suggest reporting this to the QGIS bug tracker as a regression.

Cheers,

Jonathan

On 2020-02-01 07:20, Patrick Dunford wrote:

> Further investigation and comment
>
> Since recently installing Qgis 3.4.15 or 3.4.14 on some of my
> computers running Debian 10, I immediately began to see a trend of the
> software exhibiting an increased tendency to crash when working with
> certain WMTS layers that I had previously worked with extensively over
> recent weeks for mapping purposes.
>
> After attempting numerous changes including special versions of
> projects containing no file based raster layers and only containing
> one WMTS layer at a time and being unable to use the software
> regardless I created a virtual machine running Debian 9 which has the
> latest version available for that platform, 3.4.4. This has resulted
> in immediate major improvement in the reliability of the software to
> the point I can resume using it without the numerous crashes of the
> software that have been noted recently, especially in the last week
> approximately.
>
> On 27/01/20 1:19 am, Patrick Dunford wrote:
>> Good day to all.
>>
>> Not so long ago I wrote a post about issues handling large numbers of
>> raster files. This results from what appears to be an architectural
>> design limitation of Qgis in that it only has enough resources
>> available for a certain (unknown) number of raster layers and does
>> not have any effective system for dealing with the exhaustion of
>> these resources.
>>
>> I also noted that the software does not appear to have the ability to
>> use the swap (virtual memory) resources within a system to deal with
>> the apparent exhaustion of resources, as other software would. For
>> example, Gimp which I use for graphical editing is able to handle
>> certain projects using a swap space of 200 GB, which consists of 32
>> GB of physical RAM and the rest in SSD. It is not unusual with some
>> of these graphics projects to have 100 GB or more of swap space in
>> use, without crashing the system. (This being possible since Gimp
>> 2.10 with its ability to write files of more than 4 GB)
>>
>> Whilst it is possible to manage the number of layers with file based
>> rasters, this cannot be done with a WMTS server because there is no
>> way I am aware of to manage the number of layers that can be
>> downloaded from the server.
>>
>> Version 2.18 and earlier almost never crashed with excess layers,
>> usually all that would happen is a layer would be displayed with
>> invalid data. However, crashes have become the norm with 3.x versions
>> of software.
>>
>> It seems to me there is a big architectural problem in the software
>> with its inability to utilise the full resources of the system in
>> order to process the number of layers that a WMTS server is capable
>> of issuing.
>>
> _______________________________________________
> Qgis-user mailing list
> [hidden email]
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: Qgis crashes while loading WMTS layers

Patrick Dunford-2
In reply to this post by Bo Victor Thomsen

Sorry I don't know enough about virtual memory implementation to comment.

The only software I am sufficiently familiar with from experience is Gimp, which has the setting where you can specify the size of the tile cache. So for example I have a tile cache set to 200 GB. It starts with physical RAM then pushes into the swap space.

I can monitor the resource use of the system with a widget displayed on the panel on the computer. This shows CPU use, memory and swap space use. From there I can watch the resource use of the system while doing things. I can compare the resource usage being displayed for the system with the Gimp dashboard.

The assumption for me is that the operating system, in theory, presents a virtual memory model to applications, whereby the actual physical implementation is abstracted, such that when an application makes a request for more memory allocated, it is essentially transparent where the information is actually physically stored. Maybe that is not accurate?

On 27/01/20 8:18 pm, Bo Victor Thomsen wrote:

Hi Patrick -

Swap space is an internal component the underlying operating system (OS) uses for handling memory request that exceeds the ram resources of the computer. A normal user program like QGIS has no "knowledge" about this facility and can't directly manipulate the swap system.

As Jonathan describes, "swap" space for for other program is temporary file storage, usually placed in a specific directory (in windows designated by the environment variable %TEMP%). QGIS uses this directory too.

A QGIS crash can be caused by running out of memory resources and/or temp file storage. However, that is result of a bad setup of your operating system, which can limit how much swap space, that can be allocated to memory hungry programs and the size of hard-disk used for swap space and/or temp storage

So what to do:

  1. Check memory consumption when using QGIS. Is it really using all the ram - memory and starting to swap ?
  2. Check the setup of the swap space parameters. It it using a hard-disk which is running out of storage space.
  3. Check the setup of the temp storage parameters. It it using a hard-disk which  is running out of storage space ? Are the user allowed to allocate enough storage?

(I can't be more specific not knowing the operating system you use)

If the above checks out OK and you still have issues with QGIS crashing..

The shift from QGIS 2.x to QGIS 3.x meant a complete change of the underlying graphics subsystem QT from ver 4.x to ver. 5.x (and a huge amount of other changes).

One or more of those changes  might cause a failure as you described. However, you have to check the above and then write an error issue (https://github.com/qgis/QGIS/issues) where you describe error messages, your specific setup like operating system and version, QGIS version and - if possible - test-data and project to replicate the issue.  

-- 
Med venlig hilsen / Kind regards

Bo Victor Thomsen


_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: Qgis crashes while loading WMTS layers

Patrick Dunford-2
In reply to this post by Jonathan Moules-4
Hi there

Due to imminent deadlines on a project I was working on, I had to leave
the issue and just press on with a VM running an older version.

I am currently building some test VMs and projects and hope to log this
issue in the bug tracker in the next few days.

On 2/02/20 5:25 am, Jonathan Moules wrote:

> Hi Patrick,
>
> I'd suggest reporting this to the QGIS bug tracker as a regression.
>
> Cheers,
>
> Jonathan
>
> On 2020-02-01 07:20, Patrick Dunford wrote:
>> Further investigation and comment
>>
>> Since recently installing Qgis 3.4.15 or 3.4.14 on some of my
>> computers running Debian 10, I immediately began to see a trend of
>> the software exhibiting an increased tendency to crash when working
>> with certain WMTS layers that I had previously worked with
>> extensively over recent weeks for mapping purposes.
>>
>> After attempting numerous changes including special versions of
>> projects containing no file based raster layers and only containing
>> one WMTS layer at a time and being unable to use the software
>> regardless I created a virtual machine running Debian 9 which has the
>> latest version available for that platform, 3.4.4. This has resulted
>> in immediate major improvement in the reliability of the software to
>> the point I can resume using it without the numerous crashes of the
>> software that have been noted recently, especially in the last week
>> approximately.
>>
>> On 27/01/20 1:19 am, Patrick Dunford wrote:
>>> Good day to all.
>>>
>>> Not so long ago I wrote a post about issues handling large numbers
>>> of raster files. This results from what appears to be an
>>> architectural design limitation of Qgis in that it only has enough
>>> resources available for a certain (unknown) number of raster layers
>>> and does not have any effective system for dealing with the
>>> exhaustion of these resources.
>>>
>>> I also noted that the software does not appear to have the ability
>>> to use the swap (virtual memory) resources within a system to deal
>>> with the apparent exhaustion of resources, as other software would.
>>> For example, Gimp which I use for graphical editing is able to
>>> handle certain projects using a swap space of 200 GB, which consists
>>> of 32 GB of physical RAM and the rest in SSD. It is not unusual with
>>> some of these graphics projects to have 100 GB or more of swap space
>>> in use, without crashing the system. (This being possible since Gimp
>>> 2.10 with its ability to write files of more than 4 GB)
>>>
>>> Whilst it is possible to manage the number of layers with file based
>>> rasters, this cannot be done with a WMTS server because there is no
>>> way I am aware of to manage the number of layers that can be
>>> downloaded from the server.
>>>
>>> Version 2.18 and earlier almost never crashed with excess layers,
>>> usually all that would happen is a layer would be displayed with
>>> invalid data. However, crashes have become the norm with 3.x
>>> versions of software.
>>>
>>> It seems to me there is a big architectural problem in the software
>>> with its inability to utilise the full resources of the system in
>>> order to process the number of layers that a WMTS server is capable
>>> of issuing.
>>>
>> _______________________________________________
>> Qgis-user mailing list
>> [hidden email]
>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: Qgis crashes while loading WMTS layers

Jeff Hubbs-2
In reply to this post by Patrick Dunford-2
Your understanding is essentially correct for most anything you will
likely encounter. Apps don't know or care about the memory
implementation that the operating system makes available.

On 2/12/20 7:35 AM, Patrick Dunford wrote:
>
>
> The assumption for me is that the operating system, in theory,
> presents a virtual memory model to applications, whereby the actual
> physical implementation is abstracted, such that when an application
> makes a request for more memory allocated, it is essentially
> transparent where the information is actually physically stored. Maybe
> that is not accurate?
>

_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: Qgis crashes while loading WMTS layers

Patrick Dunford-2
In reply to this post by Jonathan Moules-4
In the last few weeks I have been doing a lot of testing with various
test environments.

At the moment I can't replicate the problem.There seem to be no issues
with 3.4.15, 3.10 or 3.12

On 2/02/20 5:25 am, Jonathan Moules wrote:

> Hi Patrick,
>
> I'd suggest reporting this to the QGIS bug tracker as a regression.
>
> Cheers,
>
> Jonathan
>
> On 2020-02-01 07:20, Patrick Dunford wrote:
>> Further investigation and comment
>>
>> Since recently installing Qgis 3.4.15 or 3.4.14 on some of my
>> computers running Debian 10, I immediately began to see a trend of
>> the software exhibiting an increased tendency to crash when working
>> with certain WMTS layers that I had previously worked with
>> extensively over recent weeks for mapping purposes.
>>
>> After attempting numerous changes including special versions of
>> projects containing no file based raster layers and only containing
>> one WMTS layer at a time and being unable to use the software
>> regardless I created a virtual machine running Debian 9 which has the
>> latest version available for that platform, 3.4.4. This has resulted
>> in immediate major improvement in the reliability of the software to
>> the point I can resume using it without the numerous crashes of the
>> software that have been noted recently, especially in the last week
>> approximately.
>>
>> On 27/01/20 1:19 am, Patrick Dunford wrote:
>>> Good day to all.
>>>
>>> Not so long ago I wrote a post about issues handling large numbers
>>> of raster files. This results from what appears to be an
>>> architectural design limitation of Qgis in that it only has enough
>>> resources available for a certain (unknown) number of raster layers
>>> and does not have any effective system for dealing with the
>>> exhaustion of these resources.
>>>
>>> I also noted that the software does not appear to have the ability
>>> to use the swap (virtual memory) resources within a system to deal
>>> with the apparent exhaustion of resources, as other software would.
>>> For example, Gimp which I use for graphical editing is able to
>>> handle certain projects using a swap space of 200 GB, which consists
>>> of 32 GB of physical RAM and the rest in SSD. It is not unusual with
>>> some of these graphics projects to have 100 GB or more of swap space
>>> in use, without crashing the system. (This being possible since Gimp
>>> 2.10 with its ability to write files of more than 4 GB)
>>>
>>> Whilst it is possible to manage the number of layers with file based
>>> rasters, this cannot be done with a WMTS server because there is no
>>> way I am aware of to manage the number of layers that can be
>>> downloaded from the server.
>>>
>>> Version 2.18 and earlier almost never crashed with excess layers,
>>> usually all that would happen is a layer would be displayed with
>>> invalid data. However, crashes have become the norm with 3.x
>>> versions of software.
>>>
>>> It seems to me there is a big architectural problem in the software
>>> with its inability to utilise the full resources of the system in
>>> order to process the number of layers that a WMTS server is capable
>>> of issuing.
>>>
>> _______________________________________________
>> Qgis-user mailing list
>> [hidden email]
>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: Qgis crashes while loading WMTS layers

Micha Silver-2

Hello Patrick:


Which desktop manager are you using?


Some months ago,I reinstalled my buster machine with the default Gnome, and found, like you, repeated crashed of QGIS in all kinds of unrelated situations. After trying many updates, recreating QGIS projects, etc, I decided to try KDE. And voila, all the troubles disappeared. I have since followed the upgrades thru 3.10 to 3.12 and no issues have appeared.

I can't even guess what the problem is: maybe some clash between QGIS and the gtk libraries used by Gnome? In any case I'm back on KDE and enjoying it.



On 13/03/2020 8:24, Patrick Dunford wrote:
In the last few weeks I have been doing a lot of testing with various test environments.

At the moment I can't replicate the problem.There seem to be no issues with 3.4.15, 3.10 or 3.12

On 2/02/20 5:25 am, Jonathan Moules wrote:
Hi Patrick,

I'd suggest reporting this to the QGIS bug tracker as a regression.

Cheers,

Jonathan

On 2020-02-01 07:20, Patrick Dunford wrote:
Further investigation and comment

Since recently installing Qgis 3.4.15 or 3.4.14 on some of my computers running Debian 10, I immediately began to see a trend of the software exhibiting an increased tendency to crash when working with certain WMTS layers that I had previously worked with extensively over recent weeks for mapping purposes.

After attempting numerous changes including special versions of projects containing no file based raster layers and only containing one WMTS layer at a time and being unable to use the software regardless I created a virtual machine running Debian 9 which has the latest version available for that platform, 3.4.4. This has resulted in immediate major improvement in the reliability of the software to the point I can resume using it without the numerous crashes of the software that have been noted recently, especially in the last week approximately.

On 27/01/20 1:19 am, Patrick Dunford wrote:
Good day to all.

Not so long ago I wrote a post about issues handling large numbers of raster files. This results from what appears to be an architectural design limitation of Qgis in that it only has enough resources available for a certain (unknown) number of raster layers and does not have any effective system for dealing with the exhaustion of these resources.

I also noted that the software does not appear to have the ability to use the swap (virtual memory) resources within a system to deal with the apparent exhaustion of resources, as other software would. For example, Gimp which I use for graphical editing is able to handle certain projects using a swap space of 200 GB, which consists of 32 GB of physical RAM and the rest in SSD. It is not unusual with some of these graphics projects to have 100 GB or more of swap space in use, without crashing the system. (This being possible since Gimp 2.10 with its ability to write files of more than 4 GB)

Whilst it is possible to manage the number of layers with file based rasters, this cannot be done with a WMTS server because there is no way I am aware of to manage the number of layers that can be downloaded from the server.

Version 2.18 and earlier almost never crashed with excess layers, usually all that would happen is a layer would be displayed with invalid data. However, crashes have become the norm with 3.x versions of software.

It seems to me there is a big architectural problem in the software with its inability to utilise the full resources of the system in order to process the number of layers that a WMTS server is capable of issuing.

_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
-- 
Micha Silver
Ben Gurion Univ.
Sde Boker, Remote Sensing Lab
cell: +972-523-665918
https://orcid.org/0000-0002-1128-1325

_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user