OSGeo.org › OSGeo Software Projects › QGIS › QGIS - Developer

Possible memory leak in QGIS redraws?

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

10 messages Options

Régis Haubourg

Possible memory leak in QGIS redraws?

Hi all,
I try to narrow down a production issue here.
In long sessions with lot's of redraws on complex projects, memory seems to increase regularly.

In a more simple environnement with no plugins, just sending multiple redraws using F5 key triggers it.
It grows quite fast on 2.6 up to 4Go sometimes, quite slower in master and 2.10. Every 5-6 redraws, QGIS adds 1Mo of RAM in 2.6, less often in recent versions.

If confirmed, and if nothing else is to be blamed (windows?, QT?) this is a serious issue for enterprise uses, especially if we want to build application that run all day long with many canvas redraws.

Could anyone try to reproduce this ?

Redrawing while previous redraw is not over could be at fault?

Thanks for your help.
Cheers

Nyall Dawson

Re: Possible memory leak in QGIS redraws?

On 9 Oct 2015 01:53, "Régis Haubourg" <[hidden email]> wrote:
>
> Hi all,
> I try to narrow down a production issue here.
> In long sessions with lot's of redraws on complex projects, memory seems to
> increase regularly.
>
> In a more simple environnement with no plugins, just sending multiple
> redraws using F5 key triggers it.
> It grows quite fast on 2.6 up to 4Go sometimes, quite slower in master and
> 2.10. Every 5-6 redraws, QGIS adds 1Mo of RAM in 2.6, less often in recent
> versions.
>
> If confirmed, and if nothing else is to be blamed (windows?, QT?) this is a
> serious issue for enterprise uses, especially if we want to build
> application that run all day long with many canvas redraws.
>
> Could anyone try to reproduce this ?

I've been seeing this with large atlas exports too. I can reproduce in 2.8, in 2.10 it was much worse, but in master it's back to 2.8 levels.

Tried everything I can to track this down with no luck. My suspicion at the moment is a leak in either qt or geos. I'd be interested to see if geos 3.5 has any effect...

Nyall

>
> Redrawing while previous redraw is not over could be at fault?
>
> Thanks for your help.
> Cheers
>
>
>
>
> --
> View this message in context: http://osgeo-org.1560.x6.nabble.com/Possible-memory-leak-in-QGIS-redraws-tp5228794.html
> Sent from the Quantum GIS - Developer mailing list archive at Nabble.com.
> _______________________________________________
> Qgis-developer mailing list
> [hidden email]
> http://lists.osgeo.org/mailman/listinfo/qgis-developer

_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Régis Haubourg

Re: Possible memory leak in QGIS redraws?

Thanks Nyall for this.
Any idea how we could track this down? What tools and skills are required to do so? I won't do this by myself but I a wondering how I can hire someone to do so
Cheers
Régis

Sandro Santilli-2

Re: Possible memory leak in QGIS redraws?

On Fri, Oct 09, 2015 at 04:48:28AM -0700, Régis Haubourg wrote:
> Thanks Nyall for this.
> Any idea how we could track this down? What tools and skills are required to
> do so? I won't do this by myself but I a wondering how I can hire someone to
> do so

I think the best would be to ensure an automated C/C++ test exists
in QGIS to reproduce the leak, then any memory checker tool would
do find out more. I can help, if the leaks are in GEOS.

--strk;
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Régis Haubourg

Re: Possible memory leak in QGIS redraws?

Hi,
I created a ticket here: http://hub.qgis.org/issues/13569

If anyone is available to work on this, please contact me by private mail.
Cheers
Régis

> -----Message d'origine-----
> De : Sandro Santilli [mailto:[hidden email]] De la part de Sandro
> Santilli
> Envoyé : vendredi 9 octobre 2015 14:15
> À : HAUBOURG
> Cc : [hidden email]
> Objet : Re: [Qgis-developer] Possible memory leak in QGIS redraws?
>
> On Fri, Oct 09, 2015 at 04:48:28AM -0700, Régis Haubourg wrote:
> > Thanks Nyall for this.
> > Any idea how we could track this down? What tools and skills are
> > required to do so? I won't do this by myself but I a wondering how I
> > can hire someone to do so
>
> I think the best would be to ensure an automated C/C++ test exists in QGIS
> to reproduce the leak, then any memory checker tool would do find out
> more. I can help, if the leaks are in GEOS.
>
> --strk;

_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Nyall Dawson

Re: Possible memory leak in QGIS redraws?

In reply to this post by Régis Haubourg

On 9 October 2015 at 01:45, Régis Haubourg
<[hidden email]> wrote:

> Hi all,
> I try to narrow down a production issue here.
> In long sessions with lot's of redraws on complex projects, memory seems to
> increase regularly.
>
> In a more simple environnement with no plugins, just sending multiple
> redraws using F5 key triggers it.
> It grows quite fast on 2.6 up to 4Go sometimes, quite slower in master and
> 2.10. Every 5-6 redraws, QGIS adds 1Mo of RAM in 2.6, less often in recent
> versions.
>
> If confirmed, and if nothing else is to be blamed (windows?, QT?) this is a
> serious issue for enterprise uses, especially if we want to build
> application that run all day long with many canvas redraws.
>

I've done some more testing, using valgrind and massif. Here's what I've found:

- there's a big leak coming from Qt's font database. Fixed in Qt 5.1
(https://bugreports.qt.io/browse/QTBUG-26861). About 50% of the memory
leaked is caused by this one. We could possibly work around it by
having a singleton so we only ever use one QFontDatabase in QGIS, or
just get on with the move to Qt5 and get the proper fix for free. ;)
- I also see a lot of memory used by Python objects. Specifically, a
lot coming from QgsPythonUtilsImpl::evalString and the PyRun_String
call in that function. I don't think these are leaks though (at least,
not within QGIS). But that's the second largest memory allocator I see
reported by massif
- There was a small leak of a GEOS object in labeling... don't think
this would cause too much memory usage though. I'll submit a fix after
giving it some more testing.
- Lastly, the remaining large chunk of allocated memory is used by the
images stored during rendering. I'm 90% sure these are all properly
cleaned, but I still can't throw the niggling suspicion that just
maybe there's some special condition in which these images aren't
properly deleted.

So yeah... no magic fixes to this one unfortunately. (At least, not
until QGIS 3.0!)

Nyall
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Nyall Dawson

Re: Possible memory leak in QGIS redraws?

On 21 Oct 2015 3:20 PM, "Nyall Dawson" <[hidden email]> wrote:
>
> On 9 October 2015 at 01:45, Régis Haubourg
> <[hidden email]> wrote:
> > Hi all,
> > I try to narrow down a production issue here.
> > In long sessions with lot's of redraws on complex projects, memory seems to
> > increase regularly.
> >
> > In a more simple environnement with no plugins, just sending multiple
> > redraws using F5 key triggers it.
> > It grows quite fast on 2.6 up to 4Go sometimes, quite slower in master and
> > 2.10. Every 5-6 redraws, QGIS adds 1Mo of RAM in 2.6, less often in recent
> > versions.
> >
> > If confirmed, and if nothing else is to be blamed (windows?, QT?) this is a
> > serious issue for enterprise uses, especially if we want to build
> > application that run all day long with many canvas redraws.
> >
>
> I've done some more testing, using valgrind and massif. Here's what I've found:
>
> - there's a big leak coming from Qt's font database. Fixed in Qt 5.1
> (https://bugreports.qt.io/browse/QTBUG-26861). About 50% of the memory
> leaked is caused by this one. We could possibly work around it by
> having a singleton so we only ever use one QFontDatabase in QGIS, or
> just get on with the move to Qt5 and get the proper fix for free. ;)
> - I also see a lot of memory used by Python objects. Specifically, a
> lot coming from QgsPythonUtilsImpl::evalString and the PyRun_String
> call in that function. I don't think these are leaks though (at least,
> not within QGIS). But that's the second largest memory allocator I see
> reported by massif
> - There was a small leak of a GEOS object in labeling... don't think
> this would cause too much memory usage though. I'll submit a fix after
> giving it some more testing.
> - Lastly, the remaining large chunk of allocated memory is used by the
> images stored during rendering. I'm 90% sure these are all properly
> cleaned, but I still can't throw the niggling suspicion that just
> maybe there's some special condition in which these images aren't
> properly deleted.
>
> So yeah... no magic fixes to this one unfortunately. (At least, not
> until QGIS 3.0!)

Actually... Scratch that. Just found some huge leaks in certain circumstances. Fixes in progress.

Nyall

>
> Nyall

_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Nyall Dawson

Re: Possible memory leak in QGIS redraws?

On 21 Oct 2015 6:49 PM, "Nyall Dawson" <[hidden email]> wrote:
>
>
> On 21 Oct 2015 3:20 PM, "Nyall Dawson" <[hidden email]> wrote:
> >
> > On 9 October 2015 at 01:45, Régis Haubourg
> > <[hidden email]> wrote:
> > > Hi all,
> > > I try to narrow down a production issue here.
> > > In long sessions with lot's of redraws on complex projects, memory seems to
> > > increase regularly.
> > >
> > > In a more simple environnement with no plugins, just sending multiple
> > > redraws using F5 key triggers it.
> > > It grows quite fast on 2.6 up to 4Go sometimes, quite slower in master and
> > > 2.10. Every 5-6 redraws, QGIS adds 1Mo of RAM in 2.6, less often in recent
> > > versions.
> > >
> > > If confirmed, and if nothing else is to be blamed (windows?, QT?) this is a
> > > serious issue for enterprise uses, especially if we want to build
> > > application that run all day long with many canvas redraws.
> > >
> >
> > I've done some more testing, using valgrind and massif. Here's what I've found:
> >
> > - there's a big leak coming from Qt's font database. Fixed in Qt 5.1
> > (https://bugreports.qt.io/browse/QTBUG-26861). About 50% of the memory
> > leaked is caused by this one. We could possibly work around it by
> > having a singleton so we only ever use one QFontDatabase in QGIS, or
> > just get on with the move to Qt5 and get the proper fix for free. ;)
> > - I also see a lot of memory used by Python objects. Specifically, a
> > lot coming from QgsPythonUtilsImpl::evalString and the PyRun_String
> > call in that function. I don't think these are leaks though (at least,
> > not within QGIS). But that's the second largest memory allocator I see
> > reported by massif
> > - There was a small leak of a GEOS object in labeling... don't think
> > this would cause too much memory usage though. I'll submit a fix after
> > giving it some more testing.
> > - Lastly, the remaining large chunk of allocated memory is used by the
> > images stored during rendering. I'm 90% sure these are all properly
> > cleaned, but I still can't throw the niggling suspicion that just
> > maybe there's some special condition in which these images aren't
> > properly deleted.
> >
> > So yeah... no magic fixes to this one unfortunately. (At least, not
> > until QGIS 3.0!)
>
> Actually... Scratch that. Just found some huge leaks in certain circumstances. Fixes in progress.

Try with latest master. It's made a huge difference for me. Exporting my most complex atlas (300 pages, with shapeburst, layer effects, some pretty hefty dynamic labeling):

Initially memory usage is about 320mb.
2.8: memory usage grows to 1.5 gb
2.10: memory usage to about 2 gb
Master: memory usage grows to 480 mb

Win! I suspect Qt5 will make another big difference too.

Nyall

>
> Nyall
>
>
>
> >
> > Nyall

_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer

Régis Haubourg

Re: Possible memory leak in QGIS redraws?

Sweeeeet!
I'm out of office right now. I will test next week. Anyway, thanks a lot for that impressive work. I love that community.
Cheers
Regis

Régis Haubourg

Le 22 oct. 2015 09:35, Nyall Dawson <[hidden email]> a écrit :

Try with latest master. It's made a huge difference for me. Exporting my most complex atlas (300 pages, with shapeburst, layer effects, some pretty hefty dynamic labeling):

Initially memory usage is about 320mb.
2.8: memory usage grows to 1.5 gb
2.10: memory usage to about 2 gb
Master: memory usage grows to 480 mb

Win! I suspect Qt5 will make another big difference too.

Nyall

>
> Nyall
>
>
>
> >
> > Nyall

_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer

pvanbosgeo

Re: Possible memory leak in QGIS redraws?

In reply to this post by Nyall Dawson

On 22-10-15 09:35, Nyall Dawson wrote:

On 21 Oct 2015 6:49 PM, "Nyall Dawson" <[hidden email]> wrote:
>
>
> On 21 Oct 2015 3:20 PM, "Nyall Dawson" <[hidden email]> wrote:
> >
> > On 9 October 2015 at 01:45, Régis Haubourg
> > <[hidden email]> wrote:
> > > Hi all,
> > > I try to narrow down a production issue here.
> > > In long sessions with lot's of redraws on complex projects, memory seems to
> > > increase regularly.
> > >
> > > In a more simple environnement with no plugins, just sending multiple
> > > redraws using F5 key triggers it.
> > > It grows quite fast on 2.6 up to 4Go sometimes, quite slower in master and
> > > 2.10. Every 5-6 redraws, QGIS adds 1Mo of RAM in 2.6, less often in recent
> > > versions.
> > >
> > > If confirmed, and if nothing else is to be blamed (windows?, QT?) this is a
> > > serious issue for enterprise uses, especially if we want to build
> > > application that run all day long with many canvas redraws.
> > >
> >
> > I've done some more testing, using valgrind and massif. Here's what I've found:
> >
> > - there's a big leak coming from Qt's font database. Fixed in Qt 5.1
> > (https://bugreports.qt.io/browse/QTBUG-26861). About 50% of the memory
> > leaked is caused by this one. We could possibly work around it by
> > having a singleton so we only ever use one QFontDatabase in QGIS, or
> > just get on with the move to Qt5 and get the proper fix for free. ;)
> > - I also see a lot of memory used by Python objects. Specifically, a
> > lot coming from QgsPythonUtilsImpl::evalString and the PyRun_String
> > call in that function. I don't think these are leaks though (at least,
> > not within QGIS). But that's the second largest memory allocator I see
> > reported by massif
> > - There was a small leak of a GEOS object in labeling... don't think
> > this would cause too much memory usage though. I'll submit a fix after
> > giving it some more testing.
> > - Lastly, the remaining large chunk of allocated memory is used by the
> > images stored during rendering. I'm 90% sure these are all properly
> > cleaned, but I still can't throw the niggling suspicion that just
> > maybe there's some special condition in which these images aren't
> > properly deleted.
> >
> > So yeah... no magic fixes to this one unfortunately. (At least, not
> > until QGIS 3.0!)
>
> Actually... Scratch that. Just found some huge leaks in certain circumstances. Fixes in progress.

Try with latest master. It's made a huge difference for me. Exporting my most complex atlas (300 pages, with shapeburst, layer effects, some pretty hefty dynamic labeling):

Initially memory usage is about 320mb.
2.8: memory usage grows to 1.5 gb
2.10: memory usage to about 2 gb
Master: memory usage grows to 480 mb

Impressive! Will try out a.s.a.p.

Win! I suspect Qt5 will make another big difference too.

Nyall

>
> Nyall
>
>
>
> >
> > Nyall
_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer

_______________________________________________
Qgis-developer mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/qgis-developer

---
http://pvanb.wordpress.com/