Very Slow Table Operations in QGIS 3.2.3

classic Classic list List threaded Threaded
19 messages Options
Reply | Threaded
Open this post in threaded view
|

Very Slow Table Operations in QGIS 3.2.3

Kelley Jabr
Hello,

I'm working with a geopackage on ~530k features, and just about everything I try to do in the table takes way longer than expected. 
  • Adding a field took 5 minutes (table now has 22 total fields, so not massive)
  • Field Calculator on ~32k selected records took 8 minutes (literally all I was putting in the cells was: N)
  • Field calculator on ~480k records took so many hours that I had to leave it running over the weekend and when I came back, the Field Calculator had "closed unexpectedly"
(I'm running on a linux machine with an Intel Core i7 and 32GB of memory.)

Has anyone else experience similar issues? Any advice or recommendations for how to improve performance here? 

Thank you,

--

Kelley Jabr

Analyst, Geographic Data Operations

[hidden email]



_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: Very Slow Table Operations in QGIS 3.2.3

Nicolas Cadieux
Hi,

I had similar problems with shp files in earlier version of QGIS.  I would reboot the Window machine and restart QGIS but the problem would start again after a number of edits.  I think it was a plugin that was causing a memory leak.  Try closing all the plugins and then re-starting Qgis.  Half a million is not that small.  Have you tried with a different files format just to check the speed?

Nicolas

Le 3 oct. 2018 à 12:10, Kelley Jabr <[hidden email]> a écrit :

Hello,

I'm working with a geopackage on ~530k features, and just about everything I try to do in the table takes way longer than expected. 
  • Adding a field took 5 minutes (table now has 22 total fields, so not massive)
  • Field Calculator on ~32k selected records took 8 minutes (literally all I was putting in the cells was: N)
  • Field calculator on ~480k records took so many hours that I had to leave it running over the weekend and when I came back, the Field Calculator had "closed unexpectedly"
(I'm running on a linux machine with an Intel Core i7 and 32GB of memory.)

Has anyone else experience similar issues? Any advice or recommendations for how to improve performance here? 

Thank you,

--

Kelley Jabr

Analyst, Geographic Data Operations

[hidden email]


_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: Very Slow Table Operations in QGIS 3.2.3

ginetto
In reply to this post by Kelley Jabr
can you test doing operations with field calculator without having the attribute table opened? just a workaround could be to use a virtual layer then save the entire layer later.

Luigi Pirelli

**************************************************************************************************
* LinkedIn: https://www.linkedin.com/in/luigipirelli
* Stackexchange: http://gis.stackexchange.com/users/19667/luigi-pirelli
* GitHub: https://github.com/luipir
* Mastering QGIS 2nd Edition:
* https://www.packtpub.com/big-data-and-business-intelligence/mastering-qgis-second-edition
* Hire me: http://goo.gl/BYRQKg
**************************************************************************************************


On Wed, 3 Oct 2018 at 18:11, Kelley Jabr <[hidden email]> wrote:
Hello,

I'm working with a geopackage on ~530k features, and just about everything I try to do in the table takes way longer than expected. 
  • Adding a field took 5 minutes (table now has 22 total fields, so not massive)
  • Field Calculator on ~32k selected records took 8 minutes (literally all I was putting in the cells was: N)
  • Field calculator on ~480k records took so many hours that I had to leave it running over the weekend and when I came back, the Field Calculator had "closed unexpectedly"
(I'm running on a linux machine with an Intel Core i7 and 32GB of memory.)

Has anyone else experience similar issues? Any advice or recommendations for how to improve performance here? 

Thank you,

--

Kelley Jabr

Analyst, Geographic Data Operations

[hidden email]


_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: Very Slow Table Operations in QGIS 3.2.3

Kelley Jabr
Luigi, I'm pretty sure you just solved my problem. I tried a few operations with the table closed and it was like night and day. That 400k field calculator finished in mere moments! Thank you!!

On Wed, Oct 3, 2018 at 1:42 PM Luigi Pirelli <[hidden email]> wrote:
can you test doing operations with field calculator without having the attribute table opened? just a workaround could be to use a virtual layer then save the entire layer later.

Luigi Pirelli

**************************************************************************************************
* LinkedIn: https://www.linkedin.com/in/luigipirelli
* Stackexchange: http://gis.stackexchange.com/users/19667/luigi-pirelli
* GitHub: https://github.com/luipir
* Mastering QGIS 2nd Edition:
* https://www.packtpub.com/big-data-and-business-intelligence/mastering-qgis-second-edition
* Hire me: http://goo.gl/BYRQKg
**************************************************************************************************


On Wed, 3 Oct 2018 at 18:11, Kelley Jabr <[hidden email]> wrote:
Hello,

I'm working with a geopackage on ~530k features, and just about everything I try to do in the table takes way longer than expected. 
  • Adding a field took 5 minutes (table now has 22 total fields, so not massive)
  • Field Calculator on ~32k selected records took 8 minutes (literally all I was putting in the cells was: N)
  • Field calculator on ~480k records took so many hours that I had to leave it running over the weekend and when I came back, the Field Calculator had "closed unexpectedly"
(I'm running on a linux machine with an Intel Core i7 and 32GB of memory.)

Has anyone else experience similar issues? Any advice or recommendations for how to improve performance here? 

Thank you,

--

Kelley Jabr

Analyst, Geographic Data Operations

[hidden email]


_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user


--

Kelley Jabr

Analyst, Geographic Data Operations

[hidden email]


Indeed - We help people get jobs.

Indeed.com



_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: Very Slow Table Operations in QGIS 3.2.3

ginetto
may you open an issue about this slowness with all details to reproduce? Only with opened issue the project can allocate resources to fix them... and we are just in bugfix period before the 3.4 release. So your contribution would be precious.

tnx

Luigi Pirelli

**************************************************************************************************
* LinkedIn: https://www.linkedin.com/in/luigipirelli
* Stackexchange: http://gis.stackexchange.com/users/19667/luigi-pirelli
* GitHub: https://github.com/luipir
* Mastering QGIS 2nd Edition:
* https://www.packtpub.com/big-data-and-business-intelligence/mastering-qgis-second-edition
* Hire me: http://goo.gl/BYRQKg
**************************************************************************************************


On Wed, 3 Oct 2018 at 21:20, Kelley Jabr <[hidden email]> wrote:
Luigi, I'm pretty sure you just solved my problem. I tried a few operations with the table closed and it was like night and day. That 400k field calculator finished in mere moments! Thank you!!

On Wed, Oct 3, 2018 at 1:42 PM Luigi Pirelli <[hidden email]> wrote:
can you test doing operations with field calculator without having the attribute table opened? just a workaround could be to use a virtual layer then save the entire layer later.

Luigi Pirelli

**************************************************************************************************
* LinkedIn: https://www.linkedin.com/in/luigipirelli
* Stackexchange: http://gis.stackexchange.com/users/19667/luigi-pirelli
* GitHub: https://github.com/luipir
* Mastering QGIS 2nd Edition:
* https://www.packtpub.com/big-data-and-business-intelligence/mastering-qgis-second-edition
* Hire me: http://goo.gl/BYRQKg
**************************************************************************************************


On Wed, 3 Oct 2018 at 18:11, Kelley Jabr <[hidden email]> wrote:
Hello,

I'm working with a geopackage on ~530k features, and just about everything I try to do in the table takes way longer than expected. 
  • Adding a field took 5 minutes (table now has 22 total fields, so not massive)
  • Field Calculator on ~32k selected records took 8 minutes (literally all I was putting in the cells was: N)
  • Field calculator on ~480k records took so many hours that I had to leave it running over the weekend and when I came back, the Field Calculator had "closed unexpectedly"
(I'm running on a linux machine with an Intel Core i7 and 32GB of memory.)

Has anyone else experience similar issues? Any advice or recommendations for how to improve performance here? 

Thank you,

--

Kelley Jabr

Analyst, Geographic Data Operations

[hidden email]


_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user


--

Kelley Jabr

Analyst, Geographic Data Operations

[hidden email]


Indeed - We help people get jobs.

Indeed.com



_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: Very Slow Table Operations in QGIS 3.2.3

Andreas Neumann-4
In reply to this post by Kelley Jabr

Hi,

Glad to hear that it helped to run the operations on a closed table. Nevertheless, as Luigi said, it would make sense to submit an issue report about this slowness.

The other thing one has to keep on mind:

The SQLite API is kind of limited. Only until recently, if you wanted to rename a field, one had to copy the whole table with the renamed field changed in the new table - then the old table had to be removed (all in the background), because there was API to rename columns in SQLite. I think with the very recent version, which is not rolled out into most OS, this has changed. You'd also need a very recent GDAL/OGR.

Greetings,
Andreas

Am 03.10.2018 um 21:20 schrieb Kelley Jabr:
Luigi, I'm pretty sure you just solved my problem. I tried a few operations with the table closed and it was like night and day. That 400k field calculator finished in mere moments! Thank you!!

On Wed, Oct 3, 2018 at 1:42 PM Luigi Pirelli <[hidden email]> wrote:
can you test doing operations with field calculator without having the attribute table opened? just a workaround could be to use a virtual layer then save the entire layer later.

Luigi Pirelli

**************************************************************************************************
* LinkedIn: https://www.linkedin.com/in/luigipirelli
* Stackexchange: http://gis.stackexchange.com/users/19667/luigi-pirelli
* GitHub: https://github.com/luipir
* Mastering QGIS 2nd Edition:
* https://www.packtpub.com/big-data-and-business-intelligence/mastering-qgis-second-edition
* Hire me: http://goo.gl/BYRQKg
**************************************************************************************************


On Wed, 3 Oct 2018 at 18:11, Kelley Jabr <[hidden email]> wrote:
Hello,

I'm working with a geopackage on ~530k features, and just about everything I try to do in the table takes way longer than expected. 
  • Adding a field took 5 minutes (table now has 22 total fields, so not massive)
  • Field Calculator on ~32k selected records took 8 minutes (literally all I was putting in the cells was: N)
  • Field calculator on ~480k records took so many hours that I had to leave it running over the weekend and when I came back, the Field Calculator had "closed unexpectedly"
(I'm running on a linux machine with an Intel Core i7 and 32GB of memory.)

Has anyone else experience similar issues? Any advice or recommendations for how to improve performance here? 

Thank you,

--

Kelley Jabr

Analyst, Geographic Data Operations

[hidden email]


_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user


--

Kelley Jabr

Analyst, Geographic Data Operations

[hidden email]


Indeed - We help people get jobs.

Indeed.com




_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user


_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: Very Slow Table Operations in QGIS 3.2.3

Kelley Jabr
Okay, I've created an issue for this situation. Hopefully I've done it satisfactorily. Thanks again!

On Wed, Oct 3, 2018 at 2:31 PM Andreas Neumann <[hidden email]> wrote:

Hi,

Glad to hear that it helped to run the operations on a closed table. Nevertheless, as Luigi said, it would make sense to submit an issue report about this slowness.

The other thing one has to keep on mind:

The SQLite API is kind of limited. Only until recently, if you wanted to rename a field, one had to copy the whole table with the renamed field changed in the new table - then the old table had to be removed (all in the background), because there was API to rename columns in SQLite. I think with the very recent version, which is not rolled out into most OS, this has changed. You'd also need a very recent GDAL/OGR.

Greetings,
Andreas

Am 03.10.2018 um 21:20 schrieb Kelley Jabr:
Luigi, I'm pretty sure you just solved my problem. I tried a few operations with the table closed and it was like night and day. That 400k field calculator finished in mere moments! Thank you!!

On Wed, Oct 3, 2018 at 1:42 PM Luigi Pirelli <[hidden email]> wrote:
can you test doing operations with field calculator without having the attribute table opened? just a workaround could be to use a virtual layer then save the entire layer later.

Luigi Pirelli

**************************************************************************************************
* LinkedIn: https://www.linkedin.com/in/luigipirelli
* Stackexchange: http://gis.stackexchange.com/users/19667/luigi-pirelli
* GitHub: https://github.com/luipir
* Mastering QGIS 2nd Edition:
* https://www.packtpub.com/big-data-and-business-intelligence/mastering-qgis-second-edition
* Hire me: http://goo.gl/BYRQKg
**************************************************************************************************


On Wed, 3 Oct 2018 at 18:11, Kelley Jabr <[hidden email]> wrote:
Hello,

I'm working with a geopackage on ~530k features, and just about everything I try to do in the table takes way longer than expected. 
  • Adding a field took 5 minutes (table now has 22 total fields, so not massive)
  • Field Calculator on ~32k selected records took 8 minutes (literally all I was putting in the cells was: N)
  • Field calculator on ~480k records took so many hours that I had to leave it running over the weekend and when I came back, the Field Calculator had "closed unexpectedly"
(I'm running on a linux machine with an Intel Core i7 and 32GB of memory.)

Has anyone else experience similar issues? Any advice or recommendations for how to improve performance here? 

Thank you,

--

Kelley Jabr

Analyst, Geographic Data Operations

[hidden email]


_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user


--

Kelley Jabr

Analyst, Geographic Data Operations

[hidden email]


Indeed - We help people get jobs.

Indeed.com




_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user


--

Kelley Jabr

Analyst, Geographic Data Operations

[hidden email]


Indeed - We help people get jobs.

Indeed.com



_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: Very Slow Table Operations in QGIS 3.2.3

ginetto
tnx Kelly, this is your issue and it's ok: https://issues.qgis.org/issues/20008

to everyone, please add any comment or info that can help to replicate/solve

Luigi Pirelli

**************************************************************************************************
* LinkedIn: https://www.linkedin.com/in/luigipirelli
* Stackexchange: http://gis.stackexchange.com/users/19667/luigi-pirelli
* GitHub: https://github.com/luipir
* Mastering QGIS 2nd Edition:
* https://www.packtpub.com/big-data-and-business-intelligence/mastering-qgis-second-edition
* Hire me: http://goo.gl/BYRQKg
**************************************************************************************************


On Wed, 3 Oct 2018 at 22:40, Kelley Jabr <[hidden email]> wrote:
Okay, I've created an issue for this situation. Hopefully I've done it satisfactorily. Thanks again!

On Wed, Oct 3, 2018 at 2:31 PM Andreas Neumann <[hidden email]> wrote:

Hi,

Glad to hear that it helped to run the operations on a closed table. Nevertheless, as Luigi said, it would make sense to submit an issue report about this slowness.

The other thing one has to keep on mind:

The SQLite API is kind of limited. Only until recently, if you wanted to rename a field, one had to copy the whole table with the renamed field changed in the new table - then the old table had to be removed (all in the background), because there was API to rename columns in SQLite. I think with the very recent version, which is not rolled out into most OS, this has changed. You'd also need a very recent GDAL/OGR.

Greetings,
Andreas

Am 03.10.2018 um 21:20 schrieb Kelley Jabr:
Luigi, I'm pretty sure you just solved my problem. I tried a few operations with the table closed and it was like night and day. That 400k field calculator finished in mere moments! Thank you!!

On Wed, Oct 3, 2018 at 1:42 PM Luigi Pirelli <[hidden email]> wrote:
can you test doing operations with field calculator without having the attribute table opened? just a workaround could be to use a virtual layer then save the entire layer later.

Luigi Pirelli

**************************************************************************************************
* LinkedIn: https://www.linkedin.com/in/luigipirelli
* Stackexchange: http://gis.stackexchange.com/users/19667/luigi-pirelli
* GitHub: https://github.com/luipir
* Mastering QGIS 2nd Edition:
* https://www.packtpub.com/big-data-and-business-intelligence/mastering-qgis-second-edition
* Hire me: http://goo.gl/BYRQKg
**************************************************************************************************


On Wed, 3 Oct 2018 at 18:11, Kelley Jabr <[hidden email]> wrote:
Hello,

I'm working with a geopackage on ~530k features, and just about everything I try to do in the table takes way longer than expected. 
  • Adding a field took 5 minutes (table now has 22 total fields, so not massive)
  • Field Calculator on ~32k selected records took 8 minutes (literally all I was putting in the cells was: N)
  • Field calculator on ~480k records took so many hours that I had to leave it running over the weekend and when I came back, the Field Calculator had "closed unexpectedly"
(I'm running on a linux machine with an Intel Core i7 and 32GB of memory.)

Has anyone else experience similar issues? Any advice or recommendations for how to improve performance here? 

Thank you,

--

Kelley Jabr

Analyst, Geographic Data Operations

[hidden email]


_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user


--

Kelley Jabr

Analyst, Geographic Data Operations

[hidden email]


Indeed - We help people get jobs.

Indeed.com




_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user


--

Kelley Jabr

Analyst, Geographic Data Operations

[hidden email]


Indeed - We help people get jobs.

Indeed.com


_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: Very Slow Table Operations in QGIS 3.2.3

Alexandre Neto
Hi, 

In SQL queries on databases like Postgresql it is common to use the Limit operand to ask just a sample of the output to check if all looks OK without reading the all table. 

This situation made me wonder if this couldn't be a nice feature request to the layers attribute table when working with very large tables: to only show/load the first X features. 

Obvious operations like selection or field calculator on the table should affect the all table, and not just the visible layers. 

I am not sure if it would have an impact on how fast working with the table looks. What Kelley case seems to show, is that loading and showing the table is where the problem is, not in the operations on the attributes. 

What do u guy think? 

Best regards, 

Alex Neto



A qua, 3/10/2018, 21:54, Luigi Pirelli <[hidden email]> escreveu:
tnx Kelly, this is your issue and it's ok: https://issues.qgis.org/issues/20008

to everyone, please add any comment or info that can help to replicate/solve

Luigi Pirelli

**************************************************************************************************
* LinkedIn: https://www.linkedin.com/in/luigipirelli
* Stackexchange: http://gis.stackexchange.com/users/19667/luigi-pirelli
* GitHub: https://github.com/luipir
* Mastering QGIS 2nd Edition:
* https://www.packtpub.com/big-data-and-business-intelligence/mastering-qgis-second-edition
* Hire me: http://goo.gl/BYRQKg
**************************************************************************************************


On Wed, 3 Oct 2018 at 22:40, Kelley Jabr <[hidden email]> wrote:
Okay, I've created an issue for this situation. Hopefully I've done it satisfactorily. Thanks again!

On Wed, Oct 3, 2018 at 2:31 PM Andreas Neumann <[hidden email]> wrote:

Hi,

Glad to hear that it helped to run the operations on a closed table. Nevertheless, as Luigi said, it would make sense to submit an issue report about this slowness.

The other thing one has to keep on mind:

The SQLite API is kind of limited. Only until recently, if you wanted to rename a field, one had to copy the whole table with the renamed field changed in the new table - then the old table had to be removed (all in the background), because there was API to rename columns in SQLite. I think with the very recent version, which is not rolled out into most OS, this has changed. You'd also need a very recent GDAL/OGR.

Greetings,
Andreas

Am 03.10.2018 um 21:20 schrieb Kelley Jabr:
Luigi, I'm pretty sure you just solved my problem. I tried a few operations with the table closed and it was like night and day. That 400k field calculator finished in mere moments! Thank you!!

On Wed, Oct 3, 2018 at 1:42 PM Luigi Pirelli <[hidden email]> wrote:
can you test doing operations with field calculator without having the attribute table opened? just a workaround could be to use a virtual layer then save the entire layer later.

Luigi Pirelli

**************************************************************************************************
* LinkedIn: https://www.linkedin.com/in/luigipirelli
* Stackexchange: http://gis.stackexchange.com/users/19667/luigi-pirelli
* GitHub: https://github.com/luipir
* Mastering QGIS 2nd Edition:
* https://www.packtpub.com/big-data-and-business-intelligence/mastering-qgis-second-edition
* Hire me: http://goo.gl/BYRQKg
**************************************************************************************************


On Wed, 3 Oct 2018 at 18:11, Kelley Jabr <[hidden email]> wrote:
Hello,

I'm working with a geopackage on ~530k features, and just about everything I try to do in the table takes way longer than expected. 
  • Adding a field took 5 minutes (table now has 22 total fields, so not massive)
  • Field Calculator on ~32k selected records took 8 minutes (literally all I was putting in the cells was: N)
  • Field calculator on ~480k records took so many hours that I had to leave it running over the weekend and when I came back, the Field Calculator had "closed unexpectedly"
(I'm running on a linux machine with an Intel Core i7 and 32GB of memory.)

Has anyone else experience similar issues? Any advice or recommendations for how to improve performance here? 

Thank you,

--

Kelley Jabr

Analyst, Geographic Data Operations

[hidden email]


_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user


--

Kelley Jabr

Analyst, Geographic Data Operations

[hidden email]


Indeed - We help people get jobs.

Indeed.com




_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user


--

Kelley Jabr

Analyst, Geographic Data Operations

[hidden email]


Indeed - We help people get jobs.

Indeed.com


_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
--
Alexandre Neto
---------------------
@AlexNetoGeo
http://gisunchained.wordpress.com

_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: Very Slow Table Operations in QGIS 3.2.3

Rebecca Bartlett
In reply to this post by Kelley Jabr

Hello,

 

A couple of colleagues and I have been wrestling with a similar issue to this and I’ve been hunting around to see if it’s documented, and happily saw this email string. I’ve viewed the bug reports (both Kelley’s and the related #19771) and while they capture most of what we’ve been running into, it’s not an exact match. Here’s our situation and any input would be greatly appreciated – including viable workarounds.

 

1.       Adding a .txt or .csv file to QGIS 3.2.1 as Delimited Text File (5721 features; 9 fields; no geometry)

2.       Using Field Calculator via the attribute table to add a Decimal field (length: 10; precision: 2 – the precision is important which is why the virtual field workaround without opening the attribute table hasn’t worked well with its precision of 3, plus the precision isn’t retained upon table export although we may be missing an export option somewhere) and input values from an already-existing text field

a.       Copying text field to numeric field for a join

3.       Field calculation takes less than a second

4.       Saving the table doesn’t work. I left it running for 90 minutes the first time and it has crashed QGIS on colleagues’ computers multiple times

a.       Notable that it also doesn’t work on 2.18.24 on Windows machines but it’s fine on 2.18 with Macs (ran the process in a class of 20 after we had them use 2.18, and only the Windows users had issues)

b.       Still takes a few moments to save on Mac 2.18 – but it works!

 

We’ve tried several workarounds and the best one (a.k.a. the one that works in 3.2.1 and 2.18.24 in Windows) is to use the Field Calculator via the Source Fields tab in Properties, but it still takes >6 minutes to save which seems quite long for 5700 features.

 

I’m assuming this is the same issue but if there are any reliable workarounds we’re all ears!

 

Sincerely,

Rebecca

 

Rebecca Bartlett

GIS and Digital Resources Librarian

Research Support Services

Carleton University Library

613-520-2600 x4084

[hidden email]

 


Okay, I've created an issue for this situation. Hopefully I've done it
satisfactorily. Thanks again!
 
On Wed, Oct 3, 2018 at 2:31 PM Andreas Neumann <a.neumann at carto.net> wrote:
 
> Hi,
> 
> Glad to hear that it helped to run the operations on a closed table.
> Nevertheless, as Luigi said, it would make sense to submit an issue report
> about this slowness.
> 
> The other thing one has to keep on mind:
> 
> The SQLite API is kind of limited. Only until recently, if you wanted to
> rename a field, one had to copy the whole table with the renamed field
> changed in the new table - then the old table had to be removed (all in the
> background), because there was API to rename columns in SQLite. I think
> with the very recent version, which is not rolled out into most OS, this
> has changed. You'd also need a very recent GDAL/OGR.
> Greetings,
> Andreas
> 
> Am 03.10.2018 um 21:20 schrieb Kelley Jabr:
> 
> Luigi, I'm pretty sure you just solved my problem. I tried a few
> operations with the table closed and it was like night and day. That 400k
> field calculator finished in mere moments! Thank you!!
> 
> On Wed, Oct 3, 2018 at 1:42 PM Luigi Pirelli <luipir at gmail.com> wrote:
> 
>> can you test doing operations with field calculator without having the
>> attribute table opened? just a workaround could be to use a virtual layer
>> then save the entire layer later.
>> 
>> Luigi Pirelli
>> 
>> 
>> **************************************************************************************************
>> * LinkedIn: https://www.linkedin.com/in/luigipirelli
>> * Stackexchange: http://gis.stackexchange.com/users/19667/luigi-pirelli
>> * GitHub: https://github.com/luipir
>> * Mastering QGIS 2nd Edition:
>> *
>> https://www.packtpub.com/big-data-and-business-intelligence/mastering-qgis-second-edition
>> * Hire me: http://goo.gl/BYRQKg
>> 
>> **************************************************************************************************
>> 
>> 
>> On Wed, 3 Oct 2018 at 18:11, Kelley Jabr <kjabr at indeed.com> wrote:
>> 
>>> Hello,
>>> 
>>> I'm working with a geopackage on ~530k features, and just about
>>> everything I try to do in the table takes way longer than expected.
>>> 
>>>    - Adding a field took 5 minutes (table now has 22 total fields, so
>>>    not massive)
>>>    - Field Calculator on ~32k selected records took 8 minutes
>>>    (literally all I was putting in the cells was: N)
>>>    - Field calculator on ~480k records took so many hours that I had to
>>>    leave it running over the weekend and when I came back, the Field
>>>    Calculator had "closed unexpectedly"
>>> 
>>> (I'm running on a linux machine with an Intel Core i7 and 32GB of
>>> memory.)
>>> 
>>> Has anyone else experience similar issues? Any advice or recommendations
>>> for how to improve performance here?
>>> 
>>> Thank you,
>>> 
>>> --
>>> 
>>> Kelley Jabr
>>> 
>>> Analyst, Geographic Data Operations
>>> 
>>> kjabr at indeed.com
>>> 
>>> _______________________________________________
>>> Qgis-user mailing list
>>> Qgis-user at lists.osgeo.org
>>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>> 
>> 
> 
> --
> 
> Kelley Jabr
> 
> Analyst, Geographic Data Operations
> 
> kjabr at indeed.com
> 
> Indeed - We help people get jobs.
> 
> Indeed.com <http://www.indeed.com>
> 
> 
> 
> _______________________________________________
> Qgis-user mailing listQgis-user at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
> 
> 
> _______________________________________________
> Qgis-user mailing list
> Qgis-user at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
 
 
 
-- 
 
Kelley Jabr
 
Analyst, Geographic Data Operations
 
kjabr at indeed.com
 
Indeed - We help people get jobs.
 
Indeed.com <http://www.indeed.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-user/attachments/20181003/2c54a49e/attachment-0001.html>

 

 


_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: Very Slow Table Operations in QGIS 3.2.3

Alexandre Neto
My first suggestion would be for, after importing from csv, save it to geopackages for the rest of the operations. Then, in the end export back to the necessary file format. I think working directly on text files is not optimal. 

Best regards, 

Alexandre Neto


A ter, 9/10/2018, 20:02, Rebecca Bartlett <[hidden email]> escreveu:

Hello,

 

A couple of colleagues and I have been wrestling with a similar issue to this and I’ve been hunting around to see if it’s documented, and happily saw this email string. I’ve viewed the bug reports (both Kelley’s and the related #19771) and while they capture most of what we’ve been running into, it’s not an exact match. Here’s our situation and any input would be greatly appreciated – including viable workarounds.

 

1.       Adding a .txt or .csv file to QGIS 3.2.1 as Delimited Text File (5721 features; 9 fields; no geometry)

2.       Using Field Calculator via the attribute table to add a Decimal field (length: 10; precision: 2 – the precision is important which is why the virtual field workaround without opening the attribute table hasn’t worked well with its precision of 3, plus the precision isn’t retained upon table export although we may be missing an export option somewhere) and input values from an already-existing text field

a.       Copying text field to numeric field for a join

3.       Field calculation takes less than a second

4.       Saving the table doesn’t work. I left it running for 90 minutes the first time and it has crashed QGIS on colleagues’ computers multiple times

a.       Notable that it also doesn’t work on 2.18.24 on Windows machines but it’s fine on 2.18 with Macs (ran the process in a class of 20 after we had them use 2.18, and only the Windows users had issues)

b.       Still takes a few moments to save on Mac 2.18 – but it works!

 

We’ve tried several workarounds and the best one (a.k.a. the one that works in 3.2.1 and 2.18.24 in Windows) is to use the Field Calculator via the Source Fields tab in Properties, but it still takes >6 minutes to save which seems quite long for 5700 features.

 

I’m assuming this is the same issue but if there are any reliable workarounds we’re all ears!

 

Sincerely,

Rebecca

 

Rebecca Bartlett

GIS and Digital Resources Librarian

Research Support Services

Carleton University Library

613-520-2600 x4084

[hidden email]

 


Okay, I've created an issue for this situation. Hopefully I've done it
satisfactorily. Thanks again!
 
On Wed, Oct 3, 2018 at 2:31 PM Andreas Neumann <a.neumann at carto.net> wrote:
 
> Hi,
> 
> Glad to hear that it helped to run the operations on a closed table.
> Nevertheless, as Luigi said, it would make sense to submit an issue report
> about this slowness.
> 
> The other thing one has to keep on mind:
> 
> The SQLite API is kind of limited. Only until recently, if you wanted to
> rename a field, one had to copy the whole table with the renamed field
> changed in the new table - then the old table had to be removed (all in the
> background), because there was API to rename columns in SQLite. I think
> with the very recent version, which is not rolled out into most OS, this
> has changed. You'd also need a very recent GDAL/OGR.
> Greetings,
> Andreas
> 
> Am 03.10.2018 um 21:20 schrieb Kelley Jabr:
> 
> Luigi, I'm pretty sure you just solved my problem. I tried a few
> operations with the table closed and it was like night and day. That 400k
> field calculator finished in mere moments! Thank you!!
> 
> On Wed, Oct 3, 2018 at 1:42 PM Luigi Pirelli <luipir at gmail.com> wrote:
> 
>> can you test doing operations with field calculator without having the
>> attribute table opened? just a workaround could be to use a virtual layer
>> then save the entire layer later.
>> 
>> Luigi Pirelli
>> 
>> 
>> **************************************************************************************************
>> * LinkedIn: https://www.linkedin.com/in/luigipirelli
>> * Stackexchange: http://gis.stackexchange.com/users/19667/luigi-pirelli
>> * GitHub: https://github.com/luipir
>> * Mastering QGIS 2nd Edition:
>> *
>> https://www.packtpub.com/big-data-and-business-intelligence/mastering-qgis-second-edition
>> * Hire me: http://goo.gl/BYRQKg
>> 
>> **************************************************************************************************
>> 
>> 
>> On Wed, 3 Oct 2018 at 18:11, Kelley Jabr <kjabr at indeed.com> wrote:
>> 
>>> Hello,
>>> 
>>> I'm working with a geopackage on ~530k features, and just about
>>> everything I try to do in the table takes way longer than expected.
>>> 
>>>    - Adding a field took 5 minutes (table now has 22 total fields, so
>>>    not massive)
>>>    - Field Calculator on ~32k selected records took 8 minutes
>>>    (literally all I was putting in the cells was: N)
>>>    - Field calculator on ~480k records took so many hours that I had to
>>>    leave it running over the weekend and when I came back, the Field
>>>    Calculator had "closed unexpectedly"
>>> 
>>> (I'm running on a linux machine with an Intel Core i7 and 32GB of
>>> memory.)
>>> 
>>> Has anyone else experience similar issues? Any advice or recommendations
>>> for how to improve performance here?
>>> 
>>> Thank you,
>>> 
>>> --
>>> 
>>> Kelley Jabr
>>> 
>>> Analyst, Geographic Data Operations
>>> 
>>> kjabr at indeed.com
>>> 
>>> _______________________________________________
>>> Qgis-user mailing list
>>> Qgis-user at lists.osgeo.org
>>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>> 
>> 
> 
> --
> 
> Kelley Jabr
> 
> Analyst, Geographic Data Operations
> 
> kjabr at indeed.com
> 
> Indeed - We help people get jobs.
> 
> Indeed.com <http://www.indeed.com>
> 
> 
> 
> _______________________________________________
> Qgis-user mailing listQgis-user at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
> 
> 
> _______________________________________________
> Qgis-user mailing list
> Qgis-user at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
 
 
 
-- 
 
Kelley Jabr
 
Analyst, Geographic Data Operations
 
kjabr at indeed.com
 
Indeed - We help people get jobs.
 
Indeed.com <http://www.indeed.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-user/attachments/20181003/2c54a49e/attachment-0001.html>

 

 

_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
--
Alexandre Neto
---------------------
@AlexNetoGeo
http://gisunchained.wordpress.com

_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: Very Slow Table Operations in QGIS 3.2.3

James Bambury
Hello,

I've found the refactor fields process to be a reliable and fast backup option (plus being a process it now works in the background on a separate thread and shows a progress meter so you've a good idea if it's actually working!) if you're happy to have entirely new fields added.

I've had similar issues recently, tried to add a virtual field onto a geopackage layer of about ~5000 features consisting of a attribute(get_feature()) and ended up freezing up with no sign of progress after 15min. Refactor field did 6 new fields at once in about 20 seconds!

James

On Tue, 9 Oct 2018 at 22:32, Alexandre Neto <[hidden email]> wrote:
My first suggestion would be for, after importing from csv, save it to geopackages for the rest of the operations. Then, in the end export back to the necessary file format. I think working directly on text files is not optimal. 

Best regards, 

Alexandre Neto


A ter, 9/10/2018, 20:02, Rebecca Bartlett <[hidden email]> escreveu:

Hello,

 

A couple of colleagues and I have been wrestling with a similar issue to this and I’ve been hunting around to see if it’s documented, and happily saw this email string. I’ve viewed the bug reports (both Kelley’s and the related #19771) and while they capture most of what we’ve been running into, it’s not an exact match. Here’s our situation and any input would be greatly appreciated – including viable workarounds.

 

1.       Adding a .txt or .csv file to QGIS 3.2.1 as Delimited Text File (5721 features; 9 fields; no geometry)

2.       Using Field Calculator via the attribute table to add a Decimal field (length: 10; precision: 2 – the precision is important which is why the virtual field workaround without opening the attribute table hasn’t worked well with its precision of 3, plus the precision isn’t retained upon table export although we may be missing an export option somewhere) and input values from an already-existing text field

a.       Copying text field to numeric field for a join

3.       Field calculation takes less than a second

4.       Saving the table doesn’t work. I left it running for 90 minutes the first time and it has crashed QGIS on colleagues’ computers multiple times

a.       Notable that it also doesn’t work on 2.18.24 on Windows machines but it’s fine on 2.18 with Macs (ran the process in a class of 20 after we had them use 2.18, and only the Windows users had issues)

b.       Still takes a few moments to save on Mac 2.18 – but it works!

 

We’ve tried several workarounds and the best one (a.k.a. the one that works in 3.2.1 and 2.18.24 in Windows) is to use the Field Calculator via the Source Fields tab in Properties, but it still takes >6 minutes to save which seems quite long for 5700 features.

 

I’m assuming this is the same issue but if there are any reliable workarounds we’re all ears!

 

Sincerely,

Rebecca

 

Rebecca Bartlett

GIS and Digital Resources Librarian

Research Support Services

Carleton University Library

613-520-2600 x4084

[hidden email]

 


Okay, I've created an issue for this situation. Hopefully I've done it
satisfactorily. Thanks again!
 
On Wed, Oct 3, 2018 at 2:31 PM Andreas Neumann <a.neumann at carto.net> wrote:
 
> Hi,
> 
> Glad to hear that it helped to run the operations on a closed table.
> Nevertheless, as Luigi said, it would make sense to submit an issue report
> about this slowness.
> 
> The other thing one has to keep on mind:
> 
> The SQLite API is kind of limited. Only until recently, if you wanted to
> rename a field, one had to copy the whole table with the renamed field
> changed in the new table - then the old table had to be removed (all in the
> background), because there was API to rename columns in SQLite. I think
> with the very recent version, which is not rolled out into most OS, this
> has changed. You'd also need a very recent GDAL/OGR.
> Greetings,
> Andreas
> 
> Am 03.10.2018 um 21:20 schrieb Kelley Jabr:
> 
> Luigi, I'm pretty sure you just solved my problem. I tried a few
> operations with the table closed and it was like night and day. That 400k
> field calculator finished in mere moments! Thank you!!
> 
> On Wed, Oct 3, 2018 at 1:42 PM Luigi Pirelli <luipir at gmail.com> wrote:
> 
>> can you test doing operations with field calculator without having the
>> attribute table opened? just a workaround could be to use a virtual layer
>> then save the entire layer later.
>> 
>> Luigi Pirelli
>> 
>> 
>> **************************************************************************************************
>> * LinkedIn: https://www.linkedin.com/in/luigipirelli
>> * Stackexchange: http://gis.stackexchange.com/users/19667/luigi-pirelli
>> * GitHub: https://github.com/luipir
>> * Mastering QGIS 2nd Edition:
>> *
>> https://www.packtpub.com/big-data-and-business-intelligence/mastering-qgis-second-edition
>> * Hire me: http://goo.gl/BYRQKg
>> 
>> **************************************************************************************************
>> 
>> 
>> On Wed, 3 Oct 2018 at 18:11, Kelley Jabr <kjabr at indeed.com> wrote:
>> 
>>> Hello,
>>> 
>>> I'm working with a geopackage on ~530k features, and just about
>>> everything I try to do in the table takes way longer than expected.
>>> 
>>>    - Adding a field took 5 minutes (table now has 22 total fields, so
>>>    not massive)
>>>    - Field Calculator on ~32k selected records took 8 minutes
>>>    (literally all I was putting in the cells was: N)
>>>    - Field calculator on ~480k records took so many hours that I had to
>>>    leave it running over the weekend and when I came back, the Field
>>>    Calculator had "closed unexpectedly"
>>> 
>>> (I'm running on a linux machine with an Intel Core i7 and 32GB of
>>> memory.)
>>> 
>>> Has anyone else experience similar issues? Any advice or recommendations
>>> for how to improve performance here?
>>> 
>>> Thank you,
>>> 
>>> --
>>> 
>>> Kelley Jabr
>>> 
>>> Analyst, Geographic Data Operations
>>> 
>>> kjabr at indeed.com
>>> 
>>> _______________________________________________
>>> Qgis-user mailing list
>>> Qgis-user at lists.osgeo.org
>>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>> 
>> 
> 
> --
> 
> Kelley Jabr
> 
> Analyst, Geographic Data Operations
> 
> kjabr at indeed.com
> 
> Indeed - We help people get jobs.
> 
> Indeed.com <http://www.indeed.com>
> 
> 
> 
> _______________________________________________
> Qgis-user mailing listQgis-user at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
> 
> 
> _______________________________________________
> Qgis-user mailing list
> Qgis-user at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
 
 
 
-- 
 
Kelley Jabr
 
Analyst, Geographic Data Operations
 
kjabr at indeed.com
 
Indeed - We help people get jobs.
 
Indeed.com <http://www.indeed.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-user/attachments/20181003/2c54a49e/attachment-0001.html>

 

 

_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
--
Alexandre Neto
---------------------
@AlexNetoGeo
http://gisunchained.wordpress.com
_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: Very Slow Table Operations in QGIS 3.2.3

Antongeo76
In reply to this post by Kelley Jabr
Hi,

This slowness is also present in QGIS 2.18 as well as in QGIS 3.2.3
and also occurs with fewer features in shapefiles (15000-16000 are enough),
slowness increases as the number of unsaved editing operations increases.

With the "attribute table" opened, even "merge elements" or "split feature"
operations,
cause some problems (i.e. the counting of the records and the selection of
the same does not seem synchronized with these editing operations)
and sometimes this cause freeze of QGIS





-----
Antonio
--
Sent from: http://osgeo-org.1560.x6.nabble.com/QGIS-User-f4125267.html
_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Antonio
Reply | Threaded
Open this post in threaded view
|

Re: Very Slow Table Operations in QGIS 3.2.3

Alessandro Pasotti-2

On Tue, Oct 16, 2018 at 12:50 PM Antongeo76 <[hidden email]> wrote:
Hi,

This slowness is also present in QGIS 2.18 as well as in QGIS 3.2.3
and also occurs with fewer features in shapefiles (15000-16000 are enough),
slowness increases as the number of unsaved editing operations increases.

With the "attribute table" opened, even "merge elements" or "split feature"
operations,
cause some problems (i.e. the counting of the records and the selection of
the same does not seem synchronized with these editing operations)
and sometimes this cause freeze of QGIS





-----
Antonio
--
Sent from: http://osgeo-org.1560.x6.nabble.com/QGIS-User-f4125267.html
_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user


--
Alessandro Pasotti
w3:   www.itopen.it

_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: Very Slow Table Operations in QGIS 3.2.3

Antongeo76
Alessandro Pasotti-2 wrote
> May be related to https://issues.qgis.org/issues/20094
>
> See also: https://github.com/qgis/QGIS/pull/8200
>
> On Tue, Oct 16, 2018 at 12:50 PM Antongeo76 &lt;

> antoviscomi@

> &gt; wrote:
>
>> Hi,
>>
>> This slowness is also present in QGIS 2.18 as well as in QGIS 3.2.3
>> and also occurs with fewer features in shapefiles (15000-16000 are
>> enough),
>> slowness increases as the number of unsaved editing operations increases.
>>
>> With the "attribute table" opened, even "merge elements" or "split
>> feature"
>> operations,
>> cause some problems (i.e. the counting of the records and the selection
>> of
>> the same does not seem synchronized with these editing operations)
>> and sometimes this cause freeze of QGIS
>>
>>
>>
>>
>>
>> -----
>> Antonio
>> --
>> Sent from: http://osgeo-org.1560.x6.nabble.com/QGIS-User-f4125267.html
>> _______________________________________________
>> Qgis-user mailing list
>>

> Qgis-user@.osgeo

>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>
>
>
> --
> Alessandro Pasotti
> w3:   www.itopen.it
>
> _______________________________________________
> Qgis-user mailing list

> Qgis-user@.osgeo

> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

Yes, it can be related but it does not happen only when the "field
calculator" is used
it happens even if you perform several 'manual' editing operations and save
them after about an hour.

It gives me the feeling that there is a buffer overflow



-----
Antonio
--
Sent from: http://osgeo-org.1560.x6.nabble.com/QGIS-User-f4125267.html
_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Antonio
Reply | Threaded
Open this post in threaded view
|

Re: Very Slow Table Operations in QGIS 3.2.3

DelazJ
Hi,

Any third party plugin enabled? There are some plugins that are/were known/suspected to generate this overflow. You'll find related reports in the bug tracker.
In 3.x, try to run on a clean user profile with no additional plugin.

DelazJ

Le mar. 16 oct. 2018 à 13:06, Antongeo76 <[hidden email]> a écrit :
Alessandro Pasotti-2 wrote
> May be related to https://issues.qgis.org/issues/20094
>
> See also: https://github.com/qgis/QGIS/pull/8200
>
> On Tue, Oct 16, 2018 at 12:50 PM Antongeo76 &lt;

> antoviscomi@

> &gt; wrote:
>
>> Hi,
>>
>> This slowness is also present in QGIS 2.18 as well as in QGIS 3.2.3
>> and also occurs with fewer features in shapefiles (15000-16000 are
>> enough),
>> slowness increases as the number of unsaved editing operations increases.
>>
>> With the "attribute table" opened, even "merge elements" or "split
>> feature"
>> operations,
>> cause some problems (i.e. the counting of the records and the selection
>> of
>> the same does not seem synchronized with these editing operations)
>> and sometimes this cause freeze of QGIS
>>
>>
>>
>>
>>
>> -----
>> Antonio
>> --
>> Sent from: http://osgeo-org.1560.x6.nabble.com/QGIS-User-f4125267.html
>> _______________________________________________
>> Qgis-user mailing list
>>

> Qgis-user@.osgeo

>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>
>
>
> --
> Alessandro Pasotti
> w3:   www.itopen.it
>
> _______________________________________________
> Qgis-user mailing list

> Qgis-user@.osgeo

> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

Yes, it can be related but it does not happen only when the "field
calculator" is used
it happens even if you perform several 'manual' editing operations and save
them after about an hour.

It gives me the feeling that there is a buffer overflow



-----
Antonio
--
Sent from: http://osgeo-org.1560.x6.nabble.com/QGIS-User-f4125267.html
_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: Very Slow Table Operations in QGIS 3.2.3

ginetto
In reply to this post by Antongeo76
nothing related with overflows as described in shown tikets... btw, please give a check with master (not 2.18) nightly builds e.g. qgis-dev. Fix has been merged and you can check if it works on on dev version not in 3.2.x.
Luigi Pirelli

**************************************************************************************************
* LinkedIn: https://www.linkedin.com/in/luigipirelli
* Stackexchange: http://gis.stackexchange.com/users/19667/luigi-pirelli
* GitHub: https://github.com/luipir
* Mastering QGIS 2nd Edition:
* https://www.packtpub.com/big-data-and-business-intelligence/mastering-qgis-second-edition
* Hire me: http://goo.gl/BYRQKg
**************************************************************************************************


On Tue, 16 Oct 2018 at 13:06, Antongeo76 <[hidden email]> wrote:
Alessandro Pasotti-2 wrote
> May be related to https://issues.qgis.org/issues/20094
>
> See also: https://github.com/qgis/QGIS/pull/8200
>
> On Tue, Oct 16, 2018 at 12:50 PM Antongeo76 &lt;

> antoviscomi@

> &gt; wrote:
>
>> Hi,
>>
>> This slowness is also present in QGIS 2.18 as well as in QGIS 3.2.3
>> and also occurs with fewer features in shapefiles (15000-16000 are
>> enough),
>> slowness increases as the number of unsaved editing operations increases.
>>
>> With the "attribute table" opened, even "merge elements" or "split
>> feature"
>> operations,
>> cause some problems (i.e. the counting of the records and the selection
>> of
>> the same does not seem synchronized with these editing operations)
>> and sometimes this cause freeze of QGIS
>>
>>
>>
>>
>>
>> -----
>> Antonio
>> --
>> Sent from: http://osgeo-org.1560.x6.nabble.com/QGIS-User-f4125267.html
>> _______________________________________________
>> Qgis-user mailing list
>>

> Qgis-user@.osgeo

>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>
>
>
> --
> Alessandro Pasotti
> w3:   www.itopen.it
>
> _______________________________________________
> Qgis-user mailing list

> Qgis-user@.osgeo

> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

Yes, it can be related but it does not happen only when the "field
calculator" is used
it happens even if you perform several 'manual' editing operations and save
them after about an hour.

It gives me the feeling that there is a buffer overflow



-----
Antonio
--
Sent from: http://osgeo-org.1560.x6.nabble.com/QGIS-User-f4125267.html
_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: Very Slow Table Operations in QGIS 3.2.3

Alessandro Pasotti-2
In reply to this post by Antongeo76

On Tue, Oct 16, 2018 at 1:06 PM Antongeo76 <[hidden email]> wrote:
Alessandro Pasotti-2 wrote
> May be related to https://issues.qgis.org/issues/20094
>
> See also: https://github.com/qgis/QGIS/pull/8200
>
> On Tue, Oct 16, 2018 at 12:50 PM Antongeo76 &lt;

> antoviscomi@

> &gt; wrote:
>
>> Hi,
>>
>> This slowness is also present in QGIS 2.18 as well as in QGIS 3.2.3
>> and also occurs with fewer features in shapefiles (15000-16000 are
>> enough),
>> slowness increases as the number of unsaved editing operations increases.
>>
>> With the "attribute table" opened, even "merge elements" or "split
>> feature"
>> operations,
>> cause some problems (i.e. the counting of the records and the selection
>> of
>> the same does not seem synchronized with these editing operations)
>> and sometimes this cause freeze of QGIS
>>
>>
>>
>>
>>
>> -----
>> Antonio
>> --
>> Sent from: http://osgeo-org.1560.x6.nabble.com/QGIS-User-f4125267.html
>> _______________________________________________
>> Qgis-user mailing list
>>

> Qgis-user@.osgeo

>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>
>
>
> --
> Alessandro Pasotti
> w3:   www.itopen.it
>
> _______________________________________________
> Qgis-user mailing list

> Qgis-user@.osgeo

> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

Yes, it can be related but it does not happen only when the "field
calculator" is used
it happens even if you perform several 'manual' editing operations and save
them after about an hour.


The bug I pointed to is not related to field calculator (the title is misleading): field calculator just triggers it because it does a lot of changes.

 
It gives me the feeling that there is a buffer overflow

Buffer overflow would most probably lead to a crash, and this is not the case.


--
Alessandro Pasotti
w3:   www.itopen.it

_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Reply | Threaded
Open this post in threaded view
|

Re: Very Slow Table Operations in QGIS 3.2.3

Antongeo76
I've just tested in Master 9cae77f262,
It seems to works fine

many thanks for the explanation

Regards
Alessandro Pasotti-2 wrote
> On Tue, Oct 16, 2018 at 1:06 PM Antongeo76 &lt;

> antoviscomi@

> &gt; wrote:
>
>> Alessandro Pasotti-2 wrote
>> > May be related to https://issues.qgis.org/issues/20094
>> >
>> > See also: https://github.com/qgis/QGIS/pull/8200
>> >
>> > On Tue, Oct 16, 2018 at 12:50 PM Antongeo76 &lt;
>>
>> > antoviscomi@
>>
>> > &gt; wrote:
>> >
>> >> Hi,
>> >>
>> >> This slowness is also present in QGIS 2.18 as well as in QGIS 3.2.3
>> >> and also occurs with fewer features in shapefiles (15000-16000 are
>> >> enough),
>> >> slowness increases as the number of unsaved editing operations
>> increases.
>> >>
>> >> With the "attribute table" opened, even "merge elements" or "split
>> >> feature"
>> >> operations,
>> >> cause some problems (i.e. the counting of the records and the
>> selection
>> >> of
>> >> the same does not seem synchronized with these editing operations)
>> >> and sometimes this cause freeze of QGIS
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> -----
>> >> Antonio
>> >> --
>> >> Sent from: http://osgeo-org.1560.x6.nabble.com/QGIS-User-f4125267.html
>> >> _______________________________________________
>> >> Qgis-user mailing list
>> >>
>>
>> > Qgis-user@.osgeo
>>
>> >> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>> >> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>> >
>> >
>> >
>> > --
>> > Alessandro Pasotti
>> > w3:   www.itopen.it
>> >
>> > _______________________________________________
>> > Qgis-user mailing list
>>
>> > Qgis-user@.osgeo
>>
>> > List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>> > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>>
>> Yes, it can be related but it does not happen only when the "field
>> calculator" is used
>> it happens even if you perform several 'manual' editing operations and
>> save
>> them after about an hour.
>>
>>
> The bug I pointed to is not related to field calculator (the title is
> misleading): field calculator just triggers it because it does a lot of
> changes.
>
>
>
>> It gives me the feeling that there is a buffer overflow
>>
>
> Buffer overflow would most probably lead to a crash, and this is not the
> case.
>
>
> --
> Alessandro Pasotti
> w3:   www.itopen.it
>
> _______________________________________________
> Qgis-user mailing list

> Qgis-user@.osgeo

> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user





-----
Antonio
--
Sent from: http://osgeo-org.1560.x6.nabble.com/QGIS-User-f4125267.html
_______________________________________________
Qgis-user mailing list
[hidden email]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Antonio